IPv6 foot-dragging

There has been much talk about IPv6 lately, and for good reason.
Whatever your opinion on whether IPv6 is a good solution to IPv4 address
exhaustion, it's the only solution we have. Yet deployment, at least in
North America, has been ridiculously slow.

I have just been informed by a sales rep for AS852 that they are not
deploying IPv4 until 2012. 2012? Really?

I've heard statements that AS701 has deployed IPv6 on their network but
I've yet to see any evidence of that in my area of Canada. Apparently
they "forgot" Canada when they did it. Now I'm informed, unofficially,
that they might maybe have it deployed, if I'm lucky, some time before
the end of 2011.

I think the above two points illustrate precisely why so many networks
in North America simply cannot deploy IPv6 whether they want to or not.
We simply cannot obtain IPv6 transit from our upstreams. It's just not
available. And the old line about "vote with your money" doesn't work
when you have limited choices.

Apparently the need for IPv6 isn't yet high enough to consider adding a transit provider. I've seen enough press releases from NTT and HE to know there's at least two that can do this out there.

[..]

I think the above two points illustrate precisely why so many networks
in North America simply cannot deploy IPv6 whether they want to or not.
We simply cannot obtain IPv6 transit from our upstreams. It's just not
available. And the old line about "vote with your money" doesn't work
when you have limited choices.

And you have just found out why transition technologies exist.

They are called 'transition' for a reason: during the time that you
cannot get (proper) native connectivity you can set up a tunnel to an
entity that can provide you with proper IPv6.

The same way you can also set up a IPv6-only transit session with a
party that is located at an IX or such you are at. Might just be to
cover the time till your current transits do support IPv6.

It is just a way around the problem, it might not be nice but it can
work and you can get ready, and might get enough insight on why not to
use that organization any more who is causing all the feet to be dragged.

Greets,
Jeroen

Funny, I was just involved in a discussion on IPv6 in Canada yesterday, and this link came up from multiple people: http://bgpmon.net/blog/?p=382 . There's also http://www.vyncke.org/ipv6status/detailed.php?country=ca&type=ISP , but I've seen some indications that there may be some inaccuracies (Allstream announcing 2001:04c8::/33, for instance).

      Jima

I have had similar problems with our providers, and these are tier 1 companies that should have already been full deployed. These are also some of the more expensive providers on a per Mb basis. The one provider that was full IPv6 ready was Cogent. HE is also IPv6 (although we don't use them atm.)

There are a number of networks in Canada that provide v6 transit both
big and small. I have v6 transit from TATA, HE and Cogent out of
Toronto. Many Canadian networks peer at Torix which also lists their v6
status.

http://www.torix.net/peers.php

  ---Mike

That highlights another problem I have. I have no presence in Toronto,
nor do I have a business case (or resources) to build a presence there.
The same applies to Vancouver which is the other popular city for such
things.

I do currently employ a tunnel from HE's tunnel broker and, as a result,
I'm reasonably sure I can make IPv6 work when I have proper transit for
it. However, it would be impolite at best to turn up any sort of
production service over such a tunnel.

Speaking from the perspective of a *small* network with very limited
resources, adding a transit provider, even if one is available, is very
expensive. Installation costs tend to dwarf any business gain, often
running well into the 5 figure range. The same applies to switching
transit providers. (Install costs are the same in either case.)

Apparently the need for IPv6 isn't yet high enough to consider adding

a

transit provider. I've seen enough press releases from NTT and HE to
know there's at least two that can do this out there.

I believe the major holdup at this point is lack of v6 eyeballs. End
user CPE, particularly DSL CPE, has been lagging in v6 capability.

As for v6 upstreams, I have native v6 with both InterNAP (may not be
available at ALL POPs yet) and HE. Savvis has yet to deploy it in the US
at the POP pertinent to our operatons.

The big push for v6 eyeballs at the current time are the mobile
operators. We are seeing activity that would indicate there are mobile
devices out there that are native v6 at this time. Content providers
who have a lot of mobile clients might find they have more native v6
eyeballs than they think they have.

A couple of things you can do to check. First of all look for requests
to your DNS servers for AAAA records and note where those are coming
from. That doesn't prove a lot but it gives some indication of who
might have v6 someplace in their network. If you are seeing a
significant number of these, the next thing I would do is get a dns
server on your network working with v6 and get that IP address in whois
even if all you are serving is v4 A records. Then note who is arriving
over v6 asking for AAAA records. Those are the best candidates for
enabling v6 services. Note which services those are asking for, pick
one, and if you have gear capable of it (say, for example, a load
balancer), configure a v6 VIP for that service balancing to v4 servers
behind it. Place the AAAA record for this service in the zone handed
out via v6 AAAA requests (ONLY!) and watch the service VIP and see if
clients are connecting.

So at this point you are handing out AAAA records for a v6 service but
ONLY for DNS requests that arrive via IPv6 asking for it. Any requests
arriving via v4 asking for an AAAA record would get the NOERROR response
and an A record for the resource (client might have IPv6 internally but
doesn't have v6 all the way to the Internet or their Internet coverage
might be "spotty" and doesn't include you coughCogentcough).

A couple of things you can do to check. First of all look for requests
to your DNS servers for AAAA records and note where those are coming
from.

Firefox has for a long time done both A and AAAA lookups even if the system doesn't have IPv6. I believe MacOS does this too, now. Don't know about other apps/OSes, but for sure you'll see tons of AAAA lookups from people who have no IPv6 connectivity.

Then note who is arriving
over v6 asking for AAAA records. Those are the best candidates for
enabling v6 services.

Now you're counting DNS servers. Because the provisioning of IPv6 DNS addresses has been such a mess and still is problematic, many dual stack systems do this over IPv4. And the DNS servers they talk to may be IPv4-only, or IPv4-only users may talk to dual stack DNS servers.

In my opinion, looking at this kind of stuff in order to draw conclusions about what you should do is a waste of time. It just means more work for everyone and it doesn't fix any of the broken stuff that's out there.

If the results of world IPv6 day are as we expect and only 0.1 - 0.2 % or less of all people have problems, I think the best way forward would be to have a second world IPv6 day where we again enable IPv6 industry-wide but this time we don't turn it off again.

A couple of things you can do to check. First of all look for requests
to your DNS servers for AAAA records and note where those are coming
from.

Firefox has for a long time done both A and AAAA lookups even if the system doesn't have IPv6. I believe MacOS does this too, now. Don't know about other apps/OSes, but for sure you'll see tons of AAAA lookups from people who have no IPv6 connectivity.

It is still a way to measure it, even if it's not that accurate.

Then note who is arriving
over v6 asking for AAAA records. Those are the best candidates for
enabling v6 services.

Now you're counting DNS servers. Because the provisioning of IPv6 DNS addresses has been such a mess and still is problematic, many dual stack systems do this over IPv4. And the DNS servers they talk to may be IPv4-only, or IPv4-only users may talk to dual stack DNS servers.

In my opinion, looking at this kind of stuff in order to draw conclusions about what you should do is a waste of time. It just means more work for everyone and it doesn't fix any of the broken stuff that's out there.

If the results of world IPv6 day are as we expect and only 0.1 - 0.2 % or less of all people have problems, I think the best way forward would be to have a second world IPv6 day where we again enable IPv6 industry-wide but this time we don't turn it off again.

I'd like to see a repeat but with a week timescale. If you parse carefully, if all the $major sites are broken in the same way at the same time, it's easier to justify leaving it broken. (eg: if Google, Yahoo and Bing all do IPv6 at once, neither has to worry about losing market share to the other due to misbehaving ipv6. That's how I read igor's email about the 182k users, even if I still think we would be served with a longer test).

The most interesting data for me is looking at the sites that have 'majorly' broken IPv6 dns. I count 600+ sites that are returning weird things like ::1 or ::ffff: addresses. My favorites are the .gov site on the list and the city of albany.

Here's a pointer to the list:

http://puck.nether.net/~jared/aaaa/very-broken-dns.txt

- Jared

* Iljitsch van Beijnum

Firefox has for a long time done both A and AAAA lookups even if the
system doesn't have IPv6.

They fixed that in version 4.0, by calling getaddrinfo() with the
AI_ADDRCONFIG flag (like most other browsers do).

https://bugzilla.mozilla.org/show_bug.cgi?id=614526

Now you're counting DNS servers. Because the provisioning of IPv6 DNS
addresses has been such a mess and still is problematic, many dual
stack systems do this over IPv4. And the DNS servers they talk to may
be IPv4-only, or IPv4-only users may talk to dual stack DNS servers.

Which is why I suggested trying it on ONE service and watching it
closely. What I have done is selected a "best candidate" for a test. I
am not implying that "this is guaranteed to work".

If the results of world IPv6 day are as we expect and only 0.1 - 0.2 %
or less of all people have problems, I think the best way forward

would

be to have a second world IPv6 day where we again enable IPv6

industry-

wide but this time we don't turn it off again.

0.1% of users is a HUGE number if you have 1,000,000 subscribers. Are
you prepared to field 1,000 helpdesk calls or lose 1,000 customers? Now
imagine 100,000,000 subscribers. Are you ready for 10,000 support calls
or the loss of 10,000 paying customers?

It isn't something you just throw out there on a whim and tell people to
like it or lump it if there are potentially a lot of people involved.

If the results of world IPv6 day are as we expect and only 0.1 - 0.2 %
or less of all people have problems, I think the best way forward would
be to have a second world IPv6 day where we again enable IPv6 industry-
wide but this time we don't turn it off again.

0.1% of users is a HUGE number if you have 1,000,000 subscribers. Are
you prepared to field 1,000 helpdesk calls or lose 1,000 customers?

Apparently "we" are, at least for the former, otherwise there wouldn't be an IPv6 day.

It isn't something you just throw out there on a whim and tell people to
like it or lump it if there are potentially a lot of people involved.

So what's the alternative? Never change anything?

Remember, this is al extremely trivial stuff: most things won't even completely stop working. And a few mouseclicks (yes, you have to know which ones so the helpdesks better start figuring that out) and you're back to normal. Compare this to turning off analog TV transmitters that have been running for decades where people have to buy converter boxes and sometimes even install antennas on their roof to keep using the service.

Unless you have a captive audience for customers, you probably have a churn
rate higher than 0.1% *anyhow*. And if you *do* have a captive audience, you
won't lose customers.

I would be interested in knowing if those people who say they can measure these
0.1% dips noticed anything due to the flooding and severe weather in the
midwest and southeast US in the past few weeks.

This argument has already been refuted many times. Let's assume that you're right about the churn rate. The issue is enterprises not wanting to take affirmative steps to knock N% *more* customers off the site than whatever the current churn rate is by enabling IPv6.

So what's the alternative? Never change anything?

Of course not. But the best course forward is going to be different for
different folks. What might work best for me might not (probably WILL
not) work best for everyone else. One has to look at their situation
and plan the best path for their business with their architecture and
the resources they have available to them. I suggested one option but
that might not work for others. Others might see a strict white
listing, or maybe some combination of the two. But there is so much
brokenness out there right now that I would hesitate to trust an AAAA
request that arrives over v4 when there is a v6 name server available.

Remember, this is al extremely trivial stuff: most things won't even
completely stop working. And a few mouseclicks (yes, you have to know
which ones so the helpdesks better start figuring that out) and you're
back to normal. Compare this to turning off analog TV transmitters

that

have been running for decades where people have to buy converter boxes
and sometimes even install antennas on their roof to keep using the
service.

It depends. There are other things to take into account. If you
increase the time it takes a mobile device to complete a transaction by
only a couple of seconds, if you multiply those couple of seconds by
all of the users in a large metro area, you end up with devices
increased use of network resources (and increased battery drain on the
devices themselves). Anything that can be done to speed transactions up
and get those transmitters shut off as quickly as possible is a win. If
you don't have a lot of mobile clients hitting your site, then maybe
that isn't a problem. Every network has their own set of resources and
their own set of challenges and all of that has to fit within the
network architecture they have deployed and their business model.

Basically, there is no "magic bullet".

It depends. There are other things to take into account. If you
increase the time it takes a mobile device to complete a transaction by
only a couple of seconds, if you multiply those couple of seconds by
all of the users in a large metro area, you end up with devices
increased use of network resources (and increased battery drain on the
devices themselves). Anything that can be done to speed transactions up
and get those transmitters shut off as quickly as possible is a win. If
you don't have a lot of mobile clients hitting your site, then maybe
that isn't a problem. Every network has their own set of resources and
their own set of challenges and all of that has to fit within the
network architecture they have deployed and their business model.

So in our environment reducing the load time on an application by a
couple seconds nets out to several human lifetimes a month, so people
count seconds and fractions of seconds like they're precious.

Basically, there is no "magic bullet".

indeed, it has to be applied systemically.

I agree that seconds sometimes matters, but the latency of a transaction
doesn't have a linear relationship with radio or battery usage on a mobile
device. Because of the timers involved in the state transitions (eg
CELL_FACH -> CELL_DCH), a few seconds of extra latency often is
inconsequential because there is a minimum duration for which the radio will
stay awake anyways. Coalescing techniques like Android's setInexactRepeating
method of the Alarm Manager also optimize radio access across multiple apps.

And if I'm not mistaken, it's the transition to/from CELL_DCH which is the
most expensive resource-wise for network operators, not the duration of
keeping this state.

The argument that IPv6-induced latency is going to affect mobile devices
disproportionally doesn't seem especially compelling.

-Nick

I agree that seconds sometimes matters, but the latency of a transaction
doesn't have a linear relationship with radio or battery usage on a
mobile device. Because of the timers involved in the state transitions
(eg CELL_FACH -> CELL_DCH), a few seconds of extra latency often is
inconsequential because there is a minimum duration for which the radio
will stay awake anyways. Coalescing techniques like Android's
setInexactRepeating method of the Alarm Manager also optimize radio
access across multiple apps.

Not every device out there is an android. Not every OS on every device
handles connections the same way. Problems can compound if several
different names must be looked up in order to get a complete page view.
Are your images served from a different name? Do you have short TTLs
that require names to be looked up frequently? Again, every network is
going to have their own unique sets of issues. But until there are more
eyeballs out there that are native v6, we aren't going to see a lot of
movement.

I find it strange that you approach this issue as one of the great questions of our time. If you don't want to enable IPv6 for your service at this time, then don't enable IPv6 for your service at this time. But you'll have to do it at some point, so doing it together with your competitors and/or big players seems like a good choice. Going through huge lengths to optimize for a problem that will only exist for a couple of years or so doesn't make sense to me. Also, all this special case logic has a nasty tendency to create all kinds of unexpected problems down the road. I'm sure that the people at Microsoft thought it was a swell idea to enable 6to4 by default. If they hadn't done that, they'd saved us all a lot of wasted time.