Yahoo and IPv6

Mailman List Mirror (Read Only)
1

From: Valdis.Kletnieks@vt.edu
Date: Mon, 09 May 2011 12:25:55 -0400

> Actually, I have just noticed a slightly more disturbing thing on the Yahoo
> IPv6 help page...
>
> I have IPv6 connectivity through a HE tunnel, and I can reach IPv6 services
> (the only issue is that my ISP's DNS is not IPv6 enabled), but I tried to
> run the "Start IPv6 Test" tool at http://help.yahoo.com/l/us/yahoo/ipv6/ and
> it says:
> "We detected an issue with your IPv6 configuration. On World IPv6 Day, you
> will have issues reaching Yahoo!, as well as your other favorite web sites.

The *really* depressing part is that it says the same thing for me, on a *known*
working IPv6 network.

And then when I retry it a few minutes later, with a tcpdump running, it works.

And then another try says it failed, though tcpdump shows it seems to work.

For what it's worth, the attempted download file is:

% wget http://v6test.yahoo.com/eng/test/eye-test.png
--2011-05-09 11:44:39-- http://v6test.yahoo.com/eng/test/eye-test.png
Resolving v6test.yahoo.com... 2001:4998:f00d:1fe::2000, 2001:4998:f00d:1fe::2002, 2001:4998:f00d:1fe::2003, ...
Connecting to v6test.yahoo.com|2001:4998:f00d:1fe::2000|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [image/png]
Saving to: `eye-test.png.1'

    [ <=> ] 2,086 --.-K/s in 0s

2011-05-09 11:44:39 (154 MB/s) - `eye-test.png.1' saved [2086]

Looking at the Javascript that drives the test, it appears the *real* problem
is that they set a 3 second timeout on the download - which basically means
that if you have to retransmit either the DNS query or the TCP SYN, you're
dead as far as the test is concerned.

I have talked to Yahoo engineers about this and they say that their
testing has shown that, if it takes more than 3 seconds for a site to
load, they start to lose significant traffic. Hence the 3 second
timeout.

Sadly, I'm afraid that they have a point, but at the same time I suspect
that they are assuring failure for almost everyone. A 5 second timeout
would probably be more reasonable, but is probably unacceptable to Yahoo
management.

2

I have done some other 'observational' looks at some IPv6 related data recently that others may have seen on ipv6-ops.

A few notes:

1) Somewhere 0.5-0.8% of sites in a list of domains (about 1 million 'top' sites) have some form of broken DNS

2) Some DNS providers (eg: OpenDNS, Google, Comcast, and those that run ISC-BIND) have varying responses with these queries. The one I find most interesting is OpenDNS, they seem to never take more than 1 second for a dns query. Seems to stick within this 3-5 second overall load rule. I've not detailed what nameservers have different operations, but BIND is certainly most likely to return a SERVFAIL while others return NOERROR. It appears BIND is just more strict about enforcing strict cname -> cname mapping with proper SOA. You can see this all over bind-users.

There seems to be some other interesting data that could be inferred regarding the sites.

I do want to present this data and some details regarding IPv6 day and our observations at the upcoming NANOG meeting, but not sure I'm going to have it all together. If you are on the PC, expect a lightning talk from me :slight_smile:

I do feel the bar that Yahoo is setting is too high. There are a lot of network elements that are broken, either DNS servers, home 'gateway/nat' devices, or other elements in the delegation chain. This leaves out any of the network elements of the packet forwarding path that may be suboptimal. While not directly comparable as one is the CPE side vs Content side, if 0.6% of sites are unreachable from a BIND resolver on a properly IPv6 enabled network, the number of sites that will appear broken will be high in aggregate. These folks need to fix their problems, 6714 of the 1 million sites are broken. If you are talking about 6714 people that are going to place a helpdesk call that day, I hope everyone is ready to work their phones. I think this is the point of Yahoo, but if nobody fixes it, it will just be permanently broken. If that's the case, it should be addressed vs papered over by not serving up for the remainder of the 99.4% that are properly maintained.

While 2 9's is not that great, aiming for 5 9's is a goal, not something I feel is realistic in the next 24 months.

- Jared

3

Publicly held corporations are responsible to their shareholders to get eyeballs on their content. *That* is their job, not promoting cool new network tech. When you have millions of users hitting your site every day losing 1/2000 is a large chunk of revenue. The fact that the big players are doing world IPv6 day at all should be celebrated, promoted, and we should all be ready to take to heart the lessons learned from it.

The content providers are not to be blamed for the giant mess that IPv6 deployment has become. If 6to4 and Teredo had never happened, in all likelihood we wouldn't be in this situation today.

4

From: Doug Barton [mailto:dougb@dougbarton.us]
Sent: Monday, May 09, 2011 12:11 PM
To: Jared Mauch
Cc: nanog@nanog.org; Arie Vayner
Subject: Re: Yahoo and IPv6

> I do feel the bar that Yahoo is setting is too high. There are a lot
of network elements that are broken, either DNS servers, home
'gateway/nat' devices, or other elements in the delegation chain.

Publicly held corporations are responsible to their shareholders to get
eyeballs on their content. *That* is their job, not promoting cool new
network tech. When you have millions of users hitting your site every
day losing 1/2000 is a large chunk of revenue. The fact that the big
players are doing world IPv6 day at all should be celebrated, promoted,
and we should all be ready to take to heart the lessons learned from
it.

The content providers are not to be blamed for the giant mess that IPv6
deployment has become. If 6to4 and Teredo had never happened, in all
likelihood we wouldn't be in this situation today.

Which situation ??? The one where the content can demonstrate how broken the
networks really are? Or the one where the content sites are exposed for
their lack of prior planning?

The entire point of those technologies you are complaining about was to
break the stalemate between content and network, because both sides will
always wait and blame the other. The fact that the content side chose to
wait until the last possible minute to start is where the approach falls
down. Expecting magic to cover for lack of proactive effort 5-10 years ago
is asking a bit much, even for the content mafia.

In any case, the content side can mitigate all of the latency related issues
they complain about in 6to4 by putting in a local 6to4 router and publishing
the corresponding 2002:: prefix based address in DNS for their content. They
choose to hold their breath and turn blue, blaming the network for the lack
of 5-9's access to the eyeballs when they hold at least part of a solution
in their own hands.

We are about the witness the most expensive, complex, blame-fest of a
transition that one could have imagined 10 years ago. This is simply due to
the lack of up-front effort that both sides have demonstrated in getting to
this point. Now that time has expired, all that is left to do is sit back
and watch the fireworks.

Tony

5

From: Doug Barton [mailto:dougb@dougbarton.us]
Sent: Monday, May 09, 2011 12:11 PM
To: Jared Mauch
Cc: nanog@nanog.org; Arie Vayner
Subject: Re: Yahoo and IPv6

I do feel the bar that Yahoo is setting is too high. There are a lot

of network elements that are broken, either DNS servers, home
'gateway/nat' devices, or other elements in the delegation chain.

Publicly held corporations are responsible to their shareholders to get
eyeballs on their content. *That* is their job, not promoting cool new
network tech. When you have millions of users hitting your site every
day losing 1/2000 is a large chunk of revenue. The fact that the big
players are doing world IPv6 day at all should be celebrated, promoted,
and we should all be ready to take to heart the lessons learned from
it.

The content providers are not to be blamed for the giant mess that IPv6
deployment has become. If 6to4 and Teredo had never happened, in all
likelihood we wouldn't be in this situation today.

Which situation ??? The one where the content can demonstrate how broken the
networks really are? Or the one where the content sites are exposed for
their lack of prior planning?

I disagree with your attempt to scope the problem. :slight_smile:

The entire point of those technologies you are complaining about was to
break the stalemate between content and network,

I also disagree with this statement, but there is very little point in arguing about it at this stage.

because both sides will
always wait and blame the other. The fact that the content side chose to
wait until the last possible minute to start is where the approach falls
down. Expecting magic to cover for lack of proactive effort 5-10 years ago
is asking a bit much, even for the content mafia.

One could also argue that the fact that the IPv6 protocol is still not fully mature, and that the IPv6 intelligentsia are only recently coming to the point where they are willing to give both the content and eyeball networks what they've been asking for all along (PI and robust DHCPv6 being top of the respective lists) has led to the situation we're in now. Of course the truth is probably somewhere in the middle, but it's definitely not all on one side.

In any case, the content side can mitigate all of the latency related issues
they complain about in 6to4 by putting in a local 6to4 router and publishing
the corresponding 2002:: prefix based address in DNS for their content. They
choose to hold their breath and turn blue, blaming the network for the lack
of 5-9's access to the eyeballs when they hold at least part of a solution
in their own hands.

Looking at that from the content provider side for a second, what is their motivation for doing it? The IETF created 6to4, and some foolish OS and/or hardware vendors enabled it by default. So you're saying that it's up to the content providers to spend money to fix a problem they didn't create, when the easy/free solution is simply not to turn on IPv6 at all? I completely fail to see an incentive for the content providers to do this, but maybe I'm missing something.

And can we please stop pretending that this is an easy thing for the content providers to do? Big content networks like Yahoo! have dozens of POPs, and hundreds of address ranges. The IETF is *still* working on tweaking 6to4, so even if the content providers put up these relays today, and somehow figure out how to test them, their work is not done.

I do agree with you that pointing fingers at this stage is really not helpful. I continue to maintain that being supportive of those content networks that are willing to wade in is the right answer.

Doug (mafia? seriously?)

6

Frankly, I think the finger is simply pointing in the wrong direction.
I have zero choices for native IPv6 at home, and I'm sure that is
true for the majority of us. SOHO CPE support barely exists because
access networks haven't been asking for it. Call centers are
certainly not equipped to evaluate "traceroute tickets" or assist
users in any practical way, which is why we see "disable IPv6 and try
again" as the cookie-cutter answer to any problem when the end-user
has IPv6.

The expectation that content providers should rush to publish AAAA
records by default (instead of white-listing, etc.) at a time when
even motivated end-users can't get IPv6 without resorting to tunnels
is ridiculous. Let's be glad that these content providers have done
all the necessary prep work, such as deploying appropriate network
infrastructure and updating their software, so that they can choose to
send AAAA responses when they want to.

This problem is, and always has been, on the access side. Point your
fingers that way.

7

I do agree with you that pointing fingers at this stage is really not
helpful. I continue to maintain that being supportive of those content
networks that are willing to wade in is the right answer.

[...]

This problem is, and always has been, on the access side. Point your
fingers that way.

While I agree with Jeff, I agree with Doug more. Unfortunately, finger-pointing will not fix the problem. We have identified many of the problems, and hopefully June 8 will shine a very bright light on any that are left. Let's work on fixing the problems, and let the historians figure out whose "fault" it was.

8

<aol>+1</aol>

I think we're in a stage where the access networks are playing catch-up.

The CPE marketplace is going to see some significant growth in sales in the short term as well.

I'll say that enabling IPv6 in the datacenter and the core is "easy". Any modern hardware worth the cash you spend on it does native IPv6. (If the hardware does not, eg: firewalls, etc, please re-read the prior statement).

Putting IPv6 in a datacenter or on a lan is easy. You put a /64 there, let the host use SLAAC and let the RA magic work. You talk to the nameserver over your dual-stack (IPv4) side and it's all set.

While there are concerns about people sending bogus RA's and other things, the same is true for anyone putting the router IP address on the main ethernet of a server too. These all cause problems.

The routing protocols are all there, either with ISIS or OSPFv3. BGP is there too. You can even skip over non-IPv6 enabled nodes by doing 6PE if you have MPLS in your network.

I'm hopeful that nobody will see the problems out there on IPv6 day. They've been dealt with in the 99%+ of the cases already.

The fact that we're talking about something past the decimal point is good IMHO. I've observed that the top-million sites can't get it better than 99.4% right for a DNS query. I can break that out by the top-10, 100, 1k, 10k, 100k and show a graph if that's useful. I think it's "good enough". I'd like to call out those with broken network elements and suggest they fix them.

I'd like to have native IPv6 at home, but my provider is not ready yet. "Soon" is my hope.

The fact that it's there in many other places means it's possible. I'd like to see more progress getting there than finger pointing. I expect the next 30 days to be a lot of fun getting to IPv6 day, and it to be far less eventful than we worry it will be.

- Jared

9

Actually, finger-pointing is very helpful at this stage. I was able
to change my local ISP's tune from "we have enough IPv4 addresses for
our customers, so we aren't going to support IPv6" (ever) to "we will
start employee beta testing soon." It ultimately took the threat of
running an Op-Ed in the business section of the local paper to get
them to realize they can't continue with their plan to offer no IPv6
support at all.

With 800,000 SOHO CPE units deployed that have no IPv6 support and no
remote firmware upgrade option on the horizon, I can understand why
they hoped they could avoid ever supporting v6 -- it will cost them,
literally, a hundred million dollars to fix their CPE situation and
deploy native IPv6 if their CPE vendor can't provide a remote update.
This is also why tunneled solutions are receiving so much effort and
attention -- truck rolls and CPE replacement are huge expenses.

If we don't start pointing fingers at these access networks, they
won't "get it" until the pain of IPv4 depletion lands squarely on
content networks who may eventually be unable to get any IPv4
addresses for their services, or who may be forced to buy transit from
networks who have large, legacy IPv4 pools sitting around just to get
a provider allocation.

10

I would, too; but one harsh reality is that vendors are driven by
RFPs, not by what they consciously know their customers will need in
the near future. Why should vendors invest money in features that
aren't needed to sell routers? If customers are dumb enough to buy
them anyway, they'll buy *another* router to get those features in the
future.

I do take issue with your suggestion that /64 LANs are in any way
smart in the datacenter. They are not. I have some slides on this
topic: http://inconcepts.biz/~jsw/IPv6_NDP_Exhaustion.pdf

11

I do take issue with your suggestion that /64 LANs are in any way

smart in the datacenter. They are not. I have some slides on this
topic: http://inconcepts.biz/~jsw/IPv6_NDP_Exhaustion.pdf

There are ways of mitigating this (the easiest is to use ACLs or firewalls
to limit traffic into a subnet from untrusted sources so that only
legitimate traffic is allowed).

As for IPv6 in general, for other posters, I have a lot more concern about
things like missing routes in routing tables, lack of residential IPv6 (one
place where it would be *very* useful - think VoIP, P2P, etc), and lack of
any good tunnel broker options.

I've also had plenty of trouble getting IPv6 in data centers (4 different
providers of caged data center space, none of which provided it, including
one Tier 1 who advertised that all their business customers got free IPv6
with IPv4 transit from them; I guess they didn't think someone renting caged
space and redundant ethernet in one of their data centers was a business
customer, but I digress...)

I'd settle for a good tunnel at this point.

What do I mean by good tunnel option? The following:

1) Provider treats it like production, not as a tool for business leverage
or a service only used for "testing"

2) Provider that has full routing table. A provider couldn't stay in
business in the IPv4 world if they lacked connectivity to one or more
default free networks. Yet in IPv6, it's the norm.

3) Provider that provides support for it - first through top level

4) Provider has redundancy at all levels

5) Provider makes it quick and simple to sign up, with rates posted on their
website based on bandwidth desired (for residential and small business
customers). I don't want to talk to a sales guy if I'm just setting up a
tunnel for a DSL site!

6) Provider has an access location somewhere within, say, 1000 miles of my
location. Ideally at a major exchange point in the metro.

7) Provider's network doesn't route me through Tokyo or Frankfurt to get
from Denver to Chicago - regardless of who's network in Chicago I'm trying
to connect to.

This doesn't exist - it's wishful thinking. So this leaves native
connectivity. Of course native connectivity is problematic, as it doesn't
always come with full routing tables, isn't necessarily available on the
circuit you have, and doesn't really give you all that much. That's
assuming you can find it at all. After all, it's only been around for most
of the time of the web.

12

From: Doug Barton [mailto:dougb@dougbarton.us]
Sent: Monday, May 09, 2011 12:11 PM
To: Jared Mauch
Cc: nanog@nanog.org; Arie Vayner
Subject: Re: Yahoo and IPv6

I do feel the bar that Yahoo is setting is too high. There are a lot

of network elements that are broken, either DNS servers, home
'gateway/nat' devices, or other elements in the delegation chain.

Publicly held corporations are responsible to their shareholders to get
eyeballs on their content. *That* is their job, not promoting cool new
network tech. When you have millions of users hitting your site every
day losing 1/2000 is a large chunk of revenue. The fact that the big
players are doing world IPv6 day at all should be celebrated, promoted,
and we should all be ready to take to heart the lessons learned from
it.

The content providers are not to be blamed for the giant mess that IPv6
deployment has become. If 6to4 and Teredo had never happened, in all
likelihood we wouldn't be in this situation today.

Which situation ??? The one where the content can demonstrate how broken the
networks really are? Or the one where the content sites are exposed for
their lack of prior planning?

I disagree with your attempt to scope the problem. :slight_smile:

The entire point of those technologies you are complaining about was to
break the stalemate between content and network,

I also disagree with this statement, but there is very little point in arguing about it at this stage.

because both sides will
always wait and blame the other. The fact that the content side chose to
wait until the last possible minute to start is where the approach falls
down. Expecting magic to cover for lack of proactive effort 5-10 years ago
is asking a bit much, even for the content mafia.

One could also argue that the fact that the IPv6 protocol is still not fully mature, and that the IPv6 intelligentsia are only recently coming to the point where they are willing to give both the content and eyeball networks what they've been asking for all along (PI and robust DHCPv6 being top of the respective lists) has led to the situation we're in now. Of course the truth is probably somewhere in the middle, but it's definitely not all on one side.

PI was granted at least in the ARIN region in since August 30, 2006. I know, I wrote
the original policy proposal (2005-1 which was later merged with proposals from
Andrew Dul and Kevin Loch based on a cooperative effort among the three of us).

I doubt that DHCPv6 was standing in the way of content providers.

In any case, the content side can mitigate all of the latency related issues
they complain about in 6to4 by putting in a local 6to4 router and publishing
the corresponding 2002:: prefix based address in DNS for their content. They
choose to hold their breath and turn blue, blaming the network for the lack
of 5-9's access to the eyeballs when they hold at least part of a solution
in their own hands.

Looking at that from the content provider side for a second, what is their motivation for doing it? The IETF created 6to4, and some foolish OS and/or hardware vendors enabled it by default. So you're saying that it's up to the content providers to spend money to fix a problem they didn't create, when the easy/free solution is simply not to turn on IPv6 at all? I completely fail to see an incentive for the content providers to do this, but maybe I'm missing something.

While we're not directly a content provider, we do host several of them and we do
run the largest network of 6to4 relays that I am aware of. In our experience at HE,
this has dramatically improved the IPv6 experience for our clients. As such, I would
think that providing a better user experience should serve as reasonable motivation
for any rational content provider. It's not like running 6to4 relays is difficult or
expensive.

And can we please stop pretending that this is an easy thing for the content providers to do? Big content networks like Yahoo! have dozens of POPs, and hundreds of address ranges. The IETF is *still* working on tweaking 6to4, so even if the content providers put up these relays today, and somehow figure out how to test them, their work is not done.

It is relatively easy to do, even with dozens of POPs. There isn't anything special you
have to do for the hundreds of address ranges, really, so I don't buy that as being a
meaningful part of the argument.

I do agree with you that pointing fingers at this stage is really not helpful. I continue to maintain that being supportive of those content networks that are willing to wade in is the right answer.

Agreed, but, it's also important to point out when they're starting to swim in directions
that are counterproductive, such as having help sites that advise users to turn off
IPv6 with fixing their IPv6 capabilities as a secondary option.

Owen

13

Your suggestion has two main disadvantages:
1) it doesn't work on some platforms, because input ACL won't stop ND
learn/solicit -- obviously this is bad
2) it requires you to configure a potentially large input ACL on every
single interface on the box, and adjust that ACL whenever you
provision more IPv6 addresses for end-hosts -- kinda like not having a
control-plane filter, only worse

14

:: >> In any case, the content side can mitigate all of the latency related issues
:: >> they complain about in 6to4 by putting in a local 6to4 router and publishing
:: >> the corresponding 2002:: prefix based address in DNS for their content. They
:: >> choose to hold their breath and turn blue, blaming the network for the lack
:: >> of 5-9's access to the eyeballs when they hold at least part of a solution
:: >> in their own hands.
:: >
:: > Looking at that from the content provider side for a second, what is their motivation for doing it? The IETF created 6to4, and some foolish OS and/or hardware vendors enabled it by default. So you're saying that it's up to the content providers to spend money to fix a problem they didn't create, when the easy/free solution is simply not to turn on IPv6 at all? I completely fail to see an incentive for the content providers to do this, but maybe I'm missing something.
:: >

So, just for the record, I am not speaking for my employer, and am
speaking strictly for myself here, and I'm going to try to keep this my
one and only message about finger-pointing :slight_smile:

The time for finger-pointing is over, period, all we are all trying to do
now is figure out how to deal with the present (sucky) situation. The
current reality is that for a non-insignificant percentage of users when
you enable dual-stack, they are gong to drop off the face of the planet.
Now, for *you*, 0.026% may be insignificant (and, standalone, that number
is insignificant), but for a global content provider that has ~700M users,
that's 182 *thousand* users that *you*, *through your actions* just took
out.. 182,000 - that is *not* insignificant

*That* is what world ipv6 day is about to me -- getting enough attention
at the problem so that all of us can try to move the needle in the right
direction. If enough users realize that they are broken, and end up
"fixing themselves", then it will be a resounding success. And, yes, to
me, disabling broken ipv6 *is* "fixing themselves". If they turn broken
ipv6 into working ipv6, even better, I just hope all the access networks
staffed up their helpdesk to deal with the call volumes..

And, if the breakage stats remain bad, well, that's what DNS
"whitelists/blacklists" are going to be for..

:: While we're not directly a content provider, we do host several of them and we do
:: run the largest network of 6to4 relays that I am aware of. In our experience at HE,
:: this has dramatically improved the IPv6 experience for our clients. As such, I would
:: think that providing a better user experience should serve as reasonable motivation
:: for any rational content provider. It's not like running 6to4 relays is difficult or
:: expensive.

No, running *return* 6to4 relays is not difficult at all, in fact, some
content providers have a ton of them up right now. The problem is that
content providers can't control the forward relays, or protocol 41
filtering that's out in the wild. Also, not all breakage is caused by
6to4, there are still quite a few cases of other breakage, and *that* is
what's pushing us towards whitelisting.

See: http://www.getipv6.info/index.php/Customer_problems_that_could_occur

:: > And can we please stop pretending that this is an easy thing for the content providers to do? Big content networks like Yahoo! have dozens of POPs, and hundreds of address ranges. The IETF is *still* working on tweaking 6to4, so even if the content providers put up these relays today, and somehow figure out how to test them, their work is not done.
:: >
:: It is relatively easy to do, even with dozens of POPs. There isn't anything special you
:: have to do for the hundreds of address ranges, really, so I don't buy that as being a
:: meaningful part of the argument.

I think this is a red herring - return relays were never *the* problem.

:: > I do agree with you that pointing fingers at this stage is really not helpful. I continue to maintain that being supportive of those content networks that are willing to wade in is the right answer.
:: >
:: Agreed, but, it's also important to point out when they're starting to swim in directions
:: that are counterproductive, such as having help sites that advise users to turn off
:: IPv6 with fixing their IPv6 capabilities as a secondary option.

"We recommend disabling IPv6 or seeking assistance in order to fix your
system's IPv6 configuration through your ISP or computer manufacturer"

So, your problem is that a help page gives the user 2 options,
the first one of them being a quick and easy fix that a user can do
himself in less then a minute, and suggesting contacting the ISP or
manufacturer *second* (and possibly spending quite a bit of time on
hold/troubleshooting, and then saying "screw it")?!?

Honestly, I think the people who want ipv6 to work, and are willing and
capable to troubleshoot it, will; and those who don't will just
turn it off... Seems like the right outcome to me..

-igor
(man, did I pick a good day to be on an airplane, or what)

15

:: > I do agree with you that pointing fingers at this stage is really not helpful. I continue to maintain that being supportive of those content networks that are willing to wade in is the right answer.
:: >
:: Agreed, but, it's also important to point out when they're starting to swim in directions
:: that are counterproductive, such as having help sites that advise users to turn off
:: IPv6 with fixing their IPv6 capabilities as a secondary option.

"We recommend disabling IPv6 or seeking assistance in order to fix your
system's IPv6 configuration through your ISP or computer manufacturer"

So, your problem is that a help page gives the user 2 options,
the first one of them being a quick and easy fix that a user can do
himself in less then a minute, and suggesting contacting the ISP or
manufacturer *second* (and possibly spending quite a bit of time on
hold/troubleshooting, and then saying "screw it")?!?

Vs. other more useful options which I have spelled out elsewhere in this
thread, yes.

Honestly, I think the people who want ipv6 to work, and are willing and
capable to troubleshoot it, will; and those who don't will just
turn it off... Seems like the right outcome to me..

We can agree to disagree. Turning it off really isn't a good outcome because
it just postpones the inevitable. Encouraging people to call their ISPs to
troubleshoot their IPv6 problems accomplishes two things:

  1. It raises visibility of the need for IPv6 at the eyeball ISPs. It shows
    that there are users encountering things that cause them to care
    about IPv6 working.

  2. It helps users resolve their IPv6 problems and get working
    IPv6.

I applaud your employer's efforts to get IPv6 deployed and their leadership
in working towards IPv6 day. Hopefully they can eventually take a more
positive leadership position towards successful eyeball transitions as
well.

Owen

16

Publicly held corporations are responsible to their shareholders to get
eyeballs on their content. *That* is their job, not promoting cool new
network tech. When you have millions of users hitting your site every
day losing 1/2000 is a large chunk of revenue.

Nonsense. 0.05% is well below the noise margin for anything that involves humans.

The fact that the big
players are doing world IPv6 day at all should be celebrated, promoted,
and we should all be ready to take to heart the lessons learned from
it.

I applaud the first step, but I'm bothered by the fact that no second step is planned.

The content providers are not to be blamed for the giant mess that IPv6
deployment has become. If 6to4 and Teredo had never happened, in all
likelihood we wouldn't be in this situation today.

The entire point of those technologies you are complaining about was to
break the stalemate between content and network, because both sides will
always wait and blame the other.

You're both somewhat right: there's nothing wrong with having 6to4 and Teredo available as an option for people who want/need easy IPv6, which is too hard to get otherwise for most people. The big mistake was to enable it by default. That ALWAYS ends badly. (See for instance HTTP pipelining, good idea but it got tainted by buggy implementations on the client side that made it impossible to enable on the server side.)

The fact that the content side chose to
wait until the last possible minute to start is where the approach falls
down. Expecting magic to cover for lack of proactive effort 5-10 years ago
is asking a bit much, even for the content mafia.

The content people don't feel the address crunch and they have no incremental deployment: either you AAAA or you don't AAAA. The opposite is true for the eyeball people, so they are the ones that will have to get this ball rolling.

In any case, the content side can mitigate all of the latency related issues
they complain about in 6to4 by putting in a local 6to4 router and publishing
the corresponding 2002:: prefix based address in DNS for their content.

That wouldn't help people behind firewalls that block protocol 41 (which is way too common) and it's harmful to those with non-6to4 connectivity but no (good) RFC 3484 support so they connect to those 2002:: addresses. (I'm looking at you, MacOS. Try for yourself here: http://6to4test3.runningipv6.net/ )

We are about the witness the most expensive, complex, blame-fest of a
transition that one could have imagined 10 years ago. This is simply due to
the lack of up-front effort that both sides have demonstrated in getting to
this point. Now that time has expired, all that is left to do is sit back
and watch the fireworks.

I love fireworks.

I don't think it'll be all that bad, though. Pretty much all the pieces are in place now, it's mostly a question of simply enabling IPv6. Yes, people will whine but how else would we know the NANOG list is still working between operational issues?

17

I think it will be interesting when people start to look at the results. Following the delegation of someplace like a bank that has a financial interest in

a) security (ie: modern software)
b) people reaching their site

There's a lot of IPv6 brokeness in their services.

do "dig +trace aaaa www.citibank.co.uk"

You will eventually reach their load balancer dns servers that start giving out bad referrals/authority.

www.citibank.co.uk. 3600 IN NS ldefdc-egsl01-7000.nsroot2.com.
www.citibank.co.uk. 3600 IN NS lgbrdc-egsl01-7000.nsroot1.com.
;; Received 153 bytes from 192.193.214.2#53(192.193.214.2) in 36 ms

[trimmed]
. 3600000 IN NS m.root-servers.net.
;; BAD REFERRAL
;; Received 500 bytes from 199.67.203.246#53(199.67.203.246) in 100 ms

When you look at the top "25" broken sites, it quickly starts to look like something interesting. The temporary failure shows some error in the resolver library looking for an AAAA record. If you ask a non-bind nameserver you may have better luck as they seem to have relaxed SOA tracking.

www.capitalone.com.|208.80.48.112|OK|Temporary failure in name resolution
www.priceline.com.|64.6.17.1|OK|Temporary failure in name resolution
www.kitco.com.|66.38.218.33|OK|Temporary failure in name resolution
www.dmm.co.jp.|203.209.147.15|OK|Temporary failure in name resolution
www.lg.com.|174.35.24.66,174.35.24.81|OK|Temporary failure in name resolution
www.theweathernetwork.com.|207.96.160.181|OK|Temporary failure in name resolution
www.ovguide.com.|64.94.88.21|OK|Temporary failure in name resolution
www.alipay.com.|110.75.132.21|OK|Temporary failure in name resolution
www.sznews.com.|210.21.197.161|OK|Temporary failure in name resolution
www.ryanair.com.|193.95.148.90|OK|Temporary failure in name resolution
www.kbb.com.|209.67.183.100|OK|Temporary failure in name resolution
www.royalbank.com.|142.245.1.203|OK|Temporary failure in name resolution
www.opentable.com.|66.151.130.32|OK|Temporary failure in name resolution
www.bookryanair.com.|193.95.148.91|OK|Temporary failure in name resolution
aleadpay.com.|121.14.17.41|OK|Temporary failure in name resolution
www.20minutos.es.|85.62.13.190|OK|Temporary failure in name resolution
www.nzherald.co.nz.|184.154.158.58|OK|Temporary failure in name resolution
www.rbcroyalbank.com.|142.245.1.15|OK|Temporary failure in name resolution
www.hangzhou.com.cn.|218.108.127.43|OK|Temporary failure in name resolution
www.klikbca.com.|202.6.208.8|OK|Temporary failure in name resolution
www.uk.to.|195.144.11.40|OK|Temporary failure in name resolution
www.atdmt.com.|65.203.229.39,65.242.27.40|OK|Temporary failure in name resolution
www.hc360.com.|221.233.134.141,221.233.134.143|OK|Temporary failure in name resolution
www.dmm.com.|203.209.147.53|OK|Temporary failure in name resolution
www.businesswire.com.|204.8.173.52|OK|Temporary failure in name resolution

Aside from the above, it does seem that there are a fair number of sites that have enabled IPv6 and gone without notice.

take www.informationweek.com which (from my view) sits behind AS209 with their IPv6 space, very similar to their v4 address.

I'm optimistic that more people will 'just enable' ipv6. Hopefully other technical websites will do it as well, perhaps anyone that matches a regex of "ars" can influence the powers that be. If they can get people to disable adblock, maybe they can serve up some AAAA as well. :slight_smile:

- Jared

18

Igor Gashinsky wrote:

:: >> In any case, the content side can mitigate all of the latency
related issues
:: >> they complain about in 6to4 by putting in a local 6to4 router and
publishing
:: >> the corresponding 2002:: prefix based address in DNS for their
content. They
:: >> choose to hold their breath and turn blue, blaming the network
for the lack
:: >> of 5-9's access to the eyeballs when they hold at least part of a
solution
:: >> in their own hands.
:: >
:: > Looking at that from the content provider side for a second, what
is their motivation for doing it? The IETF created 6to4, and some
foolish OS and/or hardware vendors enabled it by default. So you're
saying that it's up to the content providers to spend money to fix a
problem they didn't create, when the easy/free solution is simply not
to turn on IPv6 at all? I completely fail to see an incentive for the
content providers to do this, but maybe I'm missing something.
:: >

So, just for the record, I am not speaking for my employer, and am
speaking strictly for myself here, and I'm going to try to keep this my
one and only message about finger-pointing :slight_smile:

The time for finger-pointing is over, period, all we are all trying to
do
now is figure out how to deal with the present (sucky) situation. The
current reality is that for a non-insignificant percentage of users
when
you enable dual-stack, they are gong to drop off the face of the
planet.
Now, for *you*, 0.026% may be insignificant (and, standalone, that
number
is insignificant), but for a global content provider that has ~700M
users,
that's 182 *thousand* users that *you*, *through your actions* just
took
out.. 182,000 - that is *not* insignificant

*That* is what world ipv6 day is about to me -- getting enough
attention
at the problem so that all of us can try to move the needle in the
right
direction. If enough users realize that they are broken, and end up
"fixing themselves", then it will be a resounding success. And, yes, to
me, disabling broken ipv6 *is* "fixing themselves". If they turn broken
ipv6 into working ipv6, even better, I just hope all the access
networks
staffed up their helpdesk to deal with the call volumes..

And, if the breakage stats remain bad, well, that's what DNS
"whitelists/blacklists" are going to be for..

:: While we're not directly a content provider, we do host several of
them and we do
:: run the largest network of 6to4 relays that I am aware of. In our
experience at HE,
:: this has dramatically improved the IPv6 experience for our clients.
As such, I would
:: think that providing a better user experience should serve as
reasonable motivation
:: for any rational content provider. It's not like running 6to4 relays
is difficult or
:: expensive.

No, running *return* 6to4 relays is not difficult at all, in fact, some
content providers have a ton of them up right now. The problem is that
content providers can't control the forward relays,

So take the relays out of the path by putting up a 6to4 router and a 2002::
prefix address on the content servers. Longest match will cause 6to4
connected systems to prefer that prefix while native connected systems will
prefer the current prefix. The resulting IPv4 path will be exactly what it
is today door-to-door. Forcing traffic through a third party by holding to a
purity principle for dns, and then complaining about the results is not
exactly the most productive thing one could do.

or protocol 41
filtering that's out in the wild.

Putting 2002:: in dns will not fix this, but it is not clear to me where
this comes from. The argument is that enterprise firewalls are blocking it,
but that makes no sense because many/most enterprises are in 1918 space so
6to4 will not be attempted to begin with, and for those that have public
space internally the oft-cited systems that are domain members will have
6to4 off by default. To get them to turn it on would require the IT staff to
explicitly enable it for the end systems but then turn around and block it
at the firewall ... Not exactly a likely scenario.

The most likely source of public space for non-domain joined systems would
be universities, but no one that is complaining about protocol 41 filtering
has shown that the source addresses are coming from those easily
identifiable places.

That leaves the case of networks that use public addresses internally, but
nat those at the border. This would confuse the client into thinking 6to4
should be viable, only to have protocol 41 blocked by the nat. These
networks do exist, and the only way to detect them would be to have an
instrumented 6to4 router or relay that compared the IPv4-bits in the source
address between the two headers. They don't have to match exactly because a
6to4 router would use its address as a source, but if the embedded bits said
25.25.25.25 while the external IPv4 header said 18.25.25.25 one might
suspect there was a nat in the path.

Also, not all breakage is caused by
6to4, there are still quite a few cases of other breakage, and *that*
is
what's pushing us towards whitelisting.

See:
ARIN IPv6 Wiki - ARIN's Vault

:: > And can we please stop pretending that this is an easy thing for
the content providers to do? Big content networks like Yahoo! have
dozens of POPs, and hundreds of address ranges. The IETF is *still*
working on tweaking 6to4, so even if the content providers put up these
relays today, and somehow figure out how to test them, their work is
not done.

Then stop focusing on the relays, as they are only necessary to get between
the 6to4 prefix and the non-6to4 prefix address ranges. They are explicitly
not in the path of any 6to4 -> 6to4 conversation.

In any case the relays should not be the responsibility of the content side,
they are rightly the responsibility of the access networks who wouldn't need
to deploy them if they had native access in place. The 6rd hack is nothing
more than 6to4 in a different prefix to get around the one-liner that should
be ignored in the original RFC that said to only publish the /16 into IPv6
bgp. I can already hear the screams about routing table, but there is no
difference between the impact of a 6rd specific announcement and a
deaggregate of 2002::

:: >
:: It is relatively easy to do, even with dozens of POPs. There isn't
anything special you
:: have to do for the hundreds of address ranges, really, so I don't
buy that as being a
:: meaningful part of the argument.

I think this is a red herring - return relays were never *the* problem.

:: > I do agree with you that pointing fingers at this stage is really
not helpful. I continue to maintain that being supportive of those
content networks that are willing to wade in is the right answer.
:: >
:: Agreed, but, it's also important to point out when they're starting
to swim in directions
:: that are counterproductive, such as having help sites that advise
users to turn off
:: IPv6 with fixing their IPv6 capabilities as a secondary option.

"We recommend disabling IPv6 or seeking assistance in order to fix your
system's IPv6 configuration through your ISP or computer manufacturer"

It would be most helpful to rearrange that statement to take the focus off
of 'disabling'. When the first thing people see is turn it off they will do
that rather than get to the point that there might be something that is
fixable.

So, your problem is that a help page gives the user 2 options,
the first one of them being a quick and easy fix that a user can do
himself in less then a minute, and suggesting contacting the ISP or
manufacturer *second* (and possibly spending quite a bit of time on
hold/troubleshooting, and then saying "screw it")?!?

Honestly, I think the people who want ipv6 to work, and are willing and
capable to troubleshoot it, will; and those who don't will just
turn it off... Seems like the right outcome to me..

I agree with your assessment, but the way the statement is worded make it
look like you would rather have people turn it off. If my wife read that,
should would be telling me that Yahoo said that the recommendation is to
turn it off, not that something might need fixing.

Tony

19

At any given instant, there's a *lot* more than 182,000 users who are cut off
due to various *IPv4* misconfigurations and issues.

Let's keep a sense of proportion, shall we?

20

:: On Tue, 10 May 2011 02:17:46 EDT, Igor Gashinsky said:
::
:: > The time for finger-pointing is over, period, all we are all trying to do
:: > now is figure out how to deal with the present (sucky) situation. The
:: > current reality is that for a non-insignificant percentage of users when
:: > you enable dual-stack, they are gong to drop off the face of the planet.
:: > Now, for *you*, 0.026% may be insignificant (and, standalone, that number
:: > is insignificant), but for a global content provider that has ~700M users,
:: > that's 182 *thousand* users that *you*, *through your actions* just took
:: > out.. 182,000 - that is *not* insignificant
::
:: At any given instant, there's a *lot* more than 182,000 users who are cut off
:: due to various *IPv4* misconfigurations and issues.

Yes, but *these* 182,000 users have perfectly working ipv4 connectivity,
and you are asking *me* to break them through *my* actions. Sorry, that's
simply too many to break for me, without a damn good reason to do so.

Doing that on world ipv6 day, when there is a lot of press, and most other
large content players doing the same, *is* a good reason - it may actually
has a shot of accomplishing some good, since it may get those users to
realize that they are broken, and fix their systems, but outside of flag
day, if I enabled AAAA by default for all users, all I'm going to do is
send those "broken" users to my competitors who chose not to enable AAAA
on their sites.

This is why I think automatic, measurement-based whitelisting/blacklisting
to minimize the collateral damage of enabling AAAA is going to be
inevitable (with the trigger set to something around 99.99%), and about
the only way we see wide-scale IPv6 adoption by content players, outside
events like world ipv6 day.

-igor