Apple Airport Extreme IPv6 problems?

Mailman List Mirror (Read Only)
1

Apple is nice enough to provide an automatic v6 tunnel from their new Airport Extreme units. They even get all the machines on the network to participate -- by default! At first this did not seem to be much of an issue, it was even pretty cool.

However, I noticed as I roll out more v6 services to support native v6 users, I am impacting the network performance of almost all of the Apple airport population that has an inefficient tunnel configuration. The user obviously will take the AAAA v6 published IP over the v4 A record.

Don't get me wrong v6 tunnels are great, when you opt-in and know what you are getting into. For example, my grandparents tunnel in California goes to Virginia. This impacts the user experience rather significantly with the first hop being nearly 100ms where their services to California are ~20ms. It's painful for a lot of users, especially when they don't even know what's going on.

Has anyone else ran into this? It's not pretty for a CDN or anyone trying to provide a quality service over v6, shunting users over inefficiently tunneled routes does not sit well with me. I think Apple has made a mistake by enabling this by default.

-Barrett

2

I think we will never move to IPv6 if vendors don't do things like the one
in the Airport. However, in order to make this "transition" phase where
there may be a possible degradation of the RTT, we need to cooperation of
the operators, for example deploying 6to4 relays in their networks.

As many 6to4 relays exists (even if they are closed only to the customers of
that network), less this will be problem.

I understand that sometimes is not easy to provide native IPv6 services, but
deploying a 6to4 relay (same as a Teredo Relay) is a very simple and
inexpensive step, but it helps a lot.

In regions such as Africa and LAC, where upstream b/w is expensive, I'm
helping the ISPs to setup those, in order to avoid traffic going thru the
upstream links and staying local.

Remember that even if we don't have products such the Airport, our customers
have OS which come with IPv6 enabled by default and they try 6to4 or Teredo
when they don't have native connectivity, so is not a problem that we can
hide from. We really need to move one step forward and avoid support calls,
for example.

Regards,
Jordi

3

We removed AAAA on our production hosts shortly after we deployed it, our global v6 deployment goes production next week, at which time I may re-add the AAAA to limited production. If we do this, I publish a report of the stats once I have more accurate figures.

4

[ snip ]

We removed AAAA on our production hosts shortly after we deployed it,
our global v6 deployment goes production next week, at which time I
may re-add the AAAA to limited production. If we do this, I publish
a report of the stats once I have more accurate figures.

How did you do the naming? Matching AAAA or unique?

I have no idea what to expect over time with behavior on matching AAAA
and A since I have no idea what to expect with v6 since we don't
really have any standard deployment plans or even de-facto standards
in place to move forward. Is there any de-facto or otherwise standard
around host schemes for dual stack?

-M<

5

How did you do the naming? Matching AAAA or unique?

Matched AAAA, I was thinking about doing a w6 or something more unique for now, but that somewhat defeats the point.

The other thought that occurred to me, does FF/Safari/IE have any ability to default back to v4 if v6 is not working or behaving badly? This could be a helpful transition feature but may be more trouble than it's worth.

-Barrett

6

[spam: Check IPv6 Toy Gallery :: SixXS - IPv6 Deployment & Tunnel Broker for an IPv6 Toy Gallery :)]

Somewhat long, hopefully useful content follows...

Barrett Lyon wrote:
[..]

The other thought that occurred to me, does FF/Safari/IE have any
ability to default back to v4 if v6 is not working or behaving badly?
This could be a helpful transition feature but may be more trouble than
it's worth.

The IETF recommendation is that IPv6 is tried before IPv4, BUT there is
RFC3484 (http://www.ietf.org/rfc/rfc3484.txt) which gives an extra edge
to this. In general it comes down that the resolver will, assuming there
is both an IPv4 and IPv6 address (A + AAAA) on the dns label requested
try, as a source address:

- IPv6 native (anything not 2002::/16 + 2003::/32)
- IPv4 native
- IPv6 6to4 (2002::/16)
- IPv6 Teredo (2003::/32

Of course when there is only a A or AAAA only that protocol will be
used. All applications are supposed to use getaddrinfo() which sorts
these addresses per the above specification, the app should then
connect() to them in order, fail/timeout and try the next one till it
connects correctly. The above table is re-programmable per host and
there are discussions/drafts to automate that for a complete network.

The correct way to use getaddrinfo() is described in:
Eva M. Castro Homepage by Eva Casto and of
course the almost 10 year old document by Jun-ichiro itojun Itoh at:
http://www.kame.net/newsletter/19980604/

Now the really BIG problem there is though is that when network
connectivity is broken. TCP connect will be sent, but no response comes
back or MTU is broken, then the session first has to time out.

Thus if a user has IPv6 and the server has it also but the connectivity
between them is b0rked then it will take quite some time to recover
properly from this. Apps could of course do a multi-connect and try all
in parallel but I am pretty sure that servers are not waiting for that
and for instance the Firefox programmers don't even know what
"threading" is, seeing that they can't even separate their UI from the
network and rendering code, thus don't wait for them to do it for that.
Also there is this nasty concept of deployed base and getting people to
upgrade is of course far from easy, fortunately those types won't do
IPv6 either hopefully :wink:

6to4 and Teredo are a big problem here, especially from an operator
viewpoint. This as an operator has absolutely no control over the flow
of packets from/to his/her network. When the packets flow back it might
just be that, due to BGP in the remote network, something attracts the
6to4 packets destined back for 6to4 and they mysteriously disappear or
get routed around the world. The same for the way from the user on your
network to the 6to4 relay at 192.88.99.1 (if you, like me, can't
remember the address just type "host -t any 6to4.ipv6.microsoft.com" :wink:
This one can also be situated anywhere on this planet and BGP might pull
it somewhere where you don't want it to go.

As such, if you, as an ACCESS operator want to have full control over
where your users IPv6 traffic goes to you might want to do a couple of
things to get it at least a bit in your control:
- setup a 6to4 relay + route 192.88.99.1 + 2002::/16
- setup a Teredo Server + Relay and make available the
   server information to your users and inform them of it.
- and/or the better option IMHO, to keep it in control: setup a
   tunnel broker and provide your users access to that. For instance
   Hexago sells appliances for this purpose but you can also ask SixXS
   to manage one for your customers.

For CONTENT operators, get yourself a nice chunk of RIR space from your
RIR. Then what you might want to do is setup the following little test:
http://www.braintrust.co.nz/ipv6wwwtest/ and/or mods of it, put it on
your important content sites. This will allow you to discover if your
clients are using IPv6 and if they are able to reach it. Then if you are
confident that you are up to it and that your clients are fine, you
might want to consider adding AAAA's to your site and go fully dual stack.

If you have somewhat tech savvy users you can of course also ask them to
test it for you. "Check out our Cool new toy: we got IPv6!" or something
and ask them how it works.

As for the above spammed toys URL, I have to note that especially AXIS
folks are really cool, you send them a mail asking "what products
support IPv6" and the next day you get back very nice PDF's containing
their overviews of everything that supports IPv6, they have lots of it,
nearly all their products do. The best thing of course is that the sales
reps actually KNOW what IPv6 is, wow, I like those AXIS folks! :slight_smile:

Greets,
Jeroen

7

Browsers are pretty good at falling back on a different address in general / IPv4 in particular when the initial try doesn't work, but it does take too long if the packet is silently dropped somewhere. If there is an ICMP unreachable there is no real delay. Worst case is a path MTU discovery black hole, then browsers generally don't fall back.

For some reason, I can't reach the IETF or ARIN over IPv6 from home, which means every time I want to read an RFC I have to wait for my browser to time out and retry.

Apple's Mail is worse: it has a bug that prevents it from delivering mail with SMTP over IPv6, but it won't fall back to IPv4 so you need to intervene manually.

It would be good if more ISPs deployed 6to4 gateways so the 6to4 experience would be better. Windows Vista will also try to use 6to4 out of the box (but only if it has a public IPv4 address, can't do 6to4 from behind NAT).

8

[spam: Check IPv6 Toy Gallery :: SixXS - IPv6 Deployment & Tunnel Broker for an IPv6 Toy Gallery :)]

Spam: read a good book about IPv6. :slight_smile:

The IETF recommendation is that IPv6 is tried before IPv4, BUT there is
RFC3484 (http://www.ietf.org/rfc/rfc3484.txt) which gives an extra edge
to this. In general it comes down that the resolver will, assuming there
is both an IPv4 and IPv6 address (A + AAAA) on the dns label requested
try, as a source address:

- IPv6 native (anything not 2002::/16 + 2003::/32)
- IPv4 native
- IPv6 6to4 (2002::/16)
- IPv6 Teredo (2003::/32

No, that's not true:

    If an implementation is not configurable or has not been configured,
    then it SHOULD operate according to the algorithms specified here in
    conjunction with the following default policy table:

       Prefix Precedence Label
       ::1/128 50 0
       ::/0 40 1
       2002::/16 30 2
       ::/96 20 3
       ::ffff:0:0/96 10 4

So first IPv6 loopback, then IPv6 any, then some ancient automatic tunneling that nobody uses and finally IPv4. ::ffff:0:0/96 is for IPv4-mapped IPv6 addresses (or was it the other way around??) so that prefix contains all IPv4 addresses in a way that they can be used with IPv6 APIs.

However, Windows XP wil _in_ _practice_ do what Jeroen says because of the label matching. The idea is that source and dest must have the same label value and then the highest precedence wins, this avoids using an IPv6 source address with an IPv4 destination address and the like. If you have native IPv6 on the remote end and 6to4 (2002::/16) on your end, then the labels don't match but for IPv4 on both ends they do so XP will choose that over the native/6to4 combo. Not sure what Vista or FreeBSD do, not aware of any other OSes that implement RFC 3484.

6to4 and Teredo are a big problem here, especially from an operator
viewpoint. This as an operator has absolutely no control over the flow
of packets from/to his/her network. When the packets flow back it might
just be that, due to BGP in the remote network, something attracts the
6to4 packets destined back for 6to4 and they mysteriously disappear or
get routed around the world.

Easily solved by running your own private (or public) 6to4 relay: then the packet goes directly to the other end without detours over IPv4. You can't control how the packets get from the remote 6to4 user to you, though.

9

"Pretty good" as in there is a browser standard to poke for v6 then v4
or is this a stack behavior?

-M<

10

>
> How did you do the naming? Matching AAAA or unique?

Matched AAAA, I was thinking about doing a w6 or something more
unique for now, but that somewhat defeats the point.

I tried to do it in a round robin record based on the described
behavior. My theory was that the inverse response should occur and
satisfy. My results were failure. BIND 9.3.2 accepted the record, did
not complain and properly reloaded the zone, but did not offer the v6
AAAA as the inverse. I'm probably missing something here... like "not
supported". :slight_smile:

The other thought that occurred to me, does FF/Safari/IE have any
ability to default back to v4 if v6 is not working or behaving
badly? This could be a helpful transition feature but may be more
trouble than it's worth.

Should be an operation defined by gethostbyname() no?

-M<

11

Since this conversation has already talked about behaviour when encountering AAAA vs A, I am worried that a browser running on a dual-stack laptop might cache the AAAA returned when it has some v6 connectivity, and then refuse to look again for the A when I pick it up and take it somewhere with only v4 connectivity.

We see the browser cache bite us regularly with regard to the way they dip into the cache for long-stale records today. The support burden will increase if there are stack transitionary woes as well.

Andy

12

[as this has nothing to do with Apple Airports in particular I changed
the subject again]

Martin Hannigan wrote:

Should be an operation defined by gethostbyname() no?

and in another:

"Pretty good" as in there is a browser standard to poke for v6 then v4
or is this a stack behavior?

No, it is done by getaddrinfo() and the resolver layers of the OS,
please read the somewhat longer mail from with the subject of "Going
dual-stack, how do apps behave and what to do as an operator", yes it is
long, but it explains more or less how it works and answers these questions.

Andy Davidson wrote:

Browsers are pretty good at falling back on a different address in
general / IPv4 in particular when the initial try doesn't work,

"Pretty good" as in there is a browser standard to poke for v6 then v4
or is this a stack behavior?

Since this conversation has already talked about behaviour when
encountering AAAA vs A, I am worried that a browser running on a
dual-stack laptop might cache the AAAA returned when it has some v6
connectivity, and then refuse to look again for the A when I pick it up
and take it somewhere with only v4 connectivity.

getaddrinfo() asks first for AAAA, then for A, then sorts the records
and then returns them to the application. Thus no such problem there
unless some OS implements this in the wrong way, but afaik that is not
the case.

Now indeed when you are swapping locations and thus change IP addresses
(eg loose the IPv6 connection, gain IPv4, or change from one IPv4
address to the other or one IPv6 address to another), indeed TCP
sessions will die, but that is a problem with mobility.

We see the browser cache bite us regularly with regard to the way they
dip into the cache for long-stale records today. The support burden
will increase if there are stack transitionary woes as well.

Browsers should (and afaik don't) care about the IP protocol they got a
resource from, they cache based on URI "http://www.example.com/bla.png&quot;
and do not note the IP protocol it was from.

Note that there are a couple of browsers though which have their own
internal "IPv6 off" switches, eg firefox has "network.dns.disableIPv6"
which can be turned off by the user, Safari had/has IPv6 turned off per
default and so was Opera. This as they perceived IPv6 to have problems
and thus if there is IPv4 they always first use IPv4 and then IPv6,
ignoring the ordering done by getaddrinfo().

Greets,
Jeroen

13
  • IPv6 native (anything not 2002::/16 + 2003::/32)
  • IPv4 native
  • IPv6 6to4 (2002::/16)
  • IPv6 Teredo (2003::/32

Incase anyone is using this for reference purposes, Jaroen really means 2001::/32, not 2003::/32.
Teredo was also previously on 3ffe:831f::/32, for those of you on older Windows XP machines. This prefix no longer works - upgrade.

Now the really BIG problem there is though is that when network
connectivity is broken. TCP connect will be sent, but no response comes
back or MTU is broken, then the session first has to time out.

6to4 and Teredo are a big problem here, especially from an operator
viewpoint.

Yes. Infact, especially if you have users on Vista. It does this IPv6 tunnelling thing that on the surface appears really cool. When you try and talk IPv6 to something other than link-local: (in order)

  • If you have a non-RFC1918 (ie. ‘public’) address, it fires up 6to4.
  • If you have an RFC1918 address, it fires up Teredo.
    Seems cool in theory, and you’d think that it would really help global IPv6 deployment - I’m sure that’s how it was intended, and I applaud MS for taking a first step. But in practice, however, this has essentially halted any IPv6 /content/ deployment that people want to do, as user experience is destroyed.

You can help, though - here’s the problem:
6to4 uses protocol 41 over IP. This doesn’t go through NAT, or stateful firewalls (generally). Much like GRE.
Because of this, if you’re a enterprise-esque network operator who runs non-RFC1918 addresses internally and do NAT, or you do stateful firewalling, PLEASE, run a 6to4 relay on 192.88.99.1 internally, but return ICMPv6 unreachable/admin denied/whatever to anything that tries to send data out through it. Better yet, tell your firewall vendor to allow you to inspect the contents of 6to4 packets, and optionally run your own 6to4 relay, so outgoing traffic is fast.
Even if you don’t want to deploy IPv6 for some time, do this at the very least RIGHT NOW, or you’re preventing those of us who want to deploy AAAA records alongside our A records from doing so. If you need configs for <vendor/OS B/C/J/L>, let me know and I’ll write some templates.

I see this sort of IPv4 network quite commonly at universities, where students take their personal laptops and throw them on the campus 802.11 network. While disabling the various IPv6 things in Vista at an enterprise policy level might work for some networks, it doesn’t for for a university with many external machines visiting. So, if you’re a university with a network like this (ie. most universities here in NZ, for example), please spend a day or two to fix this problem in your network - or better yet, do a full IPv6 deployment.

I’d like to get some work done to get some ‘qualification’ testing of the availability of 6to4 from a ‘client’ POV standardised, so this problem can go away. Moving city+job has hindered such things as of late.

As such, if you, as an ACCESS operator want to have full control over
where your users IPv6 traffic goes to you might want to do a couple of
things to get it at least a bit in your control:

  • setup a 6to4 relay + route 192.88.99.1 + 2002::/16
  • setup a Teredo Server + Relay and make available the
    server information to your users and inform them of it.

For those not on v6ops, I’ve got a draft right now that explains why you should (as an access provider) run a Teredo server, and proposes a standard to allow you to direct your users to your local Teredo server. I should be pushing out an update to it shortly. See above RE. moving life around.
Also, Relays are only useful if you have native IPv6 somewhere, OR if you run a 6to4 relay (which probably means you have native IPv6…). Note the distinct usage of ‘servers’ and ‘relays’, for the uninitiated.

I’m building some embedded system images that run Teredo and 6to4 relays, with pretty much zero configuration. It runs on Soekris hardware right now (ie. sub $USD300), but if people are interested I can port it to regular x86 hardware. All you need is an IPv6 tunnel from a broker somewhere - you don’t even need native transit, and you can improve the performance of IPv6 over the various tunnelling protocols for your end users. If you’re interested in this, drop me an email.

  • and/or the better option IMHO, to keep it in control: setup a
    tunnel broker and provide your users access to that. For instance
    Hexago sells appliances for this purpose but you can also ask SixXS
    to manage one for your customers.

Fine if you’ve got small numbers of high value+clue customers. Not so good if you’re a nation-wide residential provider.

For CONTENT operators, get yourself a nice chunk of RIR space from your
RIR. Then what you might want to do is setup the following little test:
http://www.braintrust.co.nz/ipv6wwwtest/ and/or mods of it, put it on
your important content sites. This will allow you to discover if your
clients are using IPv6 and if they are able to reach it. Then if you are
confident that you are up to it and that your clients are fine, you
might want to consider adding AAAA’s to your site and go fully dual stack.

If anyone does run the ipv6wwwtest code (or something similar), please talk to me, as I’d like some numbers from some larger web properties so I can rant about it soon at an operator meeting near you, and perhaps aggregate numbers and provide an “IPv6 Internet health report” regularly.

You don’t actually need any RIR space. You’ll note that the braintrust.co.nz website does the checks using 6to4, as the place that server lives can’t get native IPv6 transit. This takes less than a day to set up and does not require you to turn on an IPv6 network, and you can regularly evaluate whether enabling your content (and network!) for IPv6 is a good idea or not.

Also, if you do deploy an IPv6 network for your content, set up a Teredo relay, and point 2001::/32 at it. Your viewers/users will automatically use this relay when accessing your content, and their traffic to you will be over IPv4, all they way from their PC to your network - so, equivalent performance as IPv4. Note that I say relay here, not server.

If you have somewhat tech savvy users you can of course also ask them to
test it for you. “Check out our Cool new toy: we got IPv6!” or something
and ask them how it works.

Mozilla.org are doing this for example. Cue Matthew Zeier.

(Apologies for a dis-jointed email. It’s 1am, I’m tired and in a ranty mood)

14

6to4 uses protocol 41 over IP. This doesn't go through NAT

Those statements are both true, but they're unrelated. If your NAT box knows there is more to IP than TCP and UDP, it's possible that you can do IPv6-in-IP tunneling in general (protocol 41) through the NAT box, but that doesn't help 6to4 because your 6to4 address range is constructed from your IPv4 address which can't be done successfully using RFC 1918 addresses.

stateful firewalls (generally).

Depends on the firewall and how it's configured. This is a problem, because if you use public addresses but protocol 41 is blocked, IPv6 stuff needs to time out.

if you're a enterprise-esque network operator who runs non-RFC1918 addresses internally and do NAT, or you do stateful firewalling, PLEASE, run a 6to4 relay on 192.88.99.1 internally, but return ICMPv6 unreachable/admin denied/whatever to anything that tries to send data out through it. Better yet, tell your firewall vendor to allow you to inspect the contents of 6to4 packets, and optionally run your own 6to4 relay, so outgoing traffic is fast.

Right.

Even if you don't want to deploy IPv6 for some time, do this at the very least RIGHT NOW, or you're preventing those of us who want to deploy AAAA records alongside our A records from doing so.

Well, I don't care: you break it, you buy it. But I can see how people who make money from their content would...

15

Since this conversation has already talked about behaviour when encountering AAAA vs A, I am worried that a browser running on a dual-stack laptop might cache the AAAA returned when it has some v6 connectivity, and then refuse to look again for the A when I pick it up and take it somewhere with only v4 connectivity.

Hm, yes, that would suck. I've never seen problems with this with MacOS, though. I haven't used anything else both long and mobile enough to make a difinitive statement, but I think you'll be allright: when an application tries to do IPv6 when there is no IPv6 connectivity, MacOS/BSD/Windows detect this and return an error rather than let the attempt time out. Not 100% sure about Linux and I think Solaris had some trouble in this area in the past.

We see the browser cache bite us regularly with regard to the way they dip into the cache for long-stale records today.

Does browser caching still work these days? I thought all web admins disabled it on their servers because they can't be bothered to think about which cache directives to send along with each page. I can rarely return to a previously viewed page without the browser hitting the network, in any event.

16

Not all Web Admins do. At least, people still see ~30% byte hit rates
on Squid caches. :wink:

Besides, these are two different things - browser DNS caching and
browser content caching.

Adrian

17

I mean the dns cache sorry. Firefox definitely has one, it keeps annoying me.

18

I think we will never move to IPv6 if vendors don't do things
like the one in the Airport. However, in order to make this
"transition" phase where there may be a possible degradation
of the RTT, we need to cooperation of the operators, for
example deploying 6to4 relays in their networks.

And just what should operators do to cooperate?

Are you aware of any documents that describe how to set up 6to4 relays
in an ISP network?

--Michael Dillon

19

Iljitsch van Beijnum wrote:

Does browser caching still work these days? I thought all web admins disabled it on their servers because they can't be bothered to think about which cache directives to send along with each page. I can rarely return to a previously viewed page without the browser hitting the network, in any event.

Actually, browser caching is a function of the Web design tags, not the server. So, the decision to allow caching is on the page creator. On my own sites, I leave caching to the default unless there is a good reason to disable caching. One site I used to run, a warranty form processor, I disabled all caching -- at all levels -- because it was a database-driven site allowing updates from multiple people at the same time, so caching was highly inappropriate.

Caching use to bite me regularly when I was doing customer support. Which led to the mantra "Clear the cache!"

20

- setup a 6to4 relay + route 192.88.99.1 + 2002::/16

How?

- setup a Teredo Server + Relay and make available the

How?

- and/or the better option IMHO, to keep it in control: setup a
   tunnel broker and provide your users access to that. For instance
   Hexago sells appliances for this purpose but you can also ask SixXS
   to manage one for your customers.

Congratulations for explaining how to do it at the same time as you give
the advice.

--Michael Dillon