IPv6 and CDN's

Mailman List Mirror (Read Only)
1

Hi!

We currently live in times where is actually fun to go IPv6-only. In my case, as in: running a FreeBSD kernel compiled without the IPv4-stack.

A few years back doing such thing was mostly disappointing, but nowadays is actually quite doable and entertaining.

So, the other day I decided to take this experiment to the next level by disconnecting my local resolver from IPv4 as well.

Then things started to break. LinkedIn, Bing, Openstreetmap... Although they all work great on IPv6-only, now they no longer did.

It turns out that there underlying CDN's with domain names such as ‘l-msedge.net’ and ‘trafficmanager.net’ (Microsoft) or 'fastly.net', that reside on authoritative name servers that *only* have an IPv4 address.

I guess my question is simple: Why?

Are there good architectural reason for this? Or is it just something that is overlooked and forgotten about?

I would love to find out!

Thank you.

2

Marco Davids via NANOG <nanog@nanog.org> writes:

It turns out that there underlying CDN's with domain names such as
‘l-msedge.net’ and ‘trafficmanager.net’ (Microsoft) or 'fastly.net',
that reside on authoritative name servers that *only* have an IPv4
address.

Fastly does have IPv6 enabled authoritative DNS server but it looks like
it's not the default.

I ran into this some time ago with deb.debian.org on an IPv6 only Debian
VM with a locally installed resolver. I opened a ticket which was closed
in record time: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=961296

After some ranting and shouting it now works but a couple of days
ago I ran in the same problem while trying to install something via
pip. fles.pythonhosted.org is also using fastly.

I guess my question is simple: Why?

I'm asking myself the same question.

Are there good architectural reason for this? Or is it just something
that is overlooked and forgotten about?

I don't think it was overlooked or forgotten. More along

"We have always done it this way", "We had problems enabling IPv6 (ages
ago)" or something else you can find on https://ipv6excuses.com/.

Jens

3

Hi Jens,

I ran into this some time ago with deb.debian.org on an IPv6 only Debian
VM with a locally installed resolver. I opened a ticket which was closed
in record time: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=961296

Just for the record; your issue is slightly different:

You wrote:

"deb.debian.org is a CNAME for debian.map.fastly.net. There are no AAAA records for fastly.net so any DNS querys from an IPv6 only resolver will not work."

At the moment debian.map.fastly.net has an AAAA-record though.

The thing is; the authoritative name servers of fastly.net are only willing to hand out that AAAA-record via IPv4. So it still doesn't work with the (locally installed) IPv6-only resolver :wink:

Cheers,

4

I think it's a combination of both... they tried back in the day, it broke, and they "parked" it for later.

When Marketing and The World were happy to see that www.insert-favorite-content-url-here.com had AAAA and IPv6 PTR records, who cared whether boring, little-known FQDN's were remembered or not.

And then, all the engineers moved on to some other gig, where they did better because, why not?

Mark.

5

On second thoughts...

I seem to have been confused by the 'no AAAA records for fastly.net' (as a DNS-purist: that should have said "ns[1234].fastly.net" instead, to make it relevant). :wink:

I ran into this some time ago with deb.debian.org

Right.

So please ignore:

6

Hello,

client side IPv6-only is one thing, but IPv6-only recursive DNS
resolution is probably so niche that content providers and CDN's do
not particularly care at this point in time.

On the other hand, there is probably no good reason to run
authoritative DNS servers without IPv6 connectivity.

Lukas

7

I ran into this some time ago with deb.debian.org on an IPv6 only Debian
VM with a locally installed resolver. I opened a ticket which was closed
in record time: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=961296

After some ranting and shouting it now works

I’m going to post the relevant message here:

Sometimes I wonder why I report bugs…

But your answer was the answer I was expecting. Thanks for noting.

So I can summarize this as “The Debian Project doesn’t care if IPv6 is
working”?

Jens, you went into that ticket looking for a fight, in a place staffed by largely unpaid volunteers, choosing to belittle their efforts and then attempting to shame them into action.

You even chose to mark the bug severity higher than the default, despite you having chosen that mirror for your install. https://www.debian.org/Bugs/Developer#severities

Marco explained to you that the mirror network has plenty of selections to make, but you choose to make a fuss about one supported option not working to a standard the rest of the community pretty much agrees is nowhere near attainable at this time. Using netselect https://packages.debian.org/stable/net/netselect to choose a reachable mirror with the lowest latency would have easily mitigated this issue for you.

DNS64 and NAT64 are going to be with us for a very long time, and if you refuse to support IPv4 even through a translation layer then it is clear you are acting against the interests of further IPv6 adoption by associating IPv6 issues with zealotry. The apathy sometimes associated with IPv6 support today is because of this perceived high effort low reward nature of confrontation.

I would strongly advise you apologise to Marco for your grandstanding, and adopt a more constructive way of furthering your ideology. The NANOG code of conduct clearly states:

https://www.nanog.org/about/code-conduct/

In the spirit of mutual respect and collaboration, NANOG does not tolerate any unwelcome behavior, including but not limited to:

  • Aggressively pushing your own services, products, or causes.
    […]

Please join the rest of us in advocating for IPv6 adoption, rather than the current bullying tactics you seem to be choosing that wins the battle and loses the war. We are all friends* here. You can be a great asset in this effort we should all seek.

M

*FSVO friends, obvs.

8

Hi everyone, goedenmiddag Marco!

We currently live in times where is actually fun to go IPv6-only. In my
case, as in: running a FreeBSD kernel compiled without the IPv4-stack.

Indeed, this is fun experimentation. Shaking the (source code) trees
through excercises like these is a valuable way to identify gaps.

It turns out that there underlying CDN's with domain names such as
‘l-msedge.net’ and ‘trafficmanager.net’ (Microsoft) or 'fastly.net', that
reside on authoritative name servers that *only* have an IPv4 address.

As some observant readers noticed (hint: https://ip6.nl/#!deb.debian.org),
Fastly is working hard with select customers and friends to support IPv6
for everyone.

I guess my question is simple: Why?

Are there good architectural reason for this? Or is it just something
that is overlooked and forgotten about?

The universal deployment of IPv6 appears to be a multi-decennial
multigenerational project. Allow me to shed some light on various
aspects.

One of the challenges faced by those wishing to deploy IPv6 (compared to
IPv4) is how from a BGP Default-Free Zone perspective, IPv4 and IPv6 are
not alike at all! The Internet's IPv6 routing topology is vastly
different from the IPv4 topology.

The above phenomenon is perfectly understandable following from the fact
that IPv4 predates IPv6 - and IP networks grow as they grow. In a
perfect world the IPv6 network would grow perfectly congruent alongside
the global IPv4 network. In this perfect world indeed IPv6 can "just be
enabled", and used whenever available!

Unfortunately the reality of the situation is far more chaotic! For
example if you look at PeeringDB's 'netixlan' table, large discrepancies
between the number of absent IPv4 entries and absent IPv6 entries are
visible:

    $ curl -s https://peeringdb.com/api/netixlan | jq '.' | fgrep -c '"ipaddr4": null'
    1286

    $ curl -s https://peeringdb.com/api/netixlan | jq '.' | fgrep -c '"ipaddr6": null'
    8160

From the above it's implied that the density of the 'IPv4 mesh' is much
higher than the density and diversity of the 'IPv6 mesh', simply because
more operators present more IPv4 traffic-exchange opportunities to other
operators - compared to IPv6. This has performance implications.

Another aspect that flabbergasts me anno 2021 is how there *still* are
BGP peering disputes between (more than two) major global internet service
providers in which IPv6 is 'held hostage' as part of slow commercial
negotiations. Surely end-to-end IPv6 connectivity should be a priority?

Anyway, back to your question: content delivery networks who leverage
all possible technical knobs and buttons to increase performance (such
as BGP traffic engineering) might be reluctant to offer IPv6 services
"as if they are the same as IPv4". More study is required.

Tl;DR - work in progress! :slight_smile:

Kind regards,

Job

ps. Have you tried running an IPv6-only RPKI validator? About 1.4% of
RPKI VRPs appears to be 'missing' in IPv6-only environments :-/

9

Even the DNS root servers are not 100% reachable via IPv6. I would think IANA
would have some standard about reachability for root operators.

FWIW, I just was able to change my home office internet (I reside in the most
densely populated county of Florida). The new provider sold me a dual stack
connection, however when they came to deliver it, there was no IPv6 as
promised. After spending almost a week playing phone tag, I finally got some
one with clue. I was told they have no support if IPv6 and no plans to ever
support IPv6 as there is no way to monetize it.

This leaves me in the same position as my prior circuit via the local cable
co. (no plans to offer IPv6) but at least it's faster than the 2 meg up cable
service.

Until IPv6 becomes provides a way to make money for the ISP, I don't see it
being offered outside of the datacenter.

10

I don't think it'll ever make money, but I think it will reduce costs. CGNAT boxes cost money, operating them costs money, dealing with the support fallout from them costs money. Especially in the residential space, where essentially if the customer calls you, ever, you just blew years' worth of margin.

My residential ISP here in the UK routes me (and every other subscriber) a /56 without being asked. (Their supplied CPE router just puts the first /64 on the LAN and refuses to process PD requests to hand out any of the other /64s, but baby steps...)

Cheers,
Tim.

11

Hi again,

Tl;DR

Not at all. This was a very interesting read! Thank you.

While pondering over it, I noticed that the ns[1234].fastly.net servers are nicely anycasted throughout the globe. If anyone could turn on IPv6 on their authoritatives without therisk of loosing too much performance, I reckon it would be them... our Cloudflare. But they already did it. ;-).

> work in progress!

I have good hopes. Rumour has it that Fastly employs some very smart people. I'm sure we'll see nice things happening when the time is right.

12

It is being offered, it's just not being adopted.

We deliver an IPv6 /126 p2p and /56 or /48 onward assignment to all our DIA customers. No interest.

We deliver an IPv6 /125 p2p and eBGP session to all our IP Transit customers. 5/10 are interested.

You can only do what you can only do.

Mark.

13

The problem is accurately modelling cost reduction using native IPv6 in lieu of CG-NAT is hard when the folk that need convincing are the CFO's.

They are more used to "spend 1 to get 2". Convincing them to "save 2 by spending 1" - not as easy as one may think.

Mark.

14

Bryan,

Another aspect that flabbergasts me anno 2021 is how there *still* are
BGP peering disputes between (more than two) major global internet service
providers in which IPv6 is 'held hostage' as part of slow commercial
negotiations. Surely end-to-end IPv6 connectivity should be a priority?

Even the DNS root servers are not 100% reachable via IPv6.

Excepting temporary failures, they are as far as I am aware. Why do you think they aren’t?

I would think IANA
would have some standard about reachability for root operators.

I think you might misunderstand relationships here.

The IANA team’s standards are what the community defines. In the case of the root operators, RFC 7720 says “root service” must be available via IPv6 and RSSAC-001 (“Service Expectations of Root Servers”, https://www.icann.org/en/system/files/files/rssac-001-root-service-expectations-04dec15-en.pdf) says:

"[E.3.1-B] Individual Root Servers will deliver the service in conformance to IETF standards and requirements as described in RFC 7720 [4] and any other IETF standards-defined Internet Protocol as deemed appropriate."

So, in theory, all the root servers should be available via IPv6 and, as far as I can tell, they are.

However, the IANA team is not the enforcement arm of the Internet. If a root server operator chooses to not abide by RFC 7720, there is nothing the IANA team can do unilaterally other than make the root server operator aware of the fact.

Until IPv6 becomes provides a way to make money for the ISP, I don't see it
being offered outside of the datacenter.

Different markets, different approaches. In the areas I’ve lived in Los Angeles, commodity residential service via AT&T (1 Gbps up/down fiber) and Spectrum (varying speeds) is dual stack by default (as far as I can tell). I suspect all it would take would be one of the providers in your area to offer IPv6 and advertise the fact in their marketing to cause the others to fall into line.

Regards,
-drc

15

Another aspect that flabbergasts me anno 2021 is how there still are
BGP peering disputes between (more than two) major global internet service
providers in which IPv6 is ‘held hostage’ as part of slow commercial
negotiations. Surely end-to-end IPv6 connectivity should be a priority?

Even the DNS root servers are not 100% reachable via IPv6. I would think IANA
would have some standard about reachability for root operators.

FWIW, I just was able to change my home office internet (I reside in the most
densely populated county of Florida). The new provider sold me a dual stack
connection, however when they came to deliver it, there was no IPv6 as
promised. After spending almost a week playing phone tag, I finally got some
one with clue. I was told they have no support if IPv6 and no plans to ever
support IPv6 as there is no way to monetize it.

This leaves me in the same position as my prior circuit via the local cable
co. (no plans to offer IPv6) but at least it’s faster than the 2 meg up cable
service.

Until IPv6 becomes provides a way to make money for the ISP, I don’t see it
being offered outside of the datacenter.

87% of mobiles in the usa are ipv6

https://www.worldipv6launch.org/measurements/

16

87% of mobiles in the usa are ipv6

https://www.worldipv6launch.org/measurements/

Agreed. When they have to connect to an IPv4 only host, they do some type of AFTR. These devices have never known a world outside of this situation. That is a major difference.

17

So I’m curious how the mobile operators deploying ipv6 to the handsets are dealing with ipv4. The simplest would be to get the phone a routable ipv4 address, but that would seemingly exacerbate the reason they went to v6 in the first place. Are carriers NAT’ing somewhere along the line? If so, where? Like does the phone encapsulate v4 in 4-in-6? Or does the phone get a net 10 address and it gets NAT’d by the carrier?

It seems also for mobile carriers there is incentive for as much transit as possible for native v6 to the servers. Or is the deployment of v6 mainly within the carrier network itself and it’s NAT’d somewhere?

Basically what does a typical v6/v4 architecture look like for a mobile carrier these days?

Mike

18

So I’m curious how the mobile operators deploying ipv6 to the handsets are dealing with ipv4. The simplest would be to get the phone a routable ipv4 address, but that would seemingly exacerbate the reason they went to v6 in the first place.

First, consider that the 3 major cell carriers in the usa each have 100 million customers. Also, consider they all now have a home broadband angle. Where do 100 million ipv4 addresses come from? Not rfc 1918, not arin, … and we are just talking about customer ip addresses, not considering towers, backend systems, call centers, retail ….

So the genesis of 464xlat / rfc 6877 is that ipv4 cannot go where we need to go, the mobile architecture must be ipv6 to be comply with the e2e principle and not constrain the scaling of the customers / edge. Other cell carriers believe in operating many unique ipv4 networks … like a 10.0.0.0/8 per metro, but even that breaks down and cannot scale… and you end up with proxies / nats / sbcs everywhere just to make internal apps like ims work, which is a lot of state.

Are carriers NAT’ing somewhere along the line? If so, where? Like does the phone encapsulate v4 in 4-in-6? Or does the phone get a net 10 address and it gets NAT’d by the carrier?

~80% of traffic goes to fb, goog, yt, netflix, bing, o364, hbomax, apple tv, … all of which are ipv6. So, only 20% of traffic requires nat, when you have ipv6. I am hoping tiktoc and aws move to be default on for ipv6 soon.

The nats dont scale well and take the brunt of attacks, so services that require nat suffer. Real shame, but they have a path to improve performance by deploying ipv6. Thats why performance driven companies use ipv6 (fb, goog, akamai, …)

19

I think you will find that there are some places in which getting IPv6 network service has been difficult, and as a result even IPv6-capable equipment is not reachable by IPv6. Those are, however, few and far between.

Sent using a machine that autocorrects in interesting ways...

20

Mike