Awful quiet?

I miss the endless debates. Is *everyone* Christmas shopping?

Here's a thought to ponder....

With the thousands of datacenters that exist with IPv4 cores, what will it take to get them to move all of their infrastructure and customers to IPv6? Can it even be done or will they just run IPv6 to the core and proxy the rest?

-Jim P.

Jim Popovitch wrote:

I miss the endless debates. Is *everyone* Christmas shopping?

Here's a thought to ponder....

With the thousands of datacenters that exist with IPv4 cores, what will it take to get them to move all of their infrastructure and customers to IPv6? Can it even be done or will they just run IPv6 to the core and proxy the rest?

-Jim P.

Looking at my own "datacenter":

Unifix Linux 2.0.0
No it will never move.

Eisfair, kernel 2.2.x
My router and my dns, ftp, remote shell
No they will probably never move.

Suse Linux 8.3 (kernel 2.4.x)
my workstation
Used to have its IPv6 enabled. Gave me problems with connectivity.
I dont have IPv6 to the outside so I had to disable the stack.
Runs a lot smoother now.
It tooks me week to get the IPv6 stack running in the first place.

I tried ISODE 8.0 recently. It still works on all my computers.
I could even connect to a friend who also tried ISODE 8.0
It works through IPv4. What happened to ISO?

I guess that is what will finally happen to IPv6.

I used to have a local IPv6 network running. But with site-local
and link-local disappearing the configuration became invalid.
Not having valid IPv6 addresses any longer I did not get a headache
when I took my IPv6 stack down.

My log looks cleaner. No more complaints from my DNS server.
Now I am looking forward to what will come after IPv6.

:slight_smile:

Merry Christmess
Peter and Karin

Peter Dambier <peter@peter-dambier.de> writes:

Used to have its IPv6 enabled. Gave me problems with connectivity.
I dont have IPv6 to the outside so I had to disable the stack.
Runs a lot smoother now.
It tooks me week to get the IPv6 stack running in the first place.

You've had quite the run of bad luck. My IPv6 stuff was working
perfectly and with almost no effort. Until I lost an ethernet card in
a VXR and snagged one from the IPv6 box as a "spare", heh. Gotta get
around to fixing that, but in the meantime no IPv6 on the colo LAN is
not exactly an operational deal-killer.

I tried ISODE 8.0 recently. It still works on all my computers.

Gee, thanks. I'd managed to not think about ISODE for several years.
:stuck_out_tongue:

I could even connect to a friend who also tried ISODE 8.0
It works through IPv4. What happened to ISO?

I guess that is what will finally happen to IPv6.

I think the likelihood that people will eventually stop caring about
IPv6 like they stopped caring about OSI is fairly slim, even if the
only real problem that V6 addresses is address exhaustion.

IPv6 had more traction five years ago than OSI ever had.

                                        ---Rob

Jim Popovitch wrote:

With the thousands of datacenters that exist with IPv4 cores,
what will it take to get them to move all of their infrastructure
and customers to IPv6?

A L2 switched (or Hubbed ? :slight_smile: 'datacenter' doesn't need to do much
hardware wise, install IPv6 stacks on the hosts, configure done.
Most switches will happily run IPv4 and IPv6 on the same cabling, it's
just a different ethertype. One thing to watch out for are switches and
NIC's with broken multicast, if this is the case IPv6 Neighbour
Discovery (ND) will most likely not work. As a workaround one can put
interfaces in promisc/allmulti but that is sort of dirty, better replace
the hardware instead.

Most current OS's support this and usually the upgrade is gratuit(tm),
assuming that one isn't running Win95 as a 'server' and does do regular
upgrades of their software of course. This part mostly consists of
normal server upgrades/updates and should not be to costly except for
man hours and of course some testing. Enabling the stack of course
doesn't really enable IPv6 in the applications, thus that is a second
part one has to look at and will be the most time consuming.

The L3 part is most likely the expensive one though, Vendor J noticed
they could get a load of cash from government organizations and suddenly
introduced a nice expensive additional license for IPv6. Good part is
that most vendors seem to support it, they claim at least, and that the
software is starting to become sort of stable. Vendor J and C
implementations have been very well field tested over the years.
Of course as with many new functions, if you take the standard set it
will work, if you need something special expect nice creepy bugs, thus
keep your support contract up-to-date and keep that backup ready.

Another worthy point is of course the upgrading of management utilities.

As for getting transit connectivity, shop around there are a couple of
good transit providers who will be more than happy to get another
customer, proofing their management that it was a good idea to already
invest in a full native IPv6 network, instead of waiting for their
customers to leave to another party who did already do it a long time ago.

Can it even be done or will they just run
IPv6 to the core and proxy the rest?

The general idea of "IPv6 transition" is that one will run dual-stack
for the forseeable future, until IPv4 is not used/required anymore,
which might take a very long time.

For folks who want a native IPv6 core there are a variety of
possibilities to do this. Of course not a single one covers all
possibilities thus one has to shop around to match the best one.

Greets,
Jeroen

(And I most likely forgot to mention a lot of other problems and issues)

[snip]

There are more fundamental changes required to spark wide-spread
deployment of IPv6.

* a v6-only "killer application"

* or a real shortage of v4 addresses

* or some other real advantage of v6 over v4, financial and/or technical

However, most important is that those involved with v6 need to
understand that it's impossible to foresee and solve all future
problems. Till then IPv6 isn't mature enough to go anywhere. Any
successor to IPv4 has to provide similar ability to evolve over time,
and operational policies have to evolve with it.

IPv6 as it stand has a lot of room for improvement but that doesn't make
it useless. An important aspect of engineering is to take what you've
got and make the most of it. Where would we be today if early IPv4
implementations had been halted because there was no DNS, no IGP and no
EGP? There was even an internet before we got CIDR :wink:

//per

We've just gone through a pretty decent sized attempt to convert our infrastructure and applications to IPv4/IPv6 dual stack, and was asked by someone else to write up the successes and problems I had doing this. I'm no where near done writing everything up, or even finished with the migration attempt, but here's some bits of my notes, describing our real world experience to throw in. I'll add that I'm small potatoes to you guys out there, I'm sure those with much larger networks will face some additional challenges.

Our biggest client had a need for IPv6 for a couple of reasons.

The first was an application (which surprisingly was IPv6 ready) that required a unique IP address "per instance", they needed to be able to handle tens of thousands of "instances" to work around some brokenness in the application, and RFC1918 wouldn't cut it. Using made up IPv6 addresses with no IPv6 connectivity worked, but we wanted to do it "right".

The second was because of IRC and some Japanese students. This client has a pretty thriving chat community going, based around IRC. One niche of customers and users for this site that suddenly exploded in popularity was with some east asian(mostly Japanese) students, using these services from their dorm rooms or computer labs. The workstations themselves run IPv4, but the university's backbone was IPv6 only. The side effect of this was that all non HTTP IPv4 connections were going through some kind of proxy server, that had a time limit on how long you could keep a session open. They were getting booted off constantly, and the IT staff at the university was asking them to find an IPv6 IRC server they could use instead. (It's possible I'm misrepresenting the situation here, I didn't have direct contact with the people involved, so the technical details here might be wrong).

The third was for VPN. A new product is going to use potentially hundreds of VPNs, again RFC1918 addresses won't work for this application, and we're trying to be a good neighbor and not blow several /24s worth of space on something that didn't really need it. GRE tunnels worked fine for this, and allowed us to do IPv6 inside the tunnel over IPv4 transit and preserve our tiny IPv4 allocations.

But yeah, those are three really weak needs for requiring IPv6, but I'm guessing it's going to be niches like this that start the need for IPv6 at all. So, we decided we'd try to make our network IPv6 enabled, and see how hard it would be to run everything dual stack that would support it.

The first wakeup call that this wasn't going to be easy was that only one of our transit providers supported IPv6 in any way at all. After we got IPv6 turned up, the problems we discovered, roughly in order:

1) IPv6 on the internet overall seems a bit unreliable at the moment. Entire /32's disappear and reappear, gone for days at a time. The most common path over IPv6 from the US to Europe is US->JP->US->EU. I realize this may be specific to our connection itself, but browsing looking glasses seems to back up that it's not just us.

2) Lots of providers who are running IPv6 aren't taking it as seriously as IPv4. Manual prefix filters, NOC staff that doesn't even know they're running IPv6, etc.

3) Some key pieces of internet infrastructure are IPv6 oblivious. ARIN's route registry doesn't support the "route6" objects, for example.

4) Even though we went through our applications and software before starting this to check for IPv6 readiness, there's a huge difference between "Supports IPv6" and "actually works with IPv6".

5) Our DNS software(djbdns) supports IPv6, kind of. WIth patches you can enter AAAA records, but only by entering 32 digit hexadecimal numbers with no colons or abbreviations. We were never able to get it to respond to queries over IPv6, so of all our DNS is still IPv4.

6) Some software supports IPv6 and IPv4 at the same time by using IPv4 addresses on IPv6 sockets. (i.e. they bind to tcp6 on port 80, and an incoming ipv4 connection appears as ::ffff:192.168.0.1). Other applications want to bind to two different sockets. Others want you to run two copies of themselves, one for IPv6 an one for IPv4. However, on a BSD system, the setting net.inet6.ip6.v6only is a system-wide configuration option. If you turn it off (allow IPv4 connections to come in on IPv6 sockets) for one application running on the server that requires it off, and you're running a difference service on the same server that wants to run IPv4 and IPv6 separately, you have to make sure the IPv6 daemon doesn't start before the IPv4, or it will bind to both protocols and the IPv4 daemon won't be able to bind to the port.

7) We found that even if applications support IPv6, they default to disabling it during compiling 95% of the time. We had to go back and recompile our HTTP servers, PHP, all sorts of libraries, etc. Since IPv6 defaults to off on a lot of packages, we had issues just getting the software to BUILD with ipv6 turned on unless we were using the exact same libraries as were current when it was last released.

8) Once we got everything on the network and server side ready for and usable on IPv6 we discovered that a lot of our client's applications just had no idea what to do with IPv6 connections. Many PHP applications broke because they expected $_SERVER['REMOTE_ADDR'] to fit within 15 characters at most. Databases had to have their columns widened (if they were storing the address as text), or functionality had to be rewritten if they were storing IPs as 32 bit integers. Web server log analyzers claimed that the log was "corrupted" if it had an IPv6 address in it. Lots and lots of application logic just wasn't IPv6 aware at all, and either had serious cosmetic problems with displaying IPv6 addresses, or simply didn't work when an IPv6 address was encountered.

9) Once we started publishing AAAA records for a few sites, we started getting complaints from some users that they couldn't reach the sites. Some investigating showed that they had inadvertently enabled IPv6 on their desktop without having any IPv6 connectivity. In some cases it was traced to users reading about 6to4 and wanting to play with it but not correctly installing it, then not UNinstalling it. Others had turned on IPv6 on OS's that make it easy to do so (OS X for example) without realizing what it was. Some browsers/OSes seem better than others at figuring out that they don't have IPv6 working and falling back to IPv4.

10) Smaller than normal MTUs seem much more common on IPv6, and it is exposing PMTUD breakage on a lot of people's networks.

11) Almost without fail, the path an IPv6 user takes to reach us (and vice-versa) is less optimal than the IPv4 route. Users are being penalized for turning on IPv6, since they have no way to fall back to IPv4 on a site-by-site basis when using a web browser.

In the end, we've backed out almost all of our changes to make sites IPv6 visible for now. It broke things for far more IPv4 users than it helped IPv6 users. We've left the IRC services running on a dual stack system, since we were able to partition IPv6 off well enough not to hurt things. Everything else is back to IPv4 only.

I'm also personally a bit concerned with how IPv6 allocation and routing works with respect to small to medium sized networks like ours. I know this is still a hot topic and several proposals are being passed around to resolve some of these issues, but it seems like I *lose* functionality with IPv6 that I have with IPv4, mostly due to the "don't deaggregate your allocation" mantra, and how far the bar was raised to get PI space. We do a lot of things things in IPv4 land with regard to routing and addressing that I don't believe we can do in IPv6, which worries me more. Shim6 and other proposals are creative, but don't replace a lot of the functionality I'd be losing. This is another story though, that is getting really off topic.

Getting your network running IPv6 doesn't seem to be the challenge anymore. None of our L2 devices cared at all. Our L3 devices took some configuration, but moved pretty easily. it's the server and application software that needs a lot more work. I don't think we're even close to the point where an end-user can go to their provider and say "IPv6 me!" and get it working for more hassle than it's worth to them.

-- Kevin

[ .. snip .. ]

1) IPv6 on the internet overall seems a bit unreliable at the moment.
Entire /32's disappear and reappear, gone for days at a time. The
most common path over IPv6 from the US to Europe is US->JP->US->EU. I
realize this may be specific to our connection itself, but browsing
looking glasses seems to back up that it's not just us.

That really depends on who you are using for an upstream.

There are already *several* sane v6 networks in US who have proper
routing and high-quality US-Europe inter-connections, including
commercial networks that one can purchase transit from.

Networks that still implement legacy 6bone routing practice will find
themselves using scenic routes (i.e. US-JP-US-EU).

[ .. snip .. ]

10) Smaller than normal MTUs seem much more common on IPv6, and it is
exposing PMTUD breakage on a lot of people's networks.

This is almost all the time due to people misconfiguring their tunnels.
Popular vendors' routers tend to automatically derive tunnel payload
MTU based from physical circuit MTU instead of a 'commonly accepted'
size.

We had problems where a tunneled peer has SONET interface on their side,
basing a tunnel MTU out of 4470, while our side is manually set to 1480
for ipip tunnel. This causes problem with pmtud.

As outlined in C&W's v6 presentation[1], tunnel MTUs should be explicitly
configured wherever possible.

[1]: http://www.nanog.org/mtg-0505/steinegger.html

[ .. snip .. ]

I'm also personally a bit concerned with how IPv6 allocation and
routing works with respect to small to medium sized networks like
ours. I know this is still a hot topic and several proposals are
being passed around to resolve some of these issues, but it seems
like I *lose* functionality with IPv6 that I have with IPv4, mostly
due to the "don't deaggregate your allocation" mantra, and how far
the bar was raised to get PI space. We do a lot of things things in
IPv4 land with regard to routing and addressing that I don't believe
we can do in IPv6, which worries me more. Shim6 and other proposals
are creative, but don't replace a lot of the functionality I'd be
losing. This is another story though, that is getting really off topic.

I agree.. but this opens up the whole "Great Multihoming Debate" ;>

James

8) Once we got everything on the network and server side ready for
and usable on IPv6 we discovered that a lot of our client's
applications just had no idea what to do with IPv6 connections. Many
PHP applications broke because they expected $_SERVER['REMOTE_ADDR']
to fit within 15 characters at most. Databases had to have their
columns widened (if they were storing the address as text), or
functionality had to be rewritten if they were storing IPs as 32 bit
integers. Web server log analyzers claimed that the log was
"corrupted" if it had an IPv6 address in it. Lots and lots of
application logic just wasn't IPv6 aware at all, and either had
serious cosmetic problems with displaying IPv6 addresses, or simply
didn't work when an IPv6 address was encountered.

Just imagine what it will be like if the idea of
sticking a decimal point into 32 bit AS numbers
ends up getting deployed.

--Michael Dillon

Kevin Day wrote:

9) Once we started publishing AAAA records for a few sites, we started getting complaints from some users that they couldn't reach the sites.

It is possible that a broken 6to4 relay somewhere was causing problems.
Running your own local 6to4 relay (rfc3068) will improve performance and
reduce the chances of going through a broken one.

FWIW, I have been running some AAAA records for almost a year on some
revenue generating sites without any reachability complaints or
drop in traffic. I do run a local 6to4 relay though.

I know this is still a hot topic and several proposals are being passed around to resolve some of these issues, but it seems like I *lose* functionality with IPv6 that I have with IPv4, mostly due to the "don't deaggregate your allocation" mantra, and how far the bar was raised to get PI space.

It sounds like you are an existing ISP in the ARIN reigon. If so then you qualify for a /32 allocation. Let us know if you have any problems getting one. For non-ISP's, the policy is still being worked out. See the ARIN ppml list for discussion. As for deaggregation, it is not
recommended because some filter (/48's) and some don't which results
in sub-optimal paths, but it can be done depending on what your
peers will accept.

- Kevin

Kevin Day wrote:

>9) Once we started publishing AAAA records for a few sites, we started
>getting complaints from some users that they couldn't reach the sites.

It is possible that a broken 6to4 relay somewhere was causing problems.
Running your own local 6to4 relay (rfc3068) will improve performance and
reduce the chances of going through a broken one.

  we have been running w/ AAAA records for production systems
  for the past six years w/o complaint and no 6to4 relay.

--bill

9) Once we started publishing AAAA records for a few sites, we started getting
complaints from some users that they couldn't reach the sites. Some
investigating showed that they had inadvertently enabled IPv6 on their desktop
without having any IPv6 connectivity.

I would hazard an educated guess that the majority of these users had
actually enabled 6to4 via some OS-provided convenience, which *would* work
if it weren't for (a) IPv4 NAT already widely used in "home router"
appliances, resulting in bad 2002:0a00::/24 or 2002:c0a8::/32 addresses, and
(b) many IPv6-capable providers not providing a 2002:: route, or at least
not providing a *working* one, to the 6to4 islands.

Fixing (b) would much allieviate the following when the 6to4 address in
question would otherwise be reachable:

11) Almost without fail, the path an IPv6 user takes to reach us (and
vice-versa) is less optimal than the IPv4 route.

(If a user is implementing 6to4, it usually means that the v4 route *is*
better, so 6to4 becomes a routing policy suggestion as well.)

Kevin Day wrote:

9) Once we started publishing AAAA records for a few sites, we started getting complaints from some users that they couldn't reach the sites.

It is possible that a broken 6to4 relay somewhere was causing problems.
Running your own local 6to4 relay (rfc3068) will improve performance and
reduce the chances of going through a broken one.

FWIW, I have been running some AAAA records for almost a year on some
revenue generating sites without any reachability complaints or
drop in traffic. I do run a local 6to4 relay though.

I know this is still a hot topic and several proposals are being passed around to resolve some of these issues, but it seems like I *lose* functionality with IPv6 that I have with IPv4, mostly due to the "don't deaggregate your allocation" mantra, and how far the bar was raised to get PI space.

It sounds like you are an existing ISP in the ARIN reigon. If so then you qualify for a /32 allocation. Let us know if you have any problems getting one. For non-ISP's, the policy is still being worked out. See the ARIN ppml list for discussion.

specifically, 2005-1

http://www.arin.net/policy/proposals/2005_1.html

Comments by all are welcomed.

Regards
Marshall Eubanks

Have you used CNAME to AAAA records ?

If the customers trying to access the sites used XP with SP1, there was a
bug which make impossible the connection via IPv4 if IPv6 connectivity is
broken. That was sorted out already by Microsoft with SP2.

Otherwise ask for a traceroute6 so the trouble can be followed up.

Regards,
Jordi

Kevin Day wrote:

9) Once we started publishing AAAA records for a few sites, we started getting complaints from some users that they couldn't reach the sites.

It is possible that a broken 6to4 relay somewhere was causing problems.
Running your own local 6to4 relay (rfc3068) will improve performance and
reduce the chances of going through a broken one.

FWIW, I have been running some AAAA records for almost a year on some
revenue generating sites without any reachability complaints or
drop in traffic. I do run a local 6to4 relay though.

I will admit, the number of complaints was very small. But, I don't know how many it was affecting who didn't contact us, so it was decided it wasn't worth the risk with no perceived gain at this point.

I know this is still a hot topic and several proposals are being passed around to resolve some of these issues, but it seems like I *lose* functionality with IPv6 that I have with IPv4, mostly due to the "don't deaggregate your allocation" mantra, and how far the bar was raised to get PI space.

It sounds like you are an existing ISP in the ARIN reigon. If so then you qualify for a /32 allocation. Let us know if you have any problems getting one. For non-ISP's, the policy is still being worked out. See the ARIN ppml list for discussion. As for deaggregation, it is not
recommended because some filter (/48's) and some don't which results
in sub-optimal paths, but it can be done depending on what your
peers will accept.

Correct, we're in the ARIN region. We did qualify for a /32, but just barely, and only because of (hopeful) future growth and some new products we're launching. We do managed hosting for a small number of specialized clients, and act sort of as a content delivery network.

Not long ago we were a little smaller, and qualified for a /20 of our own in IPv4. /20 = 4096 addresses. We were able to pretty easily qualify for that between a few racks of servers, some IP based vhosting(necessary for a few applications), etc. In the IPv6 world, nothing we could do under that business model would qualify us for a direct allocation.

Back then we wouldn't have qualified for a /32 because we wouldn't have met the requirements. We wouldn't have met the proposed 2005-1 requirements for a /44 (we don't come close to 100,000 devices), and lose functionality if we're required to advertise it through a single aggregated address.

Just for the sake of argument though, let's say we didn't get a /32 and had to either get provider assigned space, or a /44 through the 2005-1 proposal.

1) We've got separate POPs in different cities, with no dedicated connectivity between them. They act as entirely independent networks. We don't even have the same transit providers in each city. Some content is replicated to each POP, so we can dump content directly on peers where they want it, or for redundancy. Some content is only available on one POP (dynamic sites, etc), so traffic destined for that POP can only go to that POP.

IPv4: We split our /20 into a /23 for each city, and announce only that.

IPv6: We have to announce the entire /44, and aren't allowed to deaggregate it. Where do we announce it? If I use transit provider Y only in city #1, and I announce to them our /44 even though I only really want a /48 worth coming there, I'm going to end up receiving traffic for city #2 at city #1. I can't even bounce it back off my router onto transit again, because I might see provider Y as the best path for it and it'll loop. Don't say we should ask for a /44 in each city, we don't need that much space in each location, and we're allocating space only to avoid deaggregating it, which is even more wasteful. :slight_smile:

2) We use anycast to select the closest server on our network to a viewer, anycasted DNS for quicker lookups, and a few other anycast services.

IPv4: We have a /24 dedicated for anycast, and announce that block in each city.

IPv6: ??? I honestly have no idea how to do this with IPv6, and still announce only a single block, without effectively anycasting everything we do.

3) We're far past the size where we can renumber every time we change transit providers.

IPv4: We were able to qualify for PI allocations as small as a /22, because we were multihomed. a /20 if we weren't. This is within the range of feasibility of any small/medium hosting company to reach in a very short time.

IPv6: Even forgetting multihoming issues, we're beyond the size where we can do a renumbering without it becoming a serious ordeal. If we're forced to take provider assigned space, we're locked into that transit provider unless we want to leave them so badly that we're willing to go through the hassle. I can't even easily transition from one provider to another by having the new provider announce my PA space from my old provider temporarily, since my provider might be forced to announce everything as a single aggregate as well.

The allocation and advertisement policies work great for small end-sites (they get many subnets, and more addresses than they'll ever need), and great for large providers (they have no problem justifying the need for a /32, and are able to break things up if needed). If you fit somewhere between those two though, I don't think the policies really address our needs.

Small to medium sized hosting providers, content networks and other non-bandwidth-supplying companies generally aren't providing transit to others, so they can't get a /32. They ARE used to being able to deaggregate when necessary though, anycast as needed, multihome easily, and not go through renumbering hell every time you change providers. In the case of hosting companies, a renumbering is especially painful when you have to coordinate renumbering with thousands of customers who are going to see it as a hassle with no benefit.

I completely understand the need to reduce table growth, slow ASN allocations and not waste IPv6 space. But, (and I honestly don't mean this as an attack or flamebait, just a different perspective) it feels like a lot of the big providers are saying "We can't keep up with table growth, we need policies to stop this." Totally understandable. Policies got written to slow the table growth and ASN allocation, but for the most part they're painless for the big providers(easy allocations, and no loss of functionality compared to IPv4), and all the concessions are being made by those on the small-medium side, with very little benefit shown to them.

I don't mean to sound so resistant to change, but IPv6 is really a step backwards for people in our position if we're giving up so much just to get a bigger block of addresses.

-- Kevin

<waits for flame war to erupt> :slight_smile:

Kevin Day wrote:

We wouldn't have met the proposed 2005-1 requirements for a /44 (we don't come close to 100,000 devices), and lose functionality if we're required to advertise it through a single aggregated address.

The high requirements of the "current" 2005-1 were so thoroughly
rejected at the last ARIN meeting that it will probably return
closer to the original 2005-1 at the next meeting. Something like
"if you have an IPv4 assignment/allocation from ARIN you can
get an end site assignement".

There was also a suggestion to eliminate the "single aggregate
announcement" requirement from both end-site and ISP sections.

- Kevin

Kevin Day wrote:
[..]

I agree with your point that currently your IPv4-solution can't be
applied to IPv6 but..(see the helpful and nice thingy part at the end :wink:

1) We've got separate POPs in different cities, with no dedicated
connectivity between them. They act as entirely independent networks. We
don't even have the same transit providers in each city. Some content is
replicated to each POP, so we can dump content directly on peers where
they want it, or for redundancy. Some content is only available on one
POP (dynamic sites, etc), so traffic destined for that POP can only go
to that POP.

IPv4: We split our /20 into a /23 for each city, and announce only that.

This means you are taking up more routing slots. -> note the routing.

The 'problem' here is that we call these allocations "IPv6 Address
Space", not "xxx Routing Space". RIR's provide Address Space and do not
guarantee routability in any way. Hence the subject.

This is actually a more fundamental problem in the current IP ideology
than anything else.

IPv6: We have to announce the entire /44, and aren't allowed to
deaggregate it.

Well, that is actually not true, you can also, in IPv6, simply announce
/48's from a /32 or /44 or whatever. The same way you do that in IPv4.

The issue with announcing say a /48 is though that networks which filter
will filter it out and will only reach you over the aggregate. Of course
that is their choice, just like yours is to try to announce the /48's in
IPv6, or the /23's for IPv4. An IPv4 /28 doesn't reach far either.

The problem here is though that your /48 will propagate through ISP's
with less strict filters and they will act as transits for your /48.
My experience tells that the ones not filtering also have a not so-good
setup thus your connectivity tends to go all around the world.

<SNIP>

2) We use anycast to select the closest server on our network to a
viewer, anycasted DNS for quicker lookups, and a few other anycast
services.

IPv4: We have a /24 dedicated for anycast, and announce that block in
each city.

IPv6: ??? I honestly have no idea how to do this with IPv6, and still
announce only a single block, without effectively anycasting everything
we do.

The same as IPv4 announce a /48. Have a fallback /32 for folks who do
filter it out.

3) We're far past the size where we can renumber every time we change
transit providers.

Indeed, as you have those _Addresses_ assigned to devices.
The issue you have is routing though.

[..]

The allocation and advertisement policies work great for small end-sites
(they get many subnets, and more addresses than they'll ever need), and
great for large providers (they have no problem justifying the need for
a /32, and are able to break things up if needed). If you fit somewhere
between those two though, I don't think the policies really address our
needs.

Here you say it your self: "Advertisement policies" this is routing
again, which has not much to do with Address Space.

Small to medium sized hosting providers, content networks and other
non-bandwidth-supplying companies generally aren't providing transit to
others, so they can't get a /32.

You don't have customers?

If you have say 201 customers, who each have a 1U box in a rack you
provide connectivity to, then give each one of them a /48, they are
endsites and maybe that endsite sets up VPN's.

As for the 201, that is what you might want to have once.
Tada current policy fulfilled. NEXT! :slight_smile:

They ARE used to being able to
deaggregate when necessary though, anycast as needed, multihome

[..]

One can deaggregate in IPv6 also, just don't expect it to be accepted.
Just like a /24 won't be accepted by most folks.
Also note that policy _suggests_ that one announces it as a single chunk.

I completely understand the need to reduce table growth, slow ASN
allocations and not waste IPv6 space.

Nobody is complaining (yet) about ASN consumption. Also 4-byte ASN's
will solve that issue quite easily, except for the pain of hard/software
upgrades, though these have a much less expected impact than going for IPv6.

-- Kevin

<waits for flame war to erupt> :slight_smile:

</me donates his asbestos suite and tin foil hat> :slight_smile:

The "helpful and nice thingy" part:

Apparently the problem faced here can be put into two parts:

*** Getting IPv6 address space from the RIR's in:
    - /44 chunks for PoPs/sites
    - /48 for anycast

This Address Space can then be used for giving Addresses to hosts.
This should not be to difficult to do, except for getting the policy
spelled out correctly and getting it passed through: politics.
One thing that should be included here and spelled out in big
letters: it is Address Space and not a /32, expect it to be filtered.
Also such blocks should come from one large block, eg a /20 per RIR,
which makes it easy to exempt them in filters ala the IX blocks etc.

Of course it might well be that for the forseeable future ISP's will
be lenient in accepting prefixes upto a /48.

This would be sort of similar to:
   http://www.arin.net/reference/micro_allocations.html

*** Multihoming / Route announcement trick that doesn't fill up slots

This is the very hard part. (political and technical)
But it might be years before we will hit the actual limits of BGP.
We still have to be nice to our kids though as they will hit it :wink:

Currently I am aware of the following possible 'solutions':
- SHIM6
- SCTP
- HIP
- NOID
- WIMP
See:
  draft-huston-multi6-proposals
  draft-savola-multi6-nowwhat

Most if not all of these have problems for some uses, eg Traffic
Engineering is supposedly not possible anymore the way it is
currently being done in BGP. Mindset changes will be required.

Here is the area that needs attention.
Only way to get that going is to currently either 'just do it' or
get along in the train with the SHIM6 folks and help them out to
make it the way you might want to see it.

Greets,
Jeroen
   (already caught a nasty cold so come on with
    those flaming arguments to heat me up :wink:

1) IPv6 on the internet overall seems a bit unreliable at the moment.
Entire /32's disappear and reappear, gone for days at a time.

That's certainly true for people not doing it "in production". But that
ain't a problem as they aren't doing it... in production. :slight_smile:

The most common path over IPv6 from the US to Europe is US->JP->US->EU.

Sorry, but that's not true anymore on grand scale. That might still be
valid for some exceptionally bad IPv6 providers who still "do it 6bone
style". Fortunately, those don't play any too significant role anymore in
global IPv6 routing (which was hard work to achieve).

I realize this may be specific to our connection itself, but browsing
looking glasses seems to back up that it's not just us.

That'd suprise me. Could you give examples?

2) Lots of providers who are running IPv6 aren't taking it as
seriously as IPv4. Manual prefix filters, NOC staff that doesn't even
know they're running IPv6, etc.

ACK, but "manual prefix filters" is often rooted in "there are no good
tools to do the job". I've seen folks trying to beat RtConfig for months
into doing sensible things. :slight_smile:

3) Some key pieces of internet infrastructure are IPv6 oblivious.
ARIN's route registry doesn't support the "route6" objects, for example.

Don't get me started about IRRs. :frowning:

5) Our DNS software(djbdns) supports IPv6, kind of. WIth patches you
can enter AAAA records, but only by entering 32 digit hexadecimal
numbers with no colons or abbreviations. We were never able to get it
to respond to queries over IPv6, so of all our DNS is still IPv4.

Then stop using incomplete and cumbersome software from authors with
strong religious believes and a disconnection from any technological
advances of the last $many years. :slight_smile:

"Use the right tools for the job".

10) Smaller than normal MTUs seem much more common on IPv6, and it is
exposing PMTUD breakage on a lot of people's networks.

It is, but we have tracked down most of them... at least the ones we
noticed. I don't experience PMTUD problems anymore since long... the
last one is prolly over half a year ago. And I use IPv6 on all my
servers, desktops and laptop. :slight_smile:

11) Almost without fail, the path an IPv6 user takes to reach us (and
vice-versa) is less optimal than the IPv4 route. Users are being
penalized for turning on IPv6, since they have no way to fall back to
IPv4 on a site-by-site basis when using a web browser.

That is indeed a problem. How big the penalty is, depends heavily on
your choice of upstream provider(s). The isle of sanity gets bigger and
bigger, and networks with bad IPv6 connectivity become more seldom
(relatively).

Thank you for sharing your experience!

BTW, what timeframe are we talking about? Things have changed massively
over the last 12-18 months.

Best regards,
Daniel

The issue with announcing say a /48 is though that networks which filter
will filter it out and will only reach you over the aggregate. Of course
that is their choice, just like yours is to try to announce the /48's in
IPv6, or the /23's for IPv4. An IPv4 /28 doesn't reach far either.

The problem here is though that your /48 will propagate through ISP's
with less strict filters and they will act as transits for your /48.
My experience tells that the ones not filtering also have a not so-good
setup thus your connectivity tends to go all around the world.

This description of the problem ain't totally correct. The problem is
that the more specific route propagates thru fewer paths, thus the
average AS_PATH (and forwarding path) length is usually (much) higher
for the more specific route than the aggregate. Routers decide on CIDR,
so packets to the more specific will travel the long way instead of the
short way.

It's NOT a matter of "the ones not filtering also have a not so-good
setup". Actually, many/most of the "big good IPv6 players" nowadays DO
allow /48s as they recognize the need for "end site multihoming". And
this also contributes to the fact that /48 multihoming is nowadays far
more feasible (as long as your upstreams are "sane and well connected")
than let's say a year ago.

Caveat: this is a EU/US centric view. I'm not sure about the development
in the ASPAC region on this matter as I didn't follow that closely
(partly because it's difficult to follow anything there as it's
network-topological "quite distant" and few hosts and accessible routers
to test from).

The same as IPv4 announce a /48. Have a fallback /32 for folks who do
filter it out.

As outline above, that will artificially impair connectivity
performance.

Here you say it your self: "Advertisement policies" this is routing
again, which has not much to do with Address Space.

It does, as long as we don't have a total decoupling between locators
and identifiers, alongside with the required tools for traffic
engineering with the locators.

One can deaggregate in IPv6 also, just don't expect it to be accepted.
Just like a /24 won't be accepted by most folks.

Uhm, sorry, but that's wrong. /24s are widely(!) accepted and only very
seldom not accepted. There are many (MANY!) folks running on /24 PI
(== no larger covering aggregate) without problems.

> -- Kevin
>
> <waits for flame war to erupt> :slight_smile:

</me donates his asbestos suite and tin foil hat> :slight_smile:

</me shows off his designer collection of asbestos>

This is the very hard part. (political and technical)
But it might be years before we will hit the actual limits of BGP.
We still have to be nice to our kids though as they will hit it :wink:

Really? Where are the limits of BGP? Can you show me any numbers?
You'd be the first. I'm not aware of any protocol inherent scaling
brickwalls like with other protocols where certain timing constraints
place limits (or thinking of L1 systems, you remember CSMA/CD?).
That doesn't mean that there are none - I'm not a scientist or
mathematician - I'd be happy to have any numbers to back the FUD about
the end of world being near. :slight_smile:

Most if not all of these have problems for some uses, eg Traffic
Engineering is supposedly not possible anymore the way it is
currently being done in BGP. Mindset changes will be required.

For all, or only for those who are not deemed worthy of entering the
market on the same terms and conditions like the existing players? :wink:

Best regards,
Daniel

> The issue with announcing say a /48 is though that networks which filter
> will filter it out and will only reach you over the aggregate. Of course
> that is their choice, just like yours is to try to announce the /48's in
> IPv6, or the /23's for IPv4. An IPv4 /28 doesn't reach far either.
>
> The problem here is though that your /48 will propagate through ISP's
> with less strict filters and they will act as transits for your /48.
> My experience tells that the ones not filtering also have a not so-good
> setup thus your connectivity tends to go all around the world.

This description of the problem ain't totally correct. The problem is
that the more specific route propagates thru fewer paths, thus the
average AS_PATH (and forwarding path) length is usually (much) higher
for the more specific route than the aggregate. Routers decide on CIDR,
so packets to the more specific will travel the long way instead of the
short way.

It's NOT a matter of "the ones not filtering also have a not so-good
setup". Actually, many/most of the "big good IPv6 players" nowadays DO
allow /48s as they recognize the need for "end site multihoming". And
this also contributes to the fact that /48 multihoming is nowadays far
more feasible (as long as your upstreams are "sane and well connected")
than let's say a year ago.

Caveat: this is a EU/US centric view. I'm not sure about the development
in the ASPAC region on this matter as I didn't follow that closely
(partly because it's difficult to follow anything there as it's
network-topological "quite distant" and few hosts and accessible routers
to test from).

> The same as IPv4 announce a /48. Have a fallback /32 for folks who do
> filter it out.

As outline above, that will artificially impair connectivity
performance.

> Here you say it your self: "Advertisement policies" this is routing
> again, which has not much to do with Address Space.

It does, as long as we don't have a total decoupling between locators
and identifiers, alongside with the required tools for traffic
engineering with the locators.

> One can deaggregate in IPv6 also, just don't expect it to be accepted.
> Just like a /24 won't be accepted by most folks.

Uhm, sorry, but that's wrong. /24s are widely(!) accepted and only very
seldom not accepted. There are many (MANY!) folks running on /24 PI
(== no larger covering aggregate) without problems.

> > -- Kevin
> >
> > <waits for flame war to erupt> :slight_smile:
>
> </me donates his asbestos suite and tin foil hat> :slight_smile:

</me shows off his designer collection of asbestos>

> This is the very hard part. (political and technical)
> But it might be years before we will hit the actual limits of BGP.
> We still have to be nice to our kids though as they will hit it :wink:

Really? Where are the limits of BGP? Can you show me any numbers?
You'd be the first. I'm not aware of any protocol inherent scaling
brickwalls like with other protocols where certain timing constraints
place limits (or thinking of L1 systems, you remember CSMA/CD?).

Last time I checked, Ethernet is still CSMA/CD.

Correct. And there you have minimum frame spacing requirements (IFG)
and (e.g. with 10Base2 networks) minimum distance between stations
attached to the bus to allow CSMA/CD work correctly.

I'm not aware of any BGP timing stuff that makes it inherently explode
at a certain amount of routes. So what we ACTUALLY talk about is the
fear that control planes (and associated forwarding plane adjacency
lookup systems) won't be able to keep up with the granularity growth in
routing information. Which is a different thing than "we have a BGP
scaling problem".

Best regards,
Daniel