IPv6 internet broken, cogent/telia/hurricane not peering

Mailman List Mirror (Read Only)
21

Randy Bush wrote:

sure would be nice if there was a diagnosis before the lynching

If this happened in v4, would customers care 'why' it happened?
Obviously not.
Why should v6 be any different? It either is or is not production
ready. I'm interested in HE's view on that.

many of us are interested in diagnosis. few in your lynch rope.

What Randy has been *hinting* at is largely relevant...

I'm a /32 holder, with clients that have /48. I would appreciate some of
the diagnostic paperwork that has been written...

Steve

ps. I'm not choosing sides in any way, nor do I want to start a flame,
but HE has been exceptionally helpful v6-wise since I got into the game.

22

Seth Mattinen wrote:

> If you are interested, I don't want to spam the list with my Verizon
> horror story, but you can read it here:
> IPv6 Changes – Roller Network

At the risk of sounding like I'm piling on, I'm in the same basically
the same boat that Seth is, except that I do know who my account rep is
and have been in touch with him.

Verizon's policy has been related to me that they will not accept or
propogate any IPv6 route advertisements with prefix lengths longer than
/32. Full stop. So that even includes those of us that have /48 PI
space from ARIN that are direct customers of Verizon.

Looks like Verizon doesn't want any IPv6 customers. If a company
has idiotic policies like this vote with your wallet.

23

Unfortunately, not everyone always has that choice.

24

If there isn't as choice then regulation is required.

Mark

25

Mark Andrews wrote:

26

Mark,

27

What about the small matter of all of the current AAAAs for the the IPv6 enabled root DNS servers?

28

Mark,

Verizon's policy has been related to me that they will not accept or
propogate any IPv6 route advertisements with prefix lengths longer than
/32. Full stop. So that even includes those of us that have /48 PI
space from ARIN that are direct customers of Verizon.

Looks like Verizon doesn't want any IPv6 customers. If a company
has idiotic policies like this vote with your wallet.

Not knowing all the details, it is difficult for me to judge, however it is worth observing that provider independent addresses, regardless of where they come from or whether they are IPv4 or IPv6 simply do not scale. In the face of everybody and their mother now being able to obtain PI prefixes from all the RIRs, any ISP that handles full routing is going to have to hope their router vendor of choice can keep buying more/bigger CAMs (passing the expense on to the ISP who will pass it on to their customers) and/or they'll start implementing the same sort of prefix length limitations that we saw back in the mid-90s.

I disagree. With IPv4 the bigger issue is that everyone and their mom has 9 different announcements behind their single ASN.

With IPv6, it probably won't be the ideal 1:1 ratio, but, it will come much closer. Even if the average drops to 1/2, you're
talking about a 70,000 route table today, and, likely growth in the 250-300,000 route range over the next 5-10 years.
CAM will probably scale faster than that.

The problematic time scale is that time where we have to support dual stack for a majority of the network. That's what will
really stress the CAM as the IPv6 table becomes meaningfully large (but not huge) and the IPv4 table cannot yet be
retired.

And, of course, we have IPv4 runout in the near future with the inevitable market which will almost certainly promote the use of longer prefixes.

There is that problem, too. Personally, I think the market was a horrible idea, but, it had way too much momentum for
me to be able to stop it.

In other words, get used to it.

Pretty much. I think eventually, we're going to have to look at moving to an ID/Locator split method
in the IDR realm.

Owen

29

From where I sit, it looks like:

a.root-servers.net has IPv6 address 2001:503:ba3e::2:30
BGP routing table entry for 2001:503:ba3e::/48

f.root-servers.net has IPv6 address 2001:500:2f::f
BGP routing table entry for 2001:500:2f::/48

h.root-servers.net has IPv6 address 2001:500:1::803f:235
BGP routing table entry for 2001:500:1::/48

j.root-servers.net has IPv6 address 2001:503:c27::2:30
BGP routing table entry for 2001:503:c27::/48

k.root-servers.net has IPv6 address 2001:7fd::1
BGP routing table entry for 2001:7fd::/32

l.root-servers.net has IPv6 address 2001:500:3::42
BGP routing table entry for 2001:500:3::/48

m.root-servers.net has IPv6 address 2001:dc3::35
BGP routing table entry for 2001:dc3::/32

b.root-servers.net has no AAAA record
c.root-servers.net has no AAAA record
d.root-servers.net has no AAAA record
e.root-servers.net has no AAAA record
g.root-servers.net has no AAAA record
i.root-servers.net has no AAAA record

So... Likely, Verizon customers can reach k and m root servers via IPv6
and not the others.

The fact that b, c, d, e, g, and i do not have AAAA records actually concerns me
more than the fact that Verizon customers can only reach two.

Owen

30

Owen,

With IPv6, it probably won't be the ideal 1:1 ratio, but, it will come much closer.

I wasn't aware people would be doing traffic engineering differently in IPv6 than in IPv4.

Even if the average drops to 1/2, you're talking about a 70,000 route table today,

How big are IPv6 objects in CAMs again?

and, likely growth in the 250-300,000 route range over the next 5-10 years.
CAM will probably scale faster than that.

I've heard differing opinions on this (e.g., router ASICs being both some of the most complicated ASICs ever made and being non-commodity parts hence not necessarily following Moore's Law, pin density in those ASICs reaching a point where you start running into crosstalk problems, cats and dogs living together, mass hysteria, etc). I'm not a hardware guy so I'll just stare blankly.

The problematic time scale is that time where we have to support dual stack for a majority of the network. That's what will
really stress the CAM as the IPv6 table becomes meaningfully large (but not huge) and the IPv4 table cannot yet be
retired.

Right. And when are we planning on retiring IPv4 again?

Interestingly, if you're an ISP and you don't want to redeploy your insanely expensive high end routers with the huge CAMs, you might look to see which prefixes you could drop that would cause the least impact to the majority of your customers. In this light, filtering the crap out of IPv6 would appear to make business sense.

I think eventually, we're going to have to look at moving to an ID/Locator split method in the IDR realm.

The big challenge with this is backwards compatibility...

Regards,
-drc

31

From where I sit, it looks like:

..snip..

So... Likely, Verizon customers can reach k and m root servers via IPv6
and not the others.

or.. vzb (is now dead, it's all vz) has holes in filters to permit
prefixes of certain lengths or certain prefixes exactly. I believe
since I can reach k-root from my uu-connected + uu-v6'd device that'd
be the case.

-chris

32

I thought Tony's preso from RAWS was available or part of the report,
no? (which seemed pretty clear to me about cam sizes and asic
capabilities not going to meet the needs within the next 5-7 years)

-chris

33

Owen DeLong wrote:

From where I sit, it looks like:

a.root-servers.net has IPv6 address 2001:503:ba3e::2:30
BGP routing table entry for 2001:503:ba3e::/48

f.root-servers.net has IPv6 address 2001:500:2f::f
BGP routing table entry for 2001:500:2f::/48

h.root-servers.net has IPv6 address 2001:500:1::803f:235
BGP routing table entry for 2001:500:1::/48

j.root-servers.net has IPv6 address 2001:503:c27::2:30
BGP routing table entry for 2001:503:c27::/48

k.root-servers.net has IPv6 address 2001:7fd::1
BGP routing table entry for 2001:7fd::/32

l.root-servers.net has IPv6 address 2001:500:3::42
BGP routing table entry for 2001:500:3::/48

m.root-servers.net has IPv6 address 2001:dc3::35
BGP routing table entry for 2001:dc3::/32

b.root-servers.net has no AAAA record
c.root-servers.net has no AAAA record
d.root-servers.net has no AAAA record
e.root-servers.net has no AAAA record
g.root-servers.net has no AAAA record
i.root-servers.net has no AAAA record

So... Likely, Verizon customers can reach k and m root servers via IPv6
and not the others.

I can see the /48's out of 2001 in Verizon's table.

~Seth

34

Owen DeLong wrote:

From where I sit, it looks like:

a.root-servers.net has IPv6 address 2001:503:ba3e::2:30
BGP routing table entry for 2001:503:ba3e::/48

f.root-servers.net has IPv6 address 2001:500:2f::f
BGP routing table entry for 2001:500:2f::/48

h.root-servers.net has IPv6 address 2001:500:1::803f:235
BGP routing table entry for 2001:500:1::/48

j.root-servers.net has IPv6 address 2001:503:c27::2:30
BGP routing table entry for 2001:503:c27::/48

k.root-servers.net has IPv6 address 2001:7fd::1
BGP routing table entry for 2001:7fd::/32

l.root-servers.net has IPv6 address 2001:500:3::42
BGP routing table entry for 2001:500:3::/48

m.root-servers.net has IPv6 address 2001:dc3::35
BGP routing table entry for 2001:dc3::/32

So... Likely, Verizon customers can reach k and m root servers via IPv6
and not the others.

I can see all of those through Verizon, so I'm not sure of how their policy applies, or if they're making an exception for these, but they are visible through Verizon.

35

David Conrad wrote:

Verizon's policy has been related to me that they will not accept
or propogate any IPv6 route advertisements with prefix lengths
longer than /32. Full stop. So that even includes those of us
that have /48 PI space from ARIN that are direct customers of
Verizon.

Looks like Verizon doesn't want any IPv6 customers. If a company has idiotic policies like this vote with your wallet.

Not knowing all the details, it is difficult for me to judge, however
it is worth observing that provider independent addresses, regardless
of where they come from or whether they are IPv4 or IPv6 simply do
not scale. In the face of everybody and their mother now being able
to obtain PI prefixes from all the RIRs, any ISP that handles full
routing is going to have to hope their router vendor of choice can
keep buying more/bigger CAMs (passing the expense on to the ISP who
will pass it on to their customers) and/or they'll start implementing
the same sort of prefix length limitations that we saw back in the
mid-90s.

And, of course, we have IPv4 runout in the near future with the
inevitable market which will almost certainly promote the use of
longer prefixes.

And I look at the other side of it. For us "mere" end-user organization (ie, not big backbone ISPs who have dominated the discussion for so long), IPv6 without PI is an utter and complete non-starter.

Despite how huge of a proponent of IPv6 deployment that I am, until PI space was available, I didn't start the work, because without PI space, its utter foolishness for a company that depends on their Internet access to move forward. I don't think its a coincidence that we've seen a big uptick in deployment of IPv6 since PI space became available. Telling end-user organizations to get a block from each upstream and multihome by putting an address from each upstream on every system is now and always has been a bad joke.

In other words, get used to it.

In other words, figure it out. Either we'll have PI space in IPv6, or IPv4 is going to get *crazy* fragmented as continually smaller blocks are bought and sold.

If you want to keep your cam tables from blowing up, I truly think the way forward is to get people to IPv6 as quickly as possible, where they can get blocks big enough to put all of their address space in 1 or 2 blocks, rather than the 4, 7 or more, blocks that they're currently using for IPv4.

And, no, not everyone deaggregates for TE purposes. No network that I've ever been in charge of has ever advertised anything but the most aggregated blocks that we could given the allocations/assignments we had. We'll have savings from that, and if you want to filter to limit deaggregating for TE purposes, I'm quite OK with that.

But if you cut out PI space, you're dead in the water, we just can't have that.

36

In a message written on Mon, Oct 12, 2009 at 05:09:41PM -0700, Owen DeLong wrote:

With IPv6, it probably won't be the ideal 1:1 ratio, but, it will come
much closer. Even if the average drops to 1/2, you're
talking about a 70,000 route table today, and, likely growth in the
250-300,000 route range over the next 5-10 years.
CAM will probably scale faster than that.

Here's a presentation from 2007.

http://www.vaf.net/~vaf/apricot-plenary.pdf

On page 13, you'll find a table. It starts with numbers in November
of 2006, and makes projections. The 5 year projections (Nov 2011)
have already been exceeded, in both IPv4 Internet Routes and Active
ASN's.

The problem isn't that we have 300,000 "global routes" on the
Internet (CIDR Report), but
that there are other things that compete for TCAM space. It's that TCAM
must hold not only the global routes, but also:

  - Internal routes. Your IGP routes, no-exported customer
    deagregations, blackhole routes, etc.
  - MPLS Labels, including:
      - MPLS Traffic Engineering
      - MPLS VPN Identifiers
  - Virtual Routing Instances for Layer 3 VPN's.
  - ARP Entries
  - Multicast Routes

Unfortunately details are hard to come by as most of the folks who
see this in any significant way (e.g. global "tier 1" full service
ISP's) tend not to release too many specific numbers for competitive
reasons.

That said, even using some basic assumptions (some of which are in
the preso) those 300,000 global routes might have added to them:

  300,000 global routes
  150,000 internal routes
   20,000 MPLS labels
  200,000 VPN/VRF Routes
    5,000 ARP Entries
   20,000 Multicast Routes

37

I get asked often enough about what's in 701's IPv6 routes so I just
dumped it to a plain text file for anyone interested:

http://www.rollernet.us/wordpress/as701-ipv6/

~Seth

38

Leo Bicknell wrote:

Worse, the problem is being made worse at an alarming rate. MPLS
VPN's are quicky replacing frame relay, ATM, and leased line circuits
adding MPLS lables and VPN/VRF routes to edge routers. Various
RIR's are pushing "PI for all" in IPv6 based on addressing availbility.
Some networks are actually finally using multicast for IPTV services,
generating much larger number of entries than the global multicast table
would otherwise indicate.

It's not the RIR's fault. IPv6 wasn't designed with any kind of workable
site multihoming. The only goal seems to have been to limit /32's to an
"ISP" but screw you if you aren't one. There was no alternative and it's
been how long now? PI, multihoming, multicast, etc. is reality because
the internet is now Very Serious Business for many, many people.

Yes, I know there's hacks like SHIM6 and I don't wish to go OT into a
debate about them, so I'll just say that if there had been a viable
alternative to multihoming as we know it I think it would have been
given a go before policy got pushed to the RIR's to allow IPv6 PI.

~Seth

39

IPv6 -policy- wasn't initially designed for any workable site multihoming.
The addressing and BGP stuff works fine for it. Its just not "different"
to the issues faced with IPv4.

adrian

40

Leo Bicknell wrote:

Worse, the problem is being made worse at an alarming rate. MPLS
VPN's are quicky replacing frame relay, ATM, and leased line circuits
adding MPLS lables and VPN/VRF routes to edge routers. Various
RIR's are pushing "PI for all" in IPv6 based on addressing availbility.
Some networks are actually finally using multicast for IPTV services,
generating much larger number of entries than the global multicast table
would otherwise indicate.

It's not the RIR's fault. IPv6 wasn't designed with any kind of workable
site multihoming. The only goal seems to have been to limit /32's to an

here's where a pointer to this dicussion of ~4yrs ago should be (on
this list no less)... that said: "Hey, this is afu, if you as
operators want this to work properly, please, please, please get your
butts on v6ops@ietf and make some noise."

I believe that'd have been from me, but marla azinger also sent out
some similar emails and presented 2-3 times at past nanog meetings
about multihoming options wrt ipv6. This ain't gonna get fixed by
nanog-kvetching.

"ISP" but screw you if you aren't one. There was no alternative and it's
been how long now? PI, multihoming, multicast, etc. is reality because
the internet is now Very Serious Business for many, many people.

v6 was designed in an era quite different than today's network. there
were a large number of assumptions made, practically none of which
hold water today. this can't get fixed here, please see
v6man/v6ops@ietf.

Alternately please see rrg@ietf or lisp@ietf, rrg's looking to make a
decision on their research 'soon', lisp is looking for active folks to
provide comment/direction...

Yes, I know there's hacks like SHIM6 and I don't wish to go OT into a

there are no (save lisp) network based 'hacks' for this...
shim6/hip/mip all basically do host-level multihoming, which is cool,
and may be useful to some folks, but they are not useful for folks
trying to do TE in the network. (which also was hashed out quite a bit
on this list)

debate about them, so I'll just say that if there had been a viable
alternative to multihoming as we know it I think it would have been
given a go before policy got pushed to the RIR's to allow IPv6 PI.

100% agreement... wanna join in the discussion and offer some
options/fixes/commentary?

-chris