"2M today, 10M with no change in technology"? An informal survey.

Mailman List Mirror (Read Only)
1

In another mailing list, someone has asserted that "noone believes
router vendors who say [they can support 2M routes today and 10M with
no change in technology]".

Haven't we gone over the CPU speed argument recently enough?

Do you believe router vendors who state they today have "capacities
on the order of 2 million ipv4 routes and they have no reason to
expect that they couldn't deliver 10 million route FIB products in a
few years given sufficient demand."?

Pointless argument, wait and see as it doesn't matter.

If they do then we're fine. If they don't then we
won't have that many routes so it won't be a problem.

- where do you believe existing routing technology will fall down?

If people sit watching the train crash rather than take avoiding
action

- what steps will you take/are you taking to limit your vulnerability?

As long as I can afford as big a RP as the next ISP we're fine.

brandon

2

As long as I can afford as big a RP as the next ISP we're fine.

and own enough router vendor stock that you can compensate for running
your business at a loss

randy

3

I think the far more interesting thing will happen long
before we see the tables explode and we have the end of the internet
as we know it (but i'm still fine).

  Vendors will come up with creative solutions and push them to the
market as fast as necessary, or we'll come up with other techniques (eg:
dampening to keep that pesky cpu usage down) to help us muddle through until
the factory builds enough 64-way route-processors to handle the updates,
fib download at 10ge speeds, etc.. that will become necessary in the platforms.

  The reason why I say this is because we're going to see a crunch on
the bandwidth side (IMHO) before we see this crunch on the CPU side. Based
on what I'm hearing from folks in the marketplace, not every network is
selling 10GE's like they're candy. Even with link aggregation and other
tricks, that can get you to what? 8x10G? 16x10G? That may help some
folks until some faster next-gen standard comes out from IEEE or ITU.
It doesn't appear that everyone has jumped on the 40G/768 bandwagon and
that n*10G is much more common because it's "cheap".

  After we all upgrade to 100GE (or the HSII - high speed internet
interface) on our routers, what's next? bundle it a up to 800G? How much
of a $VENDOR chassis will that consume? How much power will it require?
Will dispersion over the medium (assuming fiber) be so bad that it's unusable
after going more than 100 meters? How will the larger networks
interconnect?

  Ignoring how we deal with the rib updates, i care about how the bits
will get moved about. This is far more pressing in the next few years than
can $vendor slap a faster cpu on a board and call it good. I think we all
know that the cpus are underpowered in most routers compared to
the desktop and server space. Compared to the couple of million dollars it
would take to populate a nice HFR^WCRS/t-series network, the cost of a
2.9ghz processor is only $1200 (retail/wholesale), or only about some tiny
fraction of a $1m router. What will matter is how much that HSII costs and
how transport, hashing, and the ability to mux all those 10G or 40G waves
into something useful.

  So I say, Baah on the rib/fib issues for now, but since
no major networks would want to disclose their backbone traffic, it'll be
interesting to see how and when this exposes itself as the sleeping bear
that I personally believe it will become.. (my guess, sometime
2011 or so, maybe earlier).

  then again, i could be some rambling fool who is out of touch.

  - jared (opinions don't reflect those of my employer, wife
or children, or possibly even myself).

4

Maybe I'm oversimplifying. Its Saturday and entirely possible I'm not thinking on all brain cells ($clue--).

When it comes to an FIB, there are only say 100-200 destinations on a big router (outbound). Irrespective of the 2M or 20M routes it has. Even though the entire Internet isn't aggregated the way it should be, for FIB purposes, the router knows how it will route out to its 100-200 destinations (ports).

Couldn't it just aggregate before it drops routes into its FIB and only import specifics (deagg) if a destination changes for a more specific prefix [like the swamp]? We talk about default-free zones as needing every prefix, and they do (for propagation purposes) but for FIB purposes, they really don't.

So they keep a Zebra or OpenBGPd like process (thinking Juniper) that contains *everything* and handles propagation and then each FIB gets an aggregated entry dumped into each FIB on each routing card.

This scales very well on the order of a well aggregated internet, and even networks with lots of customer routes (internally or because of lots of customer transit connections) each router only deals with getting the packet to its neighbor. Server CPUs & RAM can handle lots of updates on a 10MM route table all day long. FIBs only need to get involved if the update will change their view of a route -- a central CPU (or cluster of them) can decide that before the FIB sees it.

If this isn't clear, I can probably explain it better, but basically what I am saying is that instead of aggregated the FIB based on CIDR rules or expecting total conformity, the router just aggregates from its point-of-view -- if an aggregate to multiple disparate netblocks all goes to the same place, it puts an aggregate in.

For example, a router with only 1 connection (no matter how many routes being sent by its upstream), would only have 1 route entered into its FIB -- because no matter where the route goes, it can go upstream. If it feeds a route table to another router, the downstream will see the whole route table irrespective.

I know the above degenerate case doesn't address bogons, but really we are only talking about a few dozen extra entries into an FIB for them.

Deepak Jain
AiNET

5

* Deepak Jain:

When it comes to an FIB, there are only say 100-200 destinations on a
big router (outbound). Irrespective of the 2M or 20M routes it
has. Even though the entire Internet isn't aggregated the way it
should be, for FIB purposes, the router knows how it will route out to
its 100-200 destinations (ports).

Could be more than that in some MPLS and IXP environments, but in
essence, that's true.

Couldn't it just aggregate before it drops routes into its FIB and
only import specifics (deagg) if a destination changes for a more
specific prefix [like the swamp]? We talk about default-free zones as
needing every prefix, and they do (for propagation purposes) but for
FIB purposes, they really don't.

My understanding is that there are no known algorithms for fast
updates (and particularly withdrawals) on aggregated FIBs, especially
if those FIBs are stored in CIDR form. This is the prime reason why
all those Cisco 65xx/76xx with MSFC2/PFC2 will be worthless junk in a
couple of months.

But I'm sure that this is not the end of the story.

For example, a router with only 1 connection (no matter how many
routes being sent by its upstream), would only have 1 route entered
into its FIB -- because no matter where the route goes, it can go
upstream.

This will cause routing loops for unallocated address space.

6

My understanding is that there are no known algorithms for fast
updates (and particularly withdrawals) on aggregated FIBs, especially
if those FIBs are stored in CIDR form. This is the prime reason why
all those Cisco 65xx/76xx with MSFC2/PFC2 will be worthless junk in a
couple of months.

Do we have a real date for when this occurs? If you aren't doing uRPF, I thought they ran up to 256,000 routes. (I may not recall correctly)

"Fast withdrawals".. We don't have instantaneous convergence right now, are you sure you aren't talking about "easy withdrawals" (as in, lots of compute time required per adjustment).

MPLS environments would have additional entries where they are doing something with VRF or MPLS... the nodes in between wouldn't see anything as normal.

For example, a router with only 1 connection (no matter how many
routes being sent by its upstream), would only have 1 route entered
into its FIB -- because no matter where the route goes, it can go
upstream.

This will cause routing loops for unallocated address space.

This would be addressed in the bogon case.

Deepak Jain

7

Heya,

>My understanding is that there are no known algorithms for fast
>updates (and particularly withdrawals) on aggregated FIBs, especially
>if those FIBs are stored in CIDR form. This is the prime reason why
>all those Cisco 65xx/76xx with MSFC2/PFC2 will be worthless junk in a
>couple of months.

Do we have a real date for when this occurs? If you aren't doing uRPF, I
thought they ran up to 256,000 routes. (I may not recall correctly)

We ran into this hiccup a few months ago on a Sup720-3B (well, a 3BXL which
mistakenly had a 3B card in the chassis, causing the SUP to clock down and
act like a 3B), but I think the Sup2's are in a similar situtation. Though
the box can handle up to 224k routes, they are set by default to only handle
192k IPv4 + MPLS routes plus 32k IPv6 + IP multicast routes. You can retune
this so that you can get up to 224k IPv4 routes, but I've recently seen our
Internet table bumping against this. My understanding is that this is a
hardware limit, so upgrading is your only option.

Eric :slight_smile:

8

The sup2 can actually handle a bit more ipv4 routes than the Sup720(non-3bxl). I don't know if it can go all the way to 256k routes. I can't seem to find any cisco data sheets that specify max ipv4 routes on the sup2. The output from show mls cef hardware suggests 256k is the limit.

9

According to this link, which alleges to be from cisco-nsp, an MSFC2 can hold 256,000 entries in its FIB of which 12,000 are reserved for Multicast. I do not know if the 12,000 can be set to serve the general purpose.

The MSFC2 therefore can server 244,000 routes without uRPF turned on.

Any reasonably valid way of predicting when we'll hit 244,000 routes in the default-free zone?

http://osdir.com/ml/network.nsp.cisco/2002-08/msg00283.html

Deepak

10

Um?

Real Soon Now?

According to http://www.cidr-report.org/as2.0/ we're at 233,000 routes (as seen from AS 2.0 now) and the rate of growth as seen from
http://bgp.potaroo.net/ seems pretty steep.

I must be missing something obvious (or should I be dusting off my unused Y2K survival gear?)

Thanks,
-drc

11

David Conrad wrote:

According to this link, which alleges to be from cisco-nsp, an MSFC2 can hold 256,000 entries in its FIB of which 12,000 are reserved for Multicast. I do not know if the 12,000 can be set to serve the general purpose.

The MSFC2 therefore can server 244,000 routes without uRPF turned on.

Any reasonably valid way of predicting when we'll hit 244,000 routes in the default-free zone?

Um?

Real Soon Now?

According to CIDR Report we're at 233,000 routes (as seen from AS 2.0 now) and the rate of growth as seen from
http://bgp.potaroo.net/ seems pretty steep.

I must be missing something obvious (or should I be dusting off my unused Y2K survival gear?)

I found that, eventually. I'm only seeing about 227K routes, but customer routes from wherever the CIDR report is getting data could be part of the difference.

Where do the FIBs break on older 12000 series and M-series routers? (or pick the *next* most popular piece of network equipment that is used in full-routes scenarios).

Maybe I should take a whack at my aggregation idea on an MSFC2 to see how it does. Hmmm..

Deepak

12

Any reasonably valid way of predicting when we'll hit 244,000 routes in the default-free zone?

Um?

Real Soon Now?

...

I must be missing something obvious (or should I be dusting off my unused Y2K survival gear?)

Unlike Y2K, the end of the useful service life up the Sup2 can easily be pushed further away in time.

ASnum NetsNow NetsAggr NetGain % Gain Description

Table 233651 151129 82522 35.3% All ASes

AS4134 1337 339 998 74.6% CHINANET-BACKBONE No.31,Jin-rong Street
AS18566 1020 101 919 90.1% COVAD - Covad Communications Co.
AS4323 1315 437 878 66.8% TWTC - Time Warner Telecom, Inc.
AS4755 1331 507 824 61.9% VSNL-AS Videsh Sanchar Nigam Ltd. Autonomous System

There's really only 151129 routes you need to have "full routes". Forcing just these top 4 slobs to aggregate reduces your global table by 3619 routes. Forcing the top 30 to aggregate frees up 15809 routes.

Of course there are other reasons to upgrade (better CPU, MPLS, IPv6, etc.), but if you can't upgrade, there are alternatives to stretch the old hardware. It's not like it hasn't been done before.

13

Jon,

Any reasonably valid way of predicting when we'll hit 244,000 routes in the default-free zone?

Real Soon Now?

According to Geoff, the BGP table is growing at around 3500 routes per month, so we're looking at blowing out MSFC2s in about 3 months if nothing changes.

And here I was, wondering about 2M routes...

Unlike Y2K, the end of the useful service life up the Sup2 can easily be pushed further away in time.

"Easy" is, I suspect, in the mind of the route injector.

There's really only 151129 routes you need to have "full routes". Forcing just these top 4 slobs to aggregate reduces your global table by 3619 routes.

~1 more month.

Forcing the top 30 to aggregate frees up 15809 routes.

~3 more months.

Of course there are other reasons to upgrade (better CPU, MPLS, IPv6, etc.), but if you can't upgrade, there are alternatives to stretch the old hardware.

For a few more months. What are upgrade cycles like again? How common are the MSFC2s?

It's not like it hasn't been done before.

Yep. The nice thing about repeating history is you have a good idea of the whinage that you're in store for.

"CIDR Wars 2.0: This Time It's For Real! No, really. We mean it this time."

:slight_smile:

Regards,
-drc

"I used to be disgusted, now I try to be amused ..." -- Elvis Costello

14

Now if this was a dust old MSFC2 that was like 5 years old I'd say ok. The problem is twofold:

1. Cisco is still selling the 7600 with the Sup32 bundle (which is what we bought) and saying you can take a full route table on it. I could already do MPLS and IPv6 on this box. This is pretty new hardware.

2. The only thing I could buy is the top of the line Sup720 3BXL. Ok, fine, but I don't need mega-super-d00per backplane speed. I just need more TCAM like Christoper Walken needs more cowbell. Cisco needs to have a reasonable solution to this problem - especially if they want to keep selling the 7600 as a router.

If I end up upgrading because of this it will probably be a forklift upgrade to another platform. And there's no guarantee that it would be a Cisco one.

15

That's an additional ~5 months at the current rate of new
routes (and current ratio of customers per new routed block.)

There's a lot more than 3500 new customers per month globally
and if we get to the point where they are not coming out of
hierarchical PA space, the new monthly routing growth will
increase dramatically.

/John

16

and here I always looked at the 6500 as a switch...

17

And the 7600 is a router?
:slight_smile:

18

I thought it was just a 6500 that sommeone got drunk and tipped over on
it's side, like a cow...

19

1. Cisco is still selling the 7600 with the Sup32 bundle (which is what we bought) and saying you can take a full route table on it. I could already do MPLS and IPv6 on this box. This is pretty new hardware.

Where are they saying that? The Sup32 sounded great until it became clear that it came with PFC3B (not 3BXL), and that there was no upgrade path to 3BXL. If it was/is being sold as a BGP routing solution, it was awfully short sighted.

2. The only thing I could buy is the top of the line Sup720 3BXL. Ok, fine, but I don't need mega-super-d00per backplane speed. I just need more TCAM like Christoper Walken needs more cowbell. Cisco needs to have a

We're in the same boat. According to show catalyst6000, our Sup2's are doing just fine. If there were a Sup32-3BXL, it'd be more than sufficient for our needs.

If I end up upgrading because of this it will probably be a forklift upgrade to another platform. And there's no guarantee that it would be a Cisco one.

I guess cisco wants to play chicken with us and Juniper. Will you really do the forklift, or just bite the bullet and go Sup720-3BXL? I think they're better on the latter and counting on a bunch of hardware sales in the coming months.

20

And tagged with some white paint.

Though if you've kept up with the latest IOS developments, cisco is finally differentiating the platforms we've assumed for years were only different in angle and paint. 6500's won't get to run the newest 7600 code.