Cost per prefix [was: request for help w/ ATT and terminology]

Mailman List Mirror (Read Only)
1

I see a minor problem with that in that if I don't actually need a chassis
as large as the 6500/sup2, there's a bit of a hefty jump to get to that
platform from potentially reasonable lesser platforms. If you're upgrading,
though, it's essentially a discard of the sup2 (because you lose access to
the chassis), so it may be fair to count the entire cost of the sup720-3bxl.

Punching in 720-3bxl to Froogle comes up with $29K. Since there are other
costs that may be associated with the upgrade (daughterboards, incompatible
line cards, etc), let's just pretend $30K is a reasonable figure, unless
someone else has Figures To Share.

... JG

2

Wouldn't a reasonable approach be to take the sum of a 6500/msfc2 and a 2851, and assume that the routing computation could be offloaded?

The difficulty I have with this discussion is that the cost per prefix is zero until you need to change eigenstate, where there's a big cost, and then it goes back to zero again.

Because this isn't really all that new a problem, most vendors try not to make devices which have no headroom at all - so kit in the lower category seems to be qualitatively different.
-David

Joe Greco wrote:

3

We have a winner.

The problem with William's calculation is that he is claiming the _only_ difference between X & Y is "prefix count". (He said this, more than once.)

He is dead wrong.

I cannot think of a pair of boxes where one can support a full table and one can't where the _only_ difference is prefix count. (I am excluding things like "Box X with 128MB RAM and Box X with 1 GB RAM".) Even the Sup2 & 3blx have far more differences than just prefix count. If you do not understand why, you clearly aren't competent to post to this thread.

For every network, there is equipment that will work, and equipment that will not. At the top end for very large networks, buying less than a CRS1 / T640 is simply not an option. And the amount of engineering going into those boxes to deal with a 1M BGP entry table is essentially _zero_. Many networks buying those boxes have 100s of 1000s of prefixes in their _IGP_, so the FIB / processor / RAM / etc. all have to deal.

Near the bottom end, for networks that could get away with a 3750 if they only supported a full table (which I submit is a pretty small fraction of the total boxes sold), they can still buy the 3750 and default to a pair of cheap 2/3/4-port boxes with a gig of RAM and use those for external connectivity. The problem is perfectly solvable without jumping to the 7600/3blx.

In between there are more variations than everyone here combined could possibly imagine. And the requirements are _NOT_ all about prefix count. Which means you have no basis for comparison. If you try to force a general comparison, you will be wrong.

So stop asking "what box would you compare?" The answer is "whatever I need!", not what YOU think is correct, since you are invariably wrong unless you know everything about MY network.

Anyone who thinks differently is either confused or has an agenda they are pushing. Or, possibly, trying to sell you a bigger box.

PLEASE, let the thread die.

4

Is there really any point in trying to put a $ figure on each route?

Jon,

Emphatically Yes!

Right now we rely on ARIN and the RIRs to artificially suppress the
growth of the prefix count and with it the availability of PI space.
This is a Really Bad Thing on so many levels, but absent a viable
market-based solution to the problem, authority-based rationing is
really the only thing we can do.

If we can determine the cost to announce a prefix then we could
develop a market-based solution to the problem... One where instead of
suppressing the prefix count and dealing with it as business overhead,
we GET PAID for announcing and propagating prefixes.

There are several market models that could work, but they all depend
on having a reliable metric for the cost of announcing a prefix. So,
if you think you'd like to get paid for announcing routes instead of
continuing to give the service away for free then yes, there is a
point to determining the cost of a prefix.

Wouldn't a reasonable approach be to take the sum of a 6500/msfc2 and a 2851,
and assume that the routing computation could be offloaded?

David,

No, because the FIB can't be offloaded; it has to sit on the router's
forwarding plane and it has to keep up. Between the low single-digit
gbps and the low double-digit gbps, equipment which both keeps up and
supports a large prefix count is significantly more expensive than
equipment which keeps up but does not support a large prefix count.

The difficulty I have with this discussion is that the cost per prefix is zero until you need to
change eigenstate, where there's a big cost, and then it goes back to zero again.

This is one of the harder aspects of operations cost analysis to wrap
your head around. Not only isn't the operations cost attributable to a
particular feature set bound to the associated manufacturing cost,
it's different for different scenarios in which the equipment is used.
To accurately assess the cost, you have to identify the boundary
conditions that drive equipment selection.

Let me construct two scenarios which illustrate this:

Scenario A: You have 1U LAN switches serving 500 PCs at 100mbps and a
few dozen servers at 1gbps. You want to upgrade to 1gbps to the PCs
and 10gbps to the servers. You replace the 1U switches with
7600/sup720s.

The cost driver in this scenario is the 10gbps server links. You can
get 1U switches that stack and backbone at 10gbps but to hook up all
the servers at 10gbps (to support the 1gbps workstations) you have to
step up to the next class of equipment. This is the sole reason you
need a 7600 instead of a few 1U gig-e switches. Therefore the 100% of
difference in cost between a 7600 and a few 1U gig-e switches is
attributable to the forwarding capacity required by the servers.

In scenario A, the 7600's much larger prefix carrying capacity is not
a cost driver. It plays no role in the decision to purchase a 7600
instead of 1U switches. As a result, the operations cost attributable
to the 7600's larger prefix capacity is exactly zero.

Scenario B: You have a metro ethernet POP moving 2 gbps of traffic
with a couple 1U fiber switches. Use is gradually increasing; you
project that 12 months out it will be 2.5gbps. For business reasons
you want to extend the DFZ to this POP. You replace the 1U switches
with a 7600/sup720-3bxl.

The cost driver in this scenario is the DFZ extension, specifically
the prefix count in the DFZ. At 2gbps, your existing equipment is not
even close to tapped out, but it can't even think about carrying the
DFZ's prefixes in its FIB. The sole reason you need a 7600 instead of
a few 1U gig-e switches is the prefix count. Therefore the 100% of
difference in cost between a 7600 and a few 1U gig-e switches is
attributable to the prefix count required by the DFZ.

Almost the exact same upgrade, but in one scenario the cost is
attributable to the forwarding capacity and in the other its
attributable to the prefix count.

I think this is what's giving Patrick the most trouble as well. He
wants equipment cost at an operations level to map to the component
manufacturing cost as common sense says it should, but it doesn't work
that way.

Regards,
Bill Herrin

5

Right now we rely on ARIN and the RIRs to artificially suppress the
growth of the prefix count and with it the availability of PI space.
If we can determine the cost to announce a prefix then we could
develop a market-based solution to the problem... instead of
suppressing the prefix count, we GET PAID for announcing and
propagating prefixes.

Right now, we rely on our appetites to tell us when we're full, and should stop eating tasty sammitches. If we can determine the cost of obesity, then we could develop a market-based solution to the problem... instead of eating less, we GET PAID for eating tasty sammitches!

Not sure this one holds together, when viewed at the macro-level. Note that this is my first posting of the morning, is probably crustier than average, and should most likely be ignored.

                                 -Bill

6

William Herrin wrote:

Right now we rely on ARIN and the RIRs to artificially suppress the
growth of the prefix count and with it the availability of PI space.
This is a Really Bad Thing on so many levels, but absent a viable
market-based solution to the problem, authority-based rationing is
really the only thing we can do.

If we can determine the cost to announce a prefix then we could
develop a market-based solution to the problem... One where instead of
suppressing the prefix count and dealing with it as business overhead,
we GET PAID for announcing and propagating prefixes.

Hi, I'm Google/Yahoo/Microsoft/AT&T/AOL/Sprint/etc. and I plan to annnounce
only /24's and I refuse to pay you to propagate those routes. Are you
really going to drop those routes? Bottom line here is you're going to
have trouble getting the big content providers to buy in, and you're going
to have an equally tough time convincing the major carriers that they should
essentially raise their rates for particular clients. So who exactly is
going to pay and how are you going to convince them they should? If
provider X tells me they're going to charge me $X per prefix I want them to
propagate, I'll just go with provider Y. You're going to need 100% buy-in.

Your solution here is merely a band-aid designed to disguise the actual
problem. Growing prefix count is largely a symptom of missing BGP
functionality. Fix or replace BGP in such a way that we can better control
the flow of incoming traffic without needing hacks like announcing smaller
subnets and prepending and the problem goes away without introducing extra
fees and beauracracy like you're suggesting.

Andrew Cruse

7

Hello Bill:

8

Right now we rely on ARIN and the RIRs to artificially suppress the
growth of the prefix count and with it the availability of PI space.

If by artificially suppress, you mean anyone who wants it can't just fill out a form and be handed a portable /24 or other small CIDR sure...but you don't need PI space to multihome or increase the size of the global table. Giving absolutely anyone who wants it PI space would make things much worse...so I wouldn't call that artificial supression. It's more like keeping the model sustainable.

If we can determine the cost to announce a prefix then we could
develop a market-based solution to the problem... One where instead of
suppressing the prefix count and dealing with it as business overhead,
we GET PAID for announcing and propagating prefixes.

I think you mean "get paid for accepting prefixes" or perhaps "pay into some global pool (for redistribution to the participants) to announce prefixes".

Good luck on that one. In how many languages can you say "not gonna happen"?

9

Giving absolutely anyone who wants it PI space would make things much
worse...so I wouldn't call that artificial supression. It's more like
keeping the model sustainable.

Jon,

Its kinda like gas in the 70's. There wasn't enough to go around given
the price controls so the only way to keep the consumption model
sustainable was with rationing.

Except history tells us that was kinda dumb. Had we allowed prices to
adjust (as we're doing today) the market would have taken care of
itself. Gas would have tripled in price and more folks would have
taken the bus but you wouldn't have the entire nation waiting in line
at the pump.

Right now, announcing a prefix is close to free. If my numbers are
right, it actually costs around $8k/year. A sustainable model selling
an $8k product as a free loss-leader requires some pretty harsh
rationing. Which is precisely what ARIN does, at our insistence.

I think you mean "get paid for accepting prefixes" or perhaps "pay into
some global pool (for redistribution to the participants) to announce
prefixes".

That's right.

The vendors are currently delivering routers that can handle 1M
prefixes and they promise that they can build routers that handle 10M
prefixes with today's technology if there's a demand. If folks want to
pay us more than it costs to dump another 700k prefixes into the
table, why shouldn't we take the profit? It's free money. Even at $8k
there are folks willing to pay it. All it requires is a market
structure that makes the transaction possible.

Let's engage our imaginations, roll with your "global pool" model for
a moment and see if anything interesting pops out of the box.

So, ARIN starts assigning addresses down to a /28 level. The only
requirement for a prefix longer than /20 is that they must remain in
continuous use on the Internet or they'll be revoked

Then, NRO sets up a Universal Service Fund for DFZ routing. Any
transit AS which agrees to accept and normally propagate exactly the
prefixes that are subscribed to the USF is entitled to receive monthly
payments from the USF based on some formula including that AS's
backbone speeds, number of routers and number of peers. At the other
side, anyone who wants their routes carried can make a fixed
contribution to the USF for each prefix that they want to announce,
all the way down to a /32. That only entitles them to announce exactly
that one prefix. If they want to disaggregate, they have to pay for
each of the deaggregates separately.

So now you have folks who can only justify a /28 but its worth
$8k/year to their business to have PI space so that there are no
renumbering costs. And the best part is that they're paying you for
the privilege, paying more than it costs, instead of you having to
blandly accept those prefixes for free.

But wait, it gets better. Now that there's a market structure in
place, its possible to envision different classes of service.

Maybe this market holds a niche for folks who don't want to pay
$8k/year into the USF. Suppose ARIN auctions off the right to announce
a "covering /8" for each of its IANA allocations. The winner can't use
any of the addresses for itself, but it has the right to sell tunnels
to the folks with more specific prefixes. So, if you're a small fry,
maybe you don't pay $8k/year into the USF. Instead, you pay $500/year
each to the two backbones closest to you and then you pay another
$1000/year to the tunnel provider whose /8 covers your prefix. Your
ISP gives you one address from their PA space to catch the endpoint of
that tunnel and for $2k/year you're in business with PI space. If you
picked your backbones right, there's even a decent chance that traffic
following the /8 usually wanders into one of them and redirects your
way before hitting the tunnel.

And suddenly, surprisingly, the Internet works better than ever
without everyone having to carry full routes, you get PAID for the
prefixes you do carry and everyone who wants PI or lots of TE can have
it! Its not free any more, but you can have anything you're willing to
pay for without having to justify yourself to the rationing board.

Good luck on that one. In how many languages can you say "not gonna happen"?

Do programming languages count? $paiddfz=!$happen;

Seriously, the goal may seem unachievable but that doesn't mean it's
not worth striving for. Who knows what we may find on the way?

10

People pay the RIRs.

The RIRs spend money on parties for network operators.

I think that charging for deaggregation of PA is hard to imagine. I think charging for PI as a model may have been worthy of consideration several years ago, but since we're only months away from entire product lines of deployed edge kit nolonger accepting a full table, the battle is over (and operators lost).

Andy

11

As far as I can see, the only way to solve de-aggregation and PI is to create some kind of cryptographic signing of aggregate routes sent out to DFZ.

RIPE/ARIN and other equal instances need to sign the combination of AS and prefix, and this is then used (somehow) to authenticate the prefix when received. This would also have the added benefit of stopping people from sending more specifics with other ISPs IP space (or even their own, as only the actual aggregate prefix would be signed, not more specifics that people use for "TE").

So this "certificate" or alike needs to be time limited and coupled to payment if we're going to charge for PI/PA yearly.

Yes, this increases complexity in the DFZ enormously, and I don't know if the benefit outweighs the complexity and added risks for failures.

12

andy@nosignal.org (Andy Davidson) writes:

People pay the RIRs.

The RIRs spend money on parties for network operators.
...

according to <http://www.arin.net/about_us/corp_docs/budget.html&gt; for 2007 and
<http://www.arin.net/about_us/corp_docs/annual/2006_audited_financials.pdf&gt;
for 2006 and 2005, ARIN's expenses are mostly unrelated to partying.

13

After fielding plenty of offlist mail - I'm sorry that my tongue in cheek comments caused some raised eyebrows; it was a non-serious suggestion for how the RIRs could spend an imaginary windfall, not a comment that they spend too much on socials. I recognise the value of conferencing, and the importance of injecting some socials at conference events since it's really good opportunity for delegates to spend time talking to others in the industry.

Andy