RE: Backbone IP network Economics - peering and transit

Patrick W.Gilmore wrote:
Unless they have cheap access to a free NAP (TorIX, SIX, etc.),
transit, even at higher prices, is probably be the best /
cheapest way to reach the Internet.

This is true, but there are plenty of other opportunities for peering,
such as: both parties buy DS-3 class transit from the same tier-2 or
even maybe tier-3 provider in a colo (which will likely be a BFM, other
problem) not a formal IX. In other words, peering in an IX does cost
money, but peering at a colo might not, as these messy colos are mostly
unmanaged and nobody cares about that 25ft cross-over cable :slight_smile:

Michel.

This is a classical mistake. Peering always costs money and its never free.
The question is, how much, and is it cheaper than transit?

Costs incurred in peering:

- Port Costs (capex)
- A share of a router's backplane capacity corresponding to the port
- Cross connect costs (one time or recurring)
- Operational costs such as legal review for BLPAs, NOC monitoring,
troubleshooting when it flaps, putting MD5 on, etc
- Administration
- Public Switch costs

It is difficult to defend peering strategies today unless your network is of
a fairly significant size (gigabits of traffic) and you are collocated in an
advantageous location(s). Otherwise, low cost transit is hard to beat.

Some of the low cost transit providers will not last, and there WILL be a
second round of bankruptcies and consolidations - Googin's words should be
heeded. However, if you are multihomed and have sufficient transit diversity
(and you don't assume any local loop liabilities), the low cost transit
should be exploited while it lasts.

Patrick W.Gilmore wrote:
Unless they have cheap access to a free NAP (TorIX, SIX, etc.),
transit, even at higher prices, is probably be the best /
cheapest way to reach the Internet.

This is true, but there are plenty of other opportunities for peering,
such as: both parties buy DS-3 class transit from the same tier-2 or
even maybe tier-3 provider in a colo (which will likely be a BFM, other
problem) not a formal IX. In other words, peering in an IX does cost
money, but peering at a colo might not, as these messy colos are mostly
unmanaged and nobody cares about that 25ft cross-over cable :slight_smile:

Michel.

This is a classical mistake. Peering always costs money and its never free.

Maybe "Free" is the wrong word. Perhaps "No additional cost over <transit/whatever>".

Or, for those of us who think that the time it takes to plug a patch cable into an unused switch port and do some configuration changes are irrelevant, maybe "free" is the right word.

Either way, it is not NEARLY as bad as you or many other people make it out to be. Allow me to explain....

The question is, how much, and is it cheaper than transit?

Costs incurred in peering:

- Port Costs (capex)

Pthhhhhhhhh. In many, many cases, especially for smaller providers, this is a spare FE on a switch which already exists.

For mid-sized providers, it is frequently a spare GE port on an existing switch, which means perhaps $500 for GBIC or something.

For large providers, there is a cost here. But large providers are a different beast, and there is no way a simple e-mail could possibly capture the complexities implicit in peering between Very Large Providers. So we'll let them figure out their own costs.

- A share of a router's backplane capacity corresponding to the port

Irrelevant. The traffic has to go somewhere, if it does not go out the peering port, it will go out transit, but it is definitely going across the router's backplane.

A better thing to put here would be possible use of a router which would not be used. Specifically, if I get a bit in a POP which has transit, I do not have to use the router out at the Peering Point. But how many people have router backpanes which are saturated? At worst you are running out of slots for ports in most cases. (Remember, we left the really big providers to their own devices.)

- Cross connect costs (one time or recurring)

Largely irrelevant - if you are really going to go out-of-biz for a $150/Month x-conn, you have bigger problems.

- Operational costs such as legal review for BLPAs, NOC monitoring,
troubleshooting when it flaps, putting MD5 on, etc

These costs are frequently quoted as reasons not to peer by the larger providers. Strangely enough, if you are not a Tier 1 (or hoping to be a Tier 1), peering sessions are usually "set up and forget". Networks who have 10s of gigabits of traffic but are not looking for reasons to deny peering requests see nearly no cost in these (especially compared to the overall cost of running the network).

BLPAs are only required by people who think they mean something. Putting on MD5 is a bug/unique situation, which affect peering perhaps once every half-decade or so. Most small and mid-sized providers can handle the "NOC monitoring, trouble shooting, etc." with single-digit-hours a month, max. And sometimes that time is handled by people who are sitting on their ass waiting for something to break anyway. (Read "sunk cost".)

So, unless you are looking for reasons to *not* peer, these are mostly BS.

- Administration

Think we covered this one.

- Public Switch costs

This is a cost and should be considered. Unless, of course, you are at TorIX, SIX, or any of the other very fine free NAPs available. Or if you can x-conn between your rack and someone else's rack in the same colo facility without going to a public switch. Or if you are in a FR or ATM cloud with other providers and can get uber-cheap PVCs between your routers with no additional hardware and a simple configuration change. Or....

I think you get the point. :slight_smile:

It is difficult to defend peering strategies today unless your network is of
a fairly significant size (gigabits of traffic) and you are collocated in an
advantageous location(s). Otherwise, low cost transit is hard to beat.

I think you mean "or you are colocated in an advantageous location", not "and". If I am in 151 Front street, for a small one-time fee, I can connect to TorIX. The amount of the fee and the time it takes to set up peering is probably in the noise, even for a relatively small provider.

Obviously if my entire traffic fits on a T1, things might be different, but I do not need anywhere near a gigabit of traffic to justify peering. You are probably at least an order of magnitude off.

In general, Peering is a Good Thing [tm]. It increases performance, can lower costs, and might even increase your network reliability. But all the "other" things (e.g. performance benefits) are probably nice-to-have, not requirements. I'd look at the money.

If you can break even or better, peering is probably a good idea. Most of the things analysts and Tier 1 providers talk about with peering costs (legal costs with contracts, managing sessions, NOC time, etc.) are mostly irrelevant, especially to anyone without a stock symbol and the related overhead of large corporations. (In many cases, they are irrelevant even to companies with those things.) So look at your real costs, and real savings:

Hard costs - How much is the NAP connection? How much is the line to the NAP? How much for a router at the NAP? (Etc.)

Then look at your benefits - Who will peer with me? How much traffic can I dump to them? (Etc.)

Add up all the costs in the first set of questions (one time and recurring), subtract your transit cost times the amount of traffic you will save, and see if it is positive or negative. Don't forget to factor in things like any additional cost you incur by having less traffic to commit to a transit provider. (For instance, if you are using tiered pricing, will dumping traffic to peers bring you down to a lower tier, and therefore a higher $/Mbps?)

If your monthly costs are lower with peering than transit alone, it is probably a good idea to peer and ignore the NOC costs. Everyone's situation is different, but don't put too much stock in things like the cost of a good BLPA. :slight_smile:

NISCC Vulnerability Advisory 236929

Believed by whom, is the question.

It has been clearly documented for a long time now that such larger
windows exist. They have even been documented specifically about BGP
(draft-ietf-idr-bgp-vuln-00.txt).

In many, many cases, especially for smaller providers, this is a spare FE on a
switch which already exists.

I assume Vijay meant the cost of a port for private peering, in which case if
you private with all your peers and you have a lot of small peers thats going to
be a lot of cost for a few kbps of traffic

> - Operational costs such as legal review for BLPAs, NOC monitoring,
> troubleshooting when it flaps, putting MD5 on, etc

These costs are frequently quoted as reasons not to peer by the larger
providers.

BLPAs are only required by people who think they mean something.

Well theyre a good excuse thats for certain :slight_smile: But I would say they do mean
something.. if you're BigISP-A and you are peering with BigISP-B you want to
make sure that continues reliably and that means a formal arrangement. Even if
your a small ISP its worthwhile considering a formal arrangement particularly
with the larger peers to make sure they dont ditch you without some good notice
or that they will upgrade without cost if your traffic increases....

In general, Peering is a Good Thing [tm]. It increases performance, can lower
costs, and might even increase your network reliability.

Hmm, we're fairly open on peering and have a bunch of small peers, in fact most
of our new peerings are with small peers (small is something like announcing a
single /24 and doing almost no traffic).

We occasionally see performance problems with these small peers, where they
maybe drop the session without warning raising an alarm here or do something
screwy with their config and leak or whatever.

They also tend to only have one connection, this forces how we route traffic to
them, as we're in the process of expanding I really want to have multiple equal
paths so that we can be sure the traffic is taking the best way to them.

My summary of these points is that I'm seriously considering what our policy
will be in the future and for good reason (altho it will undoubtedly continue to
be fairly relaxed).

If your monthly costs are lower with peering than transit alone, it is
probably a good idea to peer and ignore the NOC costs.

In some instances I'm willing to pay more for a connection (eg paid peering or
costs of backbone circuits) to ensure I'm receiving quality.

There are a couple other issues not raised...

One is the cost on the router in terms of memory and cpu of maintaining such a
large number of sessions (usually less of an issue with your big multiprocessor
routers)

The other is our new hot topic of security, not sure if anyone has thought of
this yet (or how interesting it is) but the nature of the bgp attack means that
if you can view a BGP session you can figure things about a peer that would
otherwise be hidden from you in particular the port numbers in use.. and I'm not
entirely clear on the details but it sounds like when you hit the first session,
you can take the rest out very easily.

We cant take BGP out of band (yet!), perhaps we can keep it better hidden from
view tho..

Steve

In many, many cases, especially for smaller providers, this is a spare FE on a
switch which already exists.

I assume Vijay meant the cost of a port for private peering, in which case if
you private with all your peers and you have a lot of small peers thats going to
be a lot of cost for a few kbps of traffic

It was Dan, not Vijay.

And clearly we are not talking about running a pair of fiber to everyone who has a modem's worth of traffic. He mentioned the cost of the port. I said many people have spare FEs / GEs on existing switches. And if they do not, a few hundred dollars will get them one.

- Operational costs such as legal review for BLPAs, NOC monitoring,
troubleshooting when it flaps, putting MD5 on, etc

These costs are frequently quoted as reasons not to peer by the larger
providers.

BLPAs are only required by people who think they mean something.

Well theyre a good excuse thats for certain :slight_smile: But I would say they do mean
something.. if you're BigISP-A and you are peering with BigISP-B you want to
make sure that continues reliably and that means a formal arrangement. Even if
your a small ISP its worthwhile considering a formal arrangement particularly
with the larger peers to make sure they dont ditch you without some good notice
or that they will upgrade without cost if your traffic increases....

I specifically left out BigISP-*. The complexities of peering on a Tier 1 network are not really describable in a single e-mail.

As for the smaller ISPs, read every peering agreement you've signed. They all say they can cancel with at most 30 days notice, for no reason, with no recourse, and nothing you can do about it. Furthermore, many include the ability to shut down peering if they even *think* you are doing something funny, and again you have no recourse.

Peering agreements are not worth anything to keep peering up. They are only worth something if you are worried about the peer doing something like pointing default.

In general, Peering is a Good Thing [tm]. It increases performance, can lower
costs, and might even increase your network reliability.

Hmm, we're fairly open on peering and have a bunch of small peers, in fact most
of our new peerings are with small peers (small is something like announcing a
single /24 and doing almost no traffic).

The second number there is important, the first is not. There are peers which announce a /24 or few and have gigabits of traffic.

We occasionally see performance problems with these small peers, where they
maybe drop the session without warning raising an alarm here or do something
screwy with their config and leak or whatever.

Nowhere was I saying it is a good idea to peer with someone who hurts your network. But most of the peers, even the small ones, can keep their network stable.

They also tend to only have one connection, this forces how we route traffic to
them, as we're in the process of expanding I really want to have multiple equal
paths so that we can be sure the traffic is taking the best way to them.

Perfectly valid concern. Which is why I specifically told people to find out who would peer with them before paying to go to a peering point. Don't count your chickens until they're hatched and all that. :slight_smile:

My summary of these points is that I'm seriously considering what our policy
will be in the future and for good reason (altho it will undoubtedly continue to
be fairly relaxed).

And I see nothing you mentioned which in any way goes against what I was saying. Your particular situation is very different than the next networks, as the next networks is unique to that network, etc. But that doesn't make peering bad.

If your monthly costs are lower with peering than transit alone, it is
probably a good idea to peer and ignore the NOC costs.

In some instances I'm willing to pay more for a connection (eg paid peering or
costs of backbone circuits) to ensure I'm receiving quality.

It is nice to ensure quality. But if quality is your primary goal, then directly peering with a network will give you better "quality" from an end user (read "paying customer") PoV than transit in most cases. Extra latency is usually not viewed as better quality.

If you are worried about the connection being flaky, well, like I said, don't peer with flaky networks.

Besides, most small to medium guys have enough headroom on their transit connections to take down many of their peers and push it over transit without congestion.

There are a couple other issues not raised...

One is the cost on the router in terms of memory and cpu of maintaining such a
large number of sessions (usually less of an issue with your big multiprocessor
routers)

Agreed. But since we are not talking to the one-T1-ISP (which I also said would not fit the model), people probably have enough CPU to handle a few extra BGP sessions.

If not, well, another cost to consider before peering.

The other is our new hot topic of security, not sure if anyone has thought of
this yet (or how interesting it is) but the nature of the bgp attack means that
if you can view a BGP session you can figure things about a peer that would
otherwise be hidden from you in particular the port numbers in use.. and I'm not
entirely clear on the details but it sounds like when you hit the first session,
you can take the rest out very easily.

Riiiiiiiiiiiiiiiiiiiight.

We cant take BGP out of band (yet!), perhaps we can keep it better hidden from
view tho..

Good idea.

Get right on that, would you? :slight_smile:

The other is our new hot topic of security, not sure if
anyone has thought of this yet (or how interesting it is) but
the nature of the bgp attack means that if you can view a BGP
session you can figure things about a peer that would
otherwise be hidden from you in particular the port numbers
in use.. and I'm not
entirely clear on the details but it sounds like when you hit
the first session,
you can take the rest out very easily.

We cant take BGP out of band (yet!), perhaps we can keep it
better hidden from
view tho..

There are more protection methods available than just MD5 (as you allude to
Steve). One mitigator is to use "non-routed" space for BGP peer
connections. If you have the ability to filter on TTL 255 you are in even
better shape (arguably perfectly secure against all but
configuration/hardware failures). You have some vulnerability with
non-routed space if you do default routing or have folks who default towards
the device doing the BGP peering though. Source routing is also a potential
hazard for the non-routed solution (does anyone have this enabled anymore?).

Apologies for the morph but this raised a great point.

Regards,

Blaine

Hmm. Interesting.

I am (here is SFO area) DSL customer and DialUp customer. But I never
received a notification from my provider(s), possible with free CD,
explaining me (if I am a homewife, not an engineer, of course) what to do
and how to prevent a problems. We have a lot of room for improvement.

In includes not only viiruses, but wireless (when I work in my friend's
home, I use his neighbour WiFi, because it have a bettr quality, withouut
/of course/ his even knowing it -:slight_smile: - or, better to say, his notebook prefer
his neighbour, for some reasons), and so on.

Hmm ok so assume for a moment that I dont want RFC1918 for my links, what are my
options? :

There isnt a "link-local" for IP altho this would be a great solution (surely
this can be written for BGP??).

Or I could use all eBGP addresses from a block which I dont route and filter
internally.. I suspect this is a non-starter, I will have to include all my
addresses given to me by peers and its gonna screw traces, monitoring etc.

Can I use secondary IP addresses and then BGP with these addresses, this would
be a form of "security by obscurity" but providing you can keep the info a
secret thats surely going to do it?

Steve

* steve@telecomplete.co.uk (Stephen J. Wilcox) [Thu 22 Apr 2004, 16:11 CEST]:

There isnt a "link-local" for IP altho this would be a great solution
(surely this can be written for BGP??).

169.254/16

Or I could use all eBGP addresses from a block which I dont route and
filter internally.. I suspect this is a non-starter, I will have to
include all my addresses given to me by peers and its gonna screw
traces, monitoring etc.

I've seen e.g. BT do this

  -- Niels.

Can I use secondary IP addresses and then BGP with these
addresses, this would
be a form of "security by obscurity" but providing you can
keep the info a
secret thats surely going to do it?

It will depend on your architecture in large part. In some cases there is
absolutely no need to route the prefixes that you use for your BGP sessions
beyond the devices doing BGP. This can reduce your exposure to MD5 related
cpu churn etc...

* steve@telecomplete.co.uk (Stephen J. Wilcox) [Thu 22 Apr 2004, 16:11 CEST]:
> There isnt a "link-local" for IP altho this would be a great solution
> (surely this can be written for BGP??).

169.254/16

is not link-local and will be routed if you try. i was thinking link-local as in
ISIS style

> Or I could use all eBGP addresses from a block which I dont route and filter
> internally.. I suspect this is a non-starter, I will have to include all my
> addresses given to me by peers and its gonna screw traces, monitoring etc.

I've seen e.g. BT do this

and public peering? or when viewed from their peers networks..?

Steve

Yes, but (1) its difficult and (2) as these are external sessions I need to
ensure my peers are doing the same, as the chances are they wont and the chances
are the attack comes in externally then I'm still at risk

Steve

There are more protection methods available than just MD5 (as you allude to
Steve). One mitigator is to use "non-routed" space for BGP peer
connections.

Hmm ok so assume for a moment that I dont want RFC1918 for my links, what are my options? :

There isnt a "link-local" for IP altho this would be a great solution (surely
this can be written for BGP??).

Who says BGP sessions must run over IP(v4)?

In theory it shouldn't be a problem to exchange IPv4 routing information over IPv6 BGP TCP sessions. (But it seems some of our favorite vendors didn't add this scenario to their regression tests.)

Or I could use all eBGP addresses from a block which I dont route and filter
internally.. I suspect this is a non-starter, I will have to include all my
addresses given to me by peers and its gonna screw traces, monitoring etc.

Can I use secondary IP addresses and then BGP with these addresses, this would
be a form of "security by obscurity" but providing you can keep the info a
secret thats surely going to do it?

If you combine the two approaches above and filter all traffic to the primary address, traceroutes et al still work but people from the outside don't get to hit the route processor.

Date: Thu, 22 Apr 2004 18:03:33 +0200
From: Iljitsch van Beijnum

Who says BGP sessions must run over IP(v4)?

NetBEUI, anyone? No bickering over RFC1918 on WAN links... :wink:

Eddy