Why don't ISPs peer with everyone?

Mailman List Mirror (Read Only)
1

Hello,

I wouldn't consider myself a network engineer, nor do I have any
formal training, but why don't ISPs peer with every other ISP? It
would only save EVERYONE money if they did this, no? Only issue I
see is with possibly hijacked / malicious AS owners, but that's not
very common to do without being caught.

All the whole "don't peer with this guy" only makes your customers
have worse latencies and paths to other people, making the Internet
less healthy.

Thanks,
Rucas

PS: sorry if I sent this twice; client lagged a bit.

2

Not necessarily. Peering with an ISP who wants to take the traffic between your network and theirs through a saturated pipe, an overloaded router, or across an MPLS pipe with 13 underlying hops (each of which could be a choke point themselves) will not make your end-to-end latencies any better.

As others have mentioned, some ISPs do have friendly peering policies. This is particularly true for ISPs that are co-located at the same IXP, because much of the opex is already baked into the ISP's relationship with the IXP.

The reason most of the larger ISPs, particularly those who live in the DFZ, have peering policies (especially for settlement-free peering) that could be construed as less friendly to smaller networks is because those guys want to sell you transit, rather than let you peer for free, or for less than a the full transit rate. It doesn't make financial sense for them to exchange bits with you for free, when they can make money off of those same bits if you buy transit instead.

jms

3

Nope.

It is because who pay the money, and somebody wants to earn the money
because they have more control.
So it is because of "money".

Welcome to the world of capitalism.

Alex

4

Network utopia.

~Jay
“Engineering is about finding the sweet spot between what's solvable and what isn't."
“Good engineering demands that we understand what we’re doing and why, keep an open mind, and learn from experience.”
                                                                                                                                                                            Radia Perlman
"If human beings are perceived as potentials rather than problems, as possessing strengths instead of weaknesses, as unlimited rather than dull and unresponsive, then they thrive and grow to their capabilities."
                                                                                                                                                                         
 Please consider the environment before printing e-mail

5

I wouldn't consider myself a network engineer, nor do I have any formal
training, but why don't ISPs peer with every other ISP? It would only save
EVERYONE money if they did this, no? Only issue I see is with possibly
hijacked / malicious AS owners, but that's not very common to do without
being caught.

Some ISPs have very friendly peering policies, but some obstacles facing even the friendliest ISPs are:

*Poor operator reputation or significantly different networking mindsets may make some peers undesirable
*Potential peer is attempting to become a tier-1 and demands paid-peering
*Potential peers do not have similar POPs or budget for transport between POPs for peering
*Some ISPs do not have the ability to easily determine the destination of their traffic and which peers would be most advantageous in terms of transit reduction
*Potential peer is lazy or reluctant to make changes

I'm sure I'm missing a few, but I believe these are a couple significant obstacles to a more 'meshy' internet.

Nathan

6

its not always about money. sometimes its reputation.

/bill

7

And also reasonably hygene, and both individual and community self defense.

There are some less competent network operators out there (and even
good ones have bad days). And some of the people out there speaking
BGP want to do really malign things with internet traffic, like hijack
and snoop, inject spam, sometimes injecting spam by hijacking someone
else's net temporarily, create malware sites, hack others, etc.

8

I'll answer with some questions:

Where should they peer?

Who should/will pay for the routers and aggregation ports? How about the power, racks, and building space?

Who should/will pay for the network engineers to do the configuration for the peering?

In short, peering isn't free for anyone. It _can_ be efficient in some cases but in others its damn pita and you never really know which one a given case will turn into. (its not always a problem of technical competence)

9

From nanog-bounces+bonomi=mail.r-bonomi.com@nanog.org Mon Jun 6 17:20:16 2011
Date: Mon, 06 Jun 2011 18:19:37 -0400
To: nanog@nanog.org
Subject: Why don't ISPs peer with everyone?
From: rucasbrown@hushmail.com

Hello,

I wouldn't consider myself a network engineer, nor do I have any
formal training, but why don't ISPs peer with every other ISP? It
would only save EVERYONE money if they did this, no? Only issue I
see is with possibly hijacked / malicious AS owners, but that's not
very common to do without being caught.

The answer to _every_ question that starts of "why don't they..." is
"money".

Who pays for the circuits to establish a 'peering connection' with an
ISP half-the world away? How much does traffic does "Joes Bait Shop
and ISP" in Painted Privvy, Nebraska have with a community ISP in
Honshu, JP?"

10

>
> Hello,
>
> I wouldn't consider myself a network engineer, nor do I have any
> formal training, but why don't ISPs peer with every other ISP? It
> would only save EVERYONE money if they did this, no? Only issue I
> see is with possibly hijacked / malicious AS owners, but that's not
> very common to do without being caught.

The answer to _every_ question that starts of "why don't they..." is
"money".

Who pays for the circuits to establish a 'peering connection' with an
ISP half-the world away? How much does traffic does "Joes Bait Shop
and ISP" in Painted Privvy, Nebraska have with a community ISP in
Honshu, JP?"

There are a lot of considerations. How many peering sessions can your
hardware support? How many peering locations are you going to need?
What will the internal network to tie all those together look like. Will
you now need to upgrade your core?

Will adding a new peer place another peering agreement at risk by
changing the traffic balance?

So even if the peering itself is "free", the infrastructure required to
support large scale peering at multiple locations can be quite
expensive. Are you going to want to haul traffic from New Jersey to
California to hand it to a peer who hauls the traffic all the way back
to New Jersey again? Does your router in Kansas City want to hand the
traffic to the peer in New York or in Seattle? For a small regional
network, peering can be easy. For a large network that spans a
continent, it can be pretty hard.

11

I wouldn't consider myself a network engineer, nor do I have any
formal training, but why don't ISPs peer with every other ISP? It

It depends on the ISP, but there are a variety of reasons for not
wanting to peer with any potential peer or in this case "every other
ISP". Also let's distinguish between paid-peering and settlement-free
peering. I think we can agree that if there were only paid-peering,
then a complete mesh would be not only technologically impractical, but
also economically as well.

Plug the terms "economics", "internet" and "peering" into your favorite
search engine and you should come up with some relevant reading
material.

Only issue I
see is with possibly hijacked / malicious AS owners, but that's not
very common to do without being caught.

Can you explain why this is a bigger issue in your scenario?

All the whole "don't peer with this guy" only makes your customers
have worse latencies and paths to other people, making the Internet
less healthy.

Certainly most ISPs care about that to some degree, but to get to the
heart of the matter, consider the mindset of any profit-motivated ISP,
especially where one is "larger" in some sense of the word than the
other who wants to peer. If I'm the larger ISP, and you're the smaller
ISP coming to me to peer settlement-free, why should I peer with you?
So our customers can get better performance to each other? Why don't
your customers just connect to me instead? What do I lose if we don't
peer? If you're small, probably not too much.

John

12

ISPs are often concerned about: Cost of Peering, and Loss of Revenue due
to peering -- ISPs usually like to charge for internet services they provide.
Free peering is only beneficial to both sides of a peering relationship when
it does reduce costs more than it reduces expected revenue.

(a) Costs of peering; both in terms of administrative overhead,
ports, circuits, cabinet space,
and system resources on existing equipment. Creating a presence in an
exchange or building
media connections from one ISP to another is not free, ISPs don't
naturally all have equipment
within range of a free patch cable.

Every peering connection a router deals with requires some computing power,
some memory, table entries on the router, and, depending on the exchange,
possibly additional physical connections.

And of course, there are man hours to maintain peering sessions.
ISPs are more likely to peer when cost is low relative to advantages
after all considered.

(B) Loss of revenue due to peering. An extreme example is a very
large ISP peering
with a small ISP, to allow the small ISP to reach large ISP's customers.
The large ISP loses revenue, if they provide the peering for free,
since it would mean
the small ISP is not paying for that transit.

Example: If Level3 peered with anyone who wanted, for free...
that would mean noone
would have to buy transit from Level3 to send traffic to Level3 customers.

There is an analogous situation for ISPs of all sizes though.
And if they do agree to peer, there is usually some stipulation about the ratio
of traffic beng sent versus received.

ISPs do not want to peer for free, if there is a chance their partner
would need to buy services from them directly, or indirectly
(without the peering),
that exceed the benefit/cost reduction of peering.
And once a customer, never a peer.

13

(a) Costs of peering; both in terms of administrative overhead,
ports, circuits, cabinet space,...

The cost of peering on an IXP is roughly the same as setup fees for a
new transit, and a BGP session to an IXP route server is not far from
what will a full view cost in RAM and CPU on your edges.

(B) Loss of revenue due to peering. An extreme example is a very
large ISP peering
with a small ISP, to allow the small ISP to reach large ISP's customers.
The large ISP loses revenue, if they provide the peering for free,
since it would mean
the small ISP is not paying for that transit.

Large ISPs do buy transit too. On a financial perspective, it can be
considered as "outsourcing the peering function", with a paid SLA for
this connectivity...

And once a customer, never a peer.

Never peer with one of your peer's customer is one basic rule of
peering agreements between tier-2 and 1 networks.

It's a shame financial pragmatism makes the Internet less "meshy", and
thus more fragile...

14

Please define ISP.

-Hank

15

FWIW, Hurricane Electric has an aggressively open peering policy and we
will peer with anyone who is willing to peer at any exchange where we are
connected. We believe as stated by Rucas that this only serves to enhance
the internet experience for our customers as well as our peers.

So far, it seems to be working pretty well for us. I encourage others to follow
our lead in this regard as it truly does make a more functional internet.

Owen

16

I agree, HE's peering policy makes them an attractive transit provider.

However, money and strategy still come into play here.

For example, ISP Z will think "I need some peering and transit. But if I get HE transit then some people may not peer with me at X-exchange because they will already see my routes via their HE peering" So then they get some transit from a network who is useless with their settlement free peering, then get the peers on the X-exchange and only when they are happily peered will they go to HE.

17

in this context, anyone who is a BGP speaker is an ISP.

/bill

18

All the whole "don't peer with this guy" only makes your customers
have worse latencies and paths to other people, making the Internet
less healthy.

Not necessarily. Peering with an ISP who wants to take the traffic between your network and theirs through a saturated pipe, an overloaded router, or across an MPLS pipe with 13 underlying hops (each of which could be a choke point themselves) will not make your end-to-end latencies any better.

As others have mentioned, some ISPs do have friendly peering policies. This is particularly true for ISPs that are co-located at the same IXP, because much of the opex is already baked into the ISP's relationship with the IXP.

The reason most of the larger ISPs, particularly those who live in the DFZ, have peering policies (especially for settlement-free peering) that could be construed as less friendly to smaller networks is because those guys want to sell you transit, rather than let you peer for free, or for less than a the full transit rate. It doesn't make financial sense for them to exchange bits with you for free, when they can make money off of those same bits if you buy transit instead.

carrying packets long distances cost more than carrying them short distances... large networks have an incentive to have the cost of that conveyance be reflected in peering relationship figuring out what if relationship makes sense in the marginal sense implies both parties see mutual benifit.

19

Peering costs money. The transit bandwidth saved by peering with another network may not be sufficient to cover the cost of installing and maintaining whatever connections are necessary to peer. Then there's the big networks who really don't want to peer with anyone other than similarly sized big networks...everyone else should be their transit customer.

I manage a network that's primarily a hosting network. There's a similar hosting network at the other end of the building. We both have multiple gigs of transit. We don't peer with each other. Perhaps we should, because the cost of the connection would be negligible (I think we already have multiple fiber pairs between our suites), but looking at my sampled netflow data, I'm guessing we average about 100kbit/s or less traffic in each direction between us. At that low a level, is it even worth the time and trouble to coordinate setting up a peering connection, much less tying up a gigE port at each end?

Anyone from hostdime reading this? :slight_smile:
If so, what are your thoughts?

20

-snip-

I manage a network that's primarily a hosting network. There's a similar
hosting network at the other end of the building. We both have multiple
gigs of transit. We don't peer with each other. Perhaps we should,
because the cost of the connection would be negligible (I think we already
have multiple fiber pairs between our suites), but looking at my sampled
netflow data, I'm guessing we average about 100kbit/s or less traffic in
each direction between us. At that low a level, is it even worth the time
and trouble to coordinate setting up a peering connection, much less
tying up a gigE port at each end?