a question about the economics of peering

Mailman List Mirror (Read Only)
1

Today, I was approached by *unnamed-ethernet-extension-company*. They
extend ethernets between several US and UK peering exchanges.

While speaking with them today, thier engineer and I got into a little bit
of a disagreement as to why people peer with each other at public exchange
points. My belief is that generally speaking, networks meet at public
exchange points (such as MAE-*, LINX, AMSIX, AADS, etc) is to exchange
traffic with each other more economically (read: save money).

His belief is that people will pay a premium to get to an exchange point,
because it's worth paying a premium to have 'less hops' between two
networks.

The problem with this idea is that public exchange points need
to be *avoided* when they get too congested. People may start
out trying to minimize number of hops, but I think they eventually
try to minimize total latency.

2

"David R. Dick" wrote:

>
> Today, I was approached by *unnamed-ethernet-extension-company*. They
> extend ethernets between several US and UK peering exchanges.
>
> While speaking with them today, thier engineer and I got into a little bit
> of a disagreement as to why people peer with each other at public exchange
> points. My belief is that generally speaking, networks meet at public
> exchange points (such as MAE-*, LINX, AMSIX, AADS, etc) is to exchange
> traffic with each other more economically (read: save money).
>
> His belief is that people will pay a premium to get to an exchange point,
> because it's worth paying a premium to have 'less hops' between two
> networks.

The problem with this idea is that public exchange points need
to be *avoided* when they get too congested. People may start
out trying to minimize number of hops, but I think they eventually
try to minimize total latency.

but what if the *unnamed-ethernet-extension-company* wasn't providing
access to public exchange points, but was rather enabling uncongested
private peering over its network? That way latency and hop count are
both mimimised.

BTW, public IXen in Europe don't tend to be congested. Whether this is
the result of better management, or of lower traffic volumes, than IXen
in the US I'm not sure...

Giles

3

Net99

4

:The problem with this idea is that public exchange points need
:to be *avoided* when they get too congested. People may start
:out trying to minimize number of hops, but I think they eventually
:try to minimize total latency.

Indeed. The notion of "hopcount" is objectively meaningless.
You can count hops by physical wires, data segments, network hops,
AS's or combinations of each (MPLS, GRE, PPP etc).

He may have been trying to suggest the value of having a low
latency path to a tier-1 provider, who may have low latency
across their backbone and through their cores.

But this is what you get with any good ISP, so I don't understand
what makes IX's more special than being connected to UUNet,
Exodus, or any other tier-1. In fact, having a connection to
a tier-1, who may be at many IX's, would leave you with the benefits
of diverse paths to other networks, without the premium of
an IX. Especially considering the tier-1 would be doing traffic
shaping to minimize latency across the backbone.

Also, I wonder if you can measure latency reliably over time
for more than 3 or 4 'hops' out, and relative to a small
number of destinations, before your results cease to be relative
to each other? I'm sure there is a principle like this acting
upon the Internet somehow.

Without latency measurements for the media you are counting hops in,
you might as well count sheep. I think the engineer mentioned at the
beginning of this thread was trying to hoodwink and bamboozle you.

5

>
> Today, I was approached by *unnamed-ethernet-extension-company*. They
> extend ethernets between several US and UK peering exchanges.
>
> While speaking with them today, thier engineer and I got into a
little bit
> of a disagreement as to why people peer with each other at
public exchange
> points. My belief is that generally speaking, networks meet at public
> exchange points (such as MAE-*, LINX, AMSIX, AADS, etc) is to exchange
> traffic with each other more economically (read: save money).
>
> His belief is that people will pay a premium to get to an
exchange point,
> because it's worth paying a premium to have 'less hops' between two
> networks.

The problem with this idea is that public exchange points need
to be *avoided* when they get too congested. People may start
out trying to minimize number of hops, but I think they eventually
try to minimize total latency.

Hmm. Congested public exchange points were a reality at some point in the
past. They are now a recurrent myth, thanks to the demise of FDDI EPs and
the rise of GigE EPs. There is little congestion, at this point. Of course,
this could be because most Internet traffic is exchanged over private
peering.

I think that most clued folks are largely uninterested in hop-count. Latency
is much more important.

- Daniel Golding

6

Daniel Golding wrote:

.....

Hmm. Congested public exchange points were a reality at some point in the
past. They are now a recurrent myth, thanks to the demise of FDDI EPs and
the rise of GigE EPs. There is little congestion, at this point. Of course,
this could be because most Internet traffic is exchanged over private
peering.

I think that most clued folks are largely uninterested in hop-count. Latency
is much more important.

that depends on your primary application, latency is not important for
streaming or downloads.
ak

7

Daniel Golding wrote:

.....

Hmm. Congested public exchange points were a reality at some point in the
past. They are now a recurrent myth, thanks to the demise of FDDI EPs and
the rise of GigE EPs. There is little congestion, at this point. Of course,
this could be because most Internet traffic is exchanged over private
peering.

I think that most clued folks are largely uninterested in hop-count. Latency
is much more important.

that depends on your primary application, latency is not important for
streaming or downloads.

heh! tell someone on the end of duplex satellite that! Streaming no, downloads very much yes.

jm

8

but what if the *unnamed-ethernet-extension-company* wasn't providing
access to public exchange points, but was rather enabling uncongested
private peering over its network? That way latency and hop count are
both mimimised.

But, *unnamed-ethernet-extension-company*, to me, only marketed access to
other exchange points.

Also, what does hop count have to do with latency and loss? Especially,
when *unnamed-ethernet-extension-company* is using MPLS/IP (presumably) to
do this?

BTW, public IXen in Europe don't tend to be congested. Whether this is
the result of better management, or of lower traffic volumes, than IXen
in the US I'm not sure...

Probably better management, is my guess.

9

heh! tell someone on the end of duplex satellite that! Streaming no,
downloads very much yes.

Ahem.

Latency doesn't affect unidirection streaming in the slightest.

10

heh! tell someone on the end of duplex satellite that! Streaming no,
downloads very much yes.

Latency doesn't affect unidirection streaming in the slightest.

folk who sell satellite links tend to under-educate customers on cwin

randy

11

batz wrote:

:The problem with this idea is that public exchange points need
:to be *avoided* when they get too congested. People may start
:out trying to minimize number of hops, but I think they eventually
:try to minimize total latency.

Indeed. The notion of "hopcount" is objectively meaningless.
You can count hops by physical wires, data segments, network hops,
AS's or combinations of each (MPLS, GRE, PPP etc).

He may have been trying to suggest the value of having a low
latency path to a tier-1 provider, who may have low latency
across their backbone and through their cores.

But this is what you get with any good ISP, so I don't understand
what makes IX's more special than being connected to UUNet,
Exodus, or any other tier-1. In fact, having a connection to
a tier-1, who may be at many IX's, would leave you with the benefits
of diverse paths to other networks, without the premium of
an IX. Especially considering the tier-1 would be doing traffic
shaping to minimize latency across the backbone.

Also, I wonder if you can measure latency reliably over time
for more than 3 or 4 'hops' out, and relative to a small
number of destinations, before your results cease to be relative
to each other? I'm sure there is a principle like this acting
upon the Internet somehow.

Without latency measurements for the media you are counting hops in,
you might as well count sheep. I think the engineer mentioned at the
beginning of this thread was trying to hoodwink and bamboozle you.

Folks,

I'm stood to one side up to now, but now this thread is drifting over to
personal abuse.

I was the engineer in question, and I was most certainly not trying to
hoodwink or bamboozle. Neither am I an ignorant sales-droid, as someone
else has said. Those of you who know me, as I think that a fair number
of people on this list do, will vouch for my honesty, and my pedigree in
the industry.

I think Alex misunderstood what I was trying to say, and since we were
shouting at each other down a very bad phone line with a loudspeaking
phone at either end, he's got a certain amount of excuse.

Firstly, I've been designing IP networks since the late 80s, and ISPs
since the early 90s. I'm not "fresh out of cable school" to quote one
correspondant. I've set peering policy for at least 2 large ISPs (BTnet
and Level3 (Europe)). Peering is my field of expertise. I'm on the
council of the LINX, which is the biggest peering point in Europe. I've
also got experience of selling internet services in Europe, and to be
frank, the customers over here have different requirements than over in
the US. The first thing you get asked is "how good is your peering".
They then ask you how much private peering you have, despite the fact
that IXPs in Europe tend to be well run and uncongested.

What I was offering him was a LAN extension service. One of the things
you can use this for is to peer with other customers. It looks like a
direct peering (so you keep customers happy) and it gets the bits from A
to C without passing through (possibly congested and oversubscribed) B.
Its not an oversubscribed service, so you get effectively private line
performance. And the costing is distance independent, which makes it
more like a peering point, in charging terms.

Hope this sets the record straight.

Note that I'm posting this from my home email address, and I'm speaking
on my own behalf, not that of my company.

All the best

Nigel

12

Actually, not. The hops which "count" are places where traffic flows are
stochastically multiplexed. Since every such place has to have buffers of
size comparable to end-to-end RTT * bandwidth (to give enough time for TCP
flow control to react), the maximal packet delay is Nhops*RTT.

Note this does not say what the median variable latency (fixed latency is
the signal propagation time) is going to be because it depends on
probability of multiple congestions along the path. If congestion events
are independent, the result isn't that bad (with long-lived congestion
probability of 50% the median variable latency is 2*RTT). Unfortunately,
this assumption is not likely to be realistic. I.e. in the "aggregation
tree" networks (like the most modern backbones which aggregate outgoing
exterior traffic at multiple points, where every aggregation point reduces
the total egress bandwith), the congestion is likely to occure
simultaneously at many aggregation points along the path.

The way to combat variable latency is to reduce the number of hops. This
effectively means either creation of meshes of fixed bit-rate circuits
(which is way way way ugly because of routing scalability issues), or
reducing the number of aggregators along the path by reducing the number
of routers. The simplest way to do that is to replace router clusters
with large parallel routers featuring non-blocking switching fabrics.

Note that non-CBR virtual-circuit techniques (tunnels, MPLS, non-CBR ATM,
FR, X.25 etc) don't do anything to reduce the _real_ hopcount; they simply
obscure it.

--vadim

13

Nigel, Nanog:

The original intent of the inquiry was not to bring into question the
intelligence of Nigel; I can't speak to that since I don't know him, nor
did I even know of him. My communication also does not speak to the
potential success of PacketExchange.

My question was simply a curiosity ping of _why_ people peer with each
other; in my mind, it had always, and never not, been a way to reduce cost
of traffic sent/rec'd. I was curious as to whether or not others had a
similar view to mine.

14

For us, I'd say there's a two-fold win.

1) Cost. We try to aim for less than $100/mbps on peering connections.
This isn't always the case when first connecting to a NAP, because
PtP circuits and NAP ports aren't charged per mbps, so the cost per
mbps is much higher for the first 10 or 20 meg, and comes down after
that. It also serves to expand our potential network capacity. This can
work right to extremes - in the UK (our primary geographical
target region), we can reach probably 99% of users without using
our transit with Level3 and others in the USA.

2) More direct relationships with end-user ISPs. Okay, there's no
formal SLAs or anything like that, but there *is* a relationship
between us and a peer. When a user complains about connectivity
issues, it's more likely that there's just two parties involved -
us and the user's ISP. We're not transitting through 2 or 3 other
ISPs in the middle - all of which could be the cause of problems.

Simon

15

Alex Rubenstein wrote:

Nigel, Nanog:

The original intent of the inquiry was not to bring into question the
intelligence of Nigel; I can't speak to that since I don't know him, nor
did I even know of him. My communication also does not speak to the
potential success of PacketExchange.

My question was simply a curiosity ping of _why_ people peer with each
other; in my mind, it had always, and never not, been a way to reduce cost
of traffic sent/rec'd. I was curious as to whether or not others had a
similar view to mine.

And I'd like to place on record that I had no real problem with Alex's
original email. It was only when the thread began to drift towards the
personal that I spoke up.

My own personal view is that there are a number of reasons for peering:
including but not limited to cost, shorter paths to desirable data
sources, "my network is bigger than yours", reduced latency etc. Mix and
match according to company policy/ personal inclination.

All the best

Nigel