CDNs should pay eyeball networks, too.

Mailman List Mirror (Read Only)
1

Yesterday I received the following mail, from a CDN:

---->8----
Greetings,

Limelight Networks periodically reviews its interconnection strategy to ensure the quality and integrity of its interconnection between all its partners. We have recently updated our requirements for settlement-free peering which are posted at http://www.as22822.net/

This letter is to notify you that yyy no longer meets our minimum requirements. If yyy would like to maintain our current interconnectivity, there will be settlement associated with doing so. If you are interested in pursuing this option, please reply back to this email indicating such.

Should your company decline this option, or if we do not have an agreement regarding the settlement in place prior to May 31st 2012, Limelight Networks will terminate the peering sessions on that day, with this letter serving as 30 day notice.

Sincerely,
----8<----

The same mail was sent out last year, about end of April 2011, to another ISP I'm working with.
They got depeered, but the ISP which received the mail above yesterday didn't meet the requirements last year either.
I totally understand that some companies might not be able to handle sub-5Mbps peering sessions, be it technical or organisational, but >=100Mbps should be worth any effort, as long as it improves the network.

In this particular case I'm talking about >=600Mbps of traffic send out by Limelight to "my" eyeballs, not mentioning their fairly small footprint in Germany in comparison to other CDNs.

These points aside, we are talking about a Content *Delivery* Network here. There are CDNs out there who burn to improve their customer experience (both the content creators and the content receiver) at high cost.
Having a Tier1 attitude and telling eyeball networks with <1Gbps of traffic exchanged to bugger off or pay is not one of the ways to improve this.

At the end of the day I'm going to charge CDNs who want to deliver their customers content to my eyeballs and make me pay (about 2USD per Mbps, with a minimum of 1Gbps).

-dominik

2

Yesterday I received the following mail, from a CDN:

**snip**

Should your company decline this option, or if we do not have an agreement regarding the settlement in place prior to May 31st 2012, Limelight Networks will terminate the peering sessions on that day, with this letter serving as 30 day notice.

Sincerely,

While I can understand having some peering requirements, the goal of any CDN should be to have the best reach possible. Without knowing if this is PI peering or just across a IX it is hard to judge what their (or your) costs are. If it is IX it would seem irrational to cut off someone who you are doing meaningful traffic with.

----8<----
In this particular case I'm talking about >=600Mbps of traffic send out by Limelight to "my" eyeballs, not mentioning their fairly small footprint in Germany in comparison to other CDNs.

600Mbps would appear to be meaningful traffic. Without the other side of the story it's hard to get a full grip on the situation. It is always possible that these are business (not network) decisions where there is a certain level of income expected from a division.

These points aside, we are talking about a Content *Delivery* Network here. There are CDNs out there who burn to improve their customer experience (both the content creators and the content receiver) at high cost.
Having a Tier1 attitude and telling eyeball networks with <1Gbps of traffic exchanged to bugger off or pay is not one of the ways to improve this.

I do agree here, again on an IX level. They still have the choice to backhaul to you or hot potato your traffic wherever they feel like it. From the data you have provide it appears to be a lose-lose situation to de-peer you.

At the end of the day I'm going to charge CDNs who want to deliver their customers content to my eyeballs and make me pay (about 2USD per Mbps, with a minimum of 1Gbps).

That may be what they are doing "if at all possible try to monetize it".

3

While I can understand having some peering requirements, the goal of any CDN should be to have the best reach possible. Without knowing if this is PI peering or just across a IX it is hard to judge what their (or your) costs are. If it is IX it would seem irrational to cut off someone who you are doing meaningful traffic with.

This is over an IXP, I should've pointed that out.

----8<----
In this particular case I'm talking about >=600Mbps of traffic send out by Limelight to "my" eyeballs, not mentioning their fairly small footprint in Germany in comparison to other CDNs.

600Mbps would appear to be meaningful traffic. Without the other side of the story it's hard to get a full grip on the situation. It is always possible that these are business (not network) decisions where there is a certain level of income expected from a division.

These points aside, we are talking about a Content *Delivery* Network here. There are CDNs out there who burn to improve their customer experience (both the content creators and the content receiver) at high cost.
Having a Tier1 attitude and telling eyeball networks with <1Gbps of traffic exchanged to bugger off or pay is not one of the ways to improve this.

I do agree here, again on an IX level. They still have the choice to backhaul to you or hot potato your traffic wherever they feel like it. From the data you have provide it appears to be a lose-lose situation to de-peer you.

At the end of the day I'm going to charge CDNs who want to deliver their customers content to my eyeballs and make me pay (about 2USD per Mbps, with a minimum of 1Gbps).

That may be what they are doing "if at all possible try to monetize it".

Agreed on all points. There's at least nothing technical which needs
solving.

4

Yesterday I received the following mail, from a CDN:

---->8----
Greetings,

Limelight Networks [has] recently updated our requirements for
settlement-free peering

This letter is to notify you that yyy no longer meets our minimum
requirements.

Proposed solution:

Greetings,

Where settlement-free peering has been offered but rejected, YYY moves
all data traffic transiting that AS through a single minimum-cost
Internet connection (cough Cogent cough) with the attendant impact to
reliability. We appreciate the notice of depeering and will endeavor
to identify and advise those making paid use of our respective
services as to the impending impact to their activities.

On the technical side you can only easily enforce that for outbound
traffic. Essentially filter the routes containing their AS except from
your minimum cost link. Intentionally degraded service can be better
than flat refusal. Communication is two-way so even though the CDN
sends more than it receives this is still a credible threat. Your
customer sees perfectly good connectivity everywhere else and it isn't
a complete disconnect so your customer assumes its "their" Internet
connection rather than his.

I totally understand that some companies might not be able to handle
sub-5Mbps peering sessions, be it technical or organisational, but >=100Mbps
should be worth any effort, as long as it improves the network.

If I'm willing to go to your location, buy the card for your router
and pay you for the staff hours to set it up, there should be *no*
situation in which I'm willing to accept your traffic from an upstream
Internet link but am unwilling to engage in otherwise settlement-free
peering with you.

Your customers have paid you to connect to me and my customers have
paid me to connect to you. Double-billing the activity by either of us
collecting money from the other is just plain wrong.

Regards,
Bill Herrin

5

Right on

Thanks,
Ameen Pishdadi

6

Yesterday I received the following mail, from a CDN:

---->8----
Greetings,

Limelight Networks [has] recently updated our requirements for
settlement-free peering

I love the fact Dominik says "from a CDN", then leaves Limelight's name in the text. :slight_smile:

If I'm willing to go to your location, buy the card for your router
and pay you for the staff hours to set it up, there should be *no*
situation in which I'm willing to accept your traffic from an upstream
Internet link but am unwilling to engage in otherwise settlement-free
peering with you.

I disagree with this. In fact, I can think of several possible cases where this would not hold, both using pure business and pure technical justifications.

Generalizations are difficult in complex situations, and this is most definitely a complex situation.

Your customers have paid you to connect to me and my customers have
paid me to connect to you. Double-billing the activity by either of us
collecting money from the other is just plain wrong.

Wrong? My rule is: Your network, your decision. (Anyone who is paying attention knows my decision, but I it would be quite silly to assume my decision is right for all networks in all situations.) Asking for settlements is not illegal, or even immoral. Moreover, this is an operational list. "Right" and "wrong" are not really part of the discussion.

But even if they were, this is not not "just plain wrong". "It's just business" is a much better way to say it, and in business, trying to make more money is the _point_, not wrong. Whether this is a good way to make money is left as an exercise to the reader.

Instead, let's focus on the operational impact. Will the reduced complexity on these networks result in improved performance? Irrelevant to performance? Decreased performance? Maybe even whether that change in performance is an acceptable trade for the lower CapEx/OpEx? This is relevant since business requirements are the foundation for operational discussions. Can't buy more 10G ports if the business doesn't support it.

Etc., etc.

But right vs. wrong in a peering dispute? I think not.

7

While it would be easier to troubleshoot, I might decrease performance
by the same or likely the same metrics.

Peering via IXP/PNI

pro:
- more, maybe better paths
- more ways to loadbalance traffic over various PNI/IXP
- added redundancy

cons:
- needs grow to maintain sessions
- maintain a contact database for the specific networks
- tools needed to pinpoint issues on node, PNI / IXP, network level

"Feeding" via some bigger peer networks oder classic transit

pro:
- single point of contact
- less sessions to maintain (say 400 sessions for all bigger europe
cities instead of 200 sessions per IXP)
- easier view on traffic flows (depends on your tools though)

cons:
- if a single network breaks, it might have a bigger impact
- not able to (easily) mitigate issues (ceasing announcements to a peer
vs. turning down a transit session)
- troubleshooting mostly outsourced to a 3rd party

It depends on what one needs, and what one wants to pay.

-dominik

8

If I'm willing to go to your location, buy the card for your router
and pay you for the staff hours to set it up, there should be *no*
situation in which I'm willing to accept your traffic from an upstream
Internet link but am unwilling to engage in otherwise settlement-free
peering with you.

I disagree with this. In fact, I can think of several possible cases where
this would not hold, both using pure business and pure technical
justifications.

Hi Patrick,

Please educate me. I'd be happy to adopt a more nuanced view.

Your customers have paid you to connect to me and my customers have
paid me to connect to you. Double-billing the activity by either of us
collecting money from the other is just plain wrong.

Wrong? My rule is: Your network, your decision.

Yes, wrong. Some decisions fall in to areas covered by general ethics.

You sell a customer a red ball when you know you can only deliver
green balls, it's a lie. Unethical. Wrong.

You work for a company (W2 salary) and in the course of your work
contract something to another company where you're an officer it's a
conflict of interest. Unethical. Wrong.

A customer pays you to build a piece of software by the hour. Another
comes along and asks for the same software. You bill both for each
hour. Double billing. Unethical. Wrong.

A customer pays you to deliver a packet to "the Internet." You talk to
the packet's destination and say, "Hey, I'll deliver it to you
directly but only if you pay me. Otherwise I'll just toss it out in a
random direction and hope it gets there." Double billing. Unethical.
Wrong.

None of these things is necessarily illegal although like spam some of
them are illegal under specific conditions. Yet all of them (and spam)
are Wrong.

Regards,
Bill Herrin

9

If I'm willing to go to your location, buy the card for your router
and pay you for the staff hours to set it up, there should be *no*
situation in which I'm willing to accept your traffic from an upstream
Internet link but am unwilling to engage in otherwise settlement-free
peering with you.

I disagree with this. In fact, I can think of several possible cases where
this would not hold, both using pure business and pure technical
justifications.

Hi Patrick,

Please educate me. I'd be happy to adopt a more nuanced view.

First, define "upstream Internet link" - i.e. upstream to whom? If I peer with your upstream, then peering with you could easily drop me below their requirements, causing me to lose a much bigger peer for what I may consider no real benefit.

Let's assume you meant upstream to me. Many people negotiate volume discounts, peering with you may drop me to a lower tier, which may cost me more than your peering saves me.

Let's further assume it is upstream to me. Perhaps I am trying to turn that upstream into a peer. This devolves into the first situation.

Irrelevant on what you meant by upstream, I may have operational issues such as space & power, that disallow me from adding another port pointed at you in the location you want the port. I don't care if you buy the card, if there is no slot or power for the card, buying another rack - if possible in the first place! - may not be worth the benefit of peering with you. And I haven't even covered the CapEx & OpEx of adding a chassis to deal with your port irrespective of the power & space.

Etc.

Hopefully you get the gist.

Your customers have paid you to connect to me and my customers have
paid me to connect to you. Double-billing the activity by either of us
collecting money from the other is just plain wrong.

Wrong? My rule is: Your network, your decision.

Yes, wrong. Some decisions fall in to areas covered by general ethics.

You sell a customer a red ball when you know you can only deliver
green balls, it's a lie. Unethical. Wrong.

You work for a company (W2 salary) and in the course of your work
contract something to another company where you're an officer it's a
conflict of interest. Unethical. Wrong.

I see your point here, but neither of those examples are peering related.

A customer pays you to build a piece of software by the hour. Another
comes along and asks for the same software. You bill both for each
hour. Double billing. Unethical. Wrong.

This is far less clear. Should the second customer get it for free just because you already wrote the code? Or is it the fact you billed "an hour" instead of "software" that bothers you?

Not that it matters, since this has nothing to do with peering.

A customer pays you to deliver a packet to "the Internet." You talk to
the packet's destination and say, "Hey, I'll deliver it to you
directly but only if you pay me. Otherwise I'll just toss it out in a
random direction and hope it gets there." Double billing. Unethical.
Wrong.

I think if you look closely at any transit contract, almost none of them (cough Akamai cough) guarantee delivery t the end user. They typically only guarantee delivery to the edge of their own network, and make zero promises about _which_ edge they will use to get to the next network. Plus those "guarantees" are weaker than almost any other guarantee in any industry.

Of course, if someone sold me "full transit" and could not reach a significant fraction of the Internet, I'd claim breach of contract. But A != B.

So the straw man above _may_ be morally wrong (I'm not 100% certain of it, need to ruminate some more), it doesn't actually exist in the real world. And certainly is not related to the original post.

Besides, "double billing" is not unethical in your example above. Using your exact straw man, if I go to the destination, make that threat, and they cave, I haven't harmed my customer at all. If they don't cave and I send them the packet directly anyway, I still haven't harmed my customer. Double-billing in-and-of itself is not morally wrong.

All that said, if a provider sucks, change providers. <Cue discussion about being unable to change providers and how wrong that is. :->

None of these things is necessarily illegal although like spam some of
them are illegal under specific conditions. Yet all of them (and spam)
are Wrong.

We can agree that spam is wrong. :slight_smile:

10

In a message written on Tue, May 01, 2012 at 08:23:07PM +0200, Dominik Bay wrote:

"Feeding" via some bigger peer networks oder classic transit

You have made the assumption that their choice is peering with your
network or sending it out transit. They may in fact peer with your
upstream.

That makes their choice peer with you, or peer with your upstream.
Peering with your upstream may allow them to reach many people like
you for cost of managing only a single peering session, as compared
to maintaining a few dozen.

Also, many networks have minimum volume amounts for peering
relationships. They may be able to get settlement free peering
with your upstream by having some minimum traffic level that they
would not have if they peer with some of the individual customers
behind that upstream. Peering with you may drop them below the
threshold, causing them to pay for transit on 10's of Gigs of
traffic.

11

In a message written on Tue, May 01, 2012 at 08:23:07PM +0200, Dominik Bay wrote:

"Feeding" via some bigger peer networks oder classic transit

You have made the assumption that their choice is peering with your
network or sending it out transit. They may in fact peer with your
upstream.

No, I also assume 'bigger peer networks', usually a smaller ISPs
upstream with a bigger regional footprint.

That makes their choice peer with you, or peer with your upstream.
Peering with your upstream may allow them to reach many people like
you for cost of managing only a single peering session, as compared
to maintaining a few dozen.

Agreed, more reach with a single session, but with pros there are cons,
as mentioned in my last mail. If they choose to accept these
limitations, that's fine for me.
But one should be aware, especially when aiming towards low latency,
high bandwidth connections to eyeballs.

Also, many networks have minimum volume amounts for peering
relationships. They may be able to get settlement free peering
with your upstream by having some minimum traffic level that they
would not have if they peer with some of the individual customers
behind that upstream. Peering with you may drop them below the
threshold, causing them to pay for transit on 10's of Gigs of
traffic.

Agreed, but this is more a monetary optimization, not a technical
optimization.
I don't agree on any peering request, especially if this would force
traffic on paths which are preferred but sub-optimal.
A situation which wouldn't be optimal can be this one:

Say a network in Sweden and Estonia reach them via one of their
upstreams which do have a direct connection.
They are both a member of an IXP in, say, Amsterdam.
Both networks exchange their prefixes, and set localpref at this IXP.
This would make the path worse than before, as traffic takes a detour,
and might cost more due to backhauling traffic.

12

"A customer pays you to build a piece of software by the hour. Another
comes along and asks for the same software. You bill both for each
hour. Double billing. Unethical. Wrong.

A customer pays you to deliver a packet to "the Internet." You talk to
the packet's destination and say, "Hey, I'll deliver it to you
directly but only if you pay me. Otherwise I'll just toss it out in a
random direction and hope it gets there." Double billing. Unethical.
Wrong."

Neither of these is unethical or wrong in any way. What are you
supposed to do, write software from scratch every time? That's just
silly.

13

Lets be honest. There are a million reasons we can all come up with to
try and justify something like this but 99% of the time it is just the
larger isp trying to throw their weight around in the name of greed.
In the end, the customers of both companies are the ones who suffer
and ultimately pay (figuratively and literally) for it.

14

1) Welcome to business 101.

2) Is that bad?

3) I think your 99% number is very inflated. But then 99% of stats are made up. :slight_smile:

15

>> If I'm willing to go to your location, buy the card for your router
>> and pay you for the staff hours to set it up, there should be *no*
>> situation in which I'm willing to accept your traffic from an upstream
>> Internet link but am unwilling to engage in otherwise settlement-free
>> peering with you.
>
> I disagree with this. In fact, I can think of several possible cases where
> this would not hold, both using pure business and pure technical
> justifications.

Hi Patrick,

Please educate me. I'd be happy to adopt a more nuanced view.

>> Your customers have paid you to connect to me and my customers have
>> paid me to connect to you. Double-billing the activity by either of us
>> collecting money from the other is just plain wrong.
>
> Wrong? My rule is: Your network, your decision.

Yes, wrong. Some decisions fall in to areas covered by general ethics.

You are high. If I've entered into contracts with multiple parties to
deliver their traffic, there is no 'double dipping' or 'double billing'.
Ignore any sort of traffic *type*. To assert that some transit entity
with two customers paying to reach the universe (happens to include each
party reciprocally) should go out of their way to discount for such
customer to customer traffic is both madness of bellhead-accounting
level and a quick route to bankruptcy. Stupid and nothing YOU would
certainly do for your customers...?

You sell a customer a red ball when you know you can only deliver
green balls, it's a lie. Unethical. Wrong.

Utterly irrelevant to the discussion.

You work for a company (W2 salary) and in the course of your work
contract something to another company where you're an officer it's a
conflict of interest. Unethical. Wrong.

Utterly irrelevant to the discussion.

A customer pays you to build a piece of software by the hour. Another
comes along and asks for the same software. You bill both for each
hour. Double billing. Unethical. Wrong.

Utterly irrelevant to the discussion.

A customer pays you to deliver a packet to "the Internet." You talk to
the packet's destination and say, "Hey, I'll deliver it to you
directly but only if you pay me. Otherwise I'll just toss it out in a
random direction and hope it gets there." Double billing. Unethical.
Wrong.

Mindset of an inexperienced small provider who believes "the Internet"
is something 'out there' and your infrastructure isn't part of it. Your
customers give you traffic at your handoff to deliver to/from others.
Some of those others amy also be your customers; are you giving them
free traffic customer-to-customer [complex accounting at your egress
to peers and transit] or are you billing them for traffic transferred
[port stats facing the customer]?

Discussion of applicability of one model or another to this or that type
of business is an interesting academic exercise, but if you aren't willing
to give your customers free transit across your service to each other, your
position is at best disingenuous. At worst, hypocritical.

Cheers!

Joe

16

Lets be honest. There are a million reasons we can all come up with to
try and justify something like this but 99% of the time it is just the
larger isp trying to throw their weight around in the name of greed.
In the end, the customers of both companies are the ones who suffer
and ultimately pay (figuratively and literally) for it.

1) Welcome to business 101.

2) Is that bad?

Can be for the end users if they wind up on a less direct network path.

3) I think your 99% number is very inflated. But then 99% of stats are made up. :slight_smile:

Probably.

17

In a message written on Tue, May 01, 2012 at 03:45:29PM -0500, Jerry Dent wrote:

Can be for the end users if they wind up on a less direct network path.

"Direct" is not the only measure.

I would take a 4-hop, 10GE, no packet loss path over a 1-hop, 1GE,
5% packet loss path any day of the week.

"Shorter" {hops, latency, as-path} does not mean a higher quality end
user experience.

18

Hi Mike,

If one of the customers happens to be the U.S. Government, it's not
only unethical it's a crime. It's usually a felony. You can do time.
The product was man hours. You've sold them once. You can't sell them
again.

The customers can agree to share the cost up front. But then you're
not billing the same hours twice, you're billing half an hour to one
customer and half to another. When you finish the software you can
sell the software to a second customer. But you *may not* tie your
price to the hours used to produce it for the first.

Regards,
Bill Herrin

19

"If one of the customers happens to be the U.S. Government, it's not
only unethical it's a crime. It's usually a felony. You can do time.
The product was man hours. You've sold them once. You can't sell them
again."
You're assuming the contract is simply for work hours. Generally
speaking, and from my experience, it isn't. The contract is for an
app that does X, not 20 hours toward building an app that does X.

"But you *may not* tie your
price to the hours used to produce it for the first."
Sure you can. How else do you determine what the software's going to
cost if you're not going to factor in development?

20

Let's keep our eye on the ball, people. Did the original post have any operational consequences?

IMHO, it has many. Some are even interesting to the wider audience. So why are we discussing how you bill the US gov't?