Observations of an Internet Middleman (Level3) (was: RIP Network Neutrality (was: Wow its been quiet here...

Observations of an Internet Middleman (Level3)
http://blog.level3.com/global-connectivity/observations-internet-middleman/

See also...

Level 3 accuses five unnamed US ISPs of abusing their market power in peering
http://gigaom.com/2014/05/05/level-3-accuses-five-unnamed-us-isps-of-abusing-their-market-power-in-peering/

"...
I’d love to see Cogent, Google and other providers release their data next, so even if the FCC doesn’t want to pursue this, a growing cry of consumer outrage could push the agency to do something about a very real and difficult problem that’s crippling access to video content on 5 U.S. broadband networks. Level 3 didn’t name names, but based on the deals Netflix has signed and the complaints it has made about AT&T, I’m confident that AT&T, Verizon and Comcast are among the five. "

=JeffH

Hi Jeff – I noticed the question posed here so thought I’d respond, perhaps at risk of stirring up a hornet’s nest given how long the last thread was. :wink: Anyway… there’s no congestion between Comcast and Level 3 connections, and we’re working collaboratively with Level 3. Given these facts, we have no reason to believe that Comcast is on their list.

- Jason
Comcast

Anyway… there’s no congestion between Comcast and Level 3 connections, and we’re working collaboratively with Level 3. Given these facts, we have no reason to believe that Comcast is on their list.

Sure, because Level 3 is already paying Comcast to deliver traffic to your paying customers, right?

That was an interesting read but it's not the whole story. Skip to the
TL;DR if you'd like but I'll attempt to explain what happened. What he
isn't saying is the roles of the companies involved have changed over the
last 10 years. Mostly gone are the days that content providers and access
networks each just gave a middleman/transit provider money to reach each
other. "Content provider" has expanded to become "content delivery network"
and "access network" has expanded their role to offer transit as well. If
these networks have a large amount of traffic between them and are able to
reach each other in multiple locations nationally what is the technical
reason a 3rd party transit network is required instead of a direct peering
relationship? From a purely technical perspective content and access at
that scale can peer directly cutting out the middle man.

The reality is an increasingly directly peered Internet doesn't sit well if
you are in the business of being the middle man. Now if you will, why do
transit companies themselves charge content companies to deliver bits? How
is it fair to be in the business of charging companies to receive their
bits and hand them to a settlement free peer on the hook to deliver them,
but not fair for content to just bypass the transit company and enter a
paid peering agreement with the company delivering the bits? In this case
paid peering is mutually beneficial to both companies involved and is
typically cheaper for the content company than it would cost to send that
traffic over transit.

What we have is a major shift in the market over the last 10 or so years.
So why are these large nationally connected "access" networks charging
Level 3 for ports? That's the elephant in the room here and to understand
that you have to go back to where (to my knowledge) this dispute first went
public. The most comprehensive description I have seen to date is the
following Youtube video: https://www.youtube.com/watch?v=tR1sLLOYxnY

I recommend the video before continuing. "Level 3" is really both Level 3
transit and Level 3 CDN. Level 3 has already had a long standing precedent
of justifying the right of an ISP to charge for content delivery. So what
happens when Level 3 greatly expands their content delivery business and
sends traffic to other ISP's over settlement free ports? The large access
networks say "hey, content delivery is a billable service, you should know"
and they ask Level 3 CDN for compensation. The middleman networks protest
and say "Charging for content delivery is only OK if we do it, but not when
you do it!" and their justification for this claim is made on the basis
that unlike access networks they a) Have a large network and b) send a full
table of prefixes.

So lets look at the first claim. Are the transit networks large? Yes, but
especially in the case of North American traffic destined for North America
they are typically smaller overall than the largest access networks who
arguably have the lions share of equipment tasked with delivering the bits
beyond just the colo.
The 2nd claim is mostly a strawman and this is why. Middlemen still carry
traffic not destined to directly connected peers but how they bill for it
is largely based on volume of traffic, not the number of prefixes
exchanged. The big content providers and the big access networks make up a
majority of the traffic on the Internet even if they don't make up a
majority of the prefixes.

TL;DR So the reason the ports are maxed out is the market has changed,
access networks have attempted to change peering agreements to match the
existing market conditions but the middleman networks are arguing they
should be exempt from the long standing tradition of charging for content
delivery they themselves helped to establish. Some middleman networks have
responded by refusing payment to access networks for delivery and as a
result, the paths have not been upgraded and remain congested.

End of TL;DR

The next part is (even) more opinion than fact so you are forgiven if you
stop here. My opinion is this is a peering dispute more than something
that should fall under net neutrality. If content companies sent letters to
"middlmen" that said "In your statements to the public you made the case
that content delivery to ISP's should be settlement free so we have decided
to take your offer and refuse any further payment to you from here forward"
how would they handle it? Likely those companies would not only find
themselves congested but depeered.

A bunch of people say charging at both ends is double dipping but really
modern access networks are now at least partly filling the role of transit
as well as last mile delivery. Where "content" "transit" and "access" all
have a presence in the same colo, paying more money to send traffic through
transit first instead of just directly to access because of some dated
definition of what the roles of those companies are supposed to be doesn't
make sense to me. Hijacking NN to attempt to bring litigation into the
matter to protect an old business model from a changing market makes even
less sense. Seeing Level 3 publish half truths in what looks like an
attempt to mislead the public on the matter is disappointing. I would
expect it from maybe Cogent but I have higher expectations of Level 3.

Broadband providers obviously aren't without some blame in the matter
either. One of them is allowing customer satisfaction to be so low that
they are easy targets for misinformation as most the comments I have seen
on the matter to date are more emotional than rational. They have other
mistakes too but for the purpose of keeping this brief and because some of
them have been heavily documented elsewhere I'll save them for another day.

This is a lot of hand waving and self justification to attempt to
validate the practice of [Access Network] trying to charge 3rd party
entities to deliver the content that [Access Network]'s paying
customers have requested over the service they already pay for,
instead of [Access Network] having to themselves pay for the bandwidth
because they know their customers can't leave them, and they know they
have a big enough market presence that they can rent seek with
impunity.

Why pay for transit connectivity expansion, when it financially
benefits you to instead let the links run over full, and charge the
world individually for uncongested access to your captive customers?

-Blake

-Blake

-Blake

Nice discussion about history & motivations. Not completely correct, but it's always fun to argue over history, and over motivations, since both are open to intepretation.

Personally, I am interested in the future, and specifically in market-driven solutions to our problems. Call me a capitalist if you like, but I believe in a functioning market, we can get a very good approximation of "fair".

If Company A and Company B have a mutual customer, and that customer needs both companies to perform a task, the market will find a way to make those two companies work together. Either that, or the customer will replace A or B, whichever the customer feels is underperforming, with Company C.

We have that situation today. Streaming Company wants to send End User of Broadband Company some content. If Streaming Company sucks - not enough titles, lousy customer service, high price, poor performance, etc., etc. - End User is free to select Streaming Company 2. And contrary to popular belief, there are plenty of "Streaming Company 2s" available. Besides NF, there is Hulu, Amazon, iTunes, iPlayer, etc. They might have different models, but they all allow you to access streaming content, so choice is available.

And here is where we get into the problem. Should End User believe Broadband Company sucks, they frequently cannot choose Broadband Company 2. I know I cannot, my choices are Comcast @ 100 Mbps or Verizon at 1.1 (yes, one-point-one) Mbps. So when Streaming Company sucks, but they suck because Broadband company is doing something I do not like, I cannot "vote with my wallet" and pick Broadband Company 2. I have no choice but to pick Streaming Company 2, even if I think the problem is Broadband Company's fault. (To be clear, I am not a NF subscriber - any more - and so this is not a NF/CC thing, I'm just talking generalities.)

Put more succinctly, there is no functioning market. therefore there cannot be a market-based solution.

Personally, I view that as about the most Un-American, Un-Capitalistic thing there is.

Lots of people have suggested a simple, if very difficult, fix to this problem. Make the underlying physical infrastructure a regulated monopoly, i.e. a Utility. Then allow anyone to run services over that physical infrastructure.

This is not pipe dream. The UK does it today. People there pick ISPs based on service, price, features, etc., not on "who paid off my local PUC".

And before anyone brings up the whole "the UK is more dense than the US", I preemptively call BS. There is more choice, faster speeds, and lower prices in the middle of no-where UK than downtown manhattan. Please just leave that argument where it belongs, in the dung heap.

Why can we not do something similar in the US? because the companies who own the lines have enough money to pay enough lobbyists to avoid even the promises they do make. (If anyone on this list is un-aware of things like the telcos promising ubiquitous high-speed BB years ago and never delivering, but never giving back their tax breaks or monopoly positions, you should be ashamed of yourselves.)

But hey, a guy can dream, right?

In the mean time, let's stop pretending that 'oh, L3 paid CC so they must be best friends'. L3 paid because They Had No Choice, and maybe because they see some long-term strategic benefit (e.g. they can charge others more later).

This is not a functioning market. This is a few players with Market Power charging Rents, which any first year econ major will explain is a _very_very_very_ bad place for the market to be.

The UK only does this with BT OpenReach since they were the telco monopoly
that originated as a government entity. Virgin Media (well all the people
who now form Virgin Media) built and operates their own fiber/HFC access
networks, the same as MSOs in the US, and does not offer wholesale access
and isn't treated as a utility. There are areas in the UK Virgin serves
where the wholesale network does not, and areas where they offer much
faster speeds, which is the same exact scenario as we have here. Just
because Verizon isn't using VDSL/VDSL2 or hasn't brought FIOS to your area
isn't Comcast's fault. The newer OpenReach wholesale fiber network is
also partially subsidized by the government.

I'm all for wholesale broadband access, but I wouldn't paint the situation
in the UK as vastly different than here. We had the same thing the UK
does now 10+ years ago with the CLECs and DSL providers like Covad, etc.
but the regulations changed and dried up access. TWC did wholesale access
during the same time; Earthlink Cable had quite a few customers back in
the day through the arrangement, but it was complicated and ultimately
your Internet pipe all still went through TWC.

Phil

I agree with your summary.

imiho think vi hart has it down simply and understandable by a lay
person. <http://vihart.com/net-neutrality-in-the-us-now-what/>. my
friends in last mile providers disagree. i take that as a good sign.

randy

It is important to consider bias and factual accuracy of the material.
George Ou was working for Comcast and AT&T as a lobbyist at the time
he produced the Youtube video.

Drive Slow,
Paul Wall

The pertinent question is what time period Level 3 was looking at /
averaging when writing the blog post.

Even if Comcast and Level 3 are not congested right at this moment,
they were most definitely congested several years following their
landmark agreement. A better question would be why that is/was.

Drive Slow,
Paul Wall

If we ignore why and how the few high speed options exist for a moment and accept that it's "the way it is," then it seems reasonable that the place to put regulation is on them. At the same time cutting out middlemen is generally good for everyone but the middlemen.

My current opinion then is to let ISPs cut out the middlemen but ensure that services which don't pay fees get reasonable access; regulate peering and transit agreements (not just for access providers but across the board). ISPs should be responsible to keep their links congestion free and have fair and reasonable terms to connect to their networks. They can sell direct access to their network to anyone as long as they aren't selling QoS.

Comcast and Verizon can sell direct access to content providers but they cannot degrade service as leverage in negotiations.

A side effect would be that if peering agreements must be public and there are stated terms for various types of peering many of the silky peering games that get played and the silky peering disagreements that cause problems would be more difficult.

We could finally answer the age old question, "is company X a 'tier 1'. "

++1

Andrew Fried
andrew.fried@gmail.com

In these situations, I find it helps to mentally implement structural
separation.

So you have level3-Transit and Level3-CDN as separate companies.

Netflix pays Level3-CDN to make content available locally in many cities.

It is up to the ISP to find the most efficient way to connect to the
Level3-CDN node(s).

As a CDN, does Level3 offer free peering with ISPs who only have to pay
for ports in a big switch ? ?

Similarly, if there were Comcast-Transit and Comcast-ISP, and I purchase
transit from Comcast-Transit, does it offer good connectivity around the
world, or is it just a shell company that serves the Comcast-ISP ?

Yeah, well, for extra credit integrate Akamai into that story.

If we ignore why and how the few high speed options exist for a moment and accept that it's "the way it is," then it seems reasonable that the place to put regulation is on them. At the same time cutting out middlemen is generally good for everyone but the middlemen.

My current opinion then is to let ISPs cut out the middlemen but ensure that services which don't pay fees get reasonable access; regulate peering and transit agreements (not just for access providers but across the board). ISPs should be responsible to keep their links congestion free and have fair and reasonable terms to connect to their networks. They can sell direct access to their network to anyone as long as they aren't selling QoS.

Comcast and Verizon can sell direct access to content providers but they cannot degrade service as leverage in negotiations.

That set of regulations would be utterly impossible to meaningfully enforce because so much of it depends on subjective evaluation.

The various law firms involved (Comcast, AT&T, Verizon, et al.) would have a field day playing in the gray areas of any such set of rules, most likely creating a situation of exactly the opposite of what is intended.

A side effect would be that if peering agreements must be public and there are stated terms for various types of peering many of the silky peering games that get played and the silky peering disagreements that cause problems would be more difficult.

More likely, costs would go up for everyone for everything and the game wouldn’t change by all that much.

We could finally answer the age old question, "is company X a 'tier 1'. “

Since nobody has a real definition for “tier 1”, it’s a fairly meaningless question to begin with.

(Yes, I am familiar with the alleged “does not pay for transit” definition, but I’ll point out that a completely disconnected network doesn’t pay for transit, either, but I doubt anyone would think they are a tier 1.)

Owen

They were already there.

Owen

I think Patrick was more trying to highlight that there is nothing stopping Openreach and Virgin Media from building their last miles in the same markets, side by side. They do so in many cases, and compete fairly equally for that business.

Or, for that matter, anyone else: Metronet[1] are busy building their own wireless infrastructure around the UK, and City Fibre[2] are running fibre up to everyone's door in a number of cities. Wholesale agreements are part and parcel of the business, as is the consumer choice to switch provider without penalty.

(And, just to clarify, you *can* buy wholesale Ethernet leased lines from Virgin Media Business, just not the DOCSIS access services.)

These examples are really only scratching the surface; the point is that you can switch providers to your heart's content. Or just build your own, should you have the means. There isn't a granted monopoly on end-user access in the UK (anything else is due to economics, and for that, see B4RN[3]).

I won't claim to hold the magic recipe for ensuring fair choice for consumers, and the UK market is far from perfect, but so far it's sounding a hell of a lot saner than what's happening in the US.

Tom

[1] http://www.metronet-uk.com
[2] http://www.cityfibre.com
[3] http://b4rn.org.uk/

Possibly interesting:

  FCC chairman will reportedly revise broadband proposal

  http://www.cnet.com/news/fcc-chairman-will-reportedly-revise-broadband-proposal/

or

  http://tinyurl.com/kfwrogs

imiho think vi hart has it down simply and understandable by a lay
person. <http://vihart.com/net-neutrality-in-the-us-now-what/>. my
friends in last mile providers disagree. i take that as a good sign.

Vi's analogy is wrong on a subtle but important point. In the analogy, the
delivery company needs to get a bunch of new trucks to handle the delivery
but as the customer is paying for each delivery instances, the delivery
company's costs are covered by increased end-user charges.

In the net neutrality debate, the last mile service providers are in a
position where they need to upgrade their access networks, but the end-user
pricing is not necessarily keeping pace.

There are lot of ways to argue this point, depending on whether you're the
user, the access provider or the content provider.

From a financial point of view, the content providers will say that access

providers need to charge their end users in a way which reflects their
usage requirements because let's face it, it's the users that are pulling
the traffic - they're not sending traffic to arbitrary IP addresses just
for the fun of it. The end users will say that they're only going to pay
market rate for their services, and they won't care whether this covers
their costs or not. The access providers will say that they're only
upgrading to deal with the additional requirements of the larger content
providers, particularly the CDNs and the video streaming services, and that
the going market rate doesn't allow them to charge the end users more.
Besides, it's a whole pile easier to chase a small number of companies for
a large amount of money than it is to chase a large number of customer for
a small amount of money. Even better, if you chase the the content sources
for cash, you can do this without increasing customer prices which means
you can stay more competitive in the sales market. So from a business
perspective it makes lots of sense to deprioritise the large companies that
don't pay in favour of the ones that do. Those who pay get better service
for their customers; seems fair, right?

From the proverbial helicopter viewpoint, we are walking towards a

situation where the short-term business actions of the individual companies
involved in the industry is going to lead towards customers being hurt and
this means that the likely long-term outcome is more regulation and
legislative control imposed on the industry. It is another tragedy of the
commons.

Nick