net neutrality peering dispute between CenturyTel/Qwest and Cogent in Dallas

I have several customers that are having packet loss issues, the packet loss appears to be associated with a Cogent router interface of 38.104.86.222. My upstream provider is telling me that the packet loss is being caused by a net neutrality peering dispute between CenturyTel/Quest and Cogent in Dallas. I did some quick googling to see if I could come up with any articles or something like that I could provide to my customers and did not see anything. Anyone know any details?

Thanks

Jordan Hamilton
Senior Telecommunications Engineer

Empire District Electric Co.
720 Schifferdecker
PO Box 127
Joplin, MO 64802

Ph: 417-625-4223
Cell: 417-388-3351

It's only partially about net neutrality. Cogent provides cheap bandwidth for content providers, and sends a lot of traffic to eyeball networks. In the past, peering partners expected symmetrical load sharing. Cogent feels that eyeball networks should be happy to carry their traffic since the customers want their services, the eyeball networks want Cogent to pay them extra. When there is congestion, neither side wants to upgrade their peeing until this is resolved, so they haven't. This has been going on for at least 5 years, and happens all over the cogent peering map.

Depending on what protocol you are using, it can be an issue or not. Our end users on eyeball networks had difficulty maintaining VPN connections. We had to drop our Cogent upstream and work with our remaining upstream provides to traffic engineer around Cogent. YMMV.

Oh, the irony of this typo of "peering"...

This issue isn’t limited to Cogent.

There is this bizarre belief by the larger eyeball networks (and CC, VZ, and TW are the worst offenders, pretty much in that order) that they are entitled to be paid by both the content provider _AND_ the eyeball user for carrying bits between the two.

In a healthy market, the eyeball providers would face competition and the content providers would simply ignore these demands and the eyeballs would buy from other eyeball providers.

Unfortunately, especially in the US, we don’t have a healthy market. In the best of circumstances, we have oligopolies and in the worst places, we have effective (or even actual) monopolies.

For example, in the area where I live, the claim you will hear is that there is competition. With my usage patterns, that’s a choice between Comcast (up to 30/7 $100/mo), AT&T DSL (1.5M/384k $40/mo+) and wireless (Up to 30/15 $500+/month).

I’m not in some rural backwater or even some second-tier metro. I’m within 10 miles of the former MAE West and also within 10 miles of Equinix SV1 (11 Great Oaks). There’s major fiber bundles within 2 miles of my house. I’m near US101 and Capitol Expressway in San Jose.

The reason that things are this way, IMHO, is because we have allowed “facilities based carriers” to leverage the monopoly on physical infrastructure into a monopoly for services over that infrastructure.

The most viable solution, IMHO, is to require a separation between physical infrastructure providers and those that provide services over that infrastructure. Breaking the tight coupling between the two and requiring physical infrastructure providers to lease facilities to operators on an equal footing for all operators will reduce the barriers to competition in the operator space. It will also make limited competition in the facilities space possible, though unlikely.

This model exists to some extent in a few areas that have municipal residential fiber services, and in most of those localities, it is working well.

That’s one of the reasons that the incumbent facilities based carriers have lobbied so hard to get laws in states where a city has done this that prevent other cities from following suit.

Fortunately, one of the big gains in recent FCC rulings is that these laws are likely to be rendered null and void.

Unfortunately, there is so much vested interest in the status quo that achieving this sort of separation is unlikely without a really strong grass roots movement. Sadly, the average sound-bite oriented citizen doesn’t know (or want to learn) enough to facilitate such a grass-roots movement, so if we want to build such a future, we have a long slog of public education and recruitment ahead of us.

In the mean time, we’ll get to continue to watch companies like CC, VZ, TW screw over their customers and the content providers their customers want to reach for the sake of extorting extra money from both sides of the transaction.

Owen

Arrogance is the only reason I can think of why the incumbents think that way. I'd be surprised if any competitive providers (regardless of their market dominance) would expect free peering.

Your reply implies that your understanding does not match my intended meaning.

(IOW, Perhaps you did not receive what I intended to transmit)

I’m saying that the incumbents in an act of unreasonable greed are demanding money for peering from providers with a lot of content providers while also collecting money from their direct customers for the sake of delivering that same content.

It would be like me standing between you and a hotdog stand and demanding that you give me 1.5x the price of the hotdog and then demanding that the hotdog stand sell me the hotdog to give to you for 0.5x the listed price.

In the more functional physical world, you simply walk around me and buy the hotdog for 1x the listed price and the only one who loses is the guy standing in the middle.

In the case of the incumbent facilities based carriers, they’ve managed to build a wall in front of the hot dog stand and a wall in front of you such that your view is limited to the window that they have to open and so is the hot dog vendor. Thus, you have no choice but to give them the extra 50% for the hot dog and the hot dog vendor has no choice but to give them half of the listed price as a “delivery charge”.

Admittedly, the fractions are not as I described, but the basic principle is exactly as I have described it.

Owen

I think we're on the same side, just saying it differently substituting greed for arrogance.

Additionally, the last mile providers are acting no differently than a carrier would, getting paid on both sides... only carriers are typically balanced ratios where as last mile\first mile are not.

In my 20+ yrs now of playing this game, "everyone" has had a turn thinking
their content/eyeballs are special and should get free "peering".

I dunno, Jim, that sounds almost like you might
think the inevitable outcome will be an "everyone
pays" model of settlements, the way telcos do
it. Unfortunately, in that model, the only winners
are the transit networks in the middle, because
no accounting department is going to want to
keep track of settlements for 4,000 other ASNs
that you peer with; their demand will be "reduce
the number of invoices, aggregate through 2 or
3 providers so we only have a small number of
invoices to reconcile."
I can see where you're coming from, but I'm not
sure I like the destination. :frowning:

Matt

Let me turn that on its head…

I don’t think anyone’s eyeballs are special.
I don’t think anyone’s content is special.

I think everyone should get free peering with any network whose customers
expect to be able to reach that other network’s customers.

Ignoring for a moment the idea of maximizing effective avarice, think how
much better it would be for eyeballs and content providers alike if they
could all just peer directly settlement free and/or pay a single layer of
transit providers all of whom peered with each other
for free.

Time and time again we have repeatedly proven that increased interconnect
density and promiscuous settlement free peering reduce
costs, improve performance, and generally make the internet better for
all concerned.

Now, ask yourself… If everyone followed that model, would it actually reduce
the viability of any of the businesses in question?

IMHO, there’s only one yes answer here… If enough of the eyeball/content
providers are able to cooperate and peer with each other directly, you might
see a significant impact (reduction in need) on transit providers as their entire
business would become largely irrelevant.

That’s called cutting out the middle man. In almost every industry that has been
able to do so, it’s been considered a really good thing for everyone except the
middle man who rarely gets much sympathy.

Owen

That's why those tired of playing the game build their own networks to
take out the middleman, for better or worse.

Mark.

This will work in a single market.

I've thought about this before too - when you start to cross nations or
continents, transit providers became a necessity; the eyeball networks
are typically not geared up to handle international or trans-continental
communications on their own.

The solution would be content providers deploying in each country to
remove the need for transit, but they still have to feed those clusters
somehow.

Ultimately, the big content players build and run their own networks,
completely bypassing the transit providers and peering with the eyeball
(and all) networks wherever they pitch tent. As it were, not all of them
have this muscle.

Mark.

There is more to it, then just being tired of it, it take, $$ and time and
bodies to build a network, even in 1 country. Its not something everyone
can do. I suspect the "game" and transit networks, will continue long
after most of us are no long "playing"

I do not disagree.

Mark.

<snip

The most viable solution, IMHO, is to require a separation between physical infrastructure providers and those that provide services over that infrastructure. Breaking the tight coupling between the two and requiring physical infrastructure providers to lease facilities to operators on an equal footing for all operators will reduce the barriers to competition in the operator space. It will also make limited competition in the facilities space possible, though unlikely.

This model exists to some extent in a few areas that have municipal residential fiber services, and in most of those localities, it is working well.

That�s one of the reasons that the incumbent facilities based carriers have lobbied so hard to get laws in states where a city has done this that prevent other cities from following suit.

Fortunately, one of the big gains in recent FCC rulings is that these laws are likely to be rendered null and void.

Unfortunately, there is so much vested interest in the status quo that achieving this sort of separation is unlikely without a really strong grass roots movement. Sadly, the average sound-bite oriented citizen doesn�t know (or want to learn) enough to facilitate such a grass-roots movement, so if we want to build such a future, we have a long slog of public education and recruitment ahead of us.

In the mean time, we�ll get to continue to watch companies like CC, VZ, TW screw over their customers and the content providers their customers want to reach for the sake of extorting extra money from both sides of the transaction.

Owen

I have talked about this idea for years, but most places seem to have a difficult time understanding the difference between layer 0, layer 2, and layer 3 networks.

IMHO the should be one residential fiber network (either passive or active, depending on the deployment and the physical layout of the area), and it should be run by an "essential" utility, such as the city/county water department, or if necessary the local electric company (I far prefer the water department). The access would be near universal, and the layer 0 and layer 2 network fees would be part of the "water" bill. Apartment complexes may have to be serviced with G.fast or other technologies to make the deployment faster and easier.

Getting IP bandwidth, technical support, voice service, and video service would be a competitive service provider model, with local ISPs, and large Cable COs and TelCOs competing on top of this physical network. You could even have providers that specialize in low income "life line" services, such as 5Mbit of IP bandwidth and local voice service with e911.

Historically services that have huge sunk costs, and high build out costs have been a natural monopoly and regulated. You would not think of trying to build out a competitive water or sewer network, and most building codes prevent the installation of a septic system if a sewer connection is at all possible. Why we are not going this way for a high cost of build out network (last mile) is beyond me.

Before this happens (ie when hell freezes over), I would like to see new home communities deploying fiber networks as part of the building of the "master plan" of the community. That way the home owners association can go out for bid every year or few years for a service provider to operate the fiber network. Around here (souther AZ) new communities tend to either alliance with CenturyLink, Cox, or Comcast depending on the location, and they DO NOT bring in the other providers. If a builder goes with Cox, you can NOT get a CenturyLink (ILEC) landline or DSL, if a builder goes with CenturyLink, Cox will not run anything into the community.

-Harry

This is exactly what communities in South Africa are starting to do.

In major suburbs around Johannesburg, neighborhoods are getting their
residents together and putting out bids for service providers to build
and operate FTTH networks on their behalf. In other parts of the
country, the local city or municipal councils are also getting involved.

In other neighborhoods, new and lean service providers are, by their own
volition, pulling fibre into various neighbors and deploying GPON FTTH
access nodes in homes regardless of whether you want a service or not.
If you want the service, ring them up and someone will come set you up.
You only pay for the setup fee, as the ONU remains theirs. This is what
happened in my neighborhood, which means I'm now sitting on a 25M/25M
FTTH service for about US$70/month. Big difference from when I had a
384k/3.2M ADSL service for about the same price.

The idea is that folk are taking matters into their own hands, as while
there is a national broadband strategy, its actual implementation is a
far cry.

Mark.