I didn't think the Internet providers were common carriers.
And Carterphone should apply to cellular networks, but I am not holding my breath.
Owen
Isn't this all predicated that our crappy last mile providers continue
with their crappy last mile
If you think prices for residential broadband are bad now if you passed a
law that says all content providers big and small must have settlement free
access to the Internet paid for by residential subscribers what do you
think it would do to the price of broadband?
Apologies that I dropped offlist as I was out for the day. I think the bulk of my thoughts on this have already been covered by others since, including e.g. Matt's poor grandmother and her phone dilemma in the "What Net Neutrality should and should not cover" thread.
Basically I think we're on the same page for the most part, with maybe some misunderstandings between us.
I covered this scenario in more detail in my post "What Net Neutrality should and should not cover" but if you expand on the assumption that paying for an internet connection also pays for the direct connection of every party who you exchange traffic with then you have a scenario where only half the people connected to the Internet should have to pay at all for their connection because any scenario where people simply buy their own pipe would be considered "double billing".
I don't think anyone on the Netflix^H^H^H^H^H^H $ContentProvider side of this was saying that $ContentProvider should get everything handed to them on a silver platter. $ContentProvider pays for transit sufficient to handle the traffic that their customers request. $EyeballNetwork's customers pay it for internet access, i.e. to deliver the content that they request, e.g. from $ContentProvider. That covers both directions here. Links between $ContentProvider's transit provider and $EyeballNetwork were getting congested, and $EyeballNetwork refuses to upgrade capacity. Where we were getting into the double-dip was $EyeballNetwork saying to $ContentProvide: "Hey, we know you already pay for transit, but you're gonna have to pay us as well if you want us to actually accept the traffic our customers requested".
The alternate arrangement between $ContentProvider and $EyeballNetwork seems to be private peering, where again it would seem to be fair for each side to bring the needed transport and ports to peering points. In recent history, though, it seems that $EyeballNetwork came out ahead in that agreement somehow. Now, Tore brought up a good point on paid peering in cases where e.g. $EyeballNetwork is already exchanging traffic with $ContentProvider through existing peering or below their CDR on existing transit, and indeed it seems that was the case for $EyeballNetwork via peering with $CheapTransitProvider that $ContentProvider was using. But it seems that $EyeballNetwork was having a pissing match with $CheapTransitProvider and refusing to upgrade ports.
"Okay", says $ContentProvider. "How about we just peer directly."
"Sounds great," says $EyeballNetwork. "Since we have to allocate capacity for this discrete from our existing peering capacity, you'll need to foot the bill for that."
"Huh?" says $ContentProvider. "This could have been fixed by you increasing your peering capacity to match the traffic volume your users are requesting, but you didn't want to do that because of your tiff with $CheapTransitProvider. Tell me again why we're paying for your side of this *in addition* to our own when we're only going this route because of a decision *you* made?"
"Because you need to reach our customers, and we're the only path to them, so we have leverage."
*blank stare*
"So you're willing to give your customers crappy service because your customers don't have alternate options and you think we need this more than you do?"
"That's a possibility."
"I hate you."
"I know; sign here please."
But, again, this is outside looking in. For now, I'll pick up a copy of Bill Norton's Internet Peering book as per Bob's suggestion, for some light Sunday night reading.
Cheers,
Here is a quote I made in the other thread around the same time you were
sending this:
"I also think the practice of paying an intermediary ISP a per Mbps rate in
order to get to a last mile ISP over a settlement free agreement is also a
bit disingenuous in cases where the amount of traffic is sufficient enough
to fill multiple links. Theoretically there are many times where the
intermediary ISP can hand off the traffic to a last mile ISP in exactly the
same building they received it in so they have very few of the costs of
actually delivering the traffic yet are the only party receiving money from
the content provider for delivery. This arrangement makes sense when the
traffic to the last mile ISP is a percentage of one link but after enough
links are involved the intermediary ISP is serving no real other purpose
than as a loophole used to circumvent paid peering fees (right or wrong)."
I think we are in agreement that $EyeballNetwork's customers pay it for
internet access and $ContentProvider should pay for their own pipes. But
where we diverge is with $CheapTransitProvider.
At least for the purpose of traffic following the path of $ContentProvider
$CheapTransitProvider > $EyeballNetwork's because there is so much
traffic involved the only real purpose of the relationship with
$CheapTransitProvider is a loophole to get around paying $EyeballNetwork.
They are able to charge ridiculously low delivery prices because traffic is
only on their network for just long enough to say it touched and should now
be considered settlement free. It's little more than a cheap trick and it
makes them sort of the Cash4Gold of the Internet. I can completely
understand why $EyeballNetwork would tell $CheapTransitProvider they no
longer choose to have a settlement free agreement and they must buy future
ports.
Here is a quote I made in the other thread around the same time you were
sending this:"I also think the practice of paying an intermediary ISP a per Mbps rate in
order to get to a last mile ISP over a settlement free agreement is also a
bit disingenuous in cases where the amount of traffic is sufficient enough
to fill multiple links. Theoretically there are many times where the
intermediary ISP can hand off the traffic to a last mile ISP in exactly the
same building they received it in so they have very few of the costs of
actually delivering the traffic yet are the only party receiving money from
the content provider for delivery. This arrangement makes sense when the
traffic to the last mile ISP is a percentage of one link but after enough
links are involved the intermediary ISP is serving no real other purpose
than as a loophole used to circumvent paid peering fees (right or wrong)."
But that scenario only applies where the
content network has carried the traffic
to the same building as the eyeball
network, such that it really does just
go from the content provider, into
the cheapTransit router, and then
right back out to the eyeball network.
At that point, the content provider and
eyeball network are paying roughly
commensurate amounts for their
infrastructure costs; the content
provider to get the data to that
common location, and the eyeball
network to get it from that location
back to the customers who requested
it. I'm not sure why you think it's a
loophole of any sort; if anything, it's
an anti-loophole, as the most efficient
answer would be for the content network
and eyeball network to directly interconnect,
having each hauled circuits to this point in
common--but instead, due to policies, an
intermediary is forced into the picture.
And your understanding of transit seems to
be tenuous at best. You say "are the only party
receiving money from the content provider for delivery"
as though it's a bad thing, or some unusual circumstance.
This is exactly what transit is. I pay an upstream provider
to carry my route advertisements and bits to the rest of
the world, regardless of how near or far it is. You do the
same thing as a broadband customer; you pay one
provider for access, regardless of how many content
providers you pull down content from. It sounds like
you would advocate a model where every content
source pays every eyeball network that requests
its content, and every broadband subscriber pays
to every network it requests content from, rather
than the current model of paying one upstream
transit provider for connectivity to the rest of
the internet. Is that really the case? Is that
really what you're advocating for?
I think we are in agreement that $EyeballNetwork's customers pay it for
internet access and $ContentProvider should pay for their own pipes. But
where we diverge is with $CheapTransitProvider.
OK, how about we substitute "NotSoCheapTransitProvider"
into the equation. Now, does that make the situation any
different in your eyes? Or do you still feel that fundamentally
transit is only something that eyeball networks can pay for,
that content networks must not pay a single upstream, but
must instead pay every eyeball network separately, regardless
of how inefficient and expensive that would be?
At least for the purpose of traffic following the path of $ContentProvider
> $CheapTransitProvider > $EyeballNetwork's because there is so much
traffic involved the only real purpose of the relationship with
$CheapTransitProvider is a loophole to get around paying $EyeballNetwork.
They are able to charge ridiculously low delivery prices because traffic is
only on their network for just long enough to say it touched and should now
be considered settlement free. It's little more than a cheap trick and it
makes them sort of the Cash4Gold of the Internet. I can completely
understand why $EyeballNetwork would tell $CheapTransitProvider they no
longer choose to have a settlement free agreement and they must buy future
ports.
And in that model, I think it would be entirely correct
for the content provider to either deny access to their
content for users of ExtortionistEyeballNetwork, or to
charge them additional for access to the content, to
offset the increased costs of paying for the ports to
that network. After all, unlike your flawed traffic flow,
it's not the content network pushing bits at the
eyeball network; it's the eyeball network sucking the
bits down from the content provider.
If the eyeball network feels that volume of traffic
is problematic for them, such that they can't afford
to augment capacity, the clear answer is for the
content providers to help out the poor, congested
eyeball networks by reducing the bitrate of content
so that those links won't be congested anymore.
Serve up HD streams to networks that work
cooperatively to augment capacity for their users;
for networks that won't cover their share of the
costs of augmenting, simply serve lower and
lower bitrate streams to fit within the available
link capacity, with a little banner across the
top alerting the consumer that their experience
is degraded due to their provider not adequately
installing infrastructure to handle the consumer's
requested traffic.
Egg of Chicken question. Did regulation arise because of marker failure
(monopoly, duopoly), did did regulation create monopolies ?
When cable cos started in canada (TV only), they went to the CRTC, as
part of obtaining their broadcasting licenses and demanded they be
granted monopoly status for the areas they served. So fairly quickly,
the country was carved up into different territories, each served by a
single cable company (a couple of exceptions for border cases etc).
residential telephone was almost always a monopoly. There may have been
many different telcos, but each operated as the incumbent in its town.
The bigger guys ended up gobbling most of them over the years.
The probvlem of net neutrality does not reside in the "internet" itself.
The transit industry is a functioning markletplace with many competitors
and dynamic pricing pushing pricing towards costs.
The problem resides in the last mile which is controlled by incumbents.
The problem is that telcos and cablecos are becoming undifferentiated.
Cablecos offer telephony, and telcos offer TV distribution.
The difference is that not all telcos have advances and those still
stuck with old DSL are becoming irrelevant, leaving only a monopoly
cableco to serve customers.
And whenever 100% of facilities based last mile providers are more
interested in protecting their legacy TV assets, you get problems with
net neutrality, just as Comcast is doing to Netflix.
For large ISPs, Netflix provides caching appliances that can be inside
their network, so it is not a question of transit costs. It has
everything to do with a company that is heavily involved in TV, and
which controls the ISP market is such a large areas of USA wanting to
replace lost TV revenus by billing whoever is stealing those revenus.
In other words, they use their market power to hurt competitors. While
the FCC is getting the news, this should have gone to the FTC because it
is clearly an anti-competitive and predatory measure that proves Comcast
is using its market power to hurt competitors.
As a side note in Canada, the "Competition Bureau" (FTC in USA) is
getting involved with CRTC (FCC in USA) and submits into processes with
arguments on competition.
When there is a clear case of abuse of marlet power and anti-competive
practices, (such as Comcast vs Netflix) then government intervention is
not only warranted, it is essential.
Netflix has no business paying Comcast. It offers caching appliances,
and CDN distribution which can be either inside Comcast's network, or
close enough to peer with. So Netflix not only pays for its own
connection to the internet, but also manages to pay for transit right up
to major ISPs.
Comcast has no business billing its suppliers. It is in business to bill
its retail customers.
Now, if Netflix were to charge more to Comcast customers and make a note
of this before every program starts "you are paying more because you are
on Comcast", you might see Comcast rethink its anti-competitive practice.
Funny how that problem was magically solved so rapidly the minute the
deal was inked. Seems to me like Comcast was rate limiting IP ranges and
removed the rate limit once the deal was inked. This was all about
politics/business, not about network management.
Also interesting how other ISPs have no problem with Netflix.
So L3 and earlier, cogent peer settlement free with Comcast and Netflix
maxes out these peerings while they're there. What then?
Larry Sheldon wrote:
For large ISPs, Netflix provides caching appliances that can be inside
their network, so it is not a question of transit costs. It has
everything to do with a company that is heavily involved in TV, and
which controls the ISP market is such a large areas of USA wanting to
replace lost TV revenus by billing whoever is stealing those revenus.
The use of the word stealing here is offensive and inaccurate. It implies a
sense of entitlement to those revenues which is, IMHO, absurd.
It’s like political candidates who complain about third party candidates
“stealing their votes”. The votes don’t belong to the candidates, they belong
to the voters.
If $CABLECO wants to preserve their television revenues, they should do so
by providing a competitive product that is attractive to customers. If another
company is able to provide a more attractive product, then that’s how competition
and a free market is supposed to work.
What is absolutely contrary to the public interest is allowing $CABLECO to
leverage their position as a monopoly or oligopoly ISP to create an operational
disadvantage in access for that competing product.
The so-called “internet fast lane” is a euphemism for allowing $CABLECO
to put competing video products into a newly developed slow-lane while
limiting the existing path to their own products and those content providers
that are able to and choose to pay these additional fees.
Once you follow the money trail to its logical conclusion, at its heart, it’s
the epitome of the kind of anti-competitive practices the Sherman act
was intended to prevent.
In other words, they use their market power to hurt competitors. While
the FCC is getting the news, this should have gone to the FTC because it
is clearly an anti-competitive and predatory measure that proves Comcast
is using its market power to hurt competitors.
This isn’t limited to $CABLECO. While they’re at the front of this effort, reality
is that if it succeeds, incumbents of all flavors will start using this tactic to
improve their revenues to the detriment of consumers.
Owen
Isn't this all predicated that our crappy last mile providers continue with their crappy last mile
* jnanog@gmail.com (Rick Astley) [Mon 28 Apr 2014, 05:08 CEST]:
If you think prices for residential broadband are bad now if you passed a law that says all content providers big and small must have settlement free access to the Internet paid for by residential subscribers what do you think it would do to the price of broadband?
Lower it?
Right now broadband providers pay a transit provider who then get paid by content providers to carry the bits, generally because broadband providers don't want to think about running IP networks because they their skills lie more in the television part of RF networks.
Content providers are offering to take out that middleman, bringing everybody's cost down. Some broadband providers think they deserve more of a free ride than others. It also happens that those broadband providers are generally already more expensive than their competitors.
-- Niels.
* jnanog@gmail.com (Rick Astley) [Mon 28 Apr 2014, 05:08 CEST]:
If you think prices for residential broadband are bad now if you passed a
law that says all content providers big and small must have settlement freeLower it?
Right now broadband providers pay a transit provider who then get paid
by content providers to carry the bits, generally because broadband
providers don't want to think about running IP networks because they
their skills lie more in the television part of RF networks.
People are never gonna give this thread up, I see. Easily one of the
longest threads in recent nanog history and I'm starting to see points
rehashed and strawmen trotted out.
Comcast sells wholesale transit -
And it has a settlement free peering policy - with a stated
requirement that traffic exchanged be symmetrical.
Applicant must maintain a traffic scale between its network and
Comcast that enables a general balance of inbound versus
outbound traffic. The network cost burden for carrying traffic
between networks shall be similar to justify SFI
Now, that big elephant in the room taken into account, where do the
middlemen come in here?
--srs
MSOs run expansive IP networks today, including national dark fiber DWDM
networks. They all have way more people with IP expertise than they do RF
expertise. Even modern STBs use IP for many functions since they require
2-way communication, the last hold-out is your traditional TV delivery.
Even then most of the MSOs have IPTV installations in at least some
markets. That pendulum tipped a long long time ago now.
Level3 actually had to pay Comcast the last time this all came around.
They gained Netflix as a customer, the ratios of traffic a "transit"
provider was sending to Comcast because way out of balance, and Level3
succumbed and paid. Mainly since most of the traffic wasn't "transit"
traffic, it was Netflix traffic coming off Level3 CDNs. Transit
providers have "double-dipped" forever when it comes to ingress/egress
traffic to their own customers.
-Phil
People seem to forget what Comcast is doing is nothing new. People have
been paying for unbalanced peering for as long as peering has been around.
It's a little different because Netflix doesn't have an end network
customer to bill to recoup those charges, they have customers on someone
else's network.
It's not like all broadband providers are anti-Netflix, some are even
starting to include NF as an app on their STB. There are also many who do
peer with Netflix settlement-free even with very unbalanced ratios. The
key in the future is moving the bandwidth closer to the users, and we will
see more edge caching exist either within the broadband provider
facilities or at more localized 3rd party datacenters.
Phil
* ops.lists@gmail.com (Suresh Ramasubramanian) [Mon 28 Apr 2014, 15:27 CEST]:
Comcast sells wholesale transit - Wholesale Internet | Comcast Technology Solutions
And it has a settlement free peering policy - with a stated requirement that traffic exchanged be symmetrical.
How is that possibly realistic? They have 22 million customers (soon to become 29) with wildly asymmetrical connections and a very typical consumption pattern.
Should Netflix change its apps that they upload an equal amount of bandwidth back to Netflix's servers to balance this out? That way lies madness.
SBC had a much saner policy, from their 2006 SFI peering guidelines document: "No requirement for a balanced traffic exchange ratio due primarily to the asymmetric nature of current broadband metallic transmission systems such as ADSL and cable modems." (Then AT&T happened.)
Now, that big elephant in the room taken into account, where do the middlemen come in here?
Middlemen as in transit providers? The world is larger than Comcast's coverage area so there is a very good market for your middlemen.
-- Niels.
People seem to forget what Comcast is doing is nothing new. People have
been paying for unbalanced peering for as long as peering has been around.
It's a little different because Netflix doesn't have an end network
customer to bill to recoup those charges, they have customers on someone
else's network.
Yeah. It's a scam. Comcast can't do balanced peering. Their customers are not symmetrical.
It's not like all broadband providers are anti-Netflix, some are even
starting to include NF as an app on their STB. There are also many who do
peer with Netflix settlement-free even with very unbalanced ratios. The
key in the future is moving the bandwidth closer to the users, and we will
see more edge caching exist either within the broadband provider
facilities or at more localized 3rd party datacenters.
Netflix is happy to assist with caching. The thing is, Comcast doesn't care about that. What they care about is that their last mile is getting saturated and they have to pay money to upgrade it. Costs are being shoved onto netflix and similar to justify that.
This is compared to the small ISP who is just happy to get a peering or cache to save money only on their transit fees.
Jack
+1
What I don't understand is why Netflix et al are not doing a PR campaign to explain this to the end users.
Doug
If it was Netflix connected to say Cogent and Comcast connected to Level3 you would have the same unbalanced ratios between Cogent/Level3 for the same reasons. Level3 would likely be wanting compensation from Cogent for it... It is such a large amount of bandwidth these days it's not made up by other traffic.
I am not saying any of it is right, but precedents in the past have led to this.
Phil