South Korean Internet service provider SK Broadband has sued Netflix to pay for costs from increased network traffic and maintenance work because of a surge of viewers to the U.S. firm's content, an SK spokesperson said on Friday.
[...]
Last year, Netflix had brought its own lawsuit on whether it had any obligation to pay SK for network usage, arguing Netflix's duty ends with creating content and leaving it accessible. It said SK's expenses were incurred while fulfilling its contractual obligations to Internet users, and delivery in the Internet world is "free of charge as a principle", according to court documents.
[...]
Yet another peering dispute ending in litigation?
I'll never understand over how ISPs see content providers as the enemy (or a rival). The content is why ISPs have customers. Don't get upset when your customer uses the service that you sold them (in a way that is precisely in accordance with the expected usage)!
Netflix, as an example, has even been willing to bear most of the cost with peering or bringing servers to ISPs to reduce the ISP's costs and improve the ISP customer's experience. It's about time Netflix played chicken with one of these ISPs and stopped offering service (or offered limited service) to the ISPs that try to extort them and other content providers: Sorry, your service provider does not believe in net neutrality and has imposed limitations on your Netflix experience. For a better Netflix experience, consider exploring one of these other nearby internet providers: x, y, z.
It's because infrastructure (that's us, the network operators), still don't get it.
We are no longer front & centre in the eyes of our users. They see us as an impediment... providers they must buy costly megabytes of mobile data from, providers they must call to fix broken fibre, providers they must shout at when a single CGN IPv4 address they sit behind breaks their Netflix, and so on and so on.
Users only care about the service they use their mobile phone, tablet, console or laptop for. They don't care how many customers their ISP has, whether the ISP is a small mom & pop or some global behemoth, or whether the ISP's CEO is was on the cover of TIME magazine last week.
As my American friend used to say, "They just want their MTV".
In the late 90's and early 2000's, when content folk wanted to work with us, infrastructure folk, to grow their businesses, we just saw easy, free money to tax toward our shiny new Lamborghinis and beach side holiday villas. Well, guess whom we are now begging for seats on their submarine cable build projects, community funding programs, and caches to be installed in our not-so-huge data centres, all for free?
The reason Google, Facebook, Microsoft, Amazon, e.t.c., all built their own global backbones is because of this nonsense that SK Broadband is trying to pull with Netflix. At some point, the content folk will get fed up, and go build it themselves. What an opportunity infrastructure cost itself!
Akamai have also quietly been building their own backbone. Wonder why.
No doubt Netflix have someone either thinking about the same, or putting a plan into motion.
The bad news now, is, there are plenty of many, small, local and regional ISP's who are willing to do whatever it takes to work with the content providers. All that's required is some network, a half-decent data centre and an exchange point. Gone are the days where customers clamored to sign up with Big Telco.
If anyone wonders why "infrastructure is dead", well, this is why.
21 years later, and we still don't get it! No wonder the mobile companies are watching their slow death, from the rosy days of billions from basic SMS, to the perils of 5G investments for diddly return.
Wake me up when all this is over. I'll be in my wine stupor until then.
Mark.
I'll never understand over how ISPs see content providers as the enemy (or a rival). The content is why ISPs have customers. Don't get upset when your customer uses the service that you sold them (in a way that is precisely in accordance with the expected usage)!
It's because infrastructure (that's us, the network operators), still don't get it.
We are no longer front & centre in the eyes of our users. They see us as an impediment... providers they must buy costly megabytes of mobile data from, providers they must call to fix broken fibre, providers they must shout at when a single CGN IPv4 address they sit behind breaks their Netflix, and so on and so on.
Users only care about the service they use their mobile phone, tablet, console or laptop for. They don't care how many customers their ISP has, whether the ISP is a small mom & pop or some global behemoth, or whether the ISP's CEO is was on the cover of TIME magazine last week.
As my American friend used to say, "They just want their MTV".
In the late 90's and early 2000's, when content folk wanted to work with us, infrastructure folk, to grow their businesses, we just saw easy, free money to tax toward our shiny new Lamborghinis and beach side holiday villas. Well, guess whom we are now begging for seats on their submarine cable build projects, community funding programs, and caches to be installed in our not-so-huge data centres, all for free?
The reason Google, Facebook, Microsoft, Amazon, e.t.c., all built their own global backbones is because of this nonsense that SK Broadband is trying to pull with Netflix. At some point, the content folk will get fed up, and go build it themselves. What an opportunity infrastructure cost itself!
Akamai have also quietly been building their own backbone. Wonder why.
No doubt Netflix have someone either thinking about the same, or putting a plan into motion.
The bad news now, is, there are plenty of many, small, local and regional ISP's who are willing to do whatever it takes to work with the content providers. All that's required is some network, a half-decent data centre and an exchange point. Gone are the days where customers clamored to sign up with Big Telco.
If anyone wonders why "infrastructure is dead", well, this is why.
21 years later, and we still don't get it! No wonder the mobile companies are watching their slow death, from the rosy days of billions from basic SMS, to the perils of 5G investments for diddly return.
Wake me up when all this is over. I'll be in my wine stupor until then.
I couldn’t agree more.
Mark.
Cheers.
Darwin-.
The bad news now, is, there are plenty of many, small, local
and regional ISP's who are willing to do whatever it takes to
work with the content providers. All that's required is some
network, a half-decent data centre and an exchange point. Gone
are the days where customers clamored to sign up with Big
Telco.
Speaking as one of those smaller ISPs willing to do whatever it takes, perhaps you could answer me this riddle.....
- PoP in one of your "half-decent data centres" ... tick.
- Connnection to one of your "exchange point" ... tick.
- $certain_large_cdn present on said "exchange point" ... tick.
And yet .....
- $certain_large_cdn publishes routes on route server ? Nope.
- $certain_large_cdn willing to establish direct peering session ? Nope.
I am well aware of the "big boys club" that operates at most exchanges where the large networks see it beneath them to peer with (or publish routes for the benefit of) the unwashed masses.
But I struggle to comprehend why $certain_large_cdn would effectively cut off their nose to spite their face ?
In the old days, postal services used to charge the recipient of a letter to deliver the letter. Then stamps were invented, and postal services charged the sender of the letter, and the recipent got free delivery.
Now there is free-shipping, and pre-paid return envelopes for DVDs.
Of course, the shipping isn't really "Free." Its built into the cost of goods sold.
There is no universal, fixed, unchangable way of allocating business costs.
Having done peering for many $big_boys_club and $small_isps, it always comes down to politics, $$ and time. The balance may change but end of day its those variables and its a painful game some days. From all sides 
-jim
- $certain_large_cdn publishes routes on route server ? Nope.
Many (most?) route servers provide little control over who your routes
are advertised toward. This can be fun where DDoS is concerned.
I've used some that did have deny-list controls for ASNs, fail to
consistently apply those rules. Again, that was a 'fun' surprise.
- $certain_large_cdn willing to establish direct peering session ? Nope.
There is a non-zero cost to peering. Many CDNs are happy to send cache
boxes/setup peering sessions for small peers, but their definition of
"small" will no doubt vary between CDNs, based on their perspective &
business costs. Some networks may fall well below each individual
network's thresholds, and indeed some networks may have different
thresholds between countries.
It's never as simple as "why don't you peer???"
Regards,
True. But when the sender has already paid the stamp to their courier to deliver the bits on their leg of the journey, and the recipient has already paid a stamp to deliver the bits on their leg of the journey, what case does the recipient's courier have to demand additional payment from the sender (lest the package get lost)? The stamp has already been paid. TWICE! Withholding service until additional payments are made just smells like extortion.
Netflix actually did pretty much exactly that with Verizon back in 2014, displaying a message that read "The Verizon network is crowded right now."
I think instances where the end ISP is peered directly with Netflix and demands more money is not valid at all. That should be normal cost of doing business to increase capacity as the consumer demand grows.
The topic of interest is instances where the ISP is not directly peered with Netflix and uses upstream providers and those providers are trying to make content providers absorb the cost of increasing peering capacity for services that traverse their infrastructure.
One could make the argument that Tier1’s should never be the choke point as they should be keeping up with the times and be proactively increasing capacity.
One could also note that it’s 2021 and Cogent and Hurricane Electric are still not peered.
Having worked on building out a global content delivery network,
I can hopefully shed some light here.
Let’s imagine a scenario where you have a CDN node attached to
a large peering location.
At that peering location, you have Nx100G direct peering links to
large eyeball networks, and Nx10G direct peering links to midsized
eyeball networks.
You have a historical connection into the peering switch as well,
which you’ve kept around and upgraded as time went by. Life
is generally good. You’ve got your own backbone between
your locations, and you capacity engineer your infrastructure
so that if your peering sessions go down at that site, users
are redirected to the next closest CDN node.
Then one day, Bad Juju™ happens. The router that your
Nx100G links with Large Eyeball Network A terminate on
goes down, doesn’t matter if it’s on their end or your end,
those direct sessions go down. But instead of your traffic
failing over to site B instead, it continues flowing–but now
it’s going through the route server sessions, across the
now grossly-undersized connection into the public peering
switch, and nobody is happy about that. You tried putting
import and export lists into the route server, but they don’t
seem to be taking effect, and the phone is ringing off the
hook, so you simply stop exporting routes into the route
server, and suddenly traffic is now failing over to site B
for all the eyeball requests from Large Eyeball Network A,
and you get to explain to your boss how those sessions
with the route servers caused the failover plan to fail badly.
Decision is made that route server peering sessions are
too far removed from direct control to be safe, and the
only way to effectively prevent this scenario from happening
again is to stop announcing routes to the route servers.
You’re a smaller network; you’re caught up in this, because you’d
been getting routes from the route server, and now you’re not.
You request direct peering; unfortunately, the ports on the existing
line cards on the CDN edge routers are full, and it’s going to be
months before the next capacity upgrade is planned; and those
line cards are all 100G cards, with no 10G ports planned. It’s
decided that letting that traffic get handled by the transit ports is
cheaper than trying to get some additional 10G peering port capacity
at the site. And so, your request for direct peering is denied; there’s
just no place to plug you in on the router on the other side, unless
you can drum up enough traffic to justify a 100G peering link.
Establishing direct peering across the peering fabric becomes
the only possible point of commonality, then; but that requires
adding explicit neighbor configuration into the router, unlike
the route-server mediated prefix exchange; and now you’re
still running into limitations on the edge routers–though in this
case it’s not physical port limitations, it’s neighbor adjacency
limits on the routers. The more neighbors you configure on an
edge router, the longer the configs are, and depending on how
the neighbors are established, the more processing power and
memory it takes on the router. At a certain point, there’s just not
enough to go around, and the CDN makes the decision to not add
any new BGP neighbors at that site. A few years down the line,
when hardware is added/upgraded, they may revisit that decision,
or it may persist, even after the upgrade, due simply to corporate
inertia–nobody goes back to revisit the decision, and see if it still
makes sense or not.
Sometimes it’s just a case of limited humans; without good automation
tools in place, adding BGP adjacencies takes a ticket being created
in a request system, and then another more highly skilled human
going into the router to configure the neighbor. If you’ve got a limited
number of skilled humans available (and every company has fewer
of those than they really need, but that’s always going to be the case),
the decision generally gets made to focus their attention on efforts
that bring in more revenue–and configuring public peering sessions
generally doesn’t fall into that bucket.
And sometimes, financial considerations come into play; the CDN
network has a default upstream transit provider to catch the traffic
that isn’t handled by direct peering, and the transit provider gives
a really nice price break if the traffic makes it up into the next tier;
so, CDN that’s just below the cheaper tier looks for traffic it can
shunt onto transit, to nudge the volume over the line, to qualify for
the lower per-mbit price point–and the easiest traffic to use to nudge
them over is the smaller public peering neighbor traffic. And so, your
request for direct peering is looked at, and your traffic volume is
deemed to be the perfect fit to help fill out the transit commitment
needed to bump them into the cheaper tier.
In a perfect world, these would all be solvable issues, and exchanging
traffic directly with the CDN across the public switches would make
sense for everybody involved.
Unfortunately, we live in an imperfect world, with route server filters
that don’t react quickly and with the level of precision needed, we
have limitations on how many BGP adjacencies routers can handle,
we have limited port counts on line cards, we have limited numbers
of humans to make changes to router configurations, and we have
economic pressures to meet volume thresholds to get better pricing
that all end up creating the scenario you find yourself in; a completely
reasonable request for direct peering nonetheless is ignored, or
denied, due to one or more of the reasons I’ve discussed.
Thanks!
Matt
I wasn't aware of that, but I think that's perfect! And completely reasonable on Netflix (or any content provider's part).
I'm sure Verizon's wordsmiths would argue that the "crowding" happened upstream of the Verizon network, but if stated another way (like "the paths into Verizon's network are full") anyone can see that this is an issue that Verizon made and only Verizon could solve. Netflix isn't, and shouldn't be, responsible for runing Verizon's network. Only Verizon runs the Verizon network, and it's up to Verizon to deliver the service they advertise and sell to consumers: "America's most reliable network" (TM).
Bugger.
You’re right.
I forgot to add “stupid human ego issues” to my list of reasons
why direct peering requests get ignored or rejected. 
Thanks for the reminder. ^_^;;
Matt
Thanks for your insight Matt, much appreciated.
Yes, this is a rather painful one, and it has been on the rise in the last few years as the major content folk struggle to keep up with peering requirements, and everything that goes along with maintaining those relationships.
They also seem to have a number of complex operational requirements between their own backbones, their own PoP's, partner PoP's, transit links, e.t.c.
There is no easy answer for this one, apart from doing the leg-work to find out who the best person at the other end to speak to is. I'd recommend working with the data centre and exchange point operators to make meaningful introductions. You may not end up where you want, but you certainly increase your chances of doing so.
Mark.
Nowadays, well-run exchange points would have solved this. The reason we don't use route servers anymore is because they sometimes break (either due to code, or from people), and the policies you thought would give a good 3AM sleep suddenly aren't working anymore.
It was cheaper for us to maintain bi-lateral sessions.
That said, route servers have their place, and I won't knock them down.
Mark.
The bottom line problem is that we have allowed vertical integration to allow
the natural monopoly that exists in last mile infrastructure in most locations to
be leveraged into an effective full-stack monopoly for those same players.
Lack of competition in the last-mile/eyeball space has allowed larger monopoly
eyeball providers to leverage double-payment for the same traffic, extorting
eyeballs that have no alternative to pay them to deliver the content while
simultaneously turning around and extorting the content providers themselves
to be allowed to reach “their customers”.
While monopoly $CABLECO/$DSLTELCO/$GPONPROVIDER can generally
get away making access problematic for a handful of content providers for
short periods of time, no content provider can really absorb the losses associated
with allowing that situation to continue, thus giving the content providers
leverage.
Unfortunately for the content providers, newer laws and lower costs to provide
higher bandwidth are making WISPs a viable competitor in both rural and
metropolitan settings. Consumers are catching on to this and telling the more
traditional ISPs that they now have a choice and the ISPs will have to up their
game to keep their business, so finally progress is being made, which is
removing that leverage from the previously monopoly providers on both
the consumer and the content provider side of that equation.
Obviously, ISPs don’t like this and the monopoly ones being not in the
communications business, but rather operating as a law firm with some
switching infrastructure will attempt to use legislation, PUC, and courts
as weapons to try and preserve the status quo.
I’m hopeful that the Korean courts will see SK’s shakedown for what it is
and toss it in summary judgment (or whatever the Korean equivalent is)
with costs and attorneys’ fees awarded to Netflix. I’m hoping that this will
also send a clear message to other ISPs contemplating such extortive
behavior.
OTOH, I admit I will watch in amusement if SK’s customers trying to play
Netflix videos are only able to watch in 480p after viewing a brief video
explaining that the video they are about to watch is degraded courtesy
of SK’s business practices (obviously with appropriate details).
Owen
Except that Facebook, Microsoft, and Amazon all caved to SK's demands:
"The popularity of the hit series "Squid Game" and other offerings have underscored Netflix's status as the country's second-largest data traffic generator after Google's YouTube, but the two are the only ones to not pay network usage fees, which other content providers such as Amazon, Apple and Facebook are paying, SK said."
Which has emboldened SK to go after the bigger fish.
One incentive I haven't seen anyone mention is that ISPs don't want to charge customers what it really costs to provide them access. If you're the only one in your market that is doing that, no one is going to sign up because your pricing would be so far out of line with your competition.
Given that issue, I have some sympathy for eyeball networks wanting to charge content providers for the increased capacity that is needed to bring in their content. The cost would be passed on to the content provider's customers (in the same way that corporations don't pay taxes, their customers do), so the people on that ISP who are creating the increased demand would be (indirectly) paying for the increased capacity. That's actually fairer for the other customers who aren't Netflix subscribers.
The reason that Netflix doesn't want to do it is the same reason that ISPs don't want to charge their customers what it really costs to provide them access.