Verizon Public Policy on Netflix

Mailman List Mirror (Read Only)
1

Here's a link to a post from VZN's public policy blog, about Netflix.

Now, just as a matter of principle, I tend to assume that anything VZN
says in public is a self-serving lie based on a poor understanding of the
Real World... but I did in fact read it.

Yup.

The money quote:

   One might wonder why Netflix and its transit providers were the only ones
   that ran into congestion issues. What it boils down to is this: these other
   transit and content providers took steps to ensure that there was adequate
   capacity for their traffic to enter our network.

"their traffic".

What, Verizon: Netflix is just sending you that traffic uninvited?

No: that's *your customers traffic*. You *knew* that there would be
asymmetrical amounts of traffic flowing downhill to your customers,
*or you wouldn't have provisioned nearly uniformly asymmetrical last
mile links to them*.

You just lost the bet on how much traffic that would be.

That's why they call it gambling: sometimes you lose. You lost.
Man up and provision usable peering. That traffic is your responsiblity.
If you decided not to charge your customers enough to provision for
it, take it out of retained earnings.

Just don't try to convince us all that, somehow, that traffic flow isn't
your customers traffic, and thus yours. Netflix's only fault is being
popular.

  http://publicpolicy.verizon.com/blog/entry/why-is-netflix-buffering-dispelling-the-congestion-myth

Cheers,
-- jra

2

Some good discussion on this at the recent Aspen Institute Net Neutrality
panel.
http://youtu.be/IEKQmVuqXsg

3

Jay Ashworth wrote:

Here's a link to a post from VZN's public policy blog, about Netflix.

Now, just as a matter of principle, I tend to assume that anything VZN
says in public is a self-serving lie based on a poor understanding of the
Real World... but I did in fact read it.

Yup.

The money quote:

    One might wonder why Netflix and its transit providers were the only ones
    that ran into congestion issues. What it boils down to is this: these other
    transit and content providers took steps to ensure that there was adequate
    capacity for their traffic to enter our network.

"their traffic".

What, Verizon: Netflix is just sending you that traffic uninvited?

No: that's *your customers traffic*. You *knew* that there would be
asymmetrical amounts of traffic flowing downhill to your customers,
*or you wouldn't have provisioned nearly uniformly asymmetrical last
mile links to them*.

Let me preface this by saying that I'm in no way an apologist for Verizon - I've spent a lot of my life in the municipal networking world, and Verizon's lobbying against muncipal fiber builds makes them the enemy in many regards. But, as a FIOS customer, I'm impressed by their levels of service, and as a network engineer and policy wonk it seems only fair to point out the following:

Verizon is claiming that delays between Netflix and FIOS customers result from a) the transit network between Netflix and Verizon being congested, and/or b) the connection between the transit network and Verizon is congested.

A little experimentation validates this: Traffic from my FIOS home router flows through alter.net and xo.net before hitting netflix. Now alter.net is now owned by Verizon, but when I run traceroutes, I see all the delays starting halfway through XO's network -- so why is nobody pointing a finger at XO?

I'll also note that traffic to/from google, and youtube (also google of course) seems to flow FIOS - alter.net - google -- with no delays. So again, why aren't Netflix and Verizon pointing their fingers at XO.

This is the classic asymmetric peering situation - which raises a legitimate question of who's responsible for paying for the costs of transit service and interconnections?

And, of course, one might ask why Netflix isn't buying a direct feed into either alter.net or FIOS POPs, and/or making use of a caching network like Akamai, as many other large traffic sources do on a routine basis.

Personally, I think Netflix is screwing the pooch on this one, and pointing the finger at Verizon as a convenient fall guy.

Miles Fidelman

4

And, of course, one might ask why Netflix isn't ... making use of a
caching network like Akamai, as many other large traffic sources do
on a routine basis.

they do. netflix rolls their own cache servers, installable in any
network

randy

5

Randy Bush wrote:

And, of course, one might ask why Netflix isn't ... making use of a
caching network like Akamai, as many other large traffic sources do
on a routine basis.

they do. netflix rolls their own cache servers, installable in any
network

At the ISPs expense, including connectivity to a peering point. Most content providers pay Akamai, Netflix wants ISPs to pay them. Hmmm....

Now I write a check every month to both Verizon and Netflix - and clearly it would be nice if some of that went to provisioning better service between the two. But I can as easily point to Netflix, as to Verizon, when it comes to which dollar stream should be going to bigger (or more efficient) pipes.

Miles Fidelman

6

Randy Bush wrote:

[snip]

At the ISPs expense, including connectivity to a peering point. Most content
providers pay Akamai, Netflix wants ISPs to pay them. Hmmm....

Netflix own website indicates otherwise.
https://www.netflix.com/openconnect

"ISPs can directly connect their networks to Open Connect for free.
ISPs can do this either by free peering with us at common Internet
exchanges, or can save even more transit costs by putting our free
storage appliances in or near their network."

7

I was going to sit on the sidelines.... but...

Take Netflix out of the equation and google things like "tf2 verizon
fios" or any other game. Who do you point the finger at then?

-Jim P.

8

Unless said tf2 server happens to be hosted within UU's own network, I'd imagine the blame would go to whichever party in the transit path refused to upgrade their commitments.

9

Randy Bush wrote:

And, of course, one might ask why Netflix isn't ... making use of a

caching network like Akamai, as many other large traffic sources do
on a routine basis.

they do. netflix rolls their own cache servers, installable in any
network

At the ISPs expense, including connectivity to a peering point. Most
content providers pay Akamai, Netflix wants ISPs to pay them. Hmmm....

Uh, yeah, you've already been corrected on that
score, no need to spank you again for that one...

Now I write a check every month to both Verizon and Netflix - and clearly
it would be nice if some of that went to provisioning better service
between the two. But I can as easily point to Netflix, as to Verizon, when
it comes to which dollar stream should be going to bigger (or more
efficient) pipes.

So, if Netflix had to pay additional money to get direct
links to Verizon, you'd be OK paying an additional
50cents/month to cover those additional costs,
right? And when Time Warner also wants Netflix
to pay for direct connections, you'd be ok paying
an additional 50cents/month to cover those costs
as well, right? And another 50cents/month for the
direct connections to Sprint? And another 50cents/month
for the direct connections to cablevision? (repeat for
whatever top list of eyeball networks you want to
reference).

At what point do you draw the line and say "wait
a minute, this model isn't scalable; if every eyeball
network charges netflix to connect directly to them,
my Netflix bill is going to be $70/month instead of
$7/month, and I'm going to end up cancelling my
subscription to them."

Miles Fidelman

Matt

10

Jimmy Hess wrote:

11

A little experimentation validates this: Traffic from my FIOS home router
flows through alter.net and xo.net before hitting netflix. Now alter.net is
now owned by Verizon, but when I run traceroutes, I see all the delays
starting halfway through XO's network -- so why is nobody pointing a finger
at XO?

Traceroute is pretty meaningless for analyzing if there is congestion or not. The presence of delays could mean many things that don't indicate congestion. Most large networks are well managed internally; congestion almost always appears at network edges.

In this case, the assertion is that XO's link to Verizon is congested. If that is in fact the case, it's because Verizon is running it hot. Verizon is (presumably) an XO customer, and it is on them to increase capacity or do network engineering such that their links are upgraded or traffic shifted elsewhere. It's worth pointing out that if Verizon is running a transit link hot like this, Netflix is not the only traffic that's going to be impacted, and that is in no way Netflix' fault. Even if it is a peering link, their dispute should be with XO.

What people seem to miss here is that there is no other out for $ISP than a) increase transit capacity, b) sufficiently peer with $CONTENT or c) allow performance to degrade (ie. Don't give customers what they are paying for). If we take c) off the table, it tells us that settlement-free peering would be the preferred alternative as it would usually cost less than buying more transit.

I'll also note that traffic to/from google, and youtube (also google of
course) seems to flow FIOS - alter.net - google -- with no delays. So again,
why aren't Netflix and Verizon pointing their fingers at XO.

Verizon (apparently) refuses to peer with Netflix, since Netflix has an open polic. They do, however, appear to peer with Google. Why?

This is the classic asymmetric peering situation - which raises a legitimate
question of who's responsible for paying for the costs of transit service and
interconnections?

If this were a question of Verizon transiting traffic for Netflix asymmetrically, then sure. However they are terminating the traffic in question, the only "transit" is to a paying Verizon customer on Verizon equipment; this is the part of the network their customer pays them to maintain.

And, of course, one might ask why Netflix isn't buying a direct feed into
either alter.net or FIOS POPs, and/or making use of a caching network like
Akamai, as many other large traffic sources do on a routine basis.

They likely can already meet easily at many points across the country, with little cost to either party. It is quite obvious that Netflix is very open to doing so. Why doesn't Verizon want to play? Apparently because they think they can successfully convince users that the problem is Netflix' and not Verizon's. Content peering with eyeballs should be a no-brainer - it saves both parties plenty of money and improves performance across the board. Netflix seems willing to bring their traffic to Verizon's edge for free, all Verizon needs to do is turn up the ports and build whatever capacity they would need to build anyway regardless of where the traffic comes from or what it is. Or, if the power and space is cheaper than the transport from where they meet (or to where they can meet), they can install Netflix' appliances. They always have the option of just buying more transit too, but the bottom line is that this expansion is required to carry their customer's traffic, it's not something they would be trying to charge content/transit for if it were organic traffic growth from diverse sources, they would simply upgrade their network like the rest of us.

Keenan

12

In what world is Verizon an XO customer?

But I think the whole premise of blaming XO is broken, just because your
traceroute shows inbound to Netflix via XO does not mean Netflix is sending
bits to you via XO. If you are sitting on AS701, Netflix certainly has
many routes with aspath length = 2 (Transit, VZB) and its going to be
pretty hard to know what path they are taking into VZB for yourself.

13

Jimmy Hess wrote:

Randy Bush wrote:

[snip]

At the ISPs expense, including connectivity to a peering point. Most
content
providers pay Akamai, Netflix wants ISPs to pay them. Hmmm....

Netflix own website indicates otherwise.
https://www.netflix.com/openconnect

"ISPs can directly connect their networks to Open Connect for free.
ISPs can do this either by free peering with us at common Internet
exchanges, or can save even more transit costs by putting our free
storage appliances in or near their network."

From another list, I think this puts it nicely (for those of you who

don't know Brett, he's been running a small ISP for years
http://www.lariat.net/)

--------

Netflix's only fault is being popular.

Alas, as an ISP who cares about his customers, I must say that this is not
at all the case.

Netflix generates huge amounts of wasteful, redundant traffic and then
refuses to allow ISPs to correct this inefficiency via caching.

I'm sorry. You cannot take that sentence...

It fails to provide adequate bandwidth for its traffic to ISPs' "front
doors" and then blames their downstream networks when in fact they are more
than adequate. It exercises market power over ISPs (one of the first
questions asked by every customer who calls us is, "How well do you stream
Netflix?") in an attempt to force them to host their servers for free

...together with this sentence, without hitting a WTF
moment.

He rants about Netflix generating huge amounts of traffic
and refusing to allow ISPs to cache it; and then goes on to
grumble that Netflix is trying to force them to host caching
boxes. Does he love caching, or hate caching? I really
can't tell. Netflix is offering to provide you the cache boxes
*for FREE* so that you can cache the data in your network;
isn't that exactly what he wanted, in his first sentence?
Why is it that two sentences later, free Netflix cache boxes
are suddenly an evil that must be avoided, no matter how
much Netflix may try to force them on you?

I'm sorry. I think someone forgot to take their coherency
meds before writing that paragraph.

If you like caching, you should be happy when someone
offers to give you caching boxes for FREE. If you don't
like caching, you shouldn't bitch about inefficient it is to
have traffic that isn't being cached.

Trying to play both sides of the issue like that in the
same paragraph is just...dizzying.

Matt

14

Similar but much smaller scale issue that I'm having trying to deliver our
content to access networks - small amount of traffic, heavily skewed
outbound from our AS but massive amounts of players on these access
networks - yet we're forced to pay said access networks to deliver our
mutual customers for an optimal experience.

So much double dipping.

15

Translation: if Netflix is paying Comcast, why not pay us(Verizon) ?

Rubens

16

Trying to play both sides of the issue like that in the same
paragraph is just...dizzying.

if we filtered or otherwise prevented conjecturbation, jumping to
conclusions based on misuse of tools, hyperbole, misinformation, fud,
and downright lying, how would we know the list exploder was working?

randy

17

Jimmy Hess wrote:
>>Randy Bush wrote:
>[snip]
>>At the ISPs expense, including connectivity to a peering point. Most content
>>providers pay Akamai, Netflix wants ISPs to pay them. Hmmm....
>Netflix own website indicates otherwise.
>https://www.netflix.com/openconnect
>
>"ISPs can directly connect their networks to Open Connect for free.
>ISPs can do this either by free peering with us at common Internet
>exchanges, or can save even more transit costs by putting our free
>storage appliances in or near their network."

From another list, I think this puts it nicely (for those of you who
don't know Brett, he's been running a small ISP for years
http://www.lariat.net/)

I've got to say, I'm not overly impressed by his commentary. It's vague and
non-specific, and doesn't provide any meat by which an impartial observer
could judge the claims. Hell, I'm a partial observer in *favour* of the
little guy, and I'm underwhelmed. The thoughts that come to mind are
inlined.

--------

Netflix generates huge amounts of wasteful, redundant traffic

I agree that most movies and TV shows aren't worth watching, but that's a
value judgment. "Redundant" perhaps, since everyone who watches the show
gets their own stream, but unless your network is entirely multicast-ready,
you're not exactly free of blame...

Have you attempted to reach out to Netflix to discuss the concerns you have
about the inefficiency? What was the response?

and then
refuses to allow ISPs to correct this inefficiency via caching.

In what ways do they "refuse" to allow ISPs to cache? I can imagine that
they don't like caching run by other people, because they can't control how
well that caching is run (the charitable interpretation) or they don't get
all of the eyeball data when data is cached (the more likely
interpretation). Do they refuse to send traffic to your AS if they discover
you caching their content? They'd actually be well within their rights to
do so (on the principle of "their network, their content, their rules"),
that seems implausible.

It fails to provide adequate bandwidth for its traffic to ISPs' "front
doors" and then blames their downstream networks when in fact they are
more than adequate.

I'd be interested in seeing more data regarding this assertion. If your
links aren't congested, and you're not buying transit that is congested
somewhere upstream, the only remaining point of congestion would be
somewhere inside Netflix. I'm not saying that isn't what's happening, but
assuming that the entire Internet (or everyone who's getting Netflix bits
out of the same upstream of Netflix) isn't seeing problems, then the problem
inside Netflix seems... optimistic.

It exercises market power over ISPs (one of the first questions
asked by every customer who calls us is, "How well do you stream
Netflix?")

OK, they're popular. Are you advocating for a limit on how large a portion
of a market a single company is allowed to service?

The bit of this sentence that I find implausible is the idea that customers
are sufficiently aware of quality of service issues to ask questions before
purchasing.

in an attempt to force them to host their servers for free

These are the OpenConnect caching boxes, I assume? If that's the case, it's
incorrect to say that Netflix "refuses to allow [...] caching", simply that
they prefer to provide caching their way. As it stands, I don't see the
problem with running Netflix cacheboxes instead of your own -- if you *were*
running the cache, you would presumably need to pay for hosting anyway (and
also machines), so I'm not sure how OpenConnect is worse. If there are
reasons why OpenConnect boxes *are* inferior to some other solution (such as
if they take up 20 times the power and space of an equivalent caching
solution), then those are what need to be talked about.

and
to build out network connections for which it should be footing the bill.
(Netflix told us that, if we wanted to improve streaming performance, we
should pay $10,000 per month for a dedicated link, spanning nearly 1,000
miles, to one of its "peering points" -- just to serve it and no other
streaming provider.)

Is this simply because there is no "common Internet exchange" closer to
Laramie, Wyoming? If so, then all I can say is that it sucks to be running
a service in a remote part of the world, but you can hardly blame Netflix
for that.

We tell prospective customers that we provide a guaranteed amount of
capacity for them to the nearest major Internet hub.

I'd be interested to see a more detailed description of this situation --
where Lariat *does* have interconnection presence, and how "major" that
"major Internet hub" is. If Netflix is, indeed, ignoring a major
interconnect point, then it should be pretty easy to make that case and call
"bullshit" on Netflix's claims. As it stands, though, there's not enough
information to make any substantive assertion.

will not build out to our ISP as it has to larger
ones such as Comcast

Is there a similar compelling commercial benefit for Netflix to build out to
Laramie Internet Access and Telecommunications as there is to Comcast? If
not, it would be extremely foolish (not to mentin potentially criminal) for
Netflix to spend money on that. They're a for-profit company, they should
only spend money on things that will make them even more money.

I feel for Brett, and everyone else, running a small shop and getting the
shaft from the big kids. I've been there, and it's no fun. The answer,
though, is to get specific with facts and figures, and *prove* that the
claims of Netflix are bullshit, rather than continue with vague assertions.
They're better at the "vague assertions" game than you are.

- Matt

18

While trying to substantiate Mr. Glass' grievance with Netflix regarding their lack of availability to peer, I happened upon this tidbit from two months ago:

http://dewaynenet.wordpress.com/2014/04/29/re-netflix-inks-deal-with-verizon-wont-talk-to-small-isps/

  As for Mr. Woodcock's point regarding a lack of http://lariat.net/peering existing, https://www.netflix.com/openconnect/locations doesn't seem to do what I'd expect, either, although I did finally find the link to http://www.peeringdb.com/view.php?asn=2906 . To Mr. Glass' point, I'm not seeing any way the listed PoPs could feasibly be less than 900 wire-miles from Laramie -- to be fair, cutting across "open land" is a bad joke at best.

  Life is rough in these "fly-over" states (in which I would include my current state of residence); the closest IXes of which I'm aware are in Denver and SLC (with only ~19 and 9 peers, respectively). Either of those would be a hard sell for Netflix, no doubt about it.

  I guess I'm just glad that my home ISP can justify anteing up for a pipe to SIX, resources for hosting OpenConnect nodes, and, for that matter, an ASN. Indeed, not everyone can.

      Jima

19

From another list, I think this puts it nicely (for those of you who don't
know Brett, he's been running a small ISP for years http://www.lariat.net/
)

--------

Netflix generates huge amounts of wasteful, redundant traffic and then

refuses to allow ISPs to correct this inefficiency via caching. It fails to
provide adequate bandwidth for its traffic to ISPs' "front doors" and then
blames their downstream networks when in fact they are more than adequate.
It exercises market power over ISPs (one of the first questions asked by
every customer who calls us is, "How well do you stream Netflix?") in an
attempt to force them to host their servers for free and to build out
network connections for which it should be footing the bill. (Netflix told
us that, if we wanted to improve streaming performance, we should pay
$10,000 per month for a dedicated link, spanning nearly 1,000 miles, to one
of its "peering points" -- just to serve it and no other streaming
provider.) It then launches misleading PR campaigns against ISPs that dare
to object to this behavior.

--Brett Glass

As I see it, Netflix seem to have provided a reasonable set of options to
provide data to an ISP's customers:

- Over a certain volume, they'll provide caches to be hosted within the
eyeball AS
- Under that volume, you can pick it up via peering IXes
- If you don't peer with them anywhere, you can get it via transit

The complaint here seems to be that Netflix won't build out to
any/every/many smaller locations and/or pay to have their caches hosted.
Appreciate that there may be different views, but I'd say Netflix provide a
reasonable set of options here for the smaller ISP. I'd have thought
factoring in the assorted costs to access Netflix content (building to a
mutual peering IX vs. transit vs. the cost to run a local cache) would fall
into the standard sort of analysis you'd make running an ISP same as when
assessing if it makes sense to hosts a Google or Akamai cache.

Sam

20

I disagree as all of this makes perfect sense.

Would it be right if Netflix comes to You and says we see you've got a lot of our customers hooked up to your backbone so to serve better service we'd like to connect to your network directly.
And you goes: so you would like to become our customer? Sure this is the monthly fee for the link and transport service that would suite your needs.
And Netflix goes: well how about you build the link to us bearing all the costs and you gonna charge us nothing for the transport you provide, deal?
What would be your answer?

Of course this "good deal" has some precursors.
If your customers fail to obey your statistical multiplexing predictions and links to your upstreams are running hot than you have several options.
a) You could pay for the upgrades of links to your upstreams.
b) You could take the "good deal" Netflix has proposed to save costs for a).
c) You could not give a damn about your customers as they have nowhere else to go anyways and use this advantage to force Netflix to become your customer (well paying customer as they would need big pipes).
What would you do?

Options a) and b) assumes of course that Netflix has good connections to their upstreams and not misusing their position into forcing the customer relationship into free peering one.

adam