Verizon Public Policy on Netflix

My customers do not want me to "creatively" find ways to extract
additional money from them so as to cover expenses that Netflix
should be covering. Nor do they want me to subsidize Netflix
subscribers from the fees from non-Netflix subscribers. They
want to pay a fair price for their Internet that does not include
paying ransom to third parties.

We currently provide that: we guarantee each subscriber a certain
minimum capacity to the Internet exchange at 1850 Pearl Street
in Denver (to which Netflix does not directly connect) with a certain
maximum duty cycle. But we can't guarantee the performance of a specific
third party service such as Netflix. If Netflix wants us to do that,
it is going to have to pay us, as it pays Comcast. That's only fair,
because we would be doing something special just for it -- something
which costs money.

If Netflix tries to use its market power to harm ISPs, or to smear
us via nasty on-screen messages as it has been smearing Verizon, ISPs have
no choice but to react. One way we could do this -- and I'm strongly
considering it -- is to start up a competing streaming service that
IS friendly to ISPs. It would use the minimum possible amount of
bandwidth, make proper use of caching, and -- most importantly --
actually PAY Internet service providers, instead of sapping their
resources, by allowing them to sell it and keep a portion of the fee.
This would provide an automatic, direct, per-customer reimbursement
to the ISP for the cost of bandwidth. ISPs would sign on so fast
that such a service could BURY Netflix in short order.

--Brett Glass

Dude. Netflix doesn't want you to do help its service.

Your customers want you to do that.

By the way, don't think you're not going to have to pay us for all for that dirt you're hurling...

These entrepreneurs, digging up dirt and depositing it everywhere. Don't they know how much it costs us to keep the place clean?!

Tom

Building new things often does involve digging up dirt. Unlike Netflix, we'd gladly pay anyone who participates in the digging. :wink:

--Brett Glass

But before Netflix made a deal with Comcast, would you be making the same
request ?

Rubens

My customers do not want me to "creatively" find ways to extract
additional money from them so as to cover *expenses that Netflix
should be covering*. Nor do they want me to subsidize Netflix
subscribers from the fees from non-Netflix subscribers. They
want to pay a fair price for their Internet that does not include
paying ransom to third parties.

(emphasis mine)

I've gotta be frank here: I really don't understand the line of reasoning from an access network's perspective that says $CONTENT needs to pay $ACCESS to accept the bits that $ACCESS's users requested from $CONTENT. I might be missing nuances here, but I've not yet come across an argument that's convinced me of why this should be the case.

Of course your customers don't want prices raised; that's a no-brainer, and similarly it's fair for Grandma that just checks her email and Facebook to not want to carry the infrastructure costs of someone who consumes more content. I think what Charles is driving at is that users don't care about whether you pull in content through 1850 Pearl Street, transit, direct peering, or whatever; if I as a customer am paying for a 50 mbps service, I want my 50 mbps service. That said, to your point of no performance/connectivity guarantees to/from third parties: experience seems to indicate that users understand that you're not responsible for ensuring Netflix can *push out* the content at a given rate. However, if Netflix is capable of delivering the bits to your doorstep (wherever that is), it becomes your problem to get those bits from your doorstep to the customer.

If Netflix users (or users of any other bandwidth-hungry service) are sucking up more than "their fair share", which is generally to say that they're over your oversubscription ratio, and they are *also* over the minimum capacity that you're guaranteeing below, you're in the clear from a business standpoint as long as your remaining users still get their minimum. If other users are starting to get impacted so that they don't get their guaranteed minumum, that's a network management problem (i.e. how do I cap the "offending" traffic so that other users' still get their guaranteed rate?). I suspect you may have a hard time explaining to the Netflix users that, while you're dropping/delaying their traffic in that case, you're still delivering the promised service because they're still getting their minimum service to 1850 Pearl as they're traffic is coming in from somewhere else, but I digress...

If those users are over the overscription ratio *but* still below their guaranteed minimum, that sucks for the provider, but it's still the provider's problem, and the math in the business plan was apparently wrong. Since not all users consume exactly the same amount of network resources, someone is *always* subsidizing someone else's service; the question is just "by how much?" If the value north of the oversubscription ratio is sufficiently large that it's becoming a problem, either the agreement with the user(s) needs to be adjusted ("I know we said x mbps, but we actually meant < x mbps"), which again sucks from a business standpoint but is the provider's problem to deal with, or capacity needs to be augmented to match. The agreement bump can be either global (which again is a difficult business maneuver) or targeted at the users sucking up the extra capacity (which is more palatable, though users still generally balk at tiered/usage pricing).

None of these are really fun to deal with from the business side, but if $CONTENT is simply getting the bits to your edge as requested, I don't see in any way how they can be blamed for the unfortunate business situation in which $ACCESS finds himself. User asks for bits; $CONTENT gets the bits to $ACCESS's edge; $CONTENT's responsibility is done.

You stated earlier:

"Open Connect" is not, in fact, a CDN. Nor is it "peering." It is merely a set
of policies for direct connection to ISPs, and for placing servers in ISPs'
facilities, that is as favorable as possible in every way to Netflix.

That's news to me. We peer settlement-free with Netflix at the SIX, and they cover that in the OpenConnect umbrella term:

"ISPs can directly connect their networks to Open Connect for free. ISPs can do this either by free peering with us at common Internet exchanges, or can save even more transit costs by putting our free storage appliances in or near their network."
-- Netflix | Open Connect

Also:

Because it requires expensive bandwidth that's dedicated solely to Netflix,
"peering" (as Netflix calls it; it's really just a dedicated link) has 0%,
not 100%, offload. The ISP is paying for all of the bandwidth, and it
cannot be used for anything else.

I don't see any requirements that this is a dedicated link; we peer with them over public peering fabric and exchange a bunch of other traffic over that link. Is there another requirement in OpenConnect peering that we've just not hit but you are subject to?

OpenConnect has a range of options, from public peering to private interconnects to caching appliances; the intention, I gather, is to provide a range of options. Exchange a bit of traffic but not really all that much? Public peering. Starting to consume a bunch of traffic but don't want a cache appliance? Private interconnect. Exchange a bunch of traffic and prefer caching? Get a free appliance.

Presumably if you're not peering with Netflix and you don't have an appliance, you're getting the traffic via transit. You're free to not do any of the above if you find your transit costs for Netflix traffic are lower than those options (or if you just don't like OpenConnect), but for a lot of people public/private peering or a caching appliance saves $$ and resources.

...(b) pay them equitably for direct connections (smaller and more remote ISPs have higher costs per customer and should get MORE per account than Comcast, rather than receiving nothing);

The Comcast and Verizon deals were made because those guys had leverage, not because it costs them more. "We should get paid because Comcast got paid" doesn't add up.

Yes. I have an e-mail trail dating back to November of last year in which I attempted to discuss this with Netflix. They were intransigent. They didn't believe that the views of a small ISP from Wyoming were worth considering.

--Brett

I confess I might be splitting hairs, but what Internet exchange exists at 1850 Pearl Street? The best I can ascertain is that it's a Level3 datacenter, which doesn't seem (to me) to be the same thing.

  It would appear that neither LARIAT nor Netflix has chosen to connect to the most prevalent (one might argue "only true") IXP in Colorado, so I'm honestly a bit puzzled as to a) your indignation and b) their decision.

      Jima

But before Netflix made a deal with Comcast, would you be making
the same request ?

Yes. I have an e-mail trail dating back to November of last year in
which I attempted to discuss this with Netflix. They were
intransigent. They didn't believe that the views of a small ISP from
Wyoming were worth considering.

i am not sure that was rubens's question. and i suspect they were
looking for the cash not the views. but that aside, when i started in a
small area, i was aware of the consequent lack of leverage i would have,
business deals are not based just on one party's needs. otoh, 25 years
ago it was far easier to supply into a completely unfulfilled market.

any measurements of what the cache hit ratios might be in a market such
as yours? of course, even 10% is a win if you can get the cost of the
cache close to zero. (but what's the financial win there for netflix?)

<aside>
but you have my sympathies. i doubt i would start a rural isp in the
states today, insufficient leverage to get the facilities it would need
and too vulnerable to cable and telco. i am also not buying into cattle
here in tokyo. :slight_smile:

<aside^2>
we are netflix consumers, and tunnel it to tokyo. for us, the content
sucks, all mainstream, but japanese movie houses do not subtitle in
english. when we were in the states, the dvd service had the long tail,
and our tastes are out on that more obscure tail. my guess is that, for
streaming, netflix is in a major tussle with the studios. i suspect
there is a parallel to this overall discussion.

randy

From: "Brett Glass" <nanog@brettglass.com>

Note that I misunderstood you to be the Verizon blog poster I started
this thread commenting on. My apology for that in a separate post,
but here are some replies that amount to "you are standing on the
same rock in the river they are". :slight_smile:

My customers do not want me to "creatively" find ways to extract
additional money from them so as to cover expenses that Netflix
should be covering. Nor do they want me to subsidize Netflix
subscribers from the fees from non-Netflix subscribers. They
want to pay a fair price for their Internet that does not include
paying ransom to third parties.

Characterizing it as "ransom" and "expenses Netflix should be covering"
is, alas, largely in doubt, from the responses I've seen here; it's
assuming facts not in evidence.

We currently provide that: we guarantee each subscriber a certain
minimum capacity to the Internet exchange at 1850 Pearl Street
in Denver (to which Netflix does not directly connect) with a certain
maximum duty cycle. But we can't guarantee the performance of a specific
third party service such as Netflix. If Netflix wants us to do that,
it is going to have to pay us, as it pays Comcast. That's only fair,
because we would be doing something special just for it -- something
which costs money.

It's not Netflix who expects you to deliver that quality.

It's your customers. Who pay you for it.

If they're not paying you enough, well... who set those prices?
Netflix?

If Netflix tries to use its market power to harm ISPs, or to smear
us via nasty on-screen messages as it has been smearing Verizon, ISPs
have no choice but to react. One way we could do this -- and I'm strongly
considering it -- is to start up a competing streaming service that
IS friendly to ISPs. It would use the minimum possible amount of
bandwidth, make proper use of caching, and -- most importantly --
actually PAY Internet service providers, instead of sapping their
resources, by allowing them to sell it and keep a portion of the fee.
This would provide an automatic, direct, per-customer reimbursement
to the ISP for the cost of bandwidth. ISPs would sign on so fast
that such a service could BURY Netflix in short order.

Alas, content providers probably would not.

But good luck with that.

Cheers,
-- jra

(Yes, yes, I know, feeding the troll, etc.)

The Netflix users either have to pay to you, or they have to pay to Netflix. Now, if you're paying $20 per megabit/s/month then you and your users are victims of lack of competition in your area.

In properly developed places in the world with working competition, bandwidth prices are around $0.5-5/megabit/s/month. With those levels, you would have much less problem covering the cost of transit and your customers could use the service as much as they want because on margin, producing more bandwidth doesn't cost too much. At $20, I can understand that you're hurting. However, you paying $20 isn't Netflix problem. I don't see how Netflix could be re-imbursing you for your bandwidth costs, because it's not their fault either.

So, the real problem you should spend your energy on is why are you paying so much for bandwidth, not going after Netflix.

Since this is probably not something you can fix short term, I see no other option than to externalise your high margin cost to customers by imposing a monthly cap on usage and charging more for the people using the service more. You need to make sure your reveue model matches your expenditure model.

> My customers do not want me to "creatively" find ways to extract
> additional money from them so as to cover expenses that Netflix should
> be covering. Nor do they want me to subsidize Netflix subscribers from
> the fees from non-Netflix subscribers. They want to pay a fair price for
> their Internet that does not include paying ransom to third parties.

The Netflix users either have to pay to you, or they have to pay to
Netflix. Now, if you're paying $20 per megabit/s/month then you and your
users are victims of lack of competition in your area.

In properly developed places in the world with working competition,
bandwidth prices are around $0.5-5/megabit/s/month. With those levels, you
would have much less problem covering the cost of transit and your
customers could use the service as much as they want because on margin,
producing more bandwidth doesn't cost too much. At $20, I can understand
that you're hurting. However, you paying $20 isn't Netflix problem. I
don't see how Netflix could be re-imbursing you for your bandwidth costs,
because it's not their fault either.

So, the real problem you should spend your energy on is why are you paying
so much for bandwidth, not going after Netflix.

Since this is probably not something you can fix short term, I see no
other option than to externalise your high margin cost to customers by
imposing a monthly cap on usage and charging more for the people using the
service more. You need to make sure your reveue model matches your
expenditure model.

And in some parts of the world bandwidth caps are the norm even for
terrestial lines. My DOCIS home line has a 120G (down + up on this
plan) limit then it is rate limited for the rest of the month. I
don't hit the 120G limit though I regularly go over 60G. If I need
more bandwidth I would go up to the next tier. This gives me a
fixed price as well as well defined service expectations.

That would be awesome!

If you find a way to obtain premium content
that subscribers will pay for that doesn't include
incredibly restrictive licensing terms that require
you to account for every stream watched (including
those streamed from downstream cache devices),
I'm right there ready.

Unfortunately, I suspect you'll find the rights holders
who own the shows aren't willing to let their videos
be served through a CDN that doesn't maintain
draconian control over every stream (ie, that
doesn't allow third party, uncontrolled caching).

So, you may be able to build such a CDN; but
the only content you may find that you can
populate it with are cute cat videos recycled
from last week's Youtube footage--which nobody
wants to pay for. :frowning:

Matt

Hi,

Here is a different tale from another small ISP. We quite like Netflix (and
HBO Nordic and all the other streaming services). We are a FTTH provider
and services like Netflix is why people are buying our service instead of
going with 4G LTE or ADSL. Without content we have nothing.

Yes we have the same problems. Netflix does not peer in our city, in fact
they do not peer in our country(!). We are too small for an appliance. Yes
I really think Netflix should peer in Copenhagen, as going to Stockholm
through 1000 km of fiber is not really an option.

But when we are done whining about that, I have to say that paying transit
to get Netflix is not prohibitive expensive. We pay 0.5 USD/Mbps. Our
margins are not so poor that we can not afford to buy the content our users
want. Yes buy content - if we were a TV network we also would have to pay
for the content. Without content there is no need for our service.

The other ISP seems to be paying 20 USD/Mbps and that is their error. They
failed to find proper transit. I notice that Hurricane Electric is present
at the IX mentioned, so I am quite sure they could get cheap transit, if
they would just care to actually get some quotes.

Regards,

Baldur

Software is... herrr.... configurable.

Maybe Netflix could be convinced so their box had a switch from
complete catalog hosting / caching most used data. I get from this
discussion thread that small ISP feel having these box download the
whole catalog is more than what their customers (<1000) need. Moving
this discussion away from "net neutrality" (that seems what netflix is
doing in public anouncements) to how these boxes handle and operate
would be better for everyone.

Nailed it, Matt, 100%

The box doesn't even download 10% of the whole catalog and churns less than
1% a day.

Obviously our demand curve is proprietary information, but I can assure you
that a lot of people - engineers, mathematicians, etc. have looked at and
improved the algorithm - but we are still constantly working to make it
better.

If you look at boxes like Qwilt, which is a universal flow-through cache,
the best they can get is ~30% offload with Netflix traffic (in the real
world, not in their lab). With multiple different encodes (driven by
differing DRM and device types) the odds of two people watching the exact
same thing are relatively low. The law of large numbers rules the game.

-Dave

What are the chances of performing transcoding on the device rather than
sending multiple copies to it?

It seems that would save bandwidth without risking any licensing issues.

Dan

In my experience the bandwidth is typically the lowest part of the cost equation.

Why transcode on 1k nodes when you can do it once and distribute it at lower cost,
including in electricity to run the host CPU.

Centralized transcoding on dedicated hardware makes sense.

- Jared