Verizon Public Policy on Netflix

[Brett_Glass]

This is Brett Glass; I have been alerted to some of the responses to my
message (which was cross-posted by a third party) and have temporarily
joined the list to chime in. The following is my response to his message,
edited slightly to include some new information.

Dave Temkin wrote:

First and foremost, we built our CDN, Open Connect,

"Open Connect" is not, in fact, a CDN. Nor is it "peering." It is merely a set
of policies for direct connection to ISPs, and for placing servers in ISPs'
facilities, that is as favorable as possible in every way to Netflix. It costs
Netflix as little as possible and the ISP as much as possible.

with the intention to
deploy it as widely as possible in order to save ISPs who are delivering
our traffic money

It does not, in fact, appear to save ISPs money. Note that Comcast
asked for, and was given, additional payments even after it did all of
the things that are part of Netflix' "Open Connect" program. Netflix,
exercising inappropriate market power, has not offered smaller ISPs such
as my own the same amount per customer. In fact, it has offered us no
money at all -- even though our costs per Netflix customer are higher.
Netflix thus discriminates against and threatens smaller ISPs, and by
doing so, harms broadband competition.

and improve our mutual customer experience. This goes for
ISPs large and small, domestic and international, big endian and little
endian. We've never demanded payment from an ISP nor have we ever charged
for an Open Connect Appliance.

The power and bandwidth consumed by an "Open Connect Appliance" (which is
really just a hosted Netflix server) are a substantial expense for any ISP.
Especially because the server is not a cache; it is stocked with content
whether it is used or not and therefore wastes bandwidth on content and
formats that will never be used even once.

When we first launched almost three years ago, we set a lower boundary for
receiving a Netflix Open Connect Appliance (which are always free) at
5Gbps. Since then we've softened that limit to 3.5Gbps due to efficiencies
of how we pre-load our appliances (more on that below).

Most small ISPs (the average in the US, in fact) have 1,000 to 2,000 accounts.
If every one of those streams at 1 Mbps at the same time, which is highly
unlikely, this still does not reach 3.5 Gbps. Therefore, most ISPs are excluded
simply by this requirement if not by others (such as the requirement that the
ISP alone pay for a dedicated connection to one of Netflix' relatively few
"peering points").

We explicitly call our "cache" an Appliance because it's not a demand
driven transparent or flow-through cache like the Akamai or Google caches.
We do this because we know what's going to be popular the next day or even
week and push a manifest to the Appliance to tell it what to download
(usually in the middle of the night, but this is configurable by the ISP).

What Netflix does not say here is that (a) it can only somewhat predict what
will be in demand or go "viral;" (b) it wastes bandwidth by sending multiple
copies of each video to its server in different formats, rather than
transcoding locally or saving bandwidth on lesser used formats via caching;
and (c) its server consumes large amounts of energy and bandwidth. A cache
can be much more efficient and can be owned and managed by the ISP.

The benefit of this architecture is that a single Appliance can get 70+%
offload on a network, and three appliances clustered together can get 90+%
offload, while consuming approximately 500 watts of power, using 4U of rack
space, and serving 14Gbps per appliance.

To put this in perspective: LARIAT builds its own caches which consume
as little as 20 watts and can saturate a 10 Gbps Ethernet port. The
Netflix servers are large, bloated power hogs compared to a well designed
cache.

The downside of this architecture
is that it requires significant bandwidth to fill; in some ISPs cases
significantly more than they consume at peak viewing time. This is why our
solution may not work well for some small ISPs and we instead suggest
peering, which has 100% offload.

Because it requires expensive bandwidth that's dedicated solely to Netflix,
"peering" (as Netflix calls it; it's really just a dedicated link) has 0%,
not 100%, offload. The ISP is paying for all of the bandwidth, and it
cannot be used for anything else.

We've put a lot of effort into localizing our peering infrastructure
worldwide. As you can see from this map (sorry for the image), we're in 49
locations around the world with the significant bulk of them in the US
(blue pins = 1 location, red pins = >1 location in a metro) - more detailed
version at <http://goo.gl/eDHpHU&gt;http://goo.gl/eDHpHU and in our PeeringDB record (
<http://as2906.peeringdb.com>http://as2906.peeringdb.com) :

Our ISP connects to the Internet in Cheyenne, Wyoming (a major Internet
"crossroads;" it's where I-80 meets I-25) and Denver, Colorado (which is,
if anyplace can make the claim, the "center" of the entire Internet).

Netflix' small and relatively sparse network of "peering" points does not
include either of those locations. (I've noted, just today, that they have
only recently listed a presence at a private data center outside of Denver
in Englewood, Colorado -- not one of the major Denver exchanges, such as
1850 Pearl, to which we're already connected at great expense.) To get to it,
we would have to spend several thousand dollars per month on an expensive
connection, with no reimbursement of this expense from Netflix.

If Netflix were a good citizen, it would (a) let ISPs cache content; (b) pay them
equitably for direct connections (smaller and more remote ISPs have higher costs
per customer and should get MORE per account than Comcast, rather than receiving
nothing); and (c) work with ISPs to develop updated technology that makes streaming
more efficient. Bandwidth is expensive, and unicast streaming without caching is by
far the most inefficient conceivable way of delivering "fat" content to the consumer.

--Brett Glass

[Steven_Tardy1]

Brett,

You've previously stated: https://www.grc.com/sn/sn-457.htm
  "$20 per mbps per month"
  "1.25 gigabits of bandwidth coming in"

Math:
$20/mb/month * 1250mb = $25,000month

If netflix is 1/3 of bandwidth... saving 1/3 of $25,000 -=> $8,000/month.
(OK, Keep 100mbps for Netflix to pre-populate, 100mbps is 30TB/month)
(Now I'm curious how many GB/month Netflix pre-populates, hmmm)

How would "4U of rent" and 500W($50) electricity *not* save money?
If it's a money thing, then you have a price...
How much would Netflix have to pay you for you to consider?

Can you elaborate on:
It costs Netflix as little as possible and the ISP as much as possible.

If your ISP isn't tall enough for Netflix, Akamai has a lower barrier of
entry.
Have you let Akamai give you a local cache? why or why not?

Steven Tardy, maybe I'm missing/overlooking something...

[Nick_Hilliard3]

it's a local cache server and of course it costs Netflix as little as
possible because they're a business not a charity.

The access service provider gets better performance, gets to choose where
in their network makes most sense to have the device installed, and gets to
save on transit / long-hauling costs. The overall traffic figures aren't
changed in any material way, so it's not obvious why you claim that "it
costs [...] the ISP as much as possible", because it just doesn't.

For local content cache services of this type, some service providers are
large enough to be able to financially muscle the content providers into
paying for hosting / local connectivity fees, etc. This is nearly always a
function of whether that sort of a deal can be strong-armed or not. The
reverse is also true: the content providers will nearly always decline to
pay hosting and connectivity fees if they feel they can get away with it.
Both situations are a reflection of the relative importance of each
business to each other.

Nick

[Aled_Morris2]

Shame Netflix can't fill their appliances using really cheap, bulk, one-way
satellite bandwidth which is useless for most other Internet applications.
Then their traffic wouldn't use any of your real, paid for, transit.

Of course siting a dish would be another expense with hosting one of their
boxes, but if it made the on-going costs go away...

Aled

[Ulf_Zimmermann]

iBeam tried to do that. If only they had used something else than Windows
Server and other Microsoft products to do the caching.

[Brett_Glass]

Ironically, I did this with my Usenet news feed (which back then was the big bandwidth hog) 20 years ago. Mark Lottor set it up. I understand that someone has just been granted a dubious patent on the same technique, despite the well known prior art.

--Brett Glass

[Brett_Glass]

How would "4U of rent" and 500W($50) electricity *not* save money?

Because, on top of that, we'd have huge bandwidth expenses. And Netflix
would refuse to cover any of that out of the billions in fees it's collecting
from subscribers. We can't raise our prices (that would not only cost us
customers but be unfair to many of them; it would be forcing the non-Netflix
users to subsidize Netflix). We simply need Netflix to pay at least some of its
freight.

If your ISP isn't tall enough for Netflix, Akamai has a lower barrier of entry.
Have you let Akamai give you a local cache? why or why not?

Akamai refused to do so when we approached them. The Akamai rep was rather rude
and dismissive about it; we were too small to be worthy of their attention.

It's important to note that the growth of rural ISPs is limited by population.
Even if we did not have rapacious cable and telephone monopolies to compete
with, our size is naturally limited by the number of possible customers. Each
of those customers is every bit as valuable as an urban customer, but Netflix
won't even give us the SAME amount per customer it gives Comcast, much less
more (it costs more to serve each one). And Netflix is particularly out of line
because it is insisting that we pay huge bandwidth bills for an exclusive
connection just to it. It is also wasting our existing bandwidth by refusing to
allow caching.

If Netflix continues on its current course, ALL ISPs -- not just rural ones,
will eventually be forced to rebel. And it will not be pretty.

Our best hope, unless Netflix changes its ways, is for a competitor to come
along which has more ISP-friendly practices. Such a competitor could easily
destroy Netflix via better relations with ISPs... and better performance and
lower costs due to caching at the ISP.

--Brett Glass

[Charles_Gucker1]

If Netflix continues on its current course, ALL ISPs -- not just rural ones,
will eventually be forced to rebel. And it will not be pretty.

I call hogwash. ALL ISPs are in the business of providing access to
the Internet. If you feel the need to rebel, then I suggest you
look at creative ways to increase revenue from your customers, not
threaten to cut off a portion of the Internet that "cost too much".

A point that seems to be missed in this whole discussion. It was
your choice to provide services in a rural area, not Netflix, Akamai
or the like. If your business model is flawed, then don't expect
somebody else to step in and fix it for you. Bandwidth is
expensive to procure in a rural area, if you wish to change that,
maybe it's time to find some investors and build your own network into
an urban area where bandwidth, and interconnections in general, are
more reasonably priced.

Also, based on your logic within this whole thread, if I was a
customer of yours, I'd expect you to pay me to use your services as
you would be looking to get paid for my use of third party services.
Also, I believe that what happened between Comcast and Netflix is
temporary, much like what happened between Comcast and Level(3).

charles

[jim_deleskie]

So it sounds like your customers want to use the service being sold, but
you can't afford to service them due to the pricing they are being
charged...Sounds like you need to raise prices. While I haven't worked for
a rural wireless ISP, I have work for wired ISP's in the days of modems,
Large transit networks and MSO's. If it costs you more to provide service
then you charge for it, your a charity, not a business.

-jim

[Matthew_Petach2]

>How would "4U of rent" and 500W($50) electricity *not* save money?

Because, on top of that, we'd have huge bandwidth expenses.

I know I'm just a dumb troll, but
don't you have the same bandwidth
demands already from your users
pulling down netflix content today?

If your users don't use netflix, then this
is a moot point, and we can end the
discussion now. If your users *do* use
netflix currently, then you already have
this bandwidth demand on your network,
and finding ways to reduce or offload it
would be an improvement.

And Netflix
would refuse to cover any of that out of the billions in fees it's
collecting
from subscribers. We can't raise our prices (that would not only cost us
customers but be unfair to many of them; it would be forcing the
non-Netflix
users to subsidize Netflix). We simply need Netflix to pay at least some
of its
freight.

Why not follow a model that other networks use
(if you've ever bought transit in Asia, you've no
doubt come across this -- you get a price of $x/mbps
for transit; but if you're exchanging traffic with China
ASes, that traffic is billed at $x*6/mbps.) Simply inform
your users that due to the heavy demands netflix places
on your infrastructure, you'll need to add a streaming
surcharge onto their monthly bill to cover the costs,
and then let the market solve your problem. Either
users really do want netflix badly enough to pay
the surchage and cover your costs, or they opt
to find a different provider (in which case their
heavy bandwidth usage is no longer impacting
your network, so problem solved), or they decide
they really didn't need netflix that badly (in which
case the heavy bandwidth usage also goes away,
and your problem is solved).

>If your ISP isn't tall enough for Netflix, Akamai has a lower barrier of
entry.
>Have you let Akamai give you a local cache? why or why not?

Akamai refused to do so when we approached them. The Akamai rep was rather
rude
and dismissive about it; we were too small to be worthy of their attention.

It's important to note that the growth of rural ISPs is limited by
population.
Even if we did not have rapacious cable and telephone monopolies to compete
with, our size is naturally limited by the number of possible customers.
Each
of those customers is every bit as valuable as an urban customer, but
Netflix
won't even give us the SAME amount per customer it gives Comcast, much less
more (it costs more to serve each one). And Netflix is particularly out of
line
because it is insisting that we pay huge bandwidth bills for an exclusive
connection just to it. It is also wasting our existing bandwidth by
refusing to
allow caching.

See, this is why I think it was a bad move for
any content player to bow to $cableco's demands;
it's a slippery slope. Once you negotiate with one
extortionist, the next blackmailer asks for even
more money. The only answer is to never negotiate
with...er, sorry, tuned into the wrong psychic channel
there for a moment.

If Netflix continues on its current course, ALL ISPs -- not just rural
ones,
will eventually be forced to rebel. And it will not be pretty.

And the rebellion will take what form, exactly?
Cutting off netflix and other alternative content
sources, leaving people with the predetermined
slop fed to them over the airwaves and by their
franchise-agreement-granted-monopoly cable
company? Seems like exactly what the cable
companies want. It's good they've managed
to recruit an army of foot soldiers to lead the
vanguard of the attack without even having to
pay them--they can sit back and keep their
hands relatively unbloodied as the battle unfolds.

Our best hope, unless Netflix changes its ways, is for a competitor to come
along which has more ISP-friendly practices. Such a competitor could easily
destroy Netflix via better relations with ISPs... and better performance
and
lower costs due to caching at the ISP.

That won't happen, because allowing content to
be freely cached at the edge without control is
tantamount to giving the content away without
restriction; and no level of premium content is
going to come with a license like that. Like it
or not, no content creator is going to give up
all rights to their content like that; it's not a
(currently) viable business model.

You might just as well ask why George Lucas
continues to charge money for showings of
his movies (oh, right--better make that Bob
Iger now, sorry. ^_^;) Content creation is
a business, and needs to make money to
stay in business. Until that end of the equation
changes, allowing content to be freely
cached, replicated, and distributed just
isn't going to happen, and to expect
otherwise is hopelessly unrealistic.

--Brett Glass

Matt

[Todd_Lyons]

This is an interesting conversation to watch as a non-important,
non-influential outsider.

Brett's calculation is the cost of:

(BW of preloading X new shows a week in multiple formats)
  is greater than
(BW of Z % of his user base watching Y streams a week)

It's not been clearly stated whether X is 100% of new shows, but I
suspect it's more along the lines of mostly what Netflix expects to be
popular.

Because that Netflix box is not an on-demand cache, it gets a bunch of
shows pushed to it that may or may not be watched by any of Brett's
customers. Then the bandwidth he must use to preload that box is
large, much larger than the sum of the streams his customers do watch.

Brett touched on this in the Security Now episode, but I don't think
he was clear so I want to explore the realities of these options.
IMHO two solutions exist that would make small people like Brett much
happier with this Netflix box:

1) Make the box an on-demand cache: the first customer who watches a
show causes the episode to stream/push_high_bw to the box, and from
the box out to the customer. Any subsequent customer gets it directly
from the box, even if the initial stream is still ongoing.
Complications do arise if the second (or third) customer tries to move
beyond the current location of the initial stream.

2) My suggestion is probably less popular because it requires a person
with (maybe more than) a few minutes, but give the list of shows
desired to be pre-pushed to the box to $ISP and give them a couple
hours to uncheck certain things that they know or suspect their users
won't watch, allowing them to reduce their bandwidth usage. And
conversely, provide a checkbox of shows that the ISP wants to never be
cached on the box.

I did agree with the comment later in the email that making content
freely cached is a non-starter because that content could be copied
too easily. However, if the Netflix box is what does all of the
on-demand caching in #1, then it leaves the power in Netflix's hands,
while not requiring the ISP to download multiple copies of shows that
its users will never watch.

A lot of this is dependent upon:
1) How many different copies of a single show are pushed to the box.
Does that number vary per show.
2) How many shows are pushed/pre-pushed to the box per week. How frequently.

...Todd

[Matthew_Petach2]

>> >How would "4U of rent" and 500W($50) electricity *not* save money?
>> Because, on top of that, we'd have huge bandwidth expenses.
>
> I know I'm just a dumb troll, but
> don't you have the same bandwidth
> demands already from your users
> pulling down netflix content today?

This is an interesting conversation to watch as a non-important,
non-influential outsider.

Brett's calculation is the cost of:

(BW of preloading X new shows a week in multiple formats)
  is greater than
(BW of Z % of his user base watching Y streams a week)

It's not been clearly stated whether X is 100% of new shows, but I
suspect it's more along the lines of mostly what Netflix expects to be
popular.

Because that Netflix box is not an on-demand cache, it gets a bunch of
shows pushed to it that may or may not be watched by any of Brett's
customers. Then the bandwidth he must use to preload that box is
large, much larger than the sum of the streams his customers do watch.

Thank you for clarifying that; I thought what
Brett was concerned about was traffic in
the downstream direction, not traffic for
populating the appliance.

Brett touched on this in the Security Now episode, but I don't think
he was clear so I want to explore the realities of these options.
IMHO two solutions exist that would make small people like Brett much
happier with this Netflix box:

1) Make the box an on-demand cache: the first customer who watches a
show causes the episode to stream/push_high_bw to the box, and from
the box out to the customer. Any subsequent customer gets it directly
from the box, even if the initial stream is still ongoing.
Complications do arise if the second (or third) customer tries to move
beyond the current location of the initial stream.

2) My suggestion is probably less popular because it requires a person
with (maybe more than) a few minutes, but give the list of shows
desired to be pre-pushed to the box to $ISP and give them a couple
hours to uncheck certain things that they know or suspect their users
won't watch, allowing them to reduce their bandwidth usage. And
conversely, provide a checkbox of shows that the ISP wants to never be
cached on the box.

What if Netflix provided a third option; give
the ISP a small UI through which they could
set a "not-to-exceed" traffic rate on the
appliance; the appliance would then seek
to fill itself with content according to its
priority-ranked listing by popularity, with
the rsync (or whatever underlying technology
it utilizes) set to rate limit itself to the value
set by the ISP. It's already clear that netflix
can handle streaming content that is *not*
within the openconnect appliances, as that's
what they do for the rest of their long tail
content; this would simply shift where in the
list of content the "long tail" began for users
of this ISP. This would allow the ISP to gain
the benefit of localized content sourcing for
the historically highly popular content, while
controlling the infeed volume to an acceptable
rate for their network. Even setting a relatively
small infeed rate of 100mb/sec would allow the
appliance to populate 1TB/day of content, which
would account for 30 DVD-sized titles/day--and
I'm sure netflix compresses its data sources down
considerably better than a 4GB DVD image file.

I think that approach would help keep both
sides happier; Netflix keeps control over the
content in its appliance, and the smaller ISPs
get the traffic offload benefit without having
to sacrifice a huge volume of the upstream
bandwidth to the appliance.

I did agree with the comment later in the email that making content
freely cached is a non-starter because that content could be copied
too easily. However, if the Netflix box is what does all of the
on-demand caching in #1, then it leaves the power in Netflix's hands,
while not requiring the ISP to download multiple copies of shows that
its users will never watch.

A lot of this is dependent upon:
1) How many different copies of a single show are pushed to the box.
Does that number vary per show.
2) How many shows are pushed/pre-pushed to the box per week. How
frequently.

...Todd
--
The total budget at all receivers for solving senders' problems is $0.
If you want them to accept your mail and manage it the way you want,
send it the way the spec says to. --John Levine

Yup--I think fundamentally the challenge here
is how to give the ISP some level of control over
the bandwidth consumption. Solving that, whether
by changing to a pure on-demand model, or by giving
a knob to change the infeed rate, would I think make
netflix considerably more popular with the smaller
sized ISPs.

Thanks!

Matt

[James_Hess]

Because that Netflix box is not an on-demand cache, it gets a bunch of
shows pushed to it that may or may not be watched by any of Brett's
customers. Then the bandwidth he must use to preload that box is
large, much larger than the sum of the streams his customers do watch.

However..... (1) There are other considerations besides bandwidth
saved: there is customer experience improvement if latency and
therefore load times decrease.

(2) You or a cache box don't know which streams your customers will
watch in advance. Although the cache units preload popular content,
not necessarily the entire catalog. Your users are most likely
watch during peak hours, which is the time at which more bandwidth is
the most expensive... at most other times, additional bandwidth
usage is $0,
so it doesn't strictly matter, necessarily, if more total transfer is
required using a cache box than not.

(3) If you don't have at least a couple Gigabits of Netflix traffic,
you are unlikely to consider undertaking the expense of the SLA
requirements before you can run a box, electricity, space in the
first place, if you even meet the traffic minimums required to get
free cache boxes.

And (4) The "pushing of shows to the units" occur during a
configured fill window, which their guides say will be defined by
the provider's network planning team in a manner and maximum bandwidth
demand over that time suited to your traffic profile, so as to not
increase the 95-th percentile traffic from your upstream.

For example: the fill window can occur during the hours of the day
when there is little interactive customer traffic. They recommend a
10 to 12 hour fill window with a maximum rate of 1.2 Gigabits.

http://oc.nflxvideo.net/docs/OpenConnect-Deployment-Guide.pdf

Therefore, in any of the cases where cache boxes have actually been
implemented properly, they are still likely to be a net benefit for
both provider and customers.

[Brett_Glass]

Because that Netflix box is not an on-demand cache, it gets a bunch of
shows pushed to it that may or may not be watched by any of Brett's
customers. Then the bandwidth he must use to preload that box is
large, much larger than the sum of the streams his customers do watch.

Yes. Especially since Netflix insists upon sending multiple copies of
each video -- not only in different formats but in different resolutions
-- to the server.

I did agree with the comment later in the email that making content
freely cached is a non-starter because that content could be copied
too easily.

The content could be copied just as easily from a Netflix server if it
were stolen from the ISP's office. However, by far the most likely place
for illicit copies to be made is at the client end, because it can
be done in a private location and attract no attention at all.

However, if there is any concern about either a Netflix server OR an
ISP's cache being used to obtain illicit copies of the video, the solution
is simple. This is a trivial problem to solve. Send and store the streams in
encrypted form, passing a decryption key to the user via a separate,
secured channel such as an HTTPS session. Then, it is not possible to obtain
usable copies of the content by stealing either a Netflix server OR an
ISP-owned cache. Problem solved.

--Brett Glass

[Barry_Shein1]

Just an observation:

I've been on the internet since dirt was rocks.

It seems to me that one theme which has come up over and over and over
is that some new-ish technology demands more bandwidth than whatever
it was people were doing previously and as it popularizes people begin
fighting.

In the early 80s it was downloading the host table, "could people
please try NOT to all download via a script at exactly midnight!!!"

Then it was free software in the eighties, did WSMR et al really have
a RIGHT to become a magnet for such popular program downloads?!

And graphic connection to remote super-computer centers. Could the
images please be generated locally and downloaded "off hours"
(whatever "off hours" meant on the internet) or even shipped via tape
etc rather than all these real-time graphical displays running???!!!

Hey, the BACKBONE was 56kb.

Then Usenet, and images, particularly, oh, explicit images because OMG
imagine if our administration found out our link was slow because
students (pick a powerless political class to pick on and declare
THEIR use wasteful) were downloading...um...you know.

And games OMG games.

I remember sitting in an asst provost's office in the 80s being
lectured about how email was a complete and total waste of the
university's resources! Computers were for COMPUTING (he had a phd in
physics which is where that was coming from.)

And the public getting on the internet (ahem.)

On and on.

Now it's video streaming.

And then the bandwidth catches up and it's no big deal anymore.

And then everyone stops arguing about it and goes on to the next thing
to argue about. Probably will be something in the realm of this
"Internet of Things" idea, too many people conversing with their
toaster-ovens.

My comment has always been the same:

   There are two kinds of people in this world: Those who try to
   figure out how bake more bread, and those who herd people into
   bread lines.

I've always tried to be the sort of person who tries to figure out how
to bake more bread. This too shall pass.

[Frank_Bulk1]

A third option is to use a transparent caching box, so it caches what's seen. At $20/Mbps I suspect all the popular vendors would find three year or less ROI.

Frank

[Jay_Ashworth]

From: nanog@brettglass.com

This is Brett Glass; I have been alerted to some of the responses to my
message (which was cross-posted by a third party) and have temporarily
joined the list to chime in. The following is my response to his
message, edited slightly to include some new information.

Well, they were actually responses to *my* message, which made a
fundamental point which you carefully don't address here at all, amongst
what our British counterparts would probably term your whinging.

If Netflix were a good citizen, it would (a) let ISPs cache content;
(b) pay them
equitably for direct connections (smaller and more remote ISPs have
higher costs
per customer and should get MORE per account than Comcast, rather than
receiving
nothing); and (c) work with ISPs to develop updated technology that
makes streaming
more efficient. Bandwidth is expensive, and unicast streaming without
caching is by
far the most inefficient conceivable way of delivering "fat" content
to the consumer.

Bandwidth is expensive. Given.

You made the wrong gamble on how asymmetrical your customers connections
would *really* be. But that doesn't make that traffic *not be* -- as your
brothers in the telco arm would phrase it -- "at your customers' instance",
rather than, as your arguments all assume, at Netflix's.

About 80% of so of the responses I've seen here agree that's a reasonable
view of the situation... so we'll for the moment assume that you didn't
address it because you *can't* address it.

Care to differ?

Cheers,
-- jra

[ As you might imagine, this is a bit of a hobby horse for me; Verizon's
behavior about municipally owned fiber, and it's attempts to convert post-
Sandy customers in NYS from regulated copper to unregulated FiOS service
leave a pretty bad taste in my mouth about VZN. ]

[Miles_Fidelman1]

Jay Ashworth wrote:

[ As you might imagine, this is a bit of a hobby horse for me; Verizon's behavior about municipally owned fiber, and it's attempts to convert post- Sandy customers in NYS from regulated copper to unregulated FiOS service leave a pretty bad taste in my mouth about VZN. ]

Jay,
Quite agree with you on this stuff. I used to spend a good part of my time working with municipalities on planning fiber builds - so VZ's behavior on those matters leave a pretty bad taste in my mouth too. But.. that's kind of a different issue, wouldn't you say?

Am I obtuse or does it all boil down to:

1. If both Netflix customers, and Netflix all connected to a single network - customers would be paying for their access connections, and Netflix would be paying for a pipe big enough to handle the aggregate demand.

2. The issue is that customers connect to one network (actually multiple networks, but lets stick with Verizon for now), and pay Verizon; Netflix buys aggregate capacity into other networks; with one or more transit networks in the middle.

3. Somebody has to pay for what's in the middle (ports into transit networks, bandwidth across them). Those are additional costs, that wouldn't exist if everyone were connected to the same network.

4. Both parties can make reasonable claims about why the other guys should pay.

5. Verizon and Comcast are big enough to say "Netflix pays" - with Netflix making a visible stink about it.

6. Netflix is important enough to end users, that Netflix can tell the little guys "you pay." And yes, they're making it a little easier by providing the CDN boxes.

7. In the absence of some reasonably balanced formal policies and regulations about settlements - we're going to keep seeing this kind of stuff.

Miles Fidelman

[Jared_Mauch]

I think here is where many of us may disagree.

While the current (public) dispute between Verizon and Netflix is fun for everyone to point fingers at saying "look here, there is a problem", the market also "mostly works". Verizon and Netflix seem to have reached (per press reports) an agreement and the largest problem today is the lack of ability to turn-up these ports quickly. Some market players move at light-speed, others at more glacial paces.

I've been paying close attention to this for a variety of reasons. I've heard stories of some incumbents taking double-digit months to provision these types of services to correct congestion. I'm expecting the resolution time-scale to be much longer than was seen with the Comcast <-> Netflix connections.

it would not surprise me if it took 18 months to provision these ports.

(I recall phoning AT&T once asking for 100m service at a commercial address and it took a swat-team of people on the phone to tell me they would be 4x/mo what I was paying.. I politely told them they were too expensive and to not schedule a 8 person conference call for a basic service level).

- Jared

[Jay_Ashworth]

From: "Miles Fidelman" <mfidelman@meetinghouse.net>

Jay Ashworth wrote:

[ As you might imagine, this is a bit of a hobby horse for me; Verizon's
behavior about municipally owned fiber, and it's attempts to convert
post- Sandy customers in NYS from regulated copper to unregulated FiOS
service leave a pretty bad taste in my mouth about VZN. ]

Jay,
Quite agree with you on this stuff. I used to spend a good part of my
time working with municipalities on planning fiber builds - so VZ's
behavior on those matters leave a pretty bad taste in my mouth too.
But.. that's kind of a different issue, wouldn't you say?

Certainly. Just full disclosure: I'm as motivated to reply to this as I
am *because* I already have a hard-on for VZN.

Am I obtuse or does it all boil down to:

1. If both Netflix customers, and Netflix all connected to a single
network - customers would be paying for their access connections, and
Netflix would be paying for a pipe big enough to handle the aggregate
demand.

Correct.

2. The issue is that customers connect to one network (actually multiple
networks, but lets stick with Verizon for now), and pay Verizon; Netflix
buys aggregate capacity into other networks; with one or more transit
networks in the middle.

3. Somebody has to pay for what's in the middle (ports into transit
networks, bandwidth across them). Those are additional costs, that
wouldn't exist if everyone were connected to the same network.

4. Both parties can make reasonable claims about why the other guys
should pay.

There's argument about whether VZN's claims are reasonable, and I
tend to fall on the "they are not, even though I don't like VZN anyway"
side; this thread was as much a sanity check as anything.

5. Verizon and Comcast are big enough to say "Netflix pays" - with
Netflix making a visible stink about it.

Yup.

6. Netflix is important enough to end users, that Netflix can tell the
little guys "you pay." And yes, they're making it a little easier by
providing the CDN boxes.

Fair amount easier I would say, but I don't think we have enough
empirical evidence either way, at least not in this thread.

7. In the absence of some reasonably balanced formal policies and
regulations about settlements - we're going to keep seeing this kind
of stuff.

I hope that it doesn't come to that. Regulation is horrible.

Cheers,
-- jra