Inevitable death, was Re: Verizon Public Policy on Netflix

bensons · July 14, 2014, 4:08pm

Thanks for adding this perspective, Barry. I think it's realistic. But I
also think it might miss an orthogonally connected issue - this isn't just
about bandwidth, but about commoditization, consolidation, size etc. It may
be that small ISPs just can't compete (at least in the broader market) as
the market evolves. Similar to how I was disappointed by the loss of my
local bookstore, but still buy all my stuff from Amazon. ... I hear Brett
essentially asking for Netflix to do more for him than it does for big
ISPs, because his small rural business model can't compete with the big
guys.

Thoughts?

Cheers,
-Benson

Scott_Helms1 · July 14, 2014, 5:15pm

Benson,

The difference, and its a large one, is that the large operators have no
interest in building in the less dense rural (and sometimes suburban)
areas. The smaller operators are often the only provider in the area and
unlike a bookstore if someone wants broadband in an area they can't drive
to a larger town and bring a bagful home the way we can with books.

There are a few potential paths forward that I can see and I'm sure there
are more that others can identify:

1) Various governmental funding sources like CAF subsidize the market
"enough" for smaller operators to continue to get by.

2) CAF and other funding make rural territories profitable enough that the
large operators buy many/most/all of the smaller providers.

3) Prices for rural customers increase to cover the increased costs.

4) Content providers contribute $some_amount to help cover the costs of
connectivity.

5) Operators in rural markets fall further behind making rural markets
even less attractive and that contributes the trend of rural to urban
migration here in the US.

Of course a combination of these is also possible or local governments
could get more involved, but these look to be the most likely in no real
order.

Scott Helms
Vice President of Technology
ZCorum
(678) 507-5000

Barry_Shein1 · July 14, 2014, 5:17pm

Thanks for adding this perspective, Barry. I think it's realistic. But I
also think it might miss an orthogonally connected issue - this isn't just
about bandwidth, but about commoditization, consolidation, size etc. It may
be that small ISPs just can't compete (at least in the broader market) as
the market evolves. Similar to how I was disappointed by the loss of my
local bookstore, but still buy all my stuff from Amazon. ... I hear Brett
essentially asking for Netflix to do more for him than it does for big
ISPs, because his small rural business model can't compete with the big
guys.

Thoughts?

But if the marginal cost of carrying netflix and similar approached
zero this wouldn't be a problem.

A big problem with being a usenet server was that it could take 50GB
of disk space, easy. How to monetize all that disk space in a day when
a GB disk cost $500? A surcharge for clients using usenet? Charge
downstream customers you fed? New protocols with less store and more
aggressive forward? Evolve to sites which specialize in usenet service
rather than expecting every mom & pop ISP to provide it as a base
measure of service?

But today I can get key fobs with 64GB for about $50, and of course
4TB disks for under $200.

So the apparent urgency of the content business models is directly
related to the costs, which tend to drop over time, usually to the
point that it becomes non-urgent (or argue that they can't.)

More importantly it tends to go through the same basic patterns:
Identify who is benefiting. Argue about what "benefiting" means. Try
to assess relative benefits and costs proportionately. Improve
technology step-wise to mitigate and possibly reallocate costs
assessing any effects on benefits. Follow the technology curve. Etc.

Video streaming seems challenging. But so did 50GB of disk once.

I suppose if I were to make a concrete suggestion it would be to try
to develop hypothetical cost curves, thresholds (at what cost does it
not matter even to the more vulnerable?), estimate dates (hah!), and
not put more energy into the problem than such an analysis merits. In
particular soas not to develop potentially disruptive new models whose
implementation and cost of implementation one might soon enough come
to regret.

Also remembering that extrapolations tend to be foiled by discrete
events. For example, Apr 1, 2017: Comcast/TW buys Netflix...

Just an observation:

I've been on the internet since dirt was rocks.

It seems to me that one theme which has come up over and over and over
is that some new-ish technology demands more bandwidth than whatever
it was people were doing previously and as it popularizes people begin
fighting.

In the early 80s it was downloading the host table, "could people
please try NOT to all download via a script at exactly midnight!!!"

Then it was free software in the eighties, did WSMR et al really have
a RIGHT to become a magnet for such popular program downloads?!

And graphic connection to remote super-computer centers. Could the
images please be generated locally and downloaded "off hours"
(whatever "off hours" meant on the internet) or even shipped via tape
etc rather than all these real-time graphical displays running???!!!

Hey, the BACKBONE was 56kb.

Then Usenet, and images, particularly, oh, explicit images because OMG
imagine if our administration found out our link was slow because
students (pick a powerless political class to pick on and declare
THEIR use wasteful) were downloading...um...you know.

And games OMG games.

I remember sitting in an asst provost's office in the 80s being
lectured about how email was a complete and total waste of the
university's resources! Computers were for COMPUTING (he had a phd in
physics which is where that was coming from.)

And the public getting on the internet (ahem.)

On and on.

Now it's video streaming.

And then the bandwidth catches up and it's no big deal anymore.

And then everyone stops arguing about it and goes on to the next thing
to argue about. Probably will be something in the realm of this
"Internet of Things" idea, too many people conversing with their
toaster-ovens.

My comment has always been the same:

   There are two kinds of people in this world: Those who try to
   figure out how bake more bread, and those who herd people into
   bread lines.

I've always tried to be the sort of person who tries to figure out how
to bake more bread. This too shall pass.

--
        -Barry Shein

The World | bzs@TheWorld.com |
http://www.TheWorld.com
Purveyors to the Trade | Voice: 800-THE-WRLD | Dial-Up: US, PR,
Canada
Software Tool & Die | Public Access Internet | SINCE 1989 *oo*

Thanks for adding this perspective, Barry. I think it's =
realistic. But I also think it might miss an orthogonally connected issue -=
this isn't just about bandwidth, but about commoditization, consolidat=
ion, size etc. It may be that small ISPs just can't compete (at least i=
n the broader market) as the market evolves. Similar to how I was disappoin=
ted by the loss of my local bookstore, but still buy all my stuff from Amaz=
on. ... I hear Brett essentially asking for Netflix to do more for him than=
it does for big ISPs, because his small rural business model can't com=
pete with the big guys.

Thoughts?
Cheers, 
-Benson 

<div class=3D"gmail_quote">On Jul 13, 2014 3:59 PM, "Barry Shein"=
<<a href=3D"mailto:bzs@world.std.com">bzs@world.std.com</a>> wrote:<=
br type=3D"attribution"><blockquote class=3D"gmail_quote" style=3D"margin:0=
0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
 
Just an observation: 
 
I've been on the internet since dirt was rocks. 
 
It seems to me that one theme which has come up over and over and over 
is that some new-ish technology demands more bandwidth than whatever 
it was people were doing previously and as it popularizes people begin 
fighting. 
 
In the early 80s it was downloading the host table, "could people 
please try NOT to all download via a script at exactly midnight!!!"<br=

Then it was free software in the eighties, did WSMR et al really have 
a RIGHT to become a magnet for such popular program downloads?! 
 
And graphic connection to remote super-computer centers. Could the 
images please be generated locally and downloaded "off hours" 
(whatever "off hours" meant on the internet) or even shipped via =
tape 
etc rather than all these real-time graphical displays running???!!! 
 
Hey, the BACKBONE was 56kb. 
 
Then Usenet, and images, particularly, oh, explicit images because OMG 
imagine if our administration found out our link was slow because 
students (pick a powerless political class to pick on and declare 
THEIR use wasteful) were downloading...um...you know. 
 
And games OMG games. 
 
I remember sitting in an asst provost's office in the 80s being 
lectured about how email was a complete and total waste of the 
university's resources! Computers were for COMPUTING (he had a phd in<b=

physics which is where that was coming from.) 
 
And the public getting on the internet (ahem.) 
 
On and on. 
 
Now it's video streaming. 
 
And then the bandwidth catches up and it's no big deal anymore. 
 
And then everyone stops arguing about it and goes on to the next thing 
to argue about. Probably will be something in the realm of this 
"Internet of Things" idea, too many people conversing with their<=

toaster-ovens. 
 
My comment has always been the same: 
 
=C2=A0 =C2=A0There are two kinds of people in this world: Those who try to<=

=C2=A0 =C2=A0figure out how bake more bread, and those who herd people into=
 
=C2=A0 =C2=A0bread lines. 
 
I've always tried to be the sort of person who tries to figure out how<=

to bake more bread. This too shall pass. 
 
-- 
=C2=A0 =C2=A0 =C2=A0 =C2=A0 -Barry Shein 
 
The World =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| bzs@TheWorld.co=
m =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 | <a href=3D"http://www.TheWorld.com" =
target=3D"_blank">http://www.TheWorld.com</a> 
Purveyors to the Trade | Voice: 800-THE-WRLD =C2=A0 =C2=A0 =C2=A0 =C2=A0| D=
ial-Up: US, PR, Canada 
Software Tool & Die =C2=A0 =C2=A0| Public Access Internet =C2=A0 =C2=A0=
> SINCE 1989 =C2=A0 =C2=A0 *oo* 
</blockquote></div>

Matthew_Petach2 · July 14, 2014, 5:41pm

Benson,

The difference, and its a large one, is that the large operators have no
interest in building in the less dense rural (and sometimes suburban)
areas. The smaller operators are often the only provider in the area and
unlike a bookstore if someone wants broadband in an area they can't drive
to a larger town and bring a bagful home the way we can with books.

But if that's the case, then Brett has no issue. As Benson
noted:

> Thanks for adding this perspective, Barry. I think it's realistic. But I
> also think it might miss an orthogonally connected issue - this isn't
just
> about bandwidth, but about commoditization, consolidation, size etc. It
may
> be that small ISPs just can't compete (at least in the broader market) as
> the market evolves. Similar to how I was disappointed by the loss of my
> local bookstore, but still buy all my stuff from Amazon. ... I hear Brett
> essentially asking for Netflix to do more for him than it does for big
> ISPs, because his small rural business model can't compete with the big
> guys.

Brett's concerns seem to center around his
ability to be cost-competitive with the big
guys in his area...which implies there *are*
big guys in his area to have to compete with.

If the big guys don't want to build into the
rural area, and aren't competing with Brett,
he can charge accordingly (the scenario
Scott outlines). If the big guys *have* built
into the area where Brett is serving users,
then we're outside of Scott's model, and into
Benson's model, and it may well be a case
of the local bookstore not being able to compete
with Amazon anymore.

While having no competitors in an area might
suck for the *consumers*, I don't think it's the
situation that Brett is facing; I think he's talking
about trying to compete with large carriers who
have indeed built out into his area, and have a
large economy of scale on their side.

I could be wrong, though; I often am.

Thanks!

Matt

George_Herbert · July 14, 2014, 8:42pm

He 's running wireless links, from web and prior info as I recall. His key business seems to be outside the cable tv / DSL wire loop ranges from wire centers. The bigger services seem to have fiber into Laramie, and Brett seems to have fiber to that Denver exchange pointlet .

Why he's not getting fiber to a bigger exchange point or better transit is unclear.

There are bandwidth reseller / BGP / interconnect specialist ISPs out there who live to fix these things, if there's anything like a viable customer base...

George William Herbert

Matthew_Petach2 · July 14, 2014, 10:12pm

>
> Brett's concerns seem to center around his
> ability to be cost-competitive with the big
> guys in his area...which implies there *are*
> big guys in his area to have to compete with.

He 's running wireless links, from web and prior info as I recall. His
key business seems to be outside the cable tv / DSL wire loop ranges from
wire centers. The bigger services seem to have fiber into Laramie, and
Brett seems to have fiber to that Denver exchange pointlet .

Why he's not getting fiber to a bigger exchange point or better transit is
unclear.

There are bandwidth reseller / BGP / interconnect specialist ISPs out
there who live to fix these things, if there's anything like a viable
customer base...

Ah--right, that was the genesis of my rant about
"if you don't have an ASN, you don't exist".
He'd first have to get an ASN before he could
engage in getting a different upstream transit,
or connect to different exchange points, etc.

As much as people insisted you can be an
ISP without an AS number, I will note that
it's much, MUCH harder, to the point where
the ARIN registration fees for the AS number
would quickly be recouped by the cost savings
of being able to shop for more competitive
connectivity options.

Matt

Scott_Helms1 · July 14, 2014, 11:32pm

Matt,

While I understand your point _and_ I agree that in most cases an ISP
should have an ASN. Having said that, I work with multiple operators
around the US that have exactly one somewhat economical choice for
connectivity to the rest of the Internet. In that case having a ASN is
nice, but serves little to no practical purpose. For clarity's sake all 6
of the ones I am thinking about specifically have more than 5k broadband
subs.

I continue to vehemently disagree with the notion that ASN = ISP since
many/most of the ASNs represent business networks that have nothing to do
with Internet access.

Scott Helms
Vice President of Technology
ZCorum
(678) 507-5000

Brett_Glass · July 14, 2014, 11:45pm

Why don't you simply ask me? There have been a huge number of incorrect, mostly speculative assertions made about my business in this thread, but I simply don't have time to correct all of them (I have a business to run and customers to help).

--Brett Glass

Matthew_Petach2 · July 15, 2014, 1:47am

Matt,

While I understand your point _and_ I agree that in most cases an ISP
should have an ASN. Having said that, I work with multiple operators
around the US that have exactly one somewhat economical choice for
connectivity to the rest of the Internet. In that case having a ASN is
nice, but serves little to no practical purpose. For clarity's sake all 6
of the ones I am thinking about specifically have more than 5k broadband
subs.

And as long as they're happy with their single upstream
connectivity picture, more power to them.

But the minute they're less than happy with
their connectivity option, it would sure be
nice to have their own ASN and their own
IP space, so that going to a different upstream
provider would be possible. Heck, even just
having it as a *bargaining point* would be
useful.

By not having it, they're essentially locking
the slave collar around their own neck, and
handing the leash to their upstream, along
with their wallet. As a freedom-of-choice
loving person, it boggles my mind why anyone
would subject their business to that level
of slavery. But I do acknowledge your
point, that for some category of people,
they are happy as clams with that
arrangement.

I continue to vehemently disagree with the notion that ASN = ISP since
many/most of the ASNs represent business networks that have nothing to do
with Internet access.

Oh, yes; totally agreed. It's a one-way relationship
in my mind; it's nigh-on impossible to be a competitive
ISP without an ASN; but in no way shape or form does
having an ASN make you an ISP.

Thanks!

Matt

Jared_Mauch · July 15, 2014, 2:01am

I think here is where you are wrong. There are many people out there that have cobbled together ISPs and have appliances that will load balance or do failover with multiple DSL or hybrid DSL/Cable/T1 solutions.

I do understand the line you have drawn, but some of these people compete against the largest companies in the world and win business because of their uptime and support. I wish they wouldn’t be doing “CGN” or CGN-lite type things but it happens and they don’t need an ASN to be competitive. And having an ASN would drive their costs up significantly. $500 in fees from ARIN represents a large number of subscribers profit.

- Jared

Hugo_Slabbert1 · July 15, 2014, 2:46am

Hi Brett,

Why don't you simply ask me?

I can only speak for myself, but I thought that's kind of what I and others were doing in replying to your messages, stating either support or counterpoints, and asking questions (?). With this being a list and your (as of recently) being a member of the list, my assumption (and I'm betting others') is that it's a conversation and in our replies, you may be inclined to respond or you may not.

There have been a huge number of incorrect, mostly speculative assertions made about my business in this thread, but I simply don't have time to correct all of them (I have a business to run and customers to help).

And that's fine; you're under zero obligation to anyone on the list. That said: finding radio silence, chances are the conversation will carry on and we're left to guessing/theorizing/extrapolating.

When I said "I really don't understand the line of reasoning..." I wasn't being flippant. I just know how things look from my own experience; I don't know the full details of your business and so I honestly don't know what led to your take on the topic. Your experience dealing with Netflix has obviously been more negative than mine, and I don't fully get why that is.

The prevailing trend seems to be that Netflix generally doesn't have trouble getting content to access providers' door steps, with several options for providers on how to receive that content that covers different traffic levels. In the same way as you don't owe them any special treatment, though, I don't see how they owe you (or any of us) special treatment either.

But, like I said: I don't know the details of your business or the specifics of how this plays out for you, but I am eager to hear it. More information is helpful, and if we only ever hear from people with the same view/experience, we're not very likely to get the whole picture...

Brett_Glass · July 15, 2014, 3:10am

You're assuming that the only way to be multi-homed is to have an ASN. That's not correct.

ARIN's fees are discriminatory; a small ISP must pay a much higher percentage of its revenues than a large one for IPs, ASNs, etc. Clever small ISPs find ways to work around that, and it makes them more competitive.

--Brett Glass

Mike_Lyon · July 15, 2014, 3:35am

So if Netflix was at 1850 Pearl, you wouldn't be able to peer with them
anyways cuz u have no ASN?

John_Curran1 · July 15, 2014, 3:40am

Interesting use of the word "discriminatory", as it is usually used in
the context of having different rates or fees for different categories
of people or things... Used as you have, nearly everything (including
equipment makers, conference fees, and the local coffee shop) all have
"discriminatory" fees. Myself, I'd call such fees to be uniform, but
do recognize that such uniform fees have a disproportionate impact on
smaller service providers.

/John

John Curran
President and CEO
ARIN

Brett_Glass · July 15, 2014, 3:51am

Netflix's arrangement isn't "peeering." (They call it that, misleadingly, as a way of attempting to characterize the connection as one that doesn't require money to change hands.)

ISPs peer to connect their mutual Internet customers. Netflix is not an ISP, so it cannot be said to be "peering." It's merely establishing a dedicated link to an ISP while trying to avoid paying the ISP for the resources used.

But regardless of the financial arrangements, such a connection doesn't require an ASN or BGP. In fact, it doesn't even require a registered IP address at either end! A simple Ethernet connection (or a leased line of any kind, in fact; it could just as well be a virtual circuit) and a static route would work just fine.

--Brett Glass

Mike_Lyon · July 15, 2014, 3:58am

So we are splitting hairs with what "peering" means? And I am sure Netflix
(or any other content / network / CDN provider) would be more than happy to
statically route to you? Doubtful.

Dude, put your big boy pants on, get an ASN, get some IP space, I am a
smaller ISP than you I am sure and I have both. It's not rocket science.
How are other networks suppose to take you seriously if you don't have an
ASN?

-Mike

Brett_Glass · July 15, 2014, 4:05am

Myself, I'd call such fees to be uniform,

Ah, but they are not. Smaller providers pay more per IP address than larger ones. And a much
larger share of their revenues as the base fee for being "in the club" to start with.

but do recognize that such uniform fees have a disproportionate impact on
smaller service providers.

If they were uniform, they would still have a bit of a disproportionate impact, but less so.
If they were on a sliding scale, it might be fair. Remember: Our average profit
is $5 per customer per month, and our customer base is limited by population. If I were
in this business just for the business, I'd find a more profitable business, but I CARE
about my community and accept a smaller return on my investment to help folks get connected.
(This doesn't mean that I'm a charity, but it does mean I'll never get the profits or the ROI
of a large, urban provider.)

It would be nice if what I do was also understood and valued by the Internet community at large.

--Brett Glass

Paul_S · July 15, 2014, 4:07am

Why would any content network (realistically) be interested in manually maintaining your prefixes in their routing table? BGP exists for a reason, you really should be using it.

The fact that you don't have an ASN means that automatically creating said static routes based on data from some IRRd is likely more trouble than it's likely to be worth as well.

Charles_Gucker1 · July 15, 2014, 4:08am

But regardless of the financial arrangements, such a connection doesn't
require an ASN or BGP. In fact, it doesn't even require a registered IP
address at either end! A simple Ethernet connection (or a leased line of any
kind, in fact; it could just as well be a virtual circuit) and a static
route would work just fine.

Anybody else feel a vendor t-shirt in the works?

"Who needs BGP to peer, a static route would work just fine!"

Time to get back into the Hot Tub Time Machine and back on point.
*hangs head in shame*

Brett_Glass · July 15, 2014, 4:21am

Mike:

An ASN is, literally, just a number. One that's used by a very awkward and primitive routing system that requires constant babysitting and tweaking and, after lo these many years, still doesn't deliver the security or robustness it should. Obtaining this token number (and a bunch of IP addresses which is no different, qualitatively, from what I already have) would be a large expense that would not produce any additional value for my customers but could force me to raise their fees -- something which I absolutely do not want to do.

Perhaps it's best to think of it this way: I'm outsourcing some backbone routing functions to my upstreams, which (generously) aren't charging me anything extra to do it. In my opinion, that's a good business move.

As for "peering:" the definition is pretty well established. ISPs do it; content providers at the edge do not.

Netflix is fighting a war of semantics and politics with ISPs. It is trying to cling to every least penny it receives and spend none of it on the resources it consumes or on making its delivery of content more efficient. We have been in conversations with it in which we've asked only for it to be equitable and pay us the same amount per customer as it pays other ISPs, such as Comcast (since, after all, they should be just as valuable to it). It has refused to do even that much. That's why talks have, for the moment, broken down and we are looking at other solutions.

--Brett Glass