Internet Video: The Next Wave of Massive Disruption to the US Peering Ecosystem (v1.2)

Mailman List Mirror (Read Only)
1

Then that wouldn't be enough since the other Tier 1's would need to
upgrade their peering infrastructure to handle the larger peering
links (n*10G), having to argue to their CFO that they need to do it so
that their competitors can support the massive BW customers.

Someone will take the business so that traffic is coming
regardless, they can either be that peer or be the source
with the cash. If they can't do either then they're not
in business, I hope they wouldn't ignore it congesting
their existing peers (I know...)

Then even if the peers all upgraded the peering gear at the same time,
the backbones would have to be upgraded as well to get that traffic
out of the IXes and out to the eyeball networks.

The Internet doesn't scale, turn it off

brandon

2

Why are folks turning away 10G orders?

I forgot to mention a couple other issues that folks brought up:
4) the 100G equipment won't be standardized for a few years yet, so
folks will continue to trunk which presents its own challenges over
time.
5) the last mile infrastructure may not be able to/willing to accept
the competing video traffic . There was some disagreement among the
group I discussed this point with however. A few of the cable
operations guys said there is BW and the biz guys don't want to 'give
it away' when there is a potential to charge or block (or rather
mitigate the traffic as they do now).

My favorite data point was from Geoff Huston who said that the cable
companies are clinging to their 1998 business model as if it were
relevent in the world where peer-2-peer for distribution of large
objects has already won. He believes that the sophisticated
peer-2-peer is encrypting and running over ports noone will shut off,
the secure shell ports that are required for VPNs.

So give up, be the best dumb pipes you can be I guess.

Bill

3

Why are folks turning away 10G orders?

I forgot to mention a couple other issues that folks brought up:
4) the 100G equipment won't be standardized for a few years yet, so
folks will continue to trunk which presents its own challenges over
time.

  well, there's a few important issues here:

  currently the "state-of-the-art" is to bundle/balance
n*10G. While it's possible to do 40G/n*40G in some places, this
is not entirely universal.

  Given the above constraint, in delivering 10G/n*10G to
"customers" requires some investment in your infrastructure
to be able to carry that traffic on your network. The cost difference
between sonet/sdh ports compared to 10GE is significant here and
continues to be a driving force, imho.

  Typically in the past, the "tier-1" isps have had a larger
circuit than the customer edge. eg: I have my OC3, but my provider
network is OC12/OC48. Now with everyone having 10G since it is
"cheap enough", this drives multihoming, routing table size, fib/tcam
and other memory consumption, including the corresponding CPU
"cost".

5) the last mile infrastructure may not be able to/willing to accept
the competing video traffic . There was some disagreement among the
group I discussed this point with however. A few of the cable
operations guys said there is BW and the biz guys don't want to 'give
it away' when there is a potential to charge or block (or rather
mitigate the traffic as they do now).

  I suspect this in varies depending on how it's done. Most of
the "cable" folks are dealing with short enough distances as long as
the fiber quality is high enough, they could do 10/40G to the
neighborhood. The issue becomes the coax side as well as the bandwidth
consumption of those "analog" users. Folks don't upgrade their TV or
set-top-box as quickly as they upgrade their computers. There's also a
significant cost associated with any change and dealing with those
grumpy users if they don't want a STB either.

My favorite data point was from Geoff Huston who said that the cable
companies are clinging to their 1998 business model as if it were
relevent in the world where peer-2-peer for distribution of large
objects has already won. He believes that the sophisticated
peer-2-peer is encrypting and running over ports noone will shut off,
the secure shell ports that are required for VPNs.

So give up, be the best dumb pipes you can be I guess.

  I suspect there's going to be continued seperation "at the top"
as folks see it. Those that can take on these new 10G and n*10G customers
and deliver the traffic and those who run into peering and their own
network issues in being able to deliver the bits. While 100G will ease
some of this, there's still those pesky colo/power issues to deal with.
unless you own your own facility, and even if you do, you may have
months if not years of slowly evolving upgrades to face. Perhaps
there will be some technology that will help us through this, but
at the same time, perhaps not, and we'll be getting out the huge rolls
of duct tape. It may not be politics that drives partial-transit/paid
peering deals, it may just be plain technology.

  - Jared

4

Quite simple...

We've found a fair number of networks with no 10GE equipment and no
budget to add it. Doubtless, some of these don't have OC192 capacity
either. Others have 10G in the offing but are still putting it through
"acceptance" testing and won't sell it for several more months.

Others will happily sell you a 10GE circuit but then limit you to some
fraction of that circuit because of internal limitations within their
nodes. (I've seen this on more than one network.)

And in any case, some of these don't have the egress capacity either
from the local node to their backbone or to their peers/customers to
be able to swallow that kind of traffic anyway.

Truth be told, there really are not that many networks out there at
present that are capable of accepting a 10G customer who actually
intends to USE 10G. And believe it or not, there are still those out
there that believe that customers aren't going to be able to pass a
full 10G and therefore see no need to offer it at the edge.

Currently, for all but the most intensive users, NxGE or NxOC48 still
seems to be preferred termination. (Often this is also partly a factor
of minimum commits and varying methods of billing.) This is changing
but it's happening more slowly than I would like to see.

My $0.37 (inflation's a pain)

-Wayne

5

In Hollywood, San Francisco and a few other cities with large concentration
of movie/entertainment industries 10G network connections have been sold for at least a year, not necessarily connected to the "Internet."

Don't we go through this every other network generation? I seem to recall the NSFNET only being able to handle 22Mbps on its T3 (45Mbps) connections
for a while. A few years later, another backbone was out of capacity and had to stop selling new connections for several months while it caught up.
Later we had the OC12, OC48 bottleneck on switches being used at some exchange points and getting GigE connections was a problem. And so on and so on.

We'll have wailing and gnashing of teeth for many months but somehow things will balance out again as technology, equipment and revenue catch
up with each other. And then it will start over again with someone wanting 100GE connections, and then 1000GE connections, etc, etc.

6

Some of this depends on how much you are willing to pay. The issue is as much 10G orders at today’s transit prices as it is the capacity. We’re used to paying less per unit for greater capacity, but when we’re asking networks to sell capacity in chunks as large as the ones they use to build their backbones, that may simply not work.

One other issue is that willingness to sell 10G is one vital competitive distinguisher in an otherwise largely commodity transit market. There have been rumors that older legacy carriers wish to punish more agile competitors for daring to “steal” 10G customers away from them, in spite of the fact that those older carriers have lots of trouble delivering 10G.

  • Dan
7

One other issue is that willingness to sell 10G is one vital competitive distinguisher in an otherwise largely commodity transit market. There have been rumors that older legacy carriers wish to punish more agile competitors for daring to "steal" 10G customers away from them, in spite of the fact that those older carriers have lots of trouble delivering 10G.

There isn't anything new to that either. I can think of (similar to Sean's story) a time when a backbone had all NxDS3 links in their network and used to be very upset at the idea of selling OC3s instead of NxDS3 ("If its good enough for *our* backbone...").

Then this network went ATM on Fore systems boxes with handoffs to Cisco routers to leapfrog their competitors... which um, didn't work...

Then they reluctantly went POS and "better" systems since...

They are still around today after about 6 name changes. I have no present-day-knowledge of their network or, really, their performance, so I don't want to mention any names.

Deepak