"potential new and different architectural approach" to solve the Comcast - L3 dispute

Since it is Friday, maybe some of peering experts have some time to
speculate what this new approach proposed by Comcast might be, as they
assert it would represent "a significant shift of Internet infrastructure."

http://www.lightreading.com/document.asp?doc_id=202121
http://blog.comcast.com/2010/12/comcast-continues-discussions-with-level-3----offers-to-trial-new-solutions.html

Well, their previous proposal was already representing quite a
significant shift, and not in a good way, but I wonder what the new
offer could be so that they figured it would be more acceptable to Level 3.

I hope due to the speculative nature of the question it will not be
considered off-topic.

-Lorand Jakab

I have no direct knowledge of the situation, but my guess: I suspect the proposal was along the lines of longest-path / best-exit routing by Level(3). In other words, if L(3) carries the traffic (most of the way) to the customer, then Comcast has no complaint--the costs can be more fairly distributed. The "modest investment" is probably in tools to evaluate traffic and routing metrics, to make this work. This isn't really *new* to the peering community, but it isn't normal either.

If anybody knows for sure, I'd be interested to hear.

Cheers,
-Benson

How effective have variations on hot potato routing been, historically?
I seem to recall Cogent made lots of noises early on about how they
could do hot potato routing to encourage peering, but over the years
that didn't seem to pan out that way.

... JG

I can't comment on Cogent... But, in general: hot-potato reduces network costs but doesn't eliminate them--more capacity is still required to carry more traffic. The goal is to balance out the cost, assuming the traffic is of adequate value (or equal value, ideally) to both networks.

Cheers,
-Benson

That is a reasonable guess, but Level3's FCC filing yesterday spells
out with certainty that Level3 did offer to "cold potato" traffic onto
Comcast (it does not mention the technical means e.g. MED honoring,
CDN smarts, or otherwise) and that Comcast refused.

I agree that the proposed Comcast solution may not be truly "new" but
instead unusual, but unless "Backdoor Santa" tells us what they really
have in mind, I suppose we won't know. If I were Comcast, I would
want to move the significant cost of detailed netflow collection and
analysis infrastructure onto backbone providers by wrapping that
accounting mechanism up into my settlement agreements with peers, as
well as the expense of a cost-ineffective network, and demand that
Level3 and Comcast really calculate how much each network spends on
each bit, and share in that cost. In theory, this is what happens
when an ILEC opens a rate case with its state regulator; and it is how
settlements for POTS calls work (at a very basic level.)

Actually, if I were Comcast, I would focus on running my business more
efficiently, as Level3 has thrown down the gauntlet with the FCC and
requested that the FCC dictate to Comcast specifically, and explicitly
all other broadband access providers, how they will interconnect with
peers and transit suppliers. Level3 must think that their business
would be better off with regulatory oversight of peering, or they
would not have taken this action. Comcast should realize that, of the
three potential motives for their recent actions I have previously
outlined, #1 and #3 are not just highly unlikely, but would be
practically impossible in a regulated environment. As such, they
should further realize that their peering committee is driven by
motive #2, ego, and find the best way to change their position without
losing too much credibility.

Nah, you're still thinking about this like it was a classic peering
dispute over ratios, when nothing could be further from the truth. First
off, by the very nature of a CDN, all of the Netflix/etc traffic is
going to be delivered to the best exit on the long-haul network already.
Second, Comcast is a FULL TRANSIT CUSTOMER of Level 3. Typically the
customer gets to dictate the handoff point to the provider, by either
advertising MEDs, or by sending inconsistent routes. The fact that the
existing Level3/Comcast routing DOESN'T make Level 3 haul all of the
bits to the best exit mean it's highly likely that Comcast agreeing to
haul the bits was part of their commercial transit agreement, probably
in exchange for lower transit prices.

Level3 must think that their business
would be better off with regulatory oversight of peering, or they
would not have taken this action. Comcast should realize that, of the
three potential motives for their recent actions I have previously
outlined, #1 and #3 are not just highly unlikely, but would be
practically impossible in a regulated environment. As such, they
should further realize that their peering committee is driven by
motive #2, ego, and find the best way to change their position without
losing too much credibility.

--
Jeff S Wheeler <jsw@inconcepts.biz>
Sr Network Operator / Innovative Network Concepts

Or maybe Level(3) thinks the entire game could potentially change and are attempting to head that off at the pass.

What if instead of the end users paying for Internet service, the content providers did. Sort of like broadcast TV where the broadcasters pay the freight and the user simply turns on their device and they get content. In that model, the providers of the traffic pay the delivery costs of the content. So you would have "consumer" access that is mainly paid for by the content providers and "business" access which would be paid by the end users but would have less "consumer" traffic such as Netflix, Hulu, Facebook, Twitter, etc.

If you look at the revenues being reported by some of these content providers, someone might be looking at those numbers saying "why *shouldn't* they pay? They are making money from the end users via ad sales just like broadcasters do, why shouldn't the model be the same?".

I am not making any statement of my opinion, simply looking at a possibility. If there were such a sea change, Level3 now being a major content provider might find its long range plans have had a wrench thrown in them.

And they might be correct in thinking that, if we assume the peering ecosystem is changing i.e. such that traditional "backbones" are being bypassed. Regulatory oversight might have the effect of locking-in today's interconnect regime, which would be ideal for Level(3).

Cheers,
-Benson

I've been following these threads with some interest, and even replying in a couple of places, but now it hits me that a sea change has already occurred, and it's the whole content provider / end user *thing* versus the original 'a host is a host is a host' IP *thing*.

But content providers already pay more for their 'service' than the typical asymmetric-towards-the-customer bandwidth user does.

Um.

I'm a content provider.

I pay a -lot- for internet service already. That's how my bits and bytes arrive in the tubes for those end users to recieve...

It's worth asking why Comcast did not accept Level3's suggestion that
they use MED as a face-saving maneuver, which would have allowed both
sides to declare victory.

A) Comcast may already have the contractual right to use MED but
chooses not to. I agree with you that this is unlikely, not for pure
reasons of economics, but because Comcast has some of the same set of
motives not to send MED to their transit provider as every other
network: prefix aggregation, quality control, and ego. I'll discount
geography, marketing, and inability to calculate useful MED values.
For argument's sake, let's say they currently can start sending MEDs
to Level3 whenever they want. This being the case, Level3's "offer"
would have amounted to Level3 telling Comcast upper management that
Comcast's engineering people are leaving a huge amount of money on the
table, that Level3 is far more cost-effective at running its long-haul
network than Comcast, and that they should leave the big networking to
the big boys. Comcast management could either react badly to this, or
go back to their network folks and ask why they can't be as
cost-effective as Level3.

B) Comcast may not be able to use MED today. In this case, management
may be asking themselves why. An essentially similar scenario can
play out; they can either react badly to Level3, or ask their own
staff why they are wasting money.

C) Comcast doesn't care about MED or the actual cost of doing
business. They are boldly moving towards a future that is opposite
the one "net neutrality" folks advocate, one that looks like my
"Comcast Motive #3."

D) Comcast does not think that beginning to use MED (whether currently
enabled or not) is enough to satisfy the federal regulators and
legislators who are now taking interest in this game of
interconnection brinkmanship, involving 17 million households, between
a major IP carrier delivering traffic from everyone including a
household name like Netflix, and a major cable company that is waiting
for government approval to purchase NBC. They feel they must demand
something very concrete to demonstrate that they are looking out for
consumers' best interest, which means they must make Level3 and/or
Netflix look like the bad guy.

E) Comcast thinks that a system of accounting for the cost of bearing
traffic and dividing it among the involved parties will actually be
good for their business, because they can over-build their
infrastructure as much as they like, perhaps even improving quality
for end-users, and only have to pay for about half of it. The cost of
being inefficient, stupid, or committing purchasing or forecasting
errors drops by half. This looks very much like my "Comcast Motive
#1."
E1) Comcast may also know a thing or two about Hollywood Accounting.
If you do not understand this reference, simply look it up on
Wikipedia. It suffices to say that cost/revenue sharing agreements of
this nature can be manipulated in gross ways to the advantage of the
party doing the bulk of the book-keeping.

F) Management has the same case of ego-driven decision-making that
their technical staff have demonstrated. I find this unlikely but
still possible. We all know this has been the case at the CEO level
in some major interconnection disputes of the past.

I believe this outlines the reasonable scenarios for Comcast avoiding
a face-saving maneuver with Level3.

What if instead of the end users paying for Internet service, the content providers did. Sort of like broadcast TV where the broadcasters

Um.

I'm a content provider.

I pay a -lot- for internet service already. That's how my bits and bytes arrive in the tubes for those end users to recieve...

+1 from here.

Regards
Marshall Eubanks
AmericaFree.TV

Comcast's latest:
http://fjallfoss.fcc.gov/ecfs/comment/view?id=6016064677

Agreed, though I think they pay less than most eyeball networks pay (the ISP, not the user), depending on where they host it (we have a lot of hauling we have to do).

I'd also note, that the Internet is continuing to push more towards blurring the lines of content provider/eyeball, as p2p continues to be deployed with more technologies and for more uses. As households are constantly on, there is benefit in the household hosting content which can be reached directly by those you are sharing it to. As the market shifts to containing a larger market share of households with symmetric bandwidth, we can expect to see this improve (asymmetric last miles has hindered many innovations).

Jack

http://fcc.gov/

NOTICE: The FCC website and related electronic filing systems and documents
(except for NORS) will be unavailable beginning 6:00 p.m. (EST) Friday,
December 17 through 6:00 a.m. (EST) Monday, December 20 for scheduled
maintenance.

:frowning:

http://blog.comcast.com/2010/12/comcasts-responds-to-level-3s-fcc-filing.html

I very much doubt whether my comment on the blog will survive their
moderation process, so here it is:

I believe they want the cat people to pay too, it's just easier to go
after Netflix first.

Lets say for a moment that Comcast's overall ratio with its customers is
approximately the same as their ratio in the leaked Tata graphs (yes I
know that this proves nothing, but lets just assume it for a moment),
i.e. 5:1. They then ask that every network who sends them traffic, even
their transit providers (in the case of Level 3) be under 2:1. What is
the point of insisting on a ratio that is not supported by the traffic
their customers actually request? Because it gives them a convenient
excuse to demand payment from nearly everyone on the Internet for being
out of ratio, and to restrict capacity to those who do not pay.

With so many transit ports running hot, and even peering ports running
hot as in the recent example where they intentionally turned down Global
Crossing capacity (which they claim is settlement free) and CAUSED
congestion, the ISP who hosts the cute cat pictures may have little
choice but to pay Comcast for access, or risk losing their cute cat
hosting business to someone else who is willing to do so.

I've also seen Comcast ignore several offers to honor MEDs or accept
more-specifics from networks who DO meet their published peering
requirements in every way except ratios, so I don't think they're
interested in technical solutions a potential transport cost imbalance
either. If it was about anything other than trying to extract a toll
from content providers, one of these technical solutions would clearly
have been better for them then continuing to force the traffic into
their congested transit ports, which they not only pay for, but then
also do the backhaul for across their own network.

BTW, they rejected my very nice comment on their blog asking if they
would be willing to share the graphs of their transit provider
interfaces (which are NOT peering relationships, and not under NDA) to
back up their claims that the published graphs are false, so I'm
positive yours isn't going to get through. :slight_smile:

If I was Comcast and I got this deal I'd set up scripts to continuously
spoof requests to Netflix, I mean hey I get paid for the traffic..

j

+1

In fact, I feel that at home, I need fast, reliable internet access. I wish I could
get that from one provider. Unfortunately, instead, I get fast internet service
from Comcast (most of the time) and I get reliable internet service from
Raw Bandwidth (DSL, 1.5mbps/768k).

Owen
(Comcast Business HSI customer)