Some truth about Comcast - WikiLeaks style

Mailman List Mirror (Read Only)
1

Ever wonder what Comcast's connections to the Internet look like? In the tradition of WikiLeaks, someone stumbled upon these graphs of their TATA links. For reference, TATA is the only other IP transit provider to Comcast after Level (3). Comcast is a customer of TATA and pays them to provide them with access to the Internet.

1 day graphs:

Image #1: http://img149.imageshack.us/img149/78/ntoday.gif
Image #1 (Alternate Site): http://www.glowfoto.com/viewimage.php?img=13-224638L&rand=6673&t=gif&m=12&y=2010&srv=img4

Image #2: http://img707.imageshack.us/img707/749/sqnday.gif
Image #2 (Alternate Site): http://www.glowfoto.com/static_image/13-205526L/4331/gif/12/2010/img6/glowfoto

Notice how those graphs flat-line at the top? That's because they're completely full for most of the day. If you were a Comcast customer attempting to stream Netflix via this connection, the movie would be completely unwatchable. This is how Comcast operates: They intentionally run their IP transit links so full that Content Providers have no other choice but to pay them (Comcast) for access. If you don't pay Comcast, your bits wont make it to their destination. Though they wont openly say that to anyone, the content providers who attempt to push bits towards their customers know it. Comcast customers however have no idea that they're being held hostage in order to extort money from content.

Another thing to notice is the ratio of inbound versus outbound. Since Comcast is primarily a broadband access network provider, they're going to have millions of eyeballs (users) downloading content. Comcast claims that a good network maintains a 1:1 with them, but that's simply not possible unless you had Comcast and another broadband access network talking to each other. In the attached graphs you can see the ratio is more along the lines of 5:1, which Comcast was complaining about with Level (3). The reality is that the ratio argument is bogus. Broadband access networks are naturally pull-heavy and it's being used as an excuse to call foul of Level (3) and other content heavy networks. But this shoulnd't surprise anyone, the ratio argument has been used for over a decade by many of the large telephone companies as an excuse to deny peering requests. Guess where most of Comcasts senior network executive people came from? Sprint and AT&T. Welcome to the new monopoly of the 21st century.

If you think the above graph is just a bad day or maybe a one off? Let us look at a 30 day graph...

Image #3: http://img823.imageshack.us/img823/8917/ntomonth.gif
Image #3 (Alternate Site): http://www.glowfoto.com/static_image/13-205958L/4767/gif/12/2010/img6/glowfoto

Comcast needs to be truthful with its customers, regulators and the public in general. The Level (3) incident only highlights the fact that Comcast is pinching content and backbone providers to force them to pay for uncongested access to Comcast customers. Otherwise, there's no way to send traffic to Comcast customers via the other paths on the Internet without hitting congested links.

Remember that this is not TATA's fault, Comcast is a CUSTOMER of TATA. TATA cannot force Comcast to upgrade its links, Comcast elects to simply not purchase enough capacity and lets them run full. When Comcast demanded that Level (3) pay them, the only choice Level (3) had was to give in or have its traffic (such as Netflix) routed via the congested TATA links. If Level (3) didn't agree to pay, that means Netflix and large portions of the Internet to browse would be simply unusable for the majority of the day for Comcast subscribers.

Love,

Backdoor Santa

2

Forgive me for being the skeptic, but I presume there is at least a traceroute with rDNS mentioning one of the 3 10G interfaces on gin-nto-icore1 from comcast?

It's not like the image lists the customer name on it; disregarding photoshop concerns. At least wikileaks documents look like they came from the government and have lots of details. :slight_smile:

Jack

3

Agreed. There's no independently verifiable detail to lend any credence to the source(s) of the data. It just shows some 10G links flat-topping due to saturation. There's not enough here to get particularly excited.

jms

4

Actually, there are plenty of access providers with 2:1 ratio (more ul than dl). It's not a matter if you're access provider or not, it's a matter if you offer decent upstream speed or not.

In my experience, someone with 10/10 megabit/s ETTH compared to someone with 24/1 ADSL will download the same amount of data on average, but the 10/10 will have four (4) times more upload usage, bringing the ratio from 2:1 (Dl:Ul) on ADSL to 1:2 (Dl:Ul) on ETTH.

So because Comcast is offering low upload speeds, they'll have low outgoing amount of traffic compared to incoming. With more and more ISPs offering more symmetric dl/ul speeds, we'll approach 1:1 ratio more and more...

5

I don't see anything listed that indicates operation that is at all different from any other service provider network.

The "capacity" issue listed is not an issue at all. It's simply inciting anger and the same rhetoric that pollutes the legitimate discussion of backbone network constraints.

When you shout "conspiracy" without offering verifiable facts, and not accounting for the cost (and time) it takes to upgrade networks (much less the fact that it requires capacity upgrades on both sides, in this case between TATA and Comcast), it makes the whole argument invalid in my opinion.

That and the "backdoor santa" thing makes me believe the whole thread is designed to flame rather than promote the discourse that is the hallmark of NANOG. I really hope that there are moderators about to verify this: With these kinds of people about I'm less likely to post anything of substance.

Sincerely,

Brian

6

gin-nto-icore1 is a Tata router at Equinix in NY. Whether or not that
port belongs to Comcast is anyone's guess.

Jeff

7

From Tata's looking glass:

  3 Vlan550.icore1.NTO-NewYork.as6453.net (209.58.26.78) 4 msec
    Vlan551.icore1.NTO-NewYork.as6453.net (209.58.26.82) 4 msec 0 msec
  4 pos-1-9-0-0-cr01.newyork.ny.ibone.comcast.net (68.86.86.41) [AS 7922] 4 msec 4 msec 4 msec

As far as I can tell their DNS doesn't expose Tata's router port names
at all:

77.26.58.209.in-addr.arpa domain name pointer Vlan550.icore1.NTO-NewYork.as6453.net.
78.26.58.209.in-addr.arpa domain name pointer Vlan550.icore1.NTO-NewYork.as6453.net.
81.26.58.209.in-addr.arpa domain name pointer Vlan551.icore1.NTO-NewYork.as6453.net.
82.26.58.209.in-addr.arpa domain name pointer Vlan551.icore1.NTO-NewYork.as6453.net.
41.86.86.68.in-addr.arpa domain name pointer pos-1-9-0-0-cr01.newyork.ny.ibone.comcast.net.
42.86.86.68.in-addr.arpa domain name pointer pos-1-0-0-0-pe01.111eighthave.ny.ibone.comcast.net.

Though I suppose if someone was photoshopping it, it would be pretty
obvious for them to stick something that does show up in DNS into the
graphs, so that doesn't exactly prove much. I'm also assuming Comcast
wouldn't be very happy to have these out in public, so there is pretty
much no way you're going to see a leaked graph that ISN'T from an
anonymous source.

FWIW these graphs pretty much reflect the massive congestion that I've
been observing between Tata and Comcast. I've also seen some third party
Smokeping graphs which visually show the rate of loss, and the pattern
looks very very similar, but I'll let someone who actually maintains
them be the one to post them.

8

* Richard A Steenbergen

FWIW these graphs pretty much reflect the massive congestion that I've
been observing between Tata and Comcast. I've also seen some third party
Smokeping graphs which visually show the rate of loss, and the pattern
looks very very similar, but I'll let someone who actually maintains
them be the one to post them.

Voxel have also reported seeing congestion to Comcast via Tata:

http://www.voxel.net/blog/2010/12/peering-disputes-comcast-level-3-and-you

Best regards,

9

On the 30-day graph, there's a flat spot in the data that corresponds with the Comcast outage on the 28th, but that's not a sure thing.

10

Isn't saturating their TATA links part of their strategy to make
people pay to peer with them ?

Rubens

11

Thanks for this, I think, as a residential customer of Comcast, the FCC
and FTC will both be receiving a letter from me. Clearly Comcast is not
making an effort to deliver their advertised service, and instead are
actually degrading my service.

Cordially

Patrick

12

I don't see anything listed that indicates operation that is at all
different from any other service provider network.

Yeah, the 30 day looks like a classic uptick in traffic toward the holidays.
Some bellhead beancounter maybe
took out capacity in the summer lull and ignored the engineers. Or they just
have stupidly-slow install intervals.
Same crap I've seen on loads of provider networks.

The "capacity" issue listed is not an issue at all. It's simply inciting
anger and the same rhetoric that pollutes the legitimate discussion of
backbone network constraints.

When you shout "conspiracy" without offering verifiable facts, and not
accounting for the cost (and time) it takes to upgrade networks (much less
the fact that it requires capacity upgrades on both sides, in this case
between TATA and Comcast), it makes the whole argument invalid in my
opinion.

If they wanted to be tru to the claim of "wikileaks style" in the subject
line, they'd have an actual memo from
some executive stating the policy of purposefully starving traffic. Never
attribute to malice* *that which is
adequately explained by stupidity.

13

Backdoor Santa wrote:

Ever wonder what Comcast's connections to the Internet look like? In the tradition of WikiLeaks, someone stumbled upon these graphs of their TATA links. For reference, TATA is the only other IP transit provider to Comcast after Level (3). Comcast is a customer of TATA and pays them to provide them with access to the Internet.

1 day graphs:

Image #1: http://img149.imageshack.us/img149/78/ntoday.gif

Another thing to notice is the ratio of inbound versus outbound. Since Comcast is primarily a broadband access network provider, they're going to have millions of eyeballs (users) downloading content. Comcast claims that a good network maintains a 1:1 with them, but that's simply not possible unless you had Comcast and another broadband access network talking to each other. In the attached graphs you can see the ratio is more along the lines of 5:1, which Comcast was complaining about with Level (3). The reality is that the ratio argument is bogus. Broadband access networks are naturally pull-heavy and it's being used as an excuse to call foul of Level (3) and other content heavy networks. But this shoulnd't surprise anyone, the ratio argument has been used for over a decade by many of the large telephone companies as an excuse to deny peering requests. Guess where most of Comcasts senior network executive people came from? Sprint and AT&T. Welcome to the new monopoly of th

e 21st century.

If you think the above graph is just a bad day or maybe a one off? Let us look at a 30 day graph...

Image #3: http://img823.imageshack.us/img823/8917/ntomonth.gif
              
This tells me two things:

1 - Don't use comcast as your ISP. Personally I prefer to use a local ISP, maybe even "ma and pa store" style, if available.

2 - If for some reason you just can't live without comcast, then plan ahead do all your leeching between 7 AM and 5 PM. Then watch your previously saved videos at your leisure without interruptions in the evening. And plan on finding a better ISP :slight_smile:

14

No ISP I've ever worked for or with has ever willingly ran their transit (or peering) links at capacity.

(Granted, I've been responsible for saturating links, but I moved user traffic off of them first.)

--Ricky

PS: TATA confirmed Comcast's behavior before anyone found any traffic graphs. We already knew they were gaming their own customer base.

15

Except that they seem to be busy actively turning down other capacity,
and forcing extra traffic through their Tata ports by blocking other
paths with BGP no-export communities.

For example, we've been observing Comcast turning down some of their
Global Crossing capacity in recent days, causing new congestion during
peak traffic times. I've even seen people contact the various NOCs
involved, and they've been told explicitly and by multiple parties that
Comcast is intentionally turning down extra capacity and running their
existing ports hot.

Everybody who deals with interconnection capacity in this industry knows
what's going on, but the graphs and interconnection details are all
under NDA, so it takes an inside source secretly leaking graphs to the
public to expose this kind of activity. Even then you'll still have
people who claim that it proves nothing because the graphs can't be
positively associated to a specific customer port, but realistically
these kinds of leaks are probably the best public info you'll ever see.

16

To what end? And who's calling the shots there these days? Comcast has
been nothing but shady for the last couple years. Spoofing resets, The L3
issue, etc. What's the speculation on the end game?

17

I believe Comcast has made clear their position that they feel content
providers should be paying them for access to their customers. I've seen
them repeatedly state that they feel networks who send them too much
traffic are "abusing their network". It isn't a ratios argument in the
classic sense, between two peers trying to maintain a fair balance of
costs and benefits, it's that they object to ANY content provider being
able to deliver to their customers without paying them for access. They
do this by trying to enforce ratios which are well beyond what their
actual end users are routing, and as in the case of Level 3, they
leverage that position to claim that other networks should be paying
them under threat of blocking uncongested access to their customers.

I would say their short term goal is to make people who currently won't
peer with them do so, so they can become transit free. This has been
seen time and time again, as they move networks who they want to peer
with but who will not peer with them into "congested transit" bucket. A
while back it was SAVVIS, now it is Tata, but the pattern is clear and
repetitive. Note that this only extends to a certain point though, as in
the case of Global Crossing, who they claim is a settlement free peer,
but who they have recently started pressuring and intentionally
congesting because of ratio imbalances.

Their long term goal seems to be to force content networks to pay them
for direct transit or on-net connectivity, by removing the available
capacity from other paths. If you are a content network, and you can't
reach them in a reliable fashion via "The Internet", your only choice
may be to buy from Comcast directly.

This is obviously not the first time that networks have used this
strategy, there are several prominent examples in recent history of
others using this exact same technique. But this is definitely one of
the worst examples in the US of a major eyeball network using access to
their customers (who may have little or no choice in their broadband
access) to force other networks to pay them, and IMHO it needs to be
called out publicly whenever possible.

18

Can you share any references on this? Everything I've seen has been typical lawyer double speak, i.e. the opposite of clear.

19

The Internet would offer lesser value by allowing access providers to hold their customers hostage. Clearly, such providers are not acting in their customer's interests when inhibiting access to desired and legitimate content. What is net neutrality expected to mean?

Providers should charge a fair price for bandwidth offered, not over sell the bandwidth, and not constrain bandwidth below advertised rates. Congestion pricing rewards bad practices that leads to the congestion.

-Doug

20

I just see this as a natural progression of what happens of a single
player with a captive audience due to mergers and attrition. They know
their customers aren't going anywhere. The only way to "fix" it would be
to go back to the days when there were a bunch of competing local providers.

~Seth