Cogent/Level 3 depeering

Mailman List Mirror (Read Only)
1

A couple weeks later than expected, but as of Oct 5 02:51AM EDT it looks
like 3356 and 174 are no longer reachable.

lg.level3.net:

Show Level 3 (Washington, DC) BGP routes for 38.9.51.20

No matching routes found for 38.9.51.20.

www.cogentco.com looking glass:

Tracing the route to www.Level3.com (209.245.19.42)

  1 f29.ba01.b005944-0.dca01.atlas.cogentco.com (66.250.56.189) 4 msec 4 msec 0 msec
  2 * * *
  3 * * *

I guess the earlier reports of (3)'s lack of testicular fortitude may have
been exagerated after all. :slight_smile:

2

A couple weeks later than expected, but as of Oct 5 02:51AM EDT it looks
like 3356 and 174 are no longer reachable.

lg.level3.net:

Show Level 3 (Washington, DC) BGP routes for 38.9.51.20

No matching routes found for 38.9.51.20.

www.cogentco.com looking glass:

Tracing the route to www.Level3.com (209.245.19.42)

1 f29.ba01.b005944-0.dca01.atlas.cogentco.com (66.250.56.189) 4 msec 4 msec 0 msec
2 * * *
3 * * *

I guess the earlier reports of (3)'s lack of testicular fortitude may have
been exagerated after all. :slight_smile:

It's sure causing a few headaches here.
(from level3 looking glass) Show Level 3 (London, England) BGP routes for 38.9.51.20

No matching routes found for 38.9.51.20

As of 16:22 BST Level3 still seems to have no routes for cogent's space. thats about 5 hours now.

Vince

3

I opened a billing/support ticket with Cogent. I'm not planning on paying my bill or continuing the contract if they cannot provide full BGP tables and full Internet transport (barring outages). Luckily I have 2 other providers so I can still reach Level 3.

Maybe I can buy the new 'Cogent - it is almost the Internet' service for less money.

-Matt

4

I opened a billing/support ticket with Cogent. I'm not planning on
paying my bill or continuing the contract if they cannot provide full
BGP tables and full Internet transport (barring outages). Luckily I
have 2 other providers so I can still reach Level 3.

We tried the same line with Level3 - and were told "Tough, we're not paying
service credit. The transit still works, just its coverage is slightly
different".

Maybe I can buy the new 'Cogent - it is almost the Internet' service
for less money.

Indeed, that's the natural next step.

Simon

5

Perhaps they took the 3 in level3 to mean 3 routes ? :wink:

cogent-gate# show ip bgp regex 174 .* 3356$

    Network Next Hop Metric LocPrf Weight Path
* 63.135.164.0/24 38.112.9.13 15001 100 110 174 701 18905 14745 1239 3356 i
* 194.32.125.0 38.112.9.13 18000 100 110 174 1273 9121 3549 3356 i
* 194.32.127.0 38.112.9.13 18000 100 110 174 1273 9121 3549 3356 i

Rather than swamping their desk with the same ticket, can you report back to NANOG what they tell you ?

         ---Mike

6

I opened a billing/support ticket with Cogent. I'm not planning on paying my bill or continuing the contract if they cannot provide full BGP tables and full Internet transport (barring outages). Luckily I have 2 other providers so I can still reach Level 3.

I'm curious where in your contract you think Cogent guaranteed you connectivity to Level 3?

Most transit contracts only guarantee packet delivery to the edge of their own networks. I'm pretty sure Cogent is doing that. (Hell, they have lots of spare capacity now. :slight_smile:

Of course, I would claim that the word "transit" has certain implications, but IANAL. (I'm not even an ISP. :slight_smile: So perhaps someone could enlighten me on how one would go about asking for credits for this ... disconnectivity.

Also, I've already heard from customers of L3 single-homed providers that L3 will _NOT_ be issuing credits. So I guess the question goes for L3 contracts as well.

Maybe I can buy the new 'Cogent - it is almost the Internet' service for less money.

Maybe. Would you pay L3 for "almost the Internet" as well?

There is nothing wrong with "partial transit". If we could get partial transit at 50% off full transit pricing, we would absolutely consider it - depending on things like which "part" of the Internet is served.

And why aren't people asking for partial transit pricing from providers who do things like filter smaller prefixes (because they are too cheap or too dumb to run a real backbone) or entire countries (say, for spam) or other things?

7

I opened a billing/support ticket with Cogent. I'm not planning on
paying my bill or continuing the contract if they cannot provide full
BGP tables and full Internet transport (barring outages). Luckily I
have 2 other providers so I can still reach Level 3.

We tried the same line with Level3 - and were told "Tough, we're not paying
service credit. The transit still works, just its coverage is slightly
different".

Cogent seem to be suggesting an alternate solution, which involves them giving away free transit to l(3) customers. This might be useful for some here who are feeling pain.

[I realise that there are probably at least nine sides to this story and that the text here represents just one of them.]

8

Ah, the problem with playing chicken is what happens when neither player blinks...

-C

9

In the end, both providers lose, as customers buy real Internet transit from someone else.

OTOH, the industry as a whole probably gains. I have a client who's massively overprovisioned, multihomed with multiple Ts each to 3 or 4 providers now after being bitten a couple years ago when singlehomed to C&W and they depeered PSI. Funny that those PSI customers are getting screwed again now.

10

I am curious - how did prior depeering "events" wind up being eventually resolved? What were the resolution times, if anyone remembers?

-C

11

This is what I just got from Cogent support. I'm still waiting on the billing dispute ticket. I've already told our payables department to not pay any Cogent invoices, this should get fun. Hell, I wish Verio never sold me to Cogent in the first place, it is all their fault :confused:

<quote>
Hello,

As of 5:30 am EDT, October 5th, Level(3) terminated peering with Cogent without cause (as permitted under its peering agreement with Cogent) even though both Cogent and Level(3) remained in full compliance with the previously existing interconnection agreement. Cogent has left the peering circuits open in the hope that Level(3) will change its mind and allow traffic to be exchanged between our networks. We are extending a special offering to single homed Level 3 customers.

Cogent will offer any Level 3 customer, who is single homed to the Level 3 network on the date of this notice, one year of full Internet transit free of charge at the same bandwidth currently being supplied by Level 3. Cogent will provide this connectivity in over 1,000 locations throughout North America and Europe.

For status updates and further information on the special offering -- please see our status page at http://status.cogentco.com
</quote>

-Matt

12

I opened a billing/support ticket with Cogent. I'm not planning on paying my bill or continuing the contract if they cannot provide full BGP tables and full Internet transport (barring outages). Luckily I have 2 other providers so I can still reach Level 3.

I'm curious where in your contract you think Cogent guaranteed you connectivity to Level 3?

My original contract was with NTT/Verio which Cogent purchased last year when Verio nuked their Boston POP. I'm having the contract dug out of the archives to look at what it says. IMHO I pay Cogent for Transit to the whole Internet, If I wanted partial transit or local peering I would order/contract and pay for that. Cogent is not currently providing me full transit service. I really don't care who pulled the plug, it is Cogents job to fix it for me as I am their customer.

Most transit contracts only guarantee packet delivery to the edge of their own networks. I'm pretty sure Cogent is doing that. (Hell, they have lots of spare capacity now. :slight_smile:

Most also have a clause to cover the inter-AS links, making sure that they are not overloaded.

Maybe I can buy the new 'Cogent - it is almost the Internet' service for less money.

Maybe. Would you pay L3 for "almost the Internet" as well?

Yes, if the price were right.

13

> I'm curious where in your contract you think Cogent guaranteed you
> connectivity to Level 3?

My original contract was with NTT/Verio which Cogent purchased last year when
Verio nuked their Boston POP. I'm having the contract dug out of the
archives to look at what it says. IMHO I pay Cogent for Transit to the whole
Internet, If I wanted partial transit or local peering I would order/contract
and pay for that. Cogent is not currently providing me full transit service.
I really don't care who pulled the plug, it is Cogents job to fix it for me as
I am their customer.

"Isn't BGP supposed to work around this sort of thing?"

This comes down to a little more than just "depeering" -- at least in the
BGP sense. There's active route filtering going on as well if connectivity
is dead; after all, I can bet the house that at least one of Cogent's
network edge peers has connectivity to Level3, and vice versa.

From where I sit, I can see a plethora of routes that transit more than one

tier1. And a few that transit three before hitting the origin. From a
couple locations I see 3356 and 174 visible in *all* paths to the prefixes
containing Level3 and Cogent in the path, respectively.

So perhaps the question you should be asking is: Why didn't routes for
these networks fall over to the other upstream peers which *are* capable of
moving the packets? Surely MCI, AT&T, Sprint, and others would carry the
packets to the right place. I can see the paths right here....

> Most transit contracts only guarantee packet delivery to the edge of their
> own networks. I'm pretty sure Cogent is doing that. (Hell, they have lots
> of spare capacity now. :slight_smile:

Most also have a clause to cover the inter-AS links, making sure that they are
not overloaded.

What nature of clause? I consider deliberately filtering prefixes or origin
ASs to be a violation of common backbone BGP use.

Too bad there aren't Equal Access laws for tier1s. <slyly evil grin>

14

Matthew Crocker wrote:

I opened a billing/support ticket with Cogent. I'm not planning on paying my bill or continuing the contract if they cannot provide full BGP tables and full Internet transport (barring outages). Luckily I have 2 other providers so I can still reach Level 3.

I'm curious where in your contract you think Cogent guaranteed you connectivity to Level 3?

Undereducated rant to follow...

While I realize that the "nuke survivable" thing is probably an old wives tale, it seems ridiculous that "the Internet" can't adjust by routing any packets that used to go directly from Cogent to Level 3 though some 3rd (and) 4th (and) 5th set of providers that are connected in some fashion to both...

Level 3 and Cogent can't be operating in a vacuums - if we can get to Kevin Bacon in 6 degrees, Level 3 and Cogent should be able to get to each other in under 30 hops through other providers.

And why isn't this apparently happening automatically? Pardon the density of my brain matter here, but I thought that was what BGP was all about?

I welcome any education the group wishes to drop on me in this matter.

15

So perhaps the question you should be asking is: Why didn't routes for
these networks fall over to the other upstream peers which *are* capable of
moving the packets? Surely MCI, AT&T, Sprint, and others would carry the
packets to the right place. I can see the paths right here....

They did, and I'm not down. I see Level 3 via Sprint and GNAPs/CENT just fine. I didn't lose any connectivity to Level 3 at all. Bits moving down different pipes, not a big deal to me technically. The fact remains that Cogent is not providing the service I'm paying them for and they need to get it fixed. If that means picking up transit from another Tier 1 to get to Level 3 or making amends with Level 3 to get the existing peering working again. It doesn't matter to me, I just don't like paying for stuff I'm not getting. In the grand scheme of things I'm paying A LOT for my Cogent bandwidth (it started off as Verio remember).

What nature of clause? I consider deliberately filtering prefixes or origin
ASs to be a violation of common backbone BGP use.

Too bad there aren't Equal Access laws for tier1s. <slyly evil grin>

Ewww, I'll put up with these occasional pissing matches and build around them to avoid any government regulations.

-Matt

16

No, I mean: Why didn't *your upstream's* routes fall back to *their* other
peers, who should be perfectly capable of transiting those packets?

The thinly veiled implication there is that "full mesh" is not a long term
effective way to run the backbone level transit, because dropping one peer
without an alternate path means that we get broken transit. Yum.

17

> > I'm curious where in your contract you think Cogent guaranteed you
> > connectivity to Level 3?
>
> My original contract was with NTT/Verio which Cogent purchased last year when
> Verio nuked their Boston POP. I'm having the contract dug out of the
> archives to look at what it says. IMHO I pay Cogent for Transit to the whole
> Internet, If I wanted partial transit or local peering I would order/contract
> and pay for that. Cogent is not currently providing me full transit service.
> I really don't care who pulled the plug, it is Cogents job to fix it for me as
> I am their customer.

"Isn't BGP supposed to work around this sort of thing?"

This comes down to a little more than just "depeering" -- at least in the
BGP sense. There's active route filtering going on as well if connectivity
is dead; after all, I can bet the house that at least one of Cogent's
network edge peers has connectivity to Level3, and vice versa.

No there really isn't.

So perhaps the question you should be asking is: Why didn't routes for
these networks fall over to the other upstream peers which *are* capable of
moving the packets? Surely MCI, AT&T, Sprint, and others would carry the
packets to the right place. I can see the paths right here....

Cogent does not buy transit from MCI, AT&T, or Sprint.
Level 3 does not buy transit from MCI, AT&T, or Sprint.

You can only be a "tier 1" and maintain global reachability if you peer
with every other tier 1. Level 3 is obviously the real thing, and Cogent
is "close enough" (at least in their own minds :P) that they won't buy
real transit, only spot routes for the few things that they are missing
(ATDN and Sprint basically). There is no route "filtering" going on, only
the lack of full propagation due to transit purchasing decisions, or in
this case the lack thereof.

18

"Isn't BGP supposed to work around this sort of thing?"

Ok, I'll state the obvious first ....

BGP is a routing protocol, the economics of its implementation bears no resemblance to implied or otherwise connectivity.

This comes down to a little more than just "depeering" -- at least in the
BGP sense. There's active route filtering going on as well if connectivity
is dead; after all, I can bet the house that at least one of Cogent's
network edge peers has connectivity to Level3, and vice versa.

That would assume that cogent is paying someone to transit their routes to L3. Which I can deduce is not the case.

What nature of clause? I consider deliberately filtering prefixes or origin
ASs to be a violation of common backbone BGP use.

I'm not familiar with the concept of a 'common backbone BGP use policy". The best analogy I can think of is ....

"A commodity appears at first sight an extremely obvious, trivial thing. But its analysis brings out that it is a very strange thing, abounding in metaphysical subtleties and theological niceties."
  -- Karl Marx.

19

Todd Vierling wrote:

I'm curious where in your contract you think Cogent guaranteed you
connectivity to Level 3?
     

My original contract was with NTT/Verio which Cogent purchased last year when
Verio nuked their Boston POP. I'm having the contract dug out of the
archives to look at what it says. IMHO I pay Cogent for Transit to the whole
Internet, If I wanted partial transit or local peering I would order/contract
and pay for that. Cogent is not currently providing me full transit service.
I really don't care who pulled the plug, it is Cogents job to fix it for me as
I am their customer.
   
"Isn't BGP supposed to work around this sort of thing?"

This comes down to a little more than just "depeering" -- at least in the
BGP sense. There's active route filtering going on as well if connectivity
is dead; after all, I can bet the house that at least one of Cogent's
network edge peers has connectivity to Level3, and vice versa.

</lurk>
  Maybe not, the depeering L3 is involved in is sort of like blackmail,
we can all thank the indicted ex-CEO of WorldCom, Bernie Ebbers,
for the modern peering "There can only be one" rule set.

   Big guys double dip, and little guys are paying half the big
guys double dip... great deal if you can con someone into
accepting it, or are big enough to -force- them into accepting it.

Case in point.

L3 wants CoGent to kneel, and kiss the ring,
nothing more, nothing less.

  "They must smell blood in the water".

From where I sit, I can see a plethora of routes that transit more than one

tier1. And a few that transit three before hitting the origin. From a
couple locations I see 3356 and 174 visible in *all* paths to the prefixes
containing Level3 and Cogent in the path, respectively.

Well, we know who -your- *transit* providers are.... * cough *

So perhaps the question you should be asking is: Why didn't routes for
these networks fall over to the other upstream peers which *are* capable of
moving the packets? Surely MCI, AT&T, Sprint, and others would carry the
packets to the right place. I can see the paths right here....

    Some providers, a legacy of course, are "transit free", and rely on direct routes.. Soon,
there won't be many of these left... and it will be a non-issue.

  "There can only be *one* !" - WorldCom chant, Circa 1995.

Most transit contracts only guarantee packet delivery to the edge of their
own networks. I'm pretty sure Cogent is doing that. (Hell, they have lots
of spare capacity now. :slight_smile:
     

Most also have a clause to cover the inter-AS links, making sure that they are
not overloaded.
   
What nature of clause? I consider deliberately filtering prefixes or origin
ASs to be a violation of common backbone BGP use.

   Anyone who provides -peering-, instead of transit, actively filters routes, as SOP.

Too bad there aren't Equal Access laws for tier1s. <slyly evil grin>

      Like I said, light a fire, and lets burn Bernie at the stake!

  "I saw him fly up into the sky with the Devil himself !" *

   :-P

  (* no GOP affiliated ex-CEO's were harmed,
     or -actually- threatened, in the making of this post.
    Like FOX news, this post is classified as "Entertainment"
     and may or may not accurately portray actual facts.. :wink:

<lurk>

20

This is what I just got from Cogent support. I'm still waiting on
the billing dispute ticket. I've already told our payables
department to not pay any Cogent invoices, this should get fun.

Or, more likely, pointless...

I have to wonder though, if your network access is this mission
critical, why are you single homed?

Hell, I wish Verio never sold me to Cogent in the first place, it
is all their fault :confused:

Wow, if I had a penny for everytime I heard "It's all Verio's fault"
I'd have left there a very rich man and would now be cruising on some
yacht in the bahamas avoiding hurricanes and figuring out where to
import the next case of Scotch. Thanks for the trip down memory
lane....

-doug