Level 3's side of the story

http://www.prnewswire.com/cgi-bin/stories.pl?ACCT=104&STORY=/www/story/10-07-2005/0004164041&EDATE=

"On October 6, Level 3, as it had repeatedly advised Cogent it would,
terminated free traffic exchange with Cogent. Because Internet users,
apparently without notice from Cogent and through no fault of their own,
have been impacted, Level 3 has, effective immediately, re-established a
free connection to Cogent. In order to allow Internet users to make
alternative arrangements, we will maintain this connection until 6:00 a.m.
ET, November 9, 2005. The effectiveness of this arrangement of course
depends on Cogent's willingness to maintain their side of the traffic
exchange.

Richard A Steenbergen wrote:

All News Releases and Press Releases from PR Newswire

"On October 6, Level 3, as it had repeatedly advised Cogent it would, terminated free traffic exchange with Cogent. Because Internet users, apparently without notice from Cogent and through no fault of their own,

I don't remember seeing this public notice from Level(3) posted....
Wouldn't that be "without notice from Level(3)"?

have been impacted, Level 3 has, effective immediately, re-established a free connection to Cogent. In order to allow Internet users to make alternative arrangements,

Splendid, that gives the world sufficient time to accept Cogent's offer
of 1 year free service.

we will maintain this connection until 6:00 a.m. ET, November 9, 2005. The effectiveness of this arrangement of course depends on Cogent's willingness to maintain their side of the traffic exchange.

At which time Level(3) will have had time to purchase transit, as it
will be a "tier 2" hoisted on its own petard.

And after November 9, what is then? Cogent/L3 depeering part 2?

BTW - it sounds like maybe somebody was required to give 30 days notice of service changes to certain customers with good laywers....

Part III: Level3 Strikes Back :slight_smile:

Depeering World Series now at:
Level3: 2, Cogent: 14

william(at)elan.net wrote:

BTW - it sounds like maybe somebody was required to give 30 days notice of service changes to certain customers with good laywers....

This is my bet. Let's see. Peering went down October 6th. Then a fedexed letter with nasty threats arrives today [with a fax copy arriving on the 6th]. Then a letter from Level3 fedexed back on the 7th for delivery on the 9th saying "We've cured the default and here is 30 days notice that we are terminating your account or changing your service."

Then 30 days from the 10th to November 9th...

Yup. Sounds about right.

Dave (I'm merging your comment in here too):

One question [referring to customers being used as pawns] -- what does ANY network have besides their customers to use in a peering negotiation?

Engineers often try to use things like prefix counts, # of routers, # of countries, blah blah blah but really these are all oblique references to the size and import of the single homed customers behind that network. Level3 thought their customers were more desirable than Cogent's. Level3's customers seem to have enough respect for Cogent's customers to make Level3 blink. I seem to remember the same thing happening with [another network] trying to depeer Cogent a few months back.

What I find interesting is that in this case, Cogent apparently allowed the sessions (whichever ones Level3 decided to fire back up) to come up cleanly. I think in that earlier instance Cogent kept the peering sessions down until some point [one speculates until the legal paperwork came through] -- notice I said "I think" I have no specific facts, just a vague recollection.

Taking this back to pricing (I suddenly found about 30 minutes I didn't expect to have), if Cogent terminated their most recent $6/mb/s or $8/mb/s promotion because of this concern with the Level3 peering and Level3 does indeed blink [comes back to the table and agrees to an indefinite term of SFI] Cogent might re-establish the promotion and with it grow by another 20 or 30 Gb/s or more.. possibly at the direct expense of Level3 and others and force more growth of their interconnections.

Another question (wow, 3 or 4 separate threads in one email). Instead of a "hard" de-peering. Wouldn't it be more effective if Level3 just stopped adding interconnection capacity to Cogent? That is an effective way of limiting the growth of losses between two networks, and if some discreet situation can be used [say a balanced traffic ratio] no new capacity will be added until that situation is resolved? Cogent [in this case] could solve the problem by adding the appropriate # and type of customers or by buying transit to add capacity [and possibly forfeiting SFI]. Then instead of using your peering partner's customers as pawns, you are using your own to drive the business case either way.

That is a much "nicer" way to put pressure on a peer without it getting blasted all over the headlines... Especially if you are the network doing the depeering. And if your customer contracts don't cover (SLA-wise) interconnections, you don't care either way.

Just a few ideas...

Deepak

In any industry, it only takes a handful of consumers impacted through no fault of their own to generate significant political & regulatory attention. Reasonable-sounding explanations cannot stand if they imply that thousands of innocent parties can be impacted without notice.

/John

You're right: administrators, regulators, and representatives are completely helpless except when making laws, and the reconnection of services (ref: "Because Internet users, apparently without notice from Cogent and through no fault of their own, have been impacted, Level 3 has, effective immediately, re-established a free connection to Cogent.") can just as easily be attributed to uncanny judgement on Level 3's part as any other explanation.

/John

What's your point? Are you seriously suggesting this should be regulated?

It doesn't appear necessary at all; industry self-regulation seems to be working just fine here...
(one might claim almost too well :slight_smile: )

.. let the customers draw their own consequences and conclusions based on the actions of the companies who serve them.
Thankfully, Internet connectivity is ubiquitous for the most part, and this is hardly the same as essential services (those who consider it that, are fools not to have diversity).

Those un-essential Internet services increasingly carry voice services that are considered essential by many consumers, and for whom diversity is not a realistic option. Leaving them completely open to the consequences of service provider actions would be quite an innovative curve in public policy.

/John

http://www.prnewswire.com/cgi-bin/stories.pl?ACCT=104&STORY=/www/story/10-07-2005/0004164041&EDATE=
"On October 6, Level 3, as it had repeatedly advised Cogent it would, terminated free traffic exchange with Cogent. Because Internet users, apparently without notice from Cogent and through no fault of their own,

I don't remember seeing this public notice from Level(3) posted....
Wouldn't that be "without notice from Level(3)"?

Every peering agreement I have seen includes an NDA. Most certainly notified Cogent, and almost certainly had no legal right to issue a press release.

have been impacted, Level 3 has, effective immediately, re-established a free connection to Cogent. In order to allow Internet users to make alternative arrangements,

Splendid, that gives the world sufficient time to accept Cogent's offer
of 1 year free service.

I note that connecting to Cogent after November 9th will still give you only partial transit to the Internet - in fact, far less of the Internet than connecting to Level 3. (Despite Cogent's claims otherwise.)

we will maintain this connection until 6:00 a.m. ET, November 9, 2005. The effectiveness of this arrangement of course depends on Cogent's willingness to maintain their side of the traffic exchange.

At which time Level(3) will have had time to purchase transit, as it
will be a "tier 2" hoisted on its own petard.

Is there any particular reason you think Level 3 will buy transit? Or that they give a rat's ass what tier _you_ think they are?

Perhaps Cogent will have had time to buy transit?

Or perhaps - just perhaps - the Internet will be bifurcated for the time being. There is nothing intrinsically wrong with an Internet which isn't fully connected. Each business will work out its own business needs and use the transit provider(s) they see fit.

I am a bit confused about why so many people think this is such a bad thing. Personally, I believe there is room in the market for partial transit providers. I guess that theory is about to be tested.

Agree that there is room in the market for attactively priced
partial transit providers. Now where does that leave Level 3
after Nov 9th? Possibly with a second strategy reconsidertion.

Ken Leland
Monmouth Telecom

My advice to these people is, if you don't like the connectivity being
provided, don't buy from that provider. Closing your wallet is a lot more
powerful than flapping your lips.

Personally I suspect there will still be an extensive market of people who
are multihomed to Cogent and someone else, and that these people will
continue to buy the product even without the 2.5% of the Internet that is
single homed to (3). The biggest complainers so far seem to be from the
people who didn't understand what they were signing on for, and a lot of
those are the Verio customers who Cogent purchased. Maybe Cogent is
willing to part with those customers, maybe their contracts are coming due
and they won't be renewing, or maybe the customers should have thought
better before signing a contract which allowed the transfer of the
contract to a completely different entity without an opportunity to
cancel. Certainly these are high-margin but low-bandwidth customers, maybe
with enough complaints Cogent will be willing to stick them on a smaller
seperate ASN which is willing to buy transit.

Or, if you really care about it that much, buy service from someone else,
unplug your port, stop paying your bill, and maybe send back a nastygram
which includes suggestions as to where Cogent can relocate the router card
that you used to be attached to, along with helpful advice as to how it
might be made to fit there. Cogent will not waste time or money on
lawsuits for disgruntled ex-Verio T1 customers.

Richard A Steenbergen wrote:

Certainly these are high-margin but low-bandwidth customers, maybe with enough complaints Cogent will be willing to stick them on a smaller seperate ASN which is willing to buy transit.

Does anyone have reachability data for c-root during this episode?

I wonder if they made separate arrangements for that or are planning to make arrangements for phase 2.

- Kevin

Sorry, I fundamentally disagree with the point of view that we need to regulate free entry & free exit and throw these basic tenets of our economy out the window to encourage continued poor decision making .

I'm not advocating that we need to regulate anything, only pointing out that the historic public policy position discourages service impacts to 3rd parties, and even more so when done without notice. If you carry essential consumer services, then there is already existing policy in this area.

There is no paradise behind that door. And I don't believe regulation should be used to prevent people from making occasionally poor decisions, which in turn will ultimately do nothing more than raise the price floor and create additional barriers of entry in the market. If you don't have those events happen, bad things never get weeded out. There really is nothing but bad news in store for what you propose, because it isn't as simple of a thing to impose regulation and have mostly good things happen.

That's a fine set of beliefs (and I might even agree with some of them). However, they're completely irrelevant to the existing school of thought which is guiding policy and legislation in this area, which is probably best represented by last month's House Telecom committee draft:

<http://energycommerce.house.gov/108/News/09152005_staff_disc.pdf&gt; [1]

There you go: mandatory ISP registration, interconnection, consumer protection, and more. Maybe these folks were too busy with other stuff to notice the Internet partition happen earlier in the week? Oh, wait, I remember now: these folks are only matter when legislating.

/John

[1] A readable (only-slightly-biased) summary: <http://www.heritage.org/Research/Regulation/wm860.cfm&gt;

Kevin Loch <kloch@hotnic.net> writes:

Richard A Steenbergen wrote:

Certainly these are high-margin but low-bandwidth customers, maybe
with enough complaints Cogent will be willing to stick them on a
smaller seperate ASN which is willing to buy transit.

Does anyone have reachability data for c-root during this episode?

I wonder if they made separate arrangements for that or are planning
to make arrangements for phase 2.

The nice thing about having 13 root nameservers is that it really
doesn't matter whether they do or not.

If it starts to become trendy to have root nameservers on
intentionally-partially-connected networks, (say, maybe four of them)
*then* we can start thinking about the consequences.

                                        ---rob

Wow... You're characterizing me as trying to prove this is rational and justified thought? That's almost as cool as your earlier suggestion that I'm advocating for regulation. I'm not. I know that can be hard to tell unless you read my messages.

What I have said that there is *significant* attention to the potential consumer impact of our "non-essential" IP services, and that's not surprising given the historic public policy in this area. I pointed to the bill under draft merely as documentation of this attention and to note that unless there is a radical shift in policy for telecom consumer protections, we are going to see some form of regulation as more voice moves to the Internet.

/John

Why is it (3)'s responsibility to handle Cogent's customers? It isn't. If that was the case we'd be required to notify all downstream's when we terminate on default of contract or other reasons (as an ISP).

No I think if (3) told Cogent then they did their job. It's absurd to say that (3) was responsible for public notice of Cogent's customers or anything of that nature. That's a Cogent internal matter that they screwed up, or intentionally withheld.

No, my prediction is based solely on the current actions of lawmakers to address the perceived social need of reliable phone service. If laws are passed, it's highly likely that the FCC will regulate accordingly, regardless of how amusing the mapping from regulation to reality turns out.

/John

Kevin Loch writes:

Does anyone have reachability data for c-root during this episode?

The RIPE NCC "DNSMON" service has some:

According to BGPlay for that particular prefix from Route-Views data
(for some reason the RIPE RIS server used by BGPlay seems to be down
at the moment), the "episode" seems to be between these times (UTC):

2005-10-05 09:49:03 Route Withdrawal ( 3356 174 2149 )
2005-10-07 19:24:13 Route Announcement 3356 174 2149

The interval in the URL above starts 72 hours before the start of the
episode and ends 72 hours after its end. I cannot see any particular
problems that would coincide with the episode, from that set of probes
(RIPE TTM).

Because we rely on default routes to our three transit providers, and
Level(3) is one of them, some of our customers must have had
connectivity issues to Cogent for a few hours, until we noticed
(thanks to Adam Rothschild and NANOG) and implemented a workaround.
But our RIPE TTM boxes (tt85 as well as the currently broken tt86)
aren't in those parts of our network.

I wonder if they made separate arrangements for that or are planning
to make arrangements for phase 2.

As someone else said, partial unreachability of a particular root
nameserver isn't that much of an issue. But it's an interesting
question nevertheless.

Kevin Loch writes:
> Does anyone have reachability data for c-root during this episode?

The RIPE NCC "DNSMON" service has some:

DNSMON — RIPE Network Coordination Centre

If there is anyone with more comprehensive data I'd like to hear about it :wink: !

According to BGPlay for that particular prefix from Route-Views data
(for some reason the RIPE RIS server used by BGPlay seems to be down
at the moment), the "episode" seems to be between these times (UTC):

It is up now. Of the two routes to c.root-servers.net via Level 3 in that
data set one stayed up during the period and the other got healed immediateely
via AS286. It flipped back later.

...

The interval in the URL above starts 72 hours before the start of the
episode and ends 72 hours after its end. I cannot see any particular
problems that would coincide with the episode, from that set of probes
(RIPE TTM).

I agree that there is nothing really significant here. It is mostly noise.
What one is looking for in these graphs is strong vertical
patterns. dnsmon did not detect anything significant concerning
reachability of c.root-servers.net.

As someone else said, partial unreachability of a particular root
nameserver isn't that much of an issue.

Indeed.

But it's an interesting question nevertheless.

A red instance of a fish from the northern seas ?

Daniel