Level 3 issues

Mailman List Mirror (Read Only)
1

is anyone having issues with Level3?

2

What country, location, where you fed from??

3

http://www.internetpulse.net/ (if you can get to it). Does not look pretty for L3.

I can't get to most web sites if I go via Level3 (Cleveland, OH). Ping/traceroute look good though.

marco wrote:

4

Yes. We just experienced an outage in Philadelphia. We shut down the circuit pending further investigation.

5

jajogega@gmail.com wrote:

Yes sir.

From: marco [mailto:marco@zero11.com]
Sent: Sunday, December 28, 2008 12:59 PM
To: nanog@nanog.org
Subject: Level 3 issues

is anyone having issues with Level3?
    

do you have any more details?

6

marco a �crit :

is anyone having issues with Level3?

hi,
theplanet.com and many websites (cnn.com ; amazon.com ; ... ) have not been accessible from France (Orange, home connection) for about 30 minutes.
Don't know if there is a link with your question, but .

Pierre-Henri

7

marco a �crit :

is anyone having issues with Level3?

hi,
theplanet.com and many websites (cnn.com ; amazon.com ; ... ) have not been accessible from France (Orange, home connection) for about 30 minutes.
Don't know if there is a link with your question, but it's strange...

Pierre-Henri

8

Ahh.. yes seeing that now here from Toronto ON - didn't see this issue when the original poster sent the first message... it's now happening here too...

Shutting down their session until something looks "better"

9

I'm showing significant latency and loss over my L3 stuff.

interetpulse.net showing the same thing too, seems to be a substantial problem.

Pierre-Henri wrote:

10

Ive got connection issues from Colorado to new York on level3 that have been restored, but still nothing from Chicago to Colorado, and way too many other places to list. Anyone have a ticket number with level3?

11

Blake Pfankuch wrote:

Ive got connection issues from Colorado to new York on level3 that have been restored, but still nothing from Chicago to Colorado, and way too many other places to list. Anyone have a ticket number with level3?

From: Pierre-Henri [mailto:phach34@gmail.com]
Sent: Sunday, December 28, 2008 11:06 AM
To: marco
Cc: nanog@nanog.org
Subject: Re: Level 3 issues

marco a �crit :
  

is anyone having issues with Level3?

hi,
theplanet.com and many websites (cnn.com ; amazon.com ; ... ) have not
been accessible from France (Orange, home connection) for about 30 minutes.
Don't know if there is a link with your question, but .

Pierre-Henri

I have a ticket in with L3, their NOC is pretty overwhelmed with phone
calls.

12

Same issue here from Chicago and Montreal. Seems anything routing through Washington.Level3 is going to null. The rest of the level3 network seems to be ok.
6 ae-32-52.ebr2.Chicago1.Level3.net (4.68.101.62) 0.976 ms 10.344 ms 0.866 ms
7 ae-5.ebr2.Chicago2.Level3.net (4.69.140.194) 1.245 ms 0.991 ms 0.978 ms
8 ae-2-2.ebr2.Washington1.Level3.net (4.69.132.70) 18.608 ms 18.961 ms 18.583 ms
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *

...
4 car1.Montreal2.Level3.net (67.215.0.146) 0.657 ms 0.791 ms 0.699 ms
5 ae-5-5.ebr4.NewYork1.Level3.net (4.69.141.6) 17.764 ms 8.490 ms 18.197 ms
6 ae-94-94.csw4.NewYork1.Level3.net (4.69.134.126) 15.541 ms 8.286 ms 17.098 ms
7 ae-93-93.ebr3.NewYork1.Level3.net (4.69.134.109) 11.384 ms ae-61-61.ebr1.NewYork1.Level3.net (4.69.134.65) 9.100 ms 8.614 ms
8 ae-3-3.ebr4.Washington1.Level3.net (4.69.132.93) 13.840 ms 15.584 ms 17.443 ms
9 ae-94-94.csw4.Washington1.Level3.net (4.69.134.190) 23.420 ms 25.569 ms 18.042 ms
10 ae-4-99.edge2.Washington4.Level3.net (4.68.17.211) 14.052 ms 14.028 ms 13.610 ms
11 * * *
12 * * *
13 * * *
14 * * *
15 *

Paul Stewart wrote:

13

Paul wrote:

Same issue here from Chicago and Montreal. Seems anything routing
through Washington.Level3 is going to null. The rest of the level3
network seems to be ok.
6 ae-32-52.ebr2.Chicago1.Level3.net (4.68.101.62) 0.976 ms 10.344
ms 0.866 ms
7 ae-5.ebr2.Chicago2.Level3.net (4.69.140.194) 1.245 ms 0.991 ms
0.978 ms
8 ae-2-2.ebr2.Washington1.Level3.net (4.69.132.70) 18.608 ms 18.961
ms 18.583 ms
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *

...
4 car1.Montreal2.Level3.net (67.215.0.146) 0.657 ms 0.791 ms 0.699 ms
5 ae-5-5.ebr4.NewYork1.Level3.net (4.69.141.6) 17.764 ms 8.490 ms
18.197 ms
6 ae-94-94.csw4.NewYork1.Level3.net (4.69.134.126) 15.541 ms 8.286
ms 17.098 ms
7 ae-93-93.ebr3.NewYork1.Level3.net (4.69.134.109) 11.384 ms
ae-61-61.ebr1.NewYork1.Level3.net (4.69.134.65) 9.100 ms 8.614 ms
8 ae-3-3.ebr4.Washington1.Level3.net (4.69.132.93) 13.840 ms 15.584
ms 17.443 ms
9 ae-94-94.csw4.Washington1.Level3.net (4.69.134.190) 23.420 ms
25.569 ms 18.042 ms
10 ae-4-99.edge2.Washington4.Level3.net (4.68.17.211) 14.052 ms
14.028 ms 13.610 ms
11 * * *
12 * * *
13 * * *
14 * * *
15 *

Paul Stewart wrote:

Ahh.. yes seeing that now here from Toronto ON - didn't see this
issue when the original poster sent the first message... it's now
happening here too...

Shutting down their session until something looks "better"

From: Pierre-Henri [mailto:phach34@gmail.com] Sent: December 28, 2008
1:06 PM
To: marco
Cc: nanog@nanog.org
Subject: Re: Level 3 issues

marco a �crit :

is anyone having issues with Level3?

hi,
theplanet.com and many websites (cnn.com ; amazon.com ; ... ) have
not been accessible from France (Orange, home connection) for about
30 minutes.
Don't know if there is a link with your question, but it's strange...

Pierre-Henri

----------------------------------------------------------------------------

"The information transmitsted is intended only for the person or
entity to which it is addressed and contains confidential and/or
privileged material. If you received this in error, please contact
the sender immediately and then destroy this transmission, including
all attachments, without copying, distributing or disclosing same.
Thank you."

According to L3, this issue should be fixed and we should start seeing
the traffic normalizing.
Can anyone confirm?

14

According to L3, this issue should be fixed and we should start seeing

the traffic normalizing.
Can anyone confirm?

Here in Richmond Virginia, everything seems to be back to normal now.
Traffic coming from my Comcast connection can get through L3 now.

  7 11 ms 13 ms 11 ms te-0-3-0-0-cr01.mclean.va.ibone.comcast.net [68.
86.91.121]
  8 10 ms 11 ms 12 ms xe-11-1-0.edge1.Washington1.Level3.net [4.79.231
.9]
  9 12 ms 17 ms 18 ms vlan89.csw3.Washington1.Level3.net [4.68.17.190]

10 12 ms 17 ms 17 ms ae-84-84.ebr4.Washington1.Level3.net [4.69.134.1
85]
11 16 ms 26 ms 16 ms ae-3-3.ebr1.NewYork1.Level3.net [4.69.132.94]
12 32 ms 30 ms 17 ms ae-81-81.csw3.NewYork1.Level3.net [4.69.134.74]

13 15 ms 19 ms 16 ms ae-3-89.edge1.NewYork1.Level3.net [4.68.16.142]

15

Everything seems to be back to normal in France

16

Confirmed here as well.

Jon

17

Seems to be normalizing here in Colorado as well, however still having occasional packet loss to NY.

18

Looks like most providers here in the east coast are routing through level3
again, and I'm not seeing any packet loss or latency anymore.

19

Any word on the actual cause of the issue?

20

Blake Pfankuch wrote:

Any word on the actual cause of the issue?

From: Derek Bodner [mailto:subscribedlists@derekbodner.com]
Sent: Sunday, December 28, 2008 11:53 AM
To: Blake Pfankuch
Cc: Jon Wolberg; Jason Cheslock; nanog@nanog.org
Subject: Re: Level 3 issues

Looks like most providers here in the east coast are routing through level3 again, and I'm not seeing any packet loss or latency anymore.
Seems to be normalizing here in Colorado as well, however still having occasional packet loss to NY.

From: Jon Wolberg [mailto:jon@defenderhosting.com]
Sent: Sunday, December 28, 2008 11:40 AM
To: Jason Cheslock
Cc: nanog@nanog.org<mailto:nanog@nanog.org>
Subject: Re: Level 3 issues

Confirmed here as well.

Jon

From: "Jason Cheslock" <sangreviento@gmail.com<mailto:sangreviento@gmail.com>>
To: "marco" <marco@zero11.com<mailto:marco@zero11.com>>
Cc: nanog@nanog.org<mailto:nanog@nanog.org>
Sent: Sunday, December 28, 2008 1:35:45 PM GMT -05:00 US/Canada Eastern
Subject: Re: Level 3 issues

According to L3, this issue should be fixed and we should start seeing

the traffic normalizing.
Can anyone confirm?
    
Here in Richmond Virginia, everything seems to be back to normal now.
Traffic coming from my Comcast connection can get through L3 now.

7 11 ms 13 ms 11 ms te-0-3-0-0-cr01.mclean.va.ibone.comcast.net<http://te-0-3-0-0-cr01.mclean.va.ibone.comcast.net> [68.
86.91.121]
8 10 ms 11 ms 12 ms xe-11-1-0.edge1.Washington1.Level3.net<http://xe-11-1-0.edge1.Washington1.Level3.net> [4.79.231
.9]
9 12 ms 17 ms 18 ms vlan89.csw3.Washington1.Level3.net<http://vlan89.csw3.Washington1.Level3.net> [4.68.17.190]

10 12 ms 17 ms 17 ms ae-84-84.ebr4.Washington1.Level3.net<http://ae-84-84.ebr4.Washington1.Level3.net> [4.69.134.1
85]
11 16 ms 26 ms 16 ms ae-3-3.ebr1.NewYork1.Level3.net<http://ae-3-3.ebr1.NewYork1.Level3.net> [4.69.132.94]
12 32 ms 30 ms 17 ms ae-81-81.csw3.NewYork1.Level3.net<http://ae-81-81.csw3.NewYork1.Level3.net> [4.69.134.74]

13 15 ms 19 ms 16 ms ae-3-89.edge1.NewYork1.Level3.net<http://ae-3-89.edge1.NewYork1.Level3.net> [4.68.16.142]

--
Derek Bodner
subscribedlists@derekbodner.com<mailto:subscribedlists@derekbodner.com>
  
From what I heard, it was some some malfunction with a router in

Washington D.C. which terminated a 100GB bundle from Paris. It was
carring about 50GB at the time of the failure.

Not sure why routes within the US would be effected.