Level3 funkiness

Mailman List Mirror (Read Only)
1

Anyone else experience sporadic funkiness via
Level3? I can't even reach the main website from who
knows how many networks I've tried. Also friends
and former colleagues have tried to reach the site
to no avail.

One of my machines on AT&T:
# traceroute level3.net
traceroute to level3.net (63.211.236.36), 30 hops max, 40 byte packets

4 cr1.n54ny.ip.att.net (12.122.105.58) 11.285 ms 21.702 ms 21.477 ms
5 ggr2.n54ny.ip.att.net (12.122.131.141) 12.712 ms 10.194 ms 16.393 ms
6 so-8-0-0.car3.NewYork1.Level3.net (4.68.127.149) 9.975 ms 10.019 ms 10.833 ms
7 vlan79.csw2.NewYork1.Level3.net (4.68.16.126) 10.162 ms 10.189 ms 14.474 ms
8 ae-71-71.ebr1.NewYork1.Level3.net (4.69.134.69) 15.763 ms 11.166 ms 9.725 ms
9 ae-3-3.ebr4.Washington1.Level3.net (4.69.132.93) 16.139 ms 30.616 ms 16.275 ms
10 ae-64-64.csw1.Washington1.Level3.net (4.69.134.178) 15.684 ms ae-74-74.csw2.Washington1.Level3.net (4.69.134.182) 21.870 ms ae-84-84.csw3.Washington1.Level3.net (4.69.134.186) 28.729 ms
11 ae-92-92.ebr2.Washington1.Level3.net (4.69.134.157) 17.035 ms ae-62-62.ebr2.Washington1.Level3.net (4.69.134.145) 17.041 ms ae-72-72.ebr2.Washington1.Level3.net (4.69.134.149) 21.940 ms
12 ae-2-2.ebr2.Chicago2.Level3.net (4.69.132.69) 31.671 ms 42.407 ms 45.774 ms
13 ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113) 31.922 ms 32.115 ms 38.135 ms
14 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) 75.265 ms 67.528 ms 67.937 ms
15 ge-9-0.hsa1.Denver1.Level3.net (4.68.107.35) 62.587 ms !H ge-9-1.hsa1.Denver1.Level3.net (4.68.107.99) 62.543 ms !H ge-9-2.hsa1.Denver1.Level3.net (4.68.107.163) 75.797 ms !H

(From Texas through Above.net)
$ traceroute level3.net|tail -n 1
traceroute to level3.net (63.211.236.36), 64 hops max, 40 byte packets
11 ge-6-2.hsa1.Denver1.Level3.net (4.68.107.131) 21.473 ms !H * ge-6-0.hsa1.Denver1.Level3.net (4.68.107.3) 21.547 ms !H

Confirmed it can't be reached from Travelers Ins, The
Hartford, none of my connections. Anyone else seeing
issues? I'm seeing drop off from clients going through
their Atlanta interconnects with Charter and two other
providers, which I can't make sense of. I DO KNOW they
experienced some sort of issue with a TDM switch or so
they said... Very broad statements: "We know teh
interwebs are down please stand by"

I know websites are one thing, but the chances of the
website going down, a TDM switch being wacky and now
clients traversing their networks complaining all at
once seems a little out of the ordinary.

2

Can't get to level3.net 63.211.236.36 or www.level3.net 4.68.95.28 from
Pittsburgh either and I peer directly with level3 with a full BGP feed.

3

Yes, I die on your hop14 with TWTelecom

4

From: J. Oquendo [mailto:sil@infiltrated.net]
Sent: Wednesday, April 15, 2009 15:36
To: nanog@nanog.org
Subject: Level3 funkiness

Anyone else experience sporadic funkiness via
Level3? I can't even reach the main website from who
knows how many networks I've tried. Also friends
and former colleagues have tried to reach the site
to no avail.

One of my machines on AT&T:
# traceroute level3.net
traceroute to level3.net (63.211.236.36), 30 hops max, 40 byte packets

4 cr1.n54ny.ip.att.net (12.122.105.58) 11.285 ms 21.702 ms 21.477

ms

5 ggr2.n54ny.ip.att.net (12.122.131.141) 12.712 ms 10.194 ms

16.393 ms

6 so-8-0-0.car3.NewYork1.Level3.net (4.68.127.149) 9.975 ms 10.019

ms 10.833 ms

7 vlan79.csw2.NewYork1.Level3.net (4.68.16.126) 10.162 ms 10.189 ms

14.474 ms

8 ae-71-71.ebr1.NewYork1.Level3.net (4.69.134.69) 15.763 ms 11.166

ms 9.725 ms

9 ae-3-3.ebr4.Washington1.Level3.net (4.69.132.93) 16.139 ms 30.616

ms 16.275 ms

10 ae-64-64.csw1.Washington1.Level3.net (4.69.134.178) 15.684 ms

ae-74->74.csw2.Washington1.Level3.net (4.69.134.182) 21.870 ms
ae-84->84.csw3.Washington1.Level3.net (4.69.134.186) 28.729 ms

11 ae-92-92.ebr2.Washington1.Level3.net (4.69.134.157) 17.035 ms

ae-62->62.ebr2.Washington1.Level3.net (4.69.134.145) 17.041 ms
ae-72->72.ebr2.Washington1.Level3.net (4.69.134.149) 21.940 ms

12 ae-2-2.ebr2.Chicago2.Level3.net (4.69.132.69) 31.671 ms 42.407 ms

45.774 ms

13 ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113) 31.922 ms 32.115

ms 38.135 ms

14 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) 75.265 ms 67.528 ms

67.937 ms

15 ge-9-0.hsa1.Denver1.Level3.net (4.68.107.35) 62.587 ms !H

ge-9->1.hsa1.Denver1.Level3.net (4.68.107.99) 62.543 ms !H
ge-9-2.hsa1.Denver1.Level3.net >(4.68.107.163) 75.797 ms !H

(From Texas through Above.net)
$ traceroute level3.net|tail -n 1
traceroute to level3.net (63.211.236.36), 64 hops max, 40 byte packets
11 ge-6-2.hsa1.Denver1.Level3.net (4.68.107.131) 21.473 ms !H *

ge-6->0.hsa1.Denver1.Level3.net (4.68.107.3) 21.547 ms !H

Confirmed it can't be reached from Travelers Ins, The
Hartford, none of my connections. Anyone else seeing
issues? I'm seeing drop off from clients going through
their Atlanta interconnects with Charter and two other
providers, which I can't make sense of. I DO KNOW they
experienced some sort of issue with a TDM switch or so
they said... Very broad statements: "We know teh
interwebs are down please stand by"

I know websites are one thing, but the chances of the
website going down, a TDM switch being wacky and now
clients traversing their networks complaining all at
once seems a little out of the ordinary.

=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+>=+=+
J. Oquendo
SGFA, SGFE, C|EH, CNDA, CHFI, OSCP

"It takes 20 years to build a reputation and five minutes to
ruin it. If you think about that, you'll do things
differently." - Warren Buffett

227C 5D35 7DCB 0893 95AA 4771 1DCE 1FD1 5CCD 6B5E
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x5CCD6B5E

Looks similar from Sprints perspective via
Sprint Portal as well...

Sprint Source Region: Anaheim, CA (sl-bb20-ana)
IP Destination: 63.211.236.36
Performing: ICMP Traceroute
Tracing the route to 63.211.236.36
  1 sl-crs2-ana-0-13-5-0.sprintlink.net (144.232.1.177) 4 msec 0 msec 0
msec
  2 144.232.19.227 4 msec 24 msec
    sl-st30-la-0-4-5-0.sprintlink.net (144.232.24.41) 0 msec
  3 xe-7-3-0.edge1.losangeles9.Level3.net (4.68.111.169) [AS 3356] 4
msec
    xe-7-2-0.edge1.losangeles9.Level3.net (4.68.111.89) [AS 3356] 0 msec
0 msec
  4 vlan69.csw1.LosAngeles1.Level3.net (4.68.20.62) [AS 3356] 16 msec
    vlan99.csw4.LosAngeles1.Level3.net (4.68.20.254) [AS 3356] 12 msec
    vlan89.csw3.LosAngeles1.Level3.net (4.68.20.190) [AS 3356] 24 msec
  5 ae-63-63.ebr3.LosAngeles1.Level3.net (4.69.137.33) [AS 3356] 4 msec
12 msec
    ae-93-93.ebr3.LosAngeles1.Level3.net (4.69.137.45) [AS 3356] 4 msec
  6 ae-2.ebr3.SanJose1.Level3.net (4.69.132.9) [AS 3356] 40 msec 16 msec
16 msec
  7 ae-73-73.csw2.SanJose1.Level3.net (4.69.134.230) [AS 3356] 16 msec
    ae-63-63.csw1.SanJose1.Level3.net (4.69.134.226) [AS 3356] 12 msec
12 msec
  8 ae-82-82.ebr2.SanJose1.Level3.net (4.69.134.217) [AS 3356] 12 msec
    ae-92-92.ebr2.SanJose1.Level3.net (4.69.134.221) [AS 3356] 32 msec
12 msec
  9 ae-3.ebr1.Denver1.Level3.net (4.69.132.58) [AS 3356] 60 msec 48 msec
52 msec
10 ge-6-1.hsa1.Denver1.Level3.net (4.68.107.67) [AS 3356] !H
    ge-6-2.hsa1.Denver1.Level3.net (4.68.107.131) [AS 3356] !H *

Completed - Wed Apr 15 15:43:21 EDT 2009

Sprint Source Region: Atlanta, GA (sl-bb20-atl)
IP Destination: 63.211.236.36
Performing: ICMP Traceroute
Tracing the route to 63.211.236.36
  1 sl-bb21-atl-14-0.sprintlink.net (144.232.12.142) 0 msec 0 msec 0
msec
  2 sl-crs1-atl-0-4-0-3.sprintlink.net (144.232.12.148) 4 msec 0 msec 0
msec
  3 sl-crs1-rly-0-0-5-0.sprintlink.net (144.232.20.177) 20 msec 16 msec
16 msec
  4 sl-crs1-dc-0-8-0-0.sprintlink.net (144.232.19.213) 20 msec 20 msec
24 msec
  5 sl-st31-ash-0-4-0-1.sprintlink.net (144.232.19.4) 20 msec 20 msec 24
msec
  6 4.68.63.205 [AS 3356] 20 msec
    te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) [AS 3356] 20
msec 24 msec
  7 vlan79.csw2.Washington1.Level3.net (4.68.17.126) [AS 3356] 24 msec
28 msec
    vlan89.csw3.Washington1.Level3.net (4.68.17.190) [AS 3356] 28 msec
  8 ae-72-72.ebr2.Washington1.Level3.net (4.69.134.149) [AS 3356] 24
msec
    ae-62-62.ebr2.Washington1.Level3.net (4.69.134.145) [AS 3356] 24
msec 24 msec
  9 ae-2-2.ebr2.Chicago2.Level3.net (4.69.132.69) [AS 3356] 28 msec 28
msec 28 msec
10 ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113) [AS 3356] 28 msec
28 msec 28 msec
11 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) [AS 3356] 48 msec 48 msec
52 msec
12 ge-9-1.hsa1.Denver1.Level3.net (4.68.107.99) [AS 3356] !H
    ge-9-2.hsa1.Denver1.Level3.net (4.68.107.163) [AS 3356] !H *
Completed - Wed Apr 15 15:44:17 EDT 2009

Sprint Source Region: Chicago, IL (sl-bb20-chi)
IP Destination: 63.211.236.36
Performing: ICMP Traceroute
Tracing the route to 63.211.236.36
  1 sl-crs1-chi-0-0-0-0.sprintlink.net (144.232.26.115) 0 msec 4 msec 0
msec
  2 sl-st20-chi-4-0-0.sprintlink.net (144.232.18.153) 0 msec
    sl-st20-chi-13-0-0.sprintlink.net (144.232.20.3) 0 msec 0 msec
  3 144.232.19.174 0 msec 0 msec 4 msec
  4 ae-31-53.ebr1.Chicago1.Level3.net (4.68.101.94) [AS 3356] 8 msec
    ae-31-51.ebr1.Chicago1.Level3.net (4.68.101.30) [AS 3356] 8 msec
    ae-31-55.ebr1.Chicago1.Level3.net (4.68.101.158) [AS 3356] 4 msec
  5 ae-6.ebr1.Chicago2.Level3.net (4.69.140.190) [AS 3356] 0 msec 4 msec
0 msec
  6 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) [AS 3356] 44 msec 44 msec
36 msec
  7 ge-9-0.hsa1.Denver1.Level3.net (4.68.107.35) [AS 3356] !H
    ge-9-2.hsa1.Denver1.Level3.net (4.68.107.163) [AS 3356] !H *

Completed - Wed Apr 15 15:44:49 EDT 2009

Sprint Source Region: Relay, MD (sl-bb20-rly)
IP Destination: 63.211.236.36
Performing: ICMP Traceroute
Tracing the route to 63.211.236.36
  1 sl-crs2-rly-0-1-0-0.sprintlink.net (144.232.3.96) 0 msec 0 msec 4
msec
  2 sl-crs2-dc-0-12-2-0.sprintlink.net (144.232.19.221) 0 msec 0 msec 0
msec
  3 sl-st31-ash-0-8-0-0.sprintlink.net (144.232.19.6) 0 msec
    sl-st31-ash-0-1-0-0.sprintlink.net (144.232.19.230) 0 msec
    sl-st31-ash-0-8-0-0.sprintlink.net (144.232.19.6) 4 msec
  4 4.68.63.205 [AS 3356] 4 msec
    te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) [AS 3356] 0
msec
    4.68.63.205 [AS 3356] 0 msec
  5 vlan69.csw1.Washington1.Level3.net (4.68.17.62) [AS 3356] 0 msec
    vlan89.csw3.Washington1.Level3.net (4.68.17.190) [AS 3356] 4 msec 12
msec
  6 ae-62-62.ebr2.Washington1.Level3.net (4.69.134.145) [AS 3356] 4 msec
    ae-92-92.ebr2.Washington1.Level3.net (4.69.134.157) [AS 3356] 0 msec
    ae-82-82.ebr2.Washington1.Level3.net (4.69.134.153) [AS 3356] 4 msec
  7 ae-2-2.ebr2.Chicago2.Level3.net (4.69.132.69) [AS 3356] 20 msec 20
msec 20 msec
  8 ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113) [AS 3356] 20 msec
24 msec 20 msec
  9 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) [AS 3356] 52 msec 64 msec
52 msec
10 ge-9-1.hsa1.Denver1.Level3.net (4.68.107.99) [AS 3356] !H * !H

Completed - Wed Apr 15 15:45:37 EDT 2009

5

Have you been able to in the past?? The site is used for other purposes,
and the front end site that you will see is www.level3.com, not net. So
which one?

Jay Murphy
IP Network Specialist
NM Department of Health
ITSD - IP Network Operations
Santa Fe, New Mexico 87502
Bus. Ph.: 505.827.2851

"We move the information that moves your world."

6

My Level3 connection is working normally, I can reach their site, and I'm
peered in Atlanta.

Jason A. Bertoch
Network Administrator
jason@electronet.net
Electronet Broadband Communications
3411 Capital Medical Blvd.
Tallahassee, FL 32308
(V) 850.222.0229 (F) 850.222.8771

7

In Chicago, traceroutes are dying in the same place (Denver). Peered out of
350 Cermak.

-brandon

8

Yes discovered that then thought about reposting full traceroute
feeds. It was the *.com I can get through now from 4 out of like
8 addresses. Actually on the phone with Level3 right now

9

destination unreachable on qwest out of St.Paul/Minneapolis.
Level3.com does work.

Dylan Ebner, Network Engineer

10

Yea the .com addr is on the same subnet, unless it has been carved into
a /30.

Jay Murphy
IP Network Specialist
NM Department of Health
ITSD - IP Network Operations
Santa Fe, New Mexico 87502
Bus. Ph.: 505.827.2851

"We move the information that moves your world."

11

Tracing to www.level3.net (4.68.95.28) dies at 4.68.94.1 for me as well.

12

Same result from Cogent in Texas. Dying at ge-6-2.hsa1.Denver1.Level3.net

# traceroute level3.net
traceroute to 63.211.236.36 (63.211.236.36), 30 hops max, 46 byte packets
  1 gi1-1.ccr01.aus02.atlas.cogentco.com (38.104.4.37) 0.493 ms 0.393 ms 0.496 ms
  2 te4-4.ccr01.aus01.atlas.cogentco.com (154.54.25.157) 1.040 ms te4-3.ccr01.aus01.atlas.cogentco.com (154.54.25.153) 0.645 ms te4-4.ccr01.aus01.atlas.cogentco.com (154.54.25.157) 0.662 ms
  3 te8-2.mpd01.iah01.atlas.cogentco.com (154.54.1.66) 3.972 ms 3.999 ms 3.926 ms
  4 te4-1.mpd01.dfw01.atlas.cogentco.com (154.54.2.14) 9.289 ms * 9.249 ms
  5 te3-3.mpd01.dfw03.atlas.cogentco.com (154.54.6.94) 9.227 ms te7-3.mpd01.dfw03.atlas.cogentco.com (154.54.6.66) 9.670 ms te3-3.mpd01.dfw03.atlas.cogentco.com (154.54.6.94) 9.806 ms
  6 te-3-2.car3.Dallas1.Level3.net (4.68.110.109) 9.670 ms 9.813 ms 9.431 ms
  7 * vlan79.csw2.Dallas1.Level3.net (4.68.19.126) 19.579 ms 20.438 ms
  8 ae-72-72.ebr2.Dallas1.Level3.net (4.69.136.141) 15.812 ms 15.417 ms 18.290 ms
  9 ae-2.ebr1.Denver1.Level3.net (4.69.132.105) 26.905 ms * 24.321 ms
10 ge-6-2.hsa1.Denver1.Level3.net (4.68.107.131) 24.518 ms !H * 24.644 ms !H

Alex Thurlow
Blastro Networks

http://www.blastro.com
http://www.roxwel.com
http://www.yallwire.com

13

2 dvr-edge-05.inet.qwest.net (72.165.27.181) 27.696 ms 27.688 ms 28.022 ms
3 dvr-core-01.inet.qwest.net (205.171.10.89) 28.010 ms 28.001 ms 27.990 ms
4 * * 67.14.2.89 (67.14.2.89) 50.773 ms
5 xe-8-2-0.edge2.dallas3.level3.net (4.68.63.53) 51.120 ms xe-8-1-0.edge2.dallas3.level3.net (4.68.63.49) 51.107 ms 51.099 ms
6 vlan79.csw2.Dallas1.Level3.net (4.68.19.126) 56.763 ms 37.806 ms vlan89.csw3.Dallas1.Level3.net (4.68.19.190) 33.368 ms
7 ae-82-82.ebr2.Dallas1.Level3.net (4.69.136.145) 35.514 ms ae-72-72.ebr2.Dallas1.Level3.net (4.69.136.141) 44.125 ms ae-62-62.ebr2.Dallas1.Level3.net (4.69.136.137) 44.120 ms
8 ae-2.ebr1.Denver1.Level3.net (4.69.132.105) 50.913 ms 50.895 ms 50.522 ms
9 ge-6-0.hsa1.Denver1.Level3.net (4.68.107.3) 45.675 ms !H ge-6-1.hsa1.Denver1.Level3.net (4.68.107.67) 46.875 ms !H *

14

Yea Jason, level3.net is unreachable....I am sure it is filtering ICMP,
or blocking certain ports, sessions, or services, dude. The level3.net
server is used for other purposes as stated in the previous thread, so
necessarily it is not a question of destination unknown, DNS, or access
for that matter....just you no touchy my server, that's it. I know.

Jay Murphy
IP Network Specialist
NM Department of Health
ITSD - IP Network Operations
Santa Fe, New Mexico 87502
Bus. Ph.: 505.827.2851

"We move the information that moves your world."

15

maybe host problem?
I can reach to www.level3.com, but not www.level3.net.
It seems both are belonging to same subnet.

Brandon Galbraith wrote:

16

Thanks to all who replied. I should have actually reposted
traces but it would have annoyed. I had experienced issues
connecting to Level3.com, couldn't reach lg.level3 from a
wide range of sites.

I have a ticket on "watch" mode of sorts concerning a TDM
switch having gone down in Connecticut. Just seems a little
strange yet another client passing through an interchange
with Level3 is now affected as well. I'll let them deal with
their own provider. As far as I can tell from the ITSP side
of things, I'm alright now. Anyone doing VoIP would have
seen the Level3 hiccup up here (Stamford area'ish).

17

Listen the two are different, level3.com, and level3.net, the two are
colo'd at the same place, thus the reason for the Denver "dying" end
point. It's .net as you can see; try surfing to 4.6 8.95.11 yes,
4.68.95.28, no...It's just how the DNS PTR for the box is set. It has
nothing to do with the naming convention of the routers or other network
elements. Are the two confused?

Jay Murphy
IP Network Specialist
NM Department of Health
ITSD - IP Network Operations
Santa Fe, New Mexico 87502
Bus. Ph.: 505.827.2851

"We move the information that moves your world."

18

As Brandon had stated earlier:

Out of Chicago on RCN onto L3.
Tracing route to level3.net [63.211.236.36]
over a maximum of 30 hops:

  1 1 ms 4 ms 1 ms 10.10.10.1 (My home)
  2 7 ms 9 ms 8 ms 10.20.0.1(RCN interior network)
  3 10 ms 8 ms 10 ms vl2.aggr1.chgo.il.rcn.net [207.229.191.130]
  4 10 ms 7 ms 10 ms tge3-1.border2.eqnx.il.rcn.net
[207.172.19.159]
  5 11 ms 8 ms 10 ms te-8-3.car3.Chicago1.Level3.net
[4.71.101.73]
  6 11 ms 17 ms 19 ms ae-31-53.ebr1.Chicago1.Level3.net
[4.68.101.94]
  7 8 ms 8 ms 7 ms ae-6.ebr1.Chicago2.Level3.net [4.69.140.190]
  8 44 ms 34 ms 36 ms ae-3.ebr2.Denver1.Level3.net [4.69.132.61]
  9 * ge-9-1.hsa1.Denver1.Level3.net [4.68.107.99] reports:
Destination host unreachable.

Trace complete.
Richard

19

* sil@infiltrated.net (J. Oquendo) [Wed 15 Apr 2009, 22:31 CEST]:

Yes discovered that then thought about reposting full traceroute
feeds. It was the *.com I can get through now from 4 out of like
8 addresses. Actually on the phone with Level3 right now

Wait, what? Are you seriously calling Level 3, a global operator of IP and transport networks, to ask them about why their *web site* is down?

Furrfu.

  -- Niels.

20

I didn't know that an unreachable A record indicated that (3) was down :slight_smile:

              http://www.level3.com/lookingglass/

I can reach Denver and www.level3.com, et. Al.

I seem to be able to reach everything. Is the NOC number still
1-877-2LEVEL3?

Martin