FYI, Level 3 issues in Dallas

Mailman List Mirror (Read Only)
1

We have some customers unable to access their websites, seeing this on
the way to them:

4 ae-0-11.bar2.Tampa1.Level3.net (4.69.137.110) 0.343 ms 0.423 ms
0.406 ms
5 ae-12-12.ebr1.Dallas1.Level3.net (4.69.137.118) 23.940 ms 23.470
ms 23.426 ms
6 ae-71-71.csw2.Dallas1.Level3.net (4.69.151.137) 24.026 ms
ae-81-81.csw3.Dallas1.Level3.net (4.69.151.149) 24.007 ms
ae-61-61.csw1.Dallas1.Level3.net (4.69.151.125) 23.855 ms
7 ae-3-80.edge10.Dallas1.Level3.net (4.69.145.146) 23.361 ms
ae-4-90.edge2.Dallas1.Level3.net (4.69.145.203) 23.329 ms
ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11) 23.936 ms
8 vlan90.csw4.Dallas1.Level3.net (4.69.145.254) 23.378 ms
vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 23.387 ms
vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 23.963 ms
9 ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11) 23.357 ms
ae-1-60.edge9.Dallas1.Level3.net (4.69.145.16) 22.779 ms
ae-4-90.edge2.Dallas1.Level3.net (4.69.145.203) 62.640 ms
10 vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 24.179 ms 23.297 ms
23.403 ms
11 ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11) 65.617 ms
ae-2-70.edge2.Dallas1.Level3.net (4.69.145.75) 23.286 ms
ae-1-60.edge10.Dallas1.Level3.net (4.69.145.18) 23.553 ms
12 vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 23.303 ms
vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 24.027 ms 24.028 ms
13 ae-3-80.edge2.Dallas1.Level3.net (4.69.145.139) 24.018 ms
ae-3-80.edge10.Dallas1.Level3.net (4.69.145.146) 23.374 ms
ae-4-90.edge2.Dallas1.Level3.net (4.69.145.203) 23.342 ms
14 vlan90.csw4.Dallas1.Level3.net (4.69.145.254) 24.068 ms
vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 23.361 ms 23.348 ms
15 ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11) 24.023 ms
ae-1-60.edge9.Dallas1.Level3.net (4.69.145.16) 22.740 ms
ae-4-90.edge10.Dallas1.Level3.net (4.69.145.210) 23.963 ms
16 vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 23.420 ms
vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 23.400 ms
vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 25.317 ms
17 ae-2-70.edge10.Dallas1.Level3.net (4.69.145.82) 23.985 ms
ae-3-80.edge2.Dallas1.Level3.net (4.69.145.139) 23.476 ms
ae-3-80.edge9.Dallas1.Level3.net (4.69.145.144) 51.762 ms
18 vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 23.416 ms
vlan90.csw4.Dallas1.Level3.net (4.69.145.254) 24.062 ms
vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 24.051 ms
19 ae-3-80.edge9.Dallas1.Level3.net (4.69.145.144) 44.798 ms
ae-3-80.edge10.Dallas1.Level3.net (4.69.145.146) 22.856 ms
ae-1-60.edge10.Dallas1.Level3.net (4.69.145.18) 22.891 ms
20 vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 23.433 ms
vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 25.333 ms
vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 23.367 ms
21 ae-2-70.edge2.Dallas1.Level3.net (4.69.145.75) 23.475 ms
ae-4-90.edge9.Dallas1.Level3.net (4.69.145.208) 23.476 ms
ae-1-60.edge9.Dallas1.Level3.net (4.69.145.16) 64.753 ms
22 vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 24.008 ms 24.043 ms
vlan90.csw4.Dallas1.Level3.net (4.69.145.254) 23.508 ms
23 ae-3-80.edge2.Dallas1.Level3.net (4.69.145.139) 23.505 ms
ae-3-80.edge9.Dallas1.Level3.net (4.69.145.144) 23.958 ms
ae-4-90.edge9.Dallas1.Level3.net (4.69.145.208) 24.058 ms
24 vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 23.429 ms
vlan90.csw4.Dallas1.Level3.net (4.69.145.254) 23.623 ms 25.547 ms
25 ae-2-70.edge10.Dallas1.Level3.net (4.69.145.82) 23.491 ms
ae-3-80.edge9.Dallas1.Level3.net (4.69.145.144) 23.551 ms
ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11) 24.078 ms
26 vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 24.472 ms 24.144 ms
24.080 ms
27 ae-1-60.edge10.Dallas1.Level3.net (4.69.145.18) 28.121 ms
ae-3-80.edge10.Dallas1.Level3.net (4.69.145.146) 22.848 ms
ae-4-90.edge9.Dallas1.Level3.net (4.69.145.208) 24.090 ms
28 vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 23.509 ms 24.056 ms
vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 23.522 ms
29 ae-2-70.edge2.Dallas1.Level3.net (4.69.145.75) 24.161 ms 24.153 ms
ae-4-90.edge9.Dallas1.Level3.net (4.69.145.208) 24.130 ms
30 vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 23.606 ms 23.482 ms
vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 23.580 ms

2

What would be the source and destination?

You got a nice routing loop there.

Greets,
Jeroen

3

Also peering problems in Chicago, had to drop BGP to them until they figure it out.
Reported it 4 hours ago, no good response yet.
Gary

4

like david below, we've been getting reports this morning from customers unable to reach various web sites.

investigating a number of these reports, the one commonality is level3 in chicago. sites are reachable from level3/cincinnati but not level3/chicago.

traceroutes make it one hop past our bgp peer and seem to die there.

for one particular remote web server (node1104.follett.com, and possibly others) traceroutes and pings from level3's looking glass (using chicago as the site) complete successfully, while neither does when ran from my gear through the same site.

5

Appears to have been resolved after seven hours. My ticket just says:

"We isolated the routing issue and resolved it.
The issue was due to a misconfiguration on one our core routers."

Now that the issue is corrected, interestingly enough, my trace goes
from Tampa to Chicago instead of Dallas.

David

6

our issue was resolved around the same time but is apparently now occurring again, since 0638 utc -- routing issue affecting (for us) traffic on level3 between chicago, illinois, and cincinnati, ohio. i imagine it's affecting other locations as well.

/chl

7

fyi:

Appears to have been resolved after seven hours. My ticket just says:

"We isolated the routing issue and resolved it.
The issue was due to a misconfiguration on one our core routers."

Now that the issue is corrected, interestingly enough, my trace goes
from Tampa to Chicago instead of Dallas.

our issue was resolved around the same time but is apparently now
occurring again, since 0638 utc -- routing issue affecting (for us)
traffic on level3 between chicago, illinois, and cincinnati, ohio. i
imagine it's affecting other locations as well.

routing issue experienced from 06:37 - 11:02 utc.

notes from level 3:

"11/20/2014 11:30:19 GMT - After investigating the issue I found that this circuit was affected by an outage on our network. A non-service affecting maintenance had unintended results causing routing problems in the Chicago area. All traffic through the Chicago area was affected by this issue. If you have any additional destinations that are still seeing issues please contact us as soon as possible as the issue may not be completely resolved yet."

/chl