RE: Level 3 TPA routing today?

We have gigE to Level3 in Orlando, and saw something happen
around 1pm today. Customers were complaining of latency and
packet loss, and our traffic to/from L3 dropped noticably if
only for a few minutes.

It sounded like based on Craig's post yesterday they
did have some known issue and were working to resolve
it around that time, ~8 pm EST. I turned our gig link
to them back on around 4 AM EST and have not had any
complaints so far today. Traffic is up compared to
yesterday afternoon too, back where it would normally
be. The tech I spoke to this morning said he had no
knowledge of any issues yesterday, of course my ticket
also had none of the information I sent in to them
yesterday or even a clear description of what the
problem was....

Dave

The speicific issue we were seeing yesterday seems to have been resolved
last night some time and we aren't seeing any instances of it any longer.

--chip

We opened a ticket for today's event and got the same response.

Most likely the issue was communication between the NOC and the service management center. The NOC deals with the core facing events versus the SMC which takes the incoming calls from the customers. In this case the issue was identified and resolved in the NOC.

Perhaps the RFO was not posted internally or whomever you talked with didn't check the status updates or something. Lot's of things could have resulted in a tech not knowing about this type of issue.

Anyway, to tie up loose ends, there was a problem on a core router that was isolated and then repaired in Atlanta.

regards
-Craig

Has anyone noticed significant Level3 transit issues this evening?

[wrl@<REDACTED> ~]$ traceroute ae-23-52.car3.Chicago1.Level3.net
traceroute to ae-23-52.car3.Chicago1.Level3.net (4.68.101.39), 30 hops max, 40 byte packets
[...]
  4 ge-6-1-101.hsa1.Cleveland1.Level3.net (64.156.66.29) 2.627 ms !H
[wrl@<REDACTED> ~]$

[wrl@<REDACTED> ~]$ traceroute vlan79.csw2.Dallas1.Level3.net
traceroute to vlan79.csw2.Dallas1.Level3.net (4.68.19.126), 30 hops max, 40 byte packets
[...]
  4 ge-6-1-101.hsa1.Cleveland1.Level3.net (64.156.66.29) 3.166 ms !H * *
[wrl@<REDACTED> ~]$

[root@<REDACTED> ~]# traceroute www.he.net
traceroute to www.he.net (216.218.186.2), 30 hops max, 40 byte packets
[...]
  3 te-3-1.car1.Cincinnati1.Level3.net (4.78.216.9) 29.279 ms 18.127 ms
     te-3-2.car1.Cincinnati1.Level3.net (4.78.216.13) 20.737 ms
  4 ae-2-5.bar1.Cincinnati1.Level3.net (4.69.132.206) 20.193 ms 24.019 ms 24.360 ms
  5 ae-10-10.ebr2.Chicago1.Level3.net (4.69.136.214) 35.255 ms 35.884 ms 35.595 ms
  6 ae-23-56.car3.Chicago1.Level3.net (4.68.101.167) 28.672 ms
     ae-23-54.car3.Chicago1.Level3.net (4.68.101.103) 30.715 ms
     ae-23-56.car3.Chicago1.Level3.net (4.68.101.167) 32.160 ms
  7 glbx-level3-te.Chicago1.level3.net (4.68.110.194) 97.377 ms 97.080 ms 96.625 ms
  8 * * *
  9 * * *
[root@<REDACTED> ~]#

Some infrastructure blocks are not routed to portions of the network but should not affect ultimate reachability as long as the correct loopbacks and directly connected networks are advertised properly.

regards