Hello,
Woke up this morning to a bunch of reports of issues with connectivity had to shut down some Level3/CTL connections to get it to return to normal.
As of right now their support portal won’t load: https://www.centurylink.com/business/login/
Just wondering what others are seeing.
Its dead JIM. I also can’t get in starting around 7 est
Same. Also, as reported on outages list, what’s even worse is that they appear to be continuing to propagate advertisements from circuits whose sessions have been turned down. I validated ours still were via a couple looking glass portals. Down Detector shows nearly every major service provider impacted.
They’re not reachable so who knows if they’re even working on it. I feel like they’ve been cutting heavily on the network ops side in recent years…
Hello,
Woke up this morning to a bunch of reports of issues with connectivity had to shut down some Level3/CTL connections to get it to return to normal.
As of right now their support portal won’t load: https://www.centurylink.com/business/login/
Just wondering what others are seeing.
Yeah, I am still seeing them announce our Ips even though we’ve shut down our sessions with them.
Well, When I tried calling I got a fast busy, so that’s nice.
BGP sessions randomly flapping or having routing issues in different cities since ~5AM EST
Finally got through on their support line and spoke to level1. The only thing the tech could say was it was an issue with BGP route reflectors and it started about 3am(pacific). They were still trying to isolate the issue. I’ve tried failing over my circuits and no go, the traffic just dies as L3 won’t stop advertising my routes.
Started about 5:05am central, started clearing up for me about 7:15am. My route from ATT in Chicago is still going through NYC to get back to Chicago but at least packet loss isn’t 70-100% anymore.
I also tried turning down sessions and still was seeing stale announcements on other LGs.
I am seeing some odd traffic deflections in the EU via 3356, but nothing in the US so far.
Some scattershot oddball reports landing in our NOC, but nothing conclusive.
Saw the flapping in Cleveland but not in Cincinnatti or Ashburn…
Flapping in Miami, Dallas, Atlanta, Los Angeles, Seattle and San Jose. It is also affecting some data centers in Europe too. but haven’t seen flaps there, just suboptimal routing.
That site seems to be just for their cloud products, is there one of these for their actual network?
We've been on hold for more than an hour trying to get an update.
We see the same behavior where they continue to announce our blocks
despite all the interfaces to them being hard down.
Scott Helms
Multiple BGP sessions with Level3 (DIA) started flapping at approx 03:00 Pacific:
Aug 30 03:05:13 rtr02 Rib: %BGP-3-NOTIFICATION: sent to neighbor 4.35.X.Y (AS 3356) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes
Aug 30 03:05:13 rtr02 Rib: %BGP-5-ADJCHANGE: peer 4.35.X.Y (AS 3356) old state Established event HoldTime new state Idle
Aug 30 03:07:37 rtr02 Rib: %BGP-5-ADJCHANGE: peer 4.35.X.Y (AS 3356) old state OpenConfirm event RecvKeepAlive new state Established
Aug 30 03:15:38 rtr02 Rib: %BGP-5-ADJCHANGE: peer 4.35.X.Y (AS 3356) old state Established event HoldTime new state Idle
Aug 30 03:17:15 rtr02 Rib: %BGP-5-ADJCHANGE: peer 4.35.X.Y (AS 3356) old state OpenConfirm event RecvKeepAlive new state Established
Aug 30 03:19:55 rtr02 Rib: %BGP-3-NOTIFICATION: sent to neighbor 4.35.X.Y+52091 (proto) 6/7 (Cease/connection collision resolution) 0 bytes
Aug 30 03:20:11 rtr02 Rib: %BGP-3-NOTIFICATION: received from neighbor 4.35.X.Y (AS 3356) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes
Aug 30 03:20:11 rtr02 Rib: %BGP-5-ADJCHANGE: peer 4.35.X.Y (AS 3356) old state Established event RecvNotify new state Idle
And incoming traffic from AS3356 and AS209 both dropped to very low volumes.
Just to confirm we're seeing this on AS3356 and not AS209, correct?
We have links to both and shut down AS3356 which seems to have cleared
"most" of the problems.
They’re not reachable so who knows if they’re even working on it.
Gonna go out on a limb here and assume that a lot of phones were ringing and people are in fact working on whatever it is. 
I don’t think it is anywhere near back to normal but I can get to 2-3 more sites than I could when I got called in.
I’m over in MA in a CL building, it’s very much still broken. I shut down the interfaces to CL and now just using Comcast.
How is that acceptable behaviour? I shall remember never to make a contract with these guys until they can prove that they won’t advertise my prefixes after I pull them. Under any circumstances.