Anyone have any insight to the nationwide CenturyLink issues/outages today? Just wondering. Know for sure that our connections to them from Florida, Iowa, and Washington State are all affected. Voice and data.
Steven Naslund
Chicago IL
Seeing the same from here in Montana. Internet traffic is seeing routing issues through them on the circuits that are up.
I’ve heard it’s a widespread DWDM issue. They have crews in Kansas City, New Orleans, and Atlanta
Seeing it in Colorado as well.
National outage since 4:33 am this morning…
Seems like things have stabilized as of about an hour ago for us.
We see slow recovery. Dallas data service came back up, Dubuque voice service still down.
Steven Naslund
Chicago IL
Today Level3/CenturyLink is having a major outage. This is a great example of how much diversity matter. How important to plan your network right. These type of outages can happen to anyone. this is time to understand better how important it is to plan your network resiliency and diversity. This is also time to unite as telecom industry to learn from this big outage that impacted many. I hope technical team will share their learnings in NANOG or blog so others will learn from it.
Good luck Engineering/Ops teams who are working on restoration.
Thank you
We have interwebs in Boise! At least, on the biz fiber w/ BGP.
720892 ipv4 routes, 62273 ipv6 routes.
Maybe light at the end of the tunnel...
latest we got from Centurylink was at 10:45pm CST.
Our engineers and technicians have identified the network element that has affected our customer services. Services are restoring, and the current estimated time for full recovery is four hours.
CenturyLink will be conducting an extensive post-incident investigation and root cause analysis to provide follow-up information to our customers impacted by this event. We will provide one more Sales Alert from this address once services are fully restored.
a message of 131 lines which said:
CenturyLink will be conducting an extensive post-incident
investigation and root cause analysis to provide follow-up
information to our customers
Is this problem also responsible for the 911 outage? If so, the
post-mortem analysis is not useful only for CenturyLink customers but
for everyone on the west coast.
Yes, there were 911 services affected. The latest word from C-link as of 1:46PM mountain is that all 911 services are restored where they are the provider. I'm not 100% sure if that's system-wide, or just my area in the northwest, however.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
And the other latest news is that the FCC is investigating the CenturyLink outage:
https://www.theinternetpatrol.com/fcc-investigating-centurylink-outage-says-unacceptable/
Ouch. Feel bad for the guys on the ground at C-link. Not a fun 24 hours.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
Looks like most time.nist.gov servers (3 x NIST sites on AS49) are
single homed on CenturyLink, anyone noticed NTP issues yesterday?
Looks like we lost sync intermittently across several of their servers last night. Cleared up around midnight mountain for me.
Let's chip in and get some carrier diversity for those guys 
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
Yep.
We are required by FINRA to verify that our clocks on our trading systems are within a certain tolerance of NIST time. We are still seeing issues with 3 of the NIST servers this morning. Since NIST is on a skeleton crew anyway due to the government shutdown, I don't expect any resolution shortly.
I have GPS-based Stratum 1 NTP appliances in my network, so I wouldn't
see any issues. I suspect many other operators are in the same situation.
We have two stratum-1 servers synced with GPS and a PTP feed from a provider that also provides PTP to market data systems, but we still have to monitor drift between system time and NIST time. Don't ask for the logic behind it, it's a regulation, not a technical requirement.
The telephone companies (I'm looking at YOU Verizon!) are bringing this
situation onto the community. I can see the FCC NPRM now:
"What percentage of E911 terminations is being serviced over VoIP with
carrier-based network switching, or third-party network switching,
interfaced to the PSTN?
"How many emergency service areas terminate E911 VoIP into an
on-premises device like a Cisco voice router with outward-facing T1/E1
cards, or even outward-facing DS0 ports?"
This is just the flip side of the problem with VoIP on the consumer
side, not being able to easily associated a location with a 911 call
without significant help from the calling device. Think cell phones on
the one hand, and the ubiquitous Cisco VoIP desk set on the other.
(Personal note: I have two copper-based DS0 lines here at my home
office. And I'll keep them until Nevada Bell pries them out of my cold,
dead hands. Now, those lines do terminate at a neighborhood SONET ring
fiber terminal with a battery, but it's not my worry. Fax works fine --
which you can't say for VoIP connections.)