We just saw one of our gig-e circuits to the Westin bounce three times and another just go flatline in the past hour. On hold for a couple of NOCs at the moment trying to figure out wtf is going on. Anyone have solid info?
--chuck
We just saw one of our gig-e circuits to the Westin bounce three times and another just go flatline in the past hour.
Answering my own question I know, but the OnFiber/Qwest guys I spoke to informed me that they heard the Westin had some sort of backup power scheduled maintenance go wrong. That was the 3 bouncer. I still wouldn't mind independent verification of that.
The flatliner was XO, and it seems that it may not even touch the Westin (instead goes to 1000 Denny.) I still don't know what happened there. It did come back after 86 minutes of eerie silence. Anyone else with XO circuits see anything odd tonight?
--chuck
Note to XO NOC: Your hold music is *awful* and on a way too short loop. It is bad enough when it is bad music, but to hear it over and over and over... for well over an hour, is customer torture.
We too had probs. I saw only two outages, one around 8PM PDT and one
around 9:45PM PDT. I called during the first one, and the people I talked
to were obviously in a state, and I had trouble hearing anyone, as they
were in an extremely loud part of the data center or something. From what
I could understand through the noise of some really loud fans or a
generator, there was a power test of some kind, and a generator flaked or
something. I've requested more detailed info, but have yet to receive it.
From what I understand, it affected more than just one provider.
--Rick Kunkel
there was a power test of some kind, and a generator flaked or
something. I've requested more detailed info, but have yet to receive it.
From what I understand, it affected more than just one provider.
Here's the RFO I saw late last night:
During a planned generator test that was being performed by Westin building maintenance, a bad battery resulted in the inability of the back up system to hold power during the transition from commercial power to generator power. This resulted in a service interruption that lasted approximately 7 minutes (22:00PT - 22:07PT).
Before the problem could be isolated, building management switched the power back from generator to commercial power resulting in another service interruption that lasted approximately 8 minutes (23:43PT - 23:51PT).
Still trying to find out what happened with XO.
--chuck
Our Minneapolis BGP session to XO dropped at Jan 26 23:08:03 CST (21:08 PST) for 52 minutes. A colo customer in a second XO facility in Minneapolis appeared to be unreachable for the same period. The reported cause was a problem during planned maintenance, requiring reboot of multiple routers. I was wondering whether XO might have hit a snag while responding to the Cisco advisory. If your XO link doesn't touch Westin, then perhaps the timing was coincidence?
--Bradley Urberg-Carlson