If anyone else lost power at Internap's Fisher Plaza facility in Seattle this afternoon and have any info related to exact duration or cause please hit me back off-list.
Cheers,
Jonathan
If anyone else lost power at Internap's Fisher Plaza facility in Seattle this afternoon and have any info related to exact duration or cause please hit me back off-list.
Isn't this like the 2nd or 3rd event at Fisher of late?
Any answers Jonathan? Has Fisher owned up to the cause yet? I did not hear of any local grid issues (we're certainly NOT broiling up here like the rest of the country.)
The *rumor* I heard was some sort of bypass switch failure. Genset(s) running, but no juice being delivered... then the UPS' ran dry. Outage lasted 45-90-some minutes depending on where in the facility your gear was.
The above is all hearsay though, from some INAP customers I ran into yesterday.
--chuck
Hello:
If anyone else lost power at Internap's Fisher Plaza facility in
Seattle this afternoon and have any info related to exact duration
or cause please hit me back off-list.Isn't this like the 2nd or 3rd event at Fisher of late?
It's the second event at Fisher Plaza - the last one happened near the end
of May.
Any answers Jonathan? Has Fisher owned up to the cause yet? I did not
hear of any local grid issues (we're certainly NOT broiling up here
like the rest of the country.)The *rumor* I heard was some sort of bypass switch failure. Genset(s)
running, but no juice being delivered... then the UPS' ran dry.
Outage lasted 45-90-some minutes depending on where in the facility
your gear was.
It was a breaker in the main bypass from city power to the generators. The
breaker failed to close so the generators happily fed power to nowhere.
Then, everyone's UPS failed and down we/they went. The outage lasted
approximately 26 minutes. The kicker is, this was the same component that
failed in the first outage.
I'm not sure if there's been an official announcement, per se, but I think
Fisher Communications will have to report to the FCC because all of their
radio and TV was down for much longer than the outage.
Regards,
Mike
Michael K. Smith wrote:
It was a breaker in the main bypass from city power to the generators. The
breaker failed to close so the generators happily fed power to nowhere.
Then, everyone's UPS failed and down we/they went. The outage lasted
approximately 26 minutes.
Nobody checked to make sure that at least one of the UPSs showed a status of "ONLINE" instead of "ONBATTERY"? Were there no UPSs configured to alert during continued and extended PF? Surely people didn't just trust the sound/vibration of the running generator.
-Jim P.
As far as I've heard there was not personnel manually interacting with the power system from the point that grid power was lost to when the UPS's finally gave out (it was a Sunday afternoon). I'm assuming that if someone was there engineers would have realized that the gen sets juice was going to /dev/null .
I believe Fisher Plaza now has maintainence staff on-site 24x7 until all the parts are fixed.
If anyone has a set of questions to ask colo providers about their power system best practices, or more simply background information about how facility power systems work for the 'smaller fish' here that have less exposure to running generators, etc... I'd greatly appreciate a reference. This event has reminded me that I can't be complacent and blindly hope that the people downstream from me (colo providers, etc...) are on their game 100%.
Jim Popovitch wrote:
Hello Jim:
Michael K. Smith wrote:
It was a breaker in the main bypass from city power to the generators. The
breaker failed to close so the generators happily fed power to nowhere.
Then, everyone's UPS failed and down we/they went. The outage lasted
approximately 26 minutes.Nobody checked to make sure that at least one of the UPSs showed a
status of "ONLINE" instead of "ONBATTERY"? Were there no UPSs
configured to alert during continued and extended PF? Surely people
didn't just trust the sound/vibration of the running generator.-Jim P.
Indeed. The problem was there wasn't an engineer on site who could manually
trip the breaker. They got onsite pretty quickly, but not quickly enough to
trip the breaker in time to avoid an outage. So we watched the UPS drain
all the way down which took about 24 minutes in our case. So close, yet so
far.
Regards,
Mike
AFAIK, you don't need to have to have someone onsite to trip a breaker....if
it doesn't do it automatically, there are a multitude of SCADA systems
available to manuaully flip them on. Unless, of course, the
electromechanical components that physically flip the breaker over have
failed.
Frank
Hello Frank: