See that big red button on the wall under the sign "Do Not Push This Button!"....
DC 911 outage caused by contractor error
http://wtop.com/dc/2016/08/dc-911-outage-caused-by-contractor-who-pulled-wrong-switch/
WASHINGTON — D.C. is now operating two separate 911 centers after a power outage caused by human error left the nation’s capital without any emergency phone service for almost an hour on a busy weekend night.
A contractor working Saturday night inadvertently pulled an emergency power shut off switch that cut electricity to the 911 phone system and a call routing system at the District’s Unified Communications Center, said the center’s Director Karima Holmes.
“Unfortunately because it was human error we weren’t prepared for it,” Holmes said.
[...]
“Unfortunately because it was human >error we weren’t prepared for it,”
Holmes said.
"But it's elementary!" Watson retorted
alan
"“Unfortunately because it was human error we weren’t prepared for it,”
Holmes said."
I'm glad to know they are prepared for errors by deities and squirrels.
-A
3 of my internet-lifetimes/startups ago, we had this happen when one of the L2
techs was doing their 'rounds' - but had a backpack on. They swung around and
hit the safety cover on the BRS - which got knocked off. They freaked
out a bit while putting the cover back on... and managed to activate it.
Dead silence followed: "Whoa! What wasn't that?!"
(A good story anyway. It wasnt clear from video exactly what happened. More
entertaining in review when sped up and backed with Yakety Sax.)
Hilarity ensued. Customers irated. Procedures were modified. SLAs were paid.
Nicknames were coined.
Could also be that it was a bit too red and shiny:
https://www.youtube.com/watch?v=NITBfc1EOBo#t=27s
/kc
At one point in one data center I dealt with a disgruntled employee hit the UPS disconnect button on the way out.
Same story, procedures modified, cover put over switch with a hammer to break the glass, lessons learned, accounts credited.
Hi,
https://www.youtube.com/watch?v=NITBfc1EOBo#t=27s
"This video contains content from B_Viacom, who has blocked it in your country on copyright grounds."
I love YouTube and copyright regional laws 
alan
Hi,
whilst we're posting YouTube clips..... maybe they'd have been better off keeping
a copy of the Internet
https://www.youtube.com/watch?v=iDbyYGrswtg
alan
Wow, since Im in Canada *WE* are the ones who usually don't get to watch
anything, and no $vendor has gone and made it available in any way to legally
purchase here either. (See stories of proxies being blocked to Netflix US from
Canada - to get the tastier US content unavailable to us - and piracy spiking
back up here.)
oh internet, you great equalizer.
this might work:
https://vimeo.com/126720159
otherwise we're off to indian and russian websites, de jure.
/kc
About the worst that ever happened to me was a security guy's
walkie-talkie setting off an instant Halon drop. Cost about $10,000 to
refill and was fairly exciting for those present. That also cut the
machine room's power.
At least it didn't set off the sprinkler system.
We sat down with the Halon system vendor to find out why that happened
after proving, on a by-passed system, that yes indeed one of these
common walkie-talkies sets the thing off.
File under: More Things To Worry About!
If public transit operators can put a breakable plexiglass shield over the
emergency door opening handle, on every bus, it's not a very high technical
barrier.
Does this mean you could drive around with a (illegal, but not difficult to
build or obtainl) 20W wide band VHF/UHF jammer radio fed into a 1 meter
parabolic dish, aim it at random buildings and set off peoples' halon
systems? Wow.
A very long time ago ("network" involved a fleet of green "wide-band" trucks, hauling tapes to contractors and other offices) the system involved 9 computer centers around the state, built over a period of years, so they had a lot of similarities but some key differences.
Many of them had wide, pneumatic sliding doors between the computer room and the unit-record rooms. Some of the doors had floor mats that would pop the doors open when stepped on (or a cart full of card trays was rolled onto).
Many of them (for what ever reason--I think I know but it isn't relevant here) had large black buttons on each side of the doors, on each side of the wall.
It happened that one had the mats at the sliding doors. but there was an ordinary door near the consoles that had a large black button next to it.
It was in this office that a conversion team was running some stuff that ran for hours (in violation of the rule that if a job ran more than thirty minutes it Must Include checkpoint-restart points every 20 minutes) was nearly finished after running all day and all night (as I recall it).
One of the team left the computer room via the ordinary door, pushing the big black button.
Which was (you saw this coming a long time ago, right?) the Emergency Power Off button.
I do not recall any lessons being learned. At all. The group leader (that refused to include checkpoint-restart) years later was conducting a conversion run in a different system but that had many of the same standards ran a job that ran many many hours in a computer center known for flaky power. Without Checkpoint-Restart. We took a power hit when the run had something like 24 records left to process.
We used to have to drive across a quarry to get to a repeater station (or to one of the cables, which was "aerial" across the quarry), and lots of folks scoffed at the "turn off two-way radios" signs as we approached the area.
I did not scoff.
Back when there were external disk drives with disc packs my boss
said "what does this switch do?" then flipped it.
The next thing that happened was the paper console started printing
as the mounted disc drive had just been powered off on the VAX 750.
oops.
We all had a unscheduled lunch as the system rebooted and the
filesystems where checked.
Mark
Does this mean you could drive around with a (illegal, but not difficult to
> build or obtainl) 20W wide band VHF/UHF jammer radio fed into a 1 meter
> parabolic dish, aim it at random buildings and set off peoples' halon
> systems? Wow.
I'd like to think it's been fixed.
Then again there are those Blasting Zones one goes thru on highways
with big signs ordering drivers: NO CELLPHONE USE! BLASTING ZONE!
Sounds like a good plan.
One day, when I ran the Harvard Chemistry computing facility, I was
greeted on my way in by panicked profs and grad students that the big
VMS VAX (8MB! two memory cabinets! we gave tours!) was behaving
strangely I forget what probably crawling.
A lot of its use was for long-running jobs, week plus, basically
inverting matrices.
So I lept into action, whipped off my heavy winter coat and tossed it
onto one of the big disk drives and wham the system just halted.
Static I guess. Came back ok. Problem solved.
If you get a safety cover for your EPO switch, make sure it is the right
kind of cover. Following an accidental EPO outage, we got a safety cover
that was actually a latch designed to ensure the switch stays pressed
until manually reset. We discoverd this when someone tried to demonstrate
that it was now a lot harder to accidentally press the EPO. (it wasn't)
Tony.