One thing constructive I would note that I haven't seen yet
is that while one can imagine many awful scenarios where your
lovely NOC is absolutely leveled (*) by a horrendous accident,
it is much more likely that your NOC will be forced to evacuate
for some "trivial" reason. Perhaps asbestos. Perhaps a gas main
leak in the street. Perhaps the fire alarms just keep going off.
Perhaps the power went off and the fire marshall won't let anyone
stay in the building without elevator power. It might very well
have been the case that had BBN had a backup generator they might
still have been forced to evacuate the building, if the fire dept
was paying attention.
You have to remember that it's harder to keep people in a given
building than it is to safeguard the equipment there. You always
have to be ready to send your NOC operators packing with a cell
phone and a laptop for up to 48 hours, in case the building has
to be evacuated for one of a dozen reasons. You should have
a backup call center with an ACD set up in some other city or
a contract with someone for a backup service.
(*) My nightmare scenario back when I ran the NOC at BBN at
10 Moulton Street was a gasoline tanker turning over in the
street out front. The gasoline runs down the stairwell into
the sunken parking lot under the building. Someone exits
the building into the parking lot and lights a cigarette ...
Next moment my NOC and 300 people are ashes and there's a
60 foot crater where the building used to be.