The power of water

Stuff happens to everyone, its how you respond. Would your company have
been able to recover as quickly?

http://media.corporate-ir.net/media_files/nsd/tess/consumermail.html

Thanks to folks for pointing this one out.

Good job.

I've had to deal with assorted water problems, thankfully not recently. Some lessons were learned. These water problems were indirectly or directly caused by firefighting.

1. When the data/comm center is on a bottom floor, consider floor drains at
    the time of installation. Might want emergency access to pumps, or even
    wet-dry shop vacs.

2. We had a repeated problem in several moderate-rise buildings where there
    were fairly frequent, although small, fires on upper floors. Water, of
    course, runs downhill. Some things done (this was the US Senate offsite
    computer center)

    a. Put waterproof gaskets on the emergency stairs that had entrances
       from the control room.
    b. When it happened often enough, we mounted rolled-up plastic tarps on
       rollers suspended from the ceiling, with a good solid handle dangling
       down on a rope. During the peak problem period, we'd hear an alarm,
       pull the tarps over the equipment, and then do a fast power-down (not
       necessarily emergency pull--this depended on heat buildup). The tarps
       were a few inches over the top of the racks, to leave some room for
       emergency airflow during cooldown.

3. Consider putting data centers not in the ground floor on the basement,
    but not too high either. Sean, I believe, knows the specific NFPA rule,
    but IIRC you can't have a UPS with acid electrolyte above the third floor.
    So, you can put a data center on the 2nd floor and both allow the UPS
    and have a place for the water to drain.

Corrolary:

If your data center is on the ground floor or in the basement, remember
that the sump pump must be on a UPS/generator protected circuit.

Not that I learned this one the hard way or anything. *cough*

-P.

Over one weekend I was part of a team of folks involved in moving a
voice/data center for a fairly sizeable regional office across the city
in Atlanta. We had pretty much everything moved, installed and working
by Saturday night.

Early Sunday morning as we were tidying up, someone called over the
portable radios that it was 'raining in the data center!' If I remember
correctly, it was a return pipe for the cooling system had come apart at
a poorly soldered connection in the ceiling above the UPS. Soon enough
a good portion of the floor tiles came down and the underneath the
raised floor a nice pond was forming. Since it was a Sunday morning, it
took some time before some building engineers could be called in to turn
off building water. By that time, the 1' raised floor was pretty well
full. We were on an upper floor, maybe 7 or so. I don't remember actual
amount of water, but there was water draining into the third level deep
parking garage below.

Some servers had been permanantly damaged, but most equipment including
some servers (remember NetFRAMEs?), hubs, routers, the cabling system
and a PBX survived to provide service until the insurance company
provided funds for everything to be replaced.

Having good vendor relationships helped significantly in my experience.
Many were overnighting equipment without requiring a lot of red tape.
Having access to a bunch of hair dryers was also useful. :slight_smile:

John