San Francisco Power Outage

Michael Dillon writes:

And the stories that the power guy I'm working with tells
about foreign facilities, particularly in middle east war
zones, are really scary...

We fundamentally do not have the facilities problem
completely nailed down to the point that things will never
drop. Level 4
datacenters can, and will, fail. Nothing you can do including
just doing 48V DC for everything are truly foolproof solutions.

A single level 4 datacenter is a Single Point of Failure!

Two of those middle-eastern style facilities is... ?
Has anyone actually kept track of all these data center failures over
the years and done some statistical analysis on it? Maybe two half-baked
data centers is better than one over the long run?

Remember that one 10-12 years ago in (Palo Alto, Mountainview?) where a
lady in a car caused a backhoe driver to move out of the way which
resulted in him cutting a gas line which resulted in the fire department
evacuating the data center, cutting off electricity in the area, and
forbidding the diesel generators to be switched on?

Santa Clara.

I was working right outside the evacuation radius.

Which exchange point was in the building? PB-NAP? CIX? I remember
we had a net-dark event associated, but not which one.

It was a bad day...

The lesson, as you point out, is that geographical redundancy
is sometimes necessary. This is as true for providers as for
datacenter end-users...

-george william herbert