Looking for advice on datacenter electrical/generator

Speaking on Deep Background, the Press Secretary whispered:

We have about 10-15 minutes of UPS capacity.

I.e. -- long enough to:

a) get metal spinning
b) realize "Oh SHIT" and start "sync sync sync halt" on every box.

The important thing with a generator is to make sure you test it
regularly. Ours (a diesel at one POP and a natural gas at another)
automatically run each week and alert us to any problems. Then, to test
the full system (transfer switch, etc.), we "pull the handle" (shut off
utility power) once a month and make sure that everything works as it is
supposed to.


You must test the things UNDER LOAD to be sure all is well. ISTM
the SOP is 30min to an hour, but that can be done less frequently
than starting and warmup.

Further gotcha's:

Diesel fuel is a Petri dish. Weird bugs grow in it. [Call Tom Ridge!]
If you don't have the right additives, your filters SHALL clog
on same when you most need same.

Cooling systems/maint -- please recall the NYC telehouse {was it
60 Hudson?} where post 9/11 the generator went down on overheat
because the radiator was clogged...

Most Diesels have a "day tank" with underground storage for the
real supply, and a transfer pump. It may sound trite, but be
sure that pump is itself an emergency load, otherwise..... [Any
inference that I may have in fact run into this issue myself on
Somebody's installation is unproven rumor.]

additionally, in cold climates, diesel fuel can and will gel if it isn't
the right mix.

the fuel delivered in the summer may not have the right additives. some
vendors of diesel fuel do a better job with their winter mix than others.


Cooling systems/maint -- please recall the NYC telehouse {was it
60 Hudson?} where post 9/11 the generator went down on overheat
because the radiator was clogged...

That was 25 Broadway (Telehouse USA). The air intake filter for their generator clogged from all the WTC dust in the air. Pretty rare circumstance...

There were some initial reports that this was the case, but I have
never seen the final report from Telehouse what the final analysis
found. If you remember there were also reports about a water pump
failure, raditor clogging, etc.

I know the clogged air filter was what we heard for 32 Old Slip. Since
the telcos in that building (ATT, Focal, GNaps) only operate their own
battery strings and not generators, they relied on the building generator.
They didn't have any pull in getting maintenance folks back into the area
to work on the genset. When power was restored initially it was not
enough to handle cooling, so stepping out onto the 5th floor landing was
like entering a blast furnace.

At Telehouse, this is what I dug out of my archives, presumably from John
Shields. Some things to note are that the initial failure was due to the
facility running out of fuel. Ooops. Once refueled, the water pump
supposedly died. If I recall correctly there were two other gensets
running for the 6th floor, which at the time was basically unoccupied. It
would have been a better design perhaps to tie both floors together and
get N+1 redundancy on the gensets. But Telehouse has a poor track record
with whomever does their electrical work. Before we moved to 25Bway, we
went through a period where a transfer switch would fail during tests at
33 Whitehall. If I recall correctly this happened at least 5 times in one