Heh. I am moving about 500 boxes out of there by end of September. Anyone want a temporary job? 
I could use the help.
j.
Jonathan Lassoff wrote:
:They claim in the video tour that they do not have any battery systems on
:the site. They rely solely on the flywheels.
And, there's nothing wrong with that...
Bottom line, regardless of the colo outage, any network that suffered
downtime did so due to their own lack of diligence. The crazy drunkard
headline seems to have been debunked, so this is nothing more than your
random run-of-the-mill outage. The obvious has been stated and restated
throughout this thread. This isn't fodder for nanog, let's move on...
cheers,
brian
Nothing quite like the sound of a whole machine
room spinning down at the same time. It gives you that lovely "oh shit" feeling in the pit of your stomach.<<
Yep. I plugged in my soldering iron and (coincidentally) the whole room at State of Calif., Franchise Tax, EPO'd. Everyone immediately started staring at me of course.
--Michael
I had an issue with exactly that 7 or 8 years ago at Via Networks.. the switchover gear shorted and died horrifically leading to an outage that lasted well through the night (something like 16hours in total). Being on a Friday evening it was difficult to get people on site promptly.
The lesson learned was 'the big switch' .. a huge thing that took the weight of two adults to move it, but did mean that should something similar occur we could transfer the whole building power manually directly to the generator.
I doubt such a beast would scale to the power loads on a large datacentre tho, but then they are generally not on a single grid/UPS feed.
Steve
It appears that 365 is using the Hytec Continuous Power System [
http://hitec.pageprocessor.nl/p3.php?RubriekID=2016], which is a motor,
generator, flywheel, clutch, and Diesel engine all on the same shaft. They
don't use batteries.
Yes. I used to work for the company that originally built the 365 Main
datacenter and remember touring it near the end of the construction phase.
The collection of power units up on the roof was impressive, as were the
seismic isolators in the basement.
But even when you try and do everythying right Murphy usually finds a way
to sneak up behind you and whisper "BOHICA" in your ear. For example, we
had a failure at another datacenter that uses Piller units, which operate
on the same basic principle as the Hitec ones. While running on generator
one of the engines overheated due to an oil-flow problem and threw a rod.
When the on-duty electrician responded to the alarm, there were red-hot
chunks of engine *outside* of the enclosure, and there was a hole in the
side of the unit large enough to stick your arm in. The facility manager
kept the damaged piston as a momento. 
I don't remember whether this was due to a design flaw, improper
installation, or what, but the important points are that (1) this is the
real world and shit happens, and (2) it wasn't until the generator was
worked long enough that the reduction in oil flow caused enough friction
to trigger a catastrophic failure. I.e., there's no guarantee that you
will catch this kind of problem in your monthly tests.
jof@thejof.com ("Jonathan Lassoff") writes:
Well, the fact still remains that operating a datacenter smack-dab in
the center of some of the most inflated real estate in recent history
is quite a castly endeavor.
yes. (speaking for both 365 main, and 529 bryant.)
I really wouldn't be all that surprised if 365 Main cut some corners
here and there behind the scenes to save costs while saving face.
no expense was spared in the conversion of this tank turret factory into
a modern data center. if there was a dark start option, MFN ordered it.
(but if it required maintainance, MFN's bankruptcy interrupted that, but
the current owner has never been bankrupt.)
As it is, they don't have remotely enough power to fill that facility
to capacity, and they've suffered some pretty nasty outages in the
recent past. I'm strongly considering the possibility of completely
moving out of there.
2mW/floor seemed like a lot at the time. ~6kW/rack wasn't contemplated.
(is it time to build out the land adjacent to 200 paul, then?)
jaitken@aitken.com (Jeff Aitken) writes:
..., we had a failure at another datacenter that uses Piller units, which
operate on the same basic principle as the Hitec ones. ...
i guess i never understood why anyone would install a piller that far from
the equator. (it spins like a top, on a vertical axis, and the angular
momentum is really quite gigantic for its size -- it's heavy and it spins
really really fast -- and i remember asking a piller tech why his machine
wasn't tipped slightly southward to account for Coriolis, and he said i was
confused. probably i am.) but for north america, whenever i had a choice,
i chose hitec. (which spins with an axis parallel to gravity.)