HE.net, Fremont-2 outage?

Jeffrey Lyon wrote:
> FWIW: Hurricane Electric Internet Services - News - Hurricane Electric Protects Against Power Outages

No date on that 'press release' but the way back machine helps put it
somewhere in 2002. A lot of good this "Alameda" sized generator has done
recently...

Wayback Machine

2MW isn't super huge or anything. I would expect that, given the size
I have been led to believe HE is, they've got a lot more than that now.

My memory is that Alameda isn't huge, but it isn't small either. I'm
not sure .. ah, here

http://www.reuters.com/article/pressRelease/idUS179594+03-Apr-2009+BW20090403

peak 70MW

I'm not sure what the basis for the claim is that a 2MW generator is
"large enough to power the entire city of Alameda" ... 2MW gensets are
common enough in this business and it's possible to burn through 2MW in
a few hundred racks. It isn't *that* much power.

A more conventional comparison might be to something like a hospital; one
of our local hospitals installed a 1.25MW generator which, IIRC, powers
all critical circuits.

http://hhenergyservices.com/electrical/photos.php?category_id=2845&subcategory_id=5027&id=196&number=7

Sometimes it is easier to picture things that way.

... JG

Joe Greco wrote:

Jeffrey Lyon wrote:
    

FWIW: Hurricane Electric Internet Services - News - Hurricane Electric Protects Against Power Outages
      

No date on that 'press release' but the way back machine helps put it
somewhere in 2002. A lot of good this "Alameda" sized generator has done
recently...

Wayback Machine
    
2MW isn't super huge or anything. I would expect that, given the size
I have been led to believe HE is, they've got a lot more than that now.

My memory is that Alameda isn't huge, but it isn't small either. I'm
not sure .. ah, here

http://www.reuters.com/article/pressRelease/idUS179594+03-Apr-2009+BW20090403

peak 70MW

I'm not sure what the basis for the claim is that a 2MW generator is
"large enough to power the entire city of Alameda" ... 2MW gensets are
common enough in this business and it's possible to burn through 2MW in
a few hundred racks. It isn't *that* much power.

A more conventional comparison might be to something like a hospital; one
of our local hospitals installed a 1.25MW generator which, IIRC, powers
all critical circuits.

http://hhenergyservices.com/electrical/photos.php?category_id=2845&subcategory_id=5027&id=196&number=7

Sometimes it is easier to picture things that way.
  
Regardless of generator sizing issues or disparities, if the ATS fails, then no amount of grid or generator power will keep the cabinets juiced up.

Since this is the second time in recent history that this building has experienced a short power outage caused by ATS flakiness, perhaps keeping a small UPS in the cabinet isn't such a bad idea? Even if the distribution switches/routers lose power, at least the servers wouldn't have to go through fscks and DB integrity checks due to unplanned power loss, and the recovery time would be significantly faster.

Hell, for a 5 minute power outage, some of my services were down for 20 minutes. I'll happily take a 75% reduction in downtime for the cost of a UPS
, though clearly redundancy across more reliable datacenters is a better solution.

It sounds like a great idea....until one of those small
UPSes smokes out, triggering the fire suppression (or at least
preaction), possibly also causing the power to be cut to the floor.

  The customer with the small UPS that smoked out generally
does not like receiving the bill for everyone else's equipment
cleaning, too.

  --msa

Although this time it was "short", the outage 5 weeks ago was about 90
minutes.

  Scott

Maybe some of us [[soon-to-be-]ex-]customers of Hurricane can bake them a
cake and beg for UPSes.
Or reliable power.
Or for someone to actually answer the voicemails much less phone calls
within even a few hours of an outage.
Or for there to be at the very least a status page notifying customers that
they are, in fact, screwed, and for how long, and that it's useless to
continue trying to get through at such time.

Who's with me?

> Regardless of generator sizing issues or disparities, if the ATS fails,
> then no amount of grid or generator power will keep the cabinets juiced up.

That is patently false.

Assume N+1 UPS, with each UPS module having its own ATS fed from a utility and emergency bus. Then you can even individually maintain each UPS module and ATS. Bonus and score.

And if it's a really good place, you have two of the above (2(n+1)) and each of your power cords goes to one each.

"Question everything, assume nothing, discuss all, and resolve quickly."

-- Alex Rubenstein, AR97, K2AHR, alex@nac.net, latency, Al Reuben --
-- Net Access Corporation, 800-NET-ME-36, http://www.nac.net --

Regardless of generator sizing issues or disparities, if the ATS fails,
then no amount of grid or generator power will keep the cabinets juiced up.

That is patently false.

At it's root it's true - if an ATS fails the power between the source and
destination will be interrupted.

Assume N+1 UPS, with each UPS module having its own ATS fed from a utility and
emergency bus. Then you can even individually maintain each UPS module and
ATS. Bonus and score.

And if it's a really good place, you have two of the above (2(n+1)) and each
of your power cords goes to one each.

Which doesn't address the failure of one piece of equipment. Of course, if
you're dual chorded from your server through fully redundant switch gear to
multiple, diverse vaults then a single ATS failure shouldn't affect you.

Regards,

Mike

The 2002 press release is talking about the Fremont 1 facility not the newer Fremont 2 facility. Fremont 1 has a fixed power availability to each cabinet of just a single 15A circuit. You can not modify or change that, and if you need more power your option is to add another cabinet. You are not allowed to route power cords between cabinets so you are forever running a single circuit and 80% of your 15A circuit max. The data center was built in a different time.

-- Nevin Lyne
-- Founder / Director of Technology
-- EngineHosting.com

nevin@enginehosting.com wrote:

The 2002 press release is talking about the Fremont 1 facility not
the newer Fremont 2 facility. Fremont 1 has a fixed power
availability to each cabinet of just a single 15A circuit. You can
not modify or change that, and if you need more power your option is
to add another cabinet. You are not allowed to route power cords
between cabinets so you are forever running a single circuit and 80%
of your 15A circuit max. The data center was built in a different
time.

The same is true of racks in most of the suites in the more recent
Freemont 2 facility.

Cheers,

Stef

Yeah, after years of dealing with them all I can say is Best of luck. While we still have some legacy systems in Fremont #1 we moved 98% of our operations out to other data centers back in 2005 because of the same lack of communications even about scheduled events (which to this day I don't believe are posted anywhere). We were rapidly expanding at the time, and given the brush off, so we moved. That was the only way to get good, timely, and details information about things taking place. Flash forward almost 5 years and it seems their flagship Fremont #2 which was just being announced when we started moving, is still the same song, different year...

-- Nevin Lyne
-- Founder / Director of Technology
-- EngineHosting.com

A different time, but obviously not that much different...

Fremont 2 is still limited to either a single 15A or a single 20A circuit
per rack.

They are rebuilding one of the Fremont 2 wings and turning it into a single
area rather than the existing suites, so it'll be interesting to see if
things are done differently there.

  Scott