Due to the hot weather in many parts of the US, there have been various
power outages. Some large outages have been caused by severe storms, but
mostly the heat has just overloaded power distribution equipment.
The good news so far is the Net has shown very few disruptions due to
the heat and some multi-day power outages. The major ISP access providers
have been indicating about average numbers of outages in their networks,
i.e. even during "normal" times, there are usually a few tens of
thousands of lines down nationwide.
While its expected for individual customers to go down during power
outages, usually because the customer does not have local backup power, it
is less common for major web sites and co-location centers to experience
downtime during power outages.
MySpace.com reported it was down tonight due to power problems, with a
temporary page up now. MySpace.com is hosted at a few Internet
co-location data centers (Equinix and CWIE).
I'm surprised nobody said anything about the (apparently regional) utility
outage in NoVA on Saturday. Equinix Ashburn was running on generator for
several hours, but apparently the SAVVIS facility down the road a few
miles in Sterling (old Exodus facility) didn't fare so well. The latest
story I heard was that they lost 14 out of 16 chillers and customers had
to send techs in in shifts because it reached over 130F inside.
Come on Sean, this "very few disruptions" stuff is below your usual
standards. The least you can do to help us pass the time in this damn heat
is to recount a few good stories about routers you could scramble eggs on.
I'm surprised nobody said anything about the (apparently regional) utility
outage in NoVA on Saturday.
There have been several around the country. Secaucus, NJ, various
cities near Los Angeles and the SF Bay Area, not to mention the power
quality goes to hell when the grid gets very hot. I've been a bit
surprised we haven't heard about more equipment failures.
Equinix Ashburn was running on generator for
several hours, but apparently the SAVVIS facility down the road a few
miles in Sterling (old Exodus facility) didn't fare so well. The latest
story I heard was that they lost 14 out of 16 chillers and customers had
to send techs in in shifts because it reached over 130F inside.
I've always been a fan of being able to force 100% economizer and chiller
loop bypass emergency operation; it won't keep you "cool" but will help
keep your data center from turning into an Easy-Bake Oven(tm). But that
failure operating mode is rarely part of the standard HVAC programming.
You are probably still going to need to replace some toasted disks
and chips afterwards. There is a reason why NEBS equipment costs more.
Come on Sean, this "very few disruptions" stuff is below your usual
standards. The least you can do to help us pass the time in this damn heat
is to recount a few good stories about routers you could scramble eggs on.
I remember back in the old days, when all we had were AGS+ routers in
wiring closets without any air conditioning .... We had to carry them
uphill, both ways.
there is a funny story of some dial devices on fire, and still passing
packets...
Robert E.Seastrom wrote:
"Christopher L. Morrow" <christopher.morrow@verizonbusiness.com> writes:
Come on Sean, this "very few disruptions" stuff is below your usual
standards. The least you can do to help us pass the time in this damn heat
is to recount a few good stories about routers you could scramble eggs on.
there is a funny story of some dial devices on fire, and still passing
packets...
and an equally funny story of said devices being held up in customs in
a particular european country because they said "TNT" on the outside
of their crates...
I ordered a new personal PC back in March(?) from Lenovo (discount overstock offering). Everything shipped immediately but was delayed in transit, due to a "Live Entity" inspection hold placed on it by the US FDA. The packing list included an item identified as "mouse" (it was right under the item "keyboard"). I'm waiting for nVidia or ATI to come out with a next-gen product named "Nuclear XForce" or "Plutonium Wonder".
-Jim P.
Except if you're in Qwest territory. Apparently they don't put any battery backup at their mini-DSLAMs and such. Every time we lose power, I'm still up, but the DSL signal goes away. Haven't checked dialtone, but I keep meaning too during the next outage.
Now I know it's not exactly fair singling out Qwest, because I'll bet Verizon and others share the same thing, and I'm pretty sure it's just their ADSL service and not the voice service (I haven't checked though) it's still becoming more and more common that as an individual user your connection to the internet, unless you're paying for something other than ADSL or Cable, will be just as affected by local power outages.
Indeed, my RoadRunner connection is the same way. All of my stuff stays up,
but "teh Interweb is broken." I'm guessing that they (DSL/CableCo's) find it
too cost-prohibitive to roll out UPSes to the customer aggregation points.
Suprisingly, my cable TV goes out as well when the power goes, so it might
just be more than the CMTS that's going out.
While hardwired (fiber/coax/copper) aggregation points usually don’t have backup power on them, most cellular towers have either batteries or generators for backup power, correct?
-brandon
While hardwired (fiber/coax/copper) aggregation points usually don’t have backup power on them, most cellular towers have either batteries or generators for backup power, correct?
We see good cable modem connectivity during power outages. Batteries must still be good in the HFC nodes in our area. Verizon POTS service, on the other hand, dies when the power does. The batteries in their SLC units are toast. I’ve got the local police chief off conversing with the Verizon E911 folks to find out why it’s OK to have no 911 service to a large part of town when there’s no power (Verizon repair kept trying to tell me it must be my equipment, despite testing at the network interface).
I am looking at moving telephone services off to Comcast or VOIP because they’re more reliable than Verizon is, in my particular neighborhood.
Depending on the state you live in, the PUC generally requires 4 to 8 hours of dialtone if it’s generated from the C.O. Dialtone generated from SLC may not be explicitly covered under the rules.
Regards,
Frank
Our small operation has outfitted our Calix shelves in the field with a
minimum 8 hours of run time. If they would run low we would re-charge them
with portable generators. We just consider it the cost of doing business.
Frank
Depending on the state you live in, the PUC generally requires 4 to 8 hours of dialtone if it’s generated from the C.O. Dialtone generated from SLC may not be explicitly covered under the rules.
So when they moved about 1/2 of the town from CO to SLC, they got out of the obligation to provide dial tone in power outage? That seems, ummm, interesting. Will be interesting to see what the police chief learns (I’m on a town board that works closely with the police, so he and I get to talk often and get along well). He was quite concerned to learn about the telephone outages that happen.
Thanks for the info. I will follow up with the PUC folks as well and see what they have to say. The Verizon service folks did tell me they expected dialtone to work during power failures, and kept claiming it must be my telephones that are at fault (plugging a very basic, known-functional POTS phone into the network interface says they’re wrong).
In the state of Iowa the PUC is called the Iowa Utility Board (IUB). According to the IUB, each CO must provide 2 hours of battery reserve (not 4, as I said), and if the C.O. serves over 4000 lines, a generator is required.
http://www.legis.state.ia.us/Rules/2003/iac/199iac/19922/19922pp1.pdf
Page 26, 22.6(5)
Frank
Now, where was youtube and portable digital camcorders back when these
things were going on? This is an excellent application for all that super
surveillance the DHS wants to do all the time! life-bloopers.com!
Sean,
Can you elaborate on what you mean by " force 100% economizer and chiller
loop bypass emergency operation"
Thanks,
Sam
My assumption is that it means “it isn’t going to keep things cold, but it will keep the air flowing to prevent a ‘server sauna’”.
-brandon
Brandon Galbraith wrote:
My assumption is that it means "it isn't going to keep things cold, but it
will keep the air flowing to prevent a 'server sauna'".
> From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of
> Sean Donelan
> Sent: 24 July 2006 13:27
> To: nanog@merit.edu
> Subject: Re: Hot weather and power outages continue
> I've always been a fan of being able to force 100% economizer and
chiller
> loop bypass emergency operation; it won't keep you "cool" but will help
> keep your data center from turning into an Easy-Bake Oven(tm). But that
> failure operating mode is rarely part of the standard HVAC programming.
Sean,
Can you elaborate on what you mean by " force 100% economizer and chiller
loop bypass emergency operation"
Thanks,
Sam
When I was in charge of such things, there was a way to circulate evaporator tower water n the chilled water loop to remove some of the heat, if you had enough power to run the two pumps.
It's your neighborhood distribution amp going out, or possibly a smaller amp between the neighborhood amp and your house (...or even one at your house...).
With the Digital Phone and similar rollouts, though, they've gotten better about putting battery backups on at least the neighborhood amps...they realized it was pointless to put batteries in the MTA if they don't put them on the amps..
...d