BBN (GTE) Suffers another major power problem.

>If you're in a 24x7, customer-driven market, you should have either:
>a) enough go-juice (via fuel contractor) to go until the grid is on-line

Did you read the previous email from jhawk? There is an
extremely large generator at one of the affected facilities
(read: cogeneration plant) but no amount of fuel helps if
the plant is damaged in the explosion.

Unfortunately, the previous mail from jhawk went to BBN's local
NANOG exploder and not to the public NANOG list :frowning:

As long as I'm taking a point to clarify publically, I'll try to be complete.

p.s. No, we've have not deployed redundant power plants (yet)

BBN has a number of facilities, some of which are colocated in telco
facilities, and some of which are not. Those that are in telco facilities
have (as you would, of course, expect) hours-to-days of backup power
in the event of a catastrophic failure.

Those that are not, however, do not.

BBN has major facilities colocated with both MIT and Stanford, both of
which have co-generation power, which is expected to be available
should utility power failure.

Of course, it is well-recognized, and recent history serves to enforce that
quite well, that this doesn't help for many kinds of failures, especially
if the distance between the backup power source and your equipment
is large, with multiple possible points of failure.

Sean Doran is perfectly correct when he points out that market economics
are central here. When engineering redundancy, there's going to always be
some number of customers that you are willing to let suffer in the event
of a catastrophy. In some cases, maybe that number is zero. In other cases,
perhaps it's some number of percentage points. It clearly doesn't make sense
to engineer for the case where a disaster takes out hundreds of customers,
leaving only one remaining :slight_smile: I'm not prepared to address what our outlook
on this is, of course, however we generally feel that facilities with
co-generation are less likely to be in horrible power positions than other
facilities. Maybe not "less by a lot", but certainly "less".

Unlike the Stanford case, however, while we did not have on-site generator
power, we were certainly more prepared for it, and it in fact arrived,
largely coincident with the return of utility power. That's the way it goes.

Attached are some exerpts of summaries from the MIT which you might
care for if you want nitty-gritty details.

Folks on nanog who commented that having an on-site generator for outages
being the way to go are certainly correct -- if your marketting economics
justify them. I can't really speak to ours or our future plans, however.

To answer some other questions, BBN does indeed have multiple POPs in the
Cambridge/Boston area, and this only affected one of them.

The suggestino that it was "physically impractical" to power the equipment
in question (Chris Masto) is simply incorrect.

Matt Ringel's comment that the power was out from 1730 through 2245 (EDT)
is not completely correct (see below), as there was quite a long period
of restored power in the middle, and the second outage was more widespread
than the first. Note, also, that UPS power caused our equipment to run for
quite a while longer than the numbers below indicate, but I don't have
precise figures on that handy.

The assertion that we had emergency generator backup at this location
(Eric Osborne) was incorrect; a generator was brought in as the outage
drew to a close, and co-generation was normally expected to be

The initial mail (Patrick Chicas) implied that power to the building
housing our POP was lost -- in reality the problem was roughly
half of the city of Cambridge, MA.

I think that just about covers all the bases.