generators, etc....

Even when you have generators, there are still a myriad of things which
can and do go wrong no matter how hard you try, even with trebly
redundant backups, etc. etc.

A fuel supplier accidently puts half a load of jet fuel instead of #2
diesel in your storage tank, which you won't find until it reloads the
run tank from the day tank. Floods happen and the water rises taller
than the 3 story snorkle on the diesels. Generators takes a direct
lightning strike and fries house DC power (even inside shielded
enclosures). A fiber transmission system goes crazy when the control system
is zapped by the lightning strike on the generator. A 200mph hurricane
gust rips a microwave system off the roof (tower and all) and throws it
down on the generators, crushing the exhaust system and the diesels
strangle. (No, I'm' not imagining these.)

We all try very, very hard to make things reliable, but the world isn't
perfect, nor are any of us. BBN will probably find some things to improve
and change the odds next time. Then it will be someone else's turn to
catch the javelins.

I suggest that neither gleefull hand-rubbing nor "obvious" pronouncements
based on partial knowledge about the real situation will seem quite so
appropriate when it's *your* turn to be downfield from the launcher.

  Peace,
  -mo

A fuel supplier accidently puts half a load of jet fuel instead of #2
diesel in your storage tank, which you won't find until it reloads the
run tank from the day tank.

Take samples from the fuel tank and test them every time tanks are filled.

Floods happen and the water rises taller
than the 3 story snorkle on the diesels.

Don't site buildings where there have been floods in the past 500 years
and don't site buildings downstream from dams. If a waterproof building is
built in a flood area, make sure that snorkles rise above the 500 year
flood level.

Generators takes a direct
lightning strike and fries house DC power (even inside shielded
enclosures). A fiber transmission system goes crazy when the control system
is zapped by the lightning strike on the generator.

Is there any way to protect against lightning?

A 200mph hurricane
gust rips a microwave system off the roof (tower and all) and throws it
down on the generators, crushing the exhaust system and the diesels
strangle. (No, I'm' not imagining these.)

When siting a diesel exhaust system make sure that there are no trees,
towers or similar things nearby that could fall on it. If in a hurricane
area, reinforce diesel snorkels and arrange for multiple paths to get air
in.

We all try very, very hard to make things reliable, but the world isn't
perfect, nor are any of us.

Yup. Now everybody else has learned from BBN's mistakes and from your
anecdotes. Thus we make the network more reliable one step at a time.

Michael Dillon - ISP & Internet Consulting
Memra Software Inc. - Fax: +1-604-546-3049
http://www.memra.com - E-mail: michael@memra.com

Greetings..

Even when you have generators, there are still a myriad of things which
can and do go wrong no matter how hard you try, even with trebly
redundant backups, etc. etc.

A fuel supplier accidently puts half a load of jet fuel instead of #2
diesel in your storage tank, which you won't find until it reloads the
run tank from the day tank.

Run the generator under load periodically. Have your gen maintenance
contractor do fuel tests quarterly. Water is just as bad in the Diesel
tank.

Floods happen and the water rises taller
than the 3 story snorkle on the diesels.

30' flood waters.. That's a bad, bad place for a network hub.

Generators takes a direct
lightning strike and fries house DC power (even inside shielded
enclosures).

Not so.. A properly grounded building with external ground halo should be
able to take a direct hit. Take a look inside a cellular telephone site
located in the lightening belt for a first hand understanding.

A fiber transmission system goes crazy when the control system
is zapped by the lightning strike on the generator.

See above..

A 200mph hurricane
gust rips a microwave system off the roof (tower and all) and throws it
down on the generators, crushing the exhaust system and the diesels
strangle. (No, I'm' not imagining these.)

Better to put the gensets inside the building.

We all try very, very hard to make things reliable, but the world isn't
perfect, nor are any of us. BBN will probably find some things to improve
and change the odds next time. Then it will be someone else's turn to
catch the javelins.

By indications apparent, BBN didn't have much in the way of disaster
planning in mind.

I suggest that neither gleefull hand-rubbing nor "obvious" pronouncements
based on partial knowledge about the real situation will seem quite so
appropriate when it's *your* turn to be downfield from the launcher.

I presently manage a very large cellular network located in the a
hurricane belt on an isolated power grid. One of our sidelines is
equipment co-location for ISP's. Before this job, I managed the cellular
switches for one of the Los Angeles carriers (Riots and Earthquakes).

There are many, many steps to accept and follow that will allow you to
maintain service in the worst situations. The're sometimes costly yet
common with the carriers that are in service when the others are not.

I see system outage reports weekly from other markets in my company.
Unfortunately, it is all to common to see that they ignored battery and
generator maintenance and rarely, if ever exercised their backup systems
under load.

Regards

Patrick J. Chicas
Email: pjc@unix.off-road.com
URL: http://www.Off-Road.com