RE: Building a NOC

Pocket protectors? Slide rules? Any other items nerd/engineers carry
in their shirt pocket that falls out when they are head-first into the
raised floor.

We've found socks and cigarette butts. Needless to say the perp is no
longer with us :slight_smile:

Like most disasters, you may not be able to prevent flooding, but you
can mitigate some of its affects. Hit the E.P.O. and unfurl the plastic
tarps you have stored & handy for just this emergency.

We also put drip pans which cover the entire ceiling over the data center -
we were burnt 2 years ago in a pre-war office building in NYC. (avoid space
with wc's above!) The pans drain outside of the machine room into a
high-traffic hallway (with a drain). Any drips get noticed very quickly,
and we don't have to test the water alarm.

I guess that flooding by the mains water is seriousl worth thinking about. For
example we just realised that because the "mens room" (and the "ladies)
lavatory cistern overflows go into the room (they are in the middle
of the building) if something goes wrong we have a little flood. So
any mains water installations (dishwashers, kitchens, showers) are
a potential hazzard - solution a) install mains water cutoff valves
with sensors in the floors b) get everyone not involved in the NOC
out of the building. We are just doing both! I suppose that all
the wiring could be in the ceiling although that would not help
in multistorey building. Rambling a bit but what about the film Inferno with
all that water stored at the top of the building. :-))


The anecdotes in this discussion thread, I think, are definitely worth
keeping somewhere, although I'm not sure of the venue. Some of us had been
vaguely chatting in Albuquerque about a documentation part of the NANOG web

But some non-obvious threads are surfacing in this discussion, at least
non-obvious if you haven't been there. Restating a few of the less obvious

  -- Define the NOC's function

  -- Define the NOC's audience, including people that will have no useful
     function but do have political influence that MUST be satisfied

  -- When considering a physical site, do a careful and paranoid threat
     of the planned space, the building, and its immediate environs. You
     will want to identify possible hazards including fire, flood, etc.

  -- Consider how you will get large equipment in and out of the site,
     especially those that might need emergency replacement. What if the
     building is on limited power and the elevator is down? Some buildings
     have to have large equipment lifted in with a crane or helicopter.
     Air conditioning equipment or major power supplies are examples of
     very hard to handle components.

  -- As part of the site survey, understand how electrical power and
     communications feeders come into the space and building. Are there
     alternatives for redundancy?

  -- Will there be a backup NOC, even consisting of a node into which staff
     can dial?

  -- Do a thorough electrical requirmements plan, and be sure the grounding
     system is up to current practices. When devices have multiple power
     supplies (e.g., Cisco 7000/7500), be sure they are plugged into circuits
     on different breakers.

  -- Think through who will be planning, installing, and inspecting signal
     cabling. If it's a union shop, or especially if non-union personnel
     will be doing any work in a union area, see if you can get the shop
     steward on your side.

  -- Be sure cellular/cordless phones will work in all your wire closets,
     or be sure there are voice jacks where a telephone set can be plugged
     in to coordinate testing.

  -- When planning cable runs, be sure you can access critical components
     that otherwise might be buried under a heavy mass of cables. Plan the
     runs so they will not interfere with cards sliding in and out of chassis.

  -- Beware of snakes, floor drains, shock hazards, etc., under raised floors.

  -- Consider backup facilities for critical people. If there was a major
     disaster that isolated the site, but still let some of its function work,
     where will staff sleep? Are there emergency food supplies (not a bad
     idea even beyond major disasters -- if people are working around the
     clock, it's good to have alternatives besides pizza, if that is

Especially if you are in earthquake or amorous rat country, think about
catastrophes that can knock out the NOC but leave parts of your network


The notes below will be a keeper for our organization for some time.

We are in the process of building a NOC. Anyone interested in hearing
the progress (or lack of) thus far, let me know and I can email some
of the information direct.

Thanks to those who contributed to the list below, I will be using it
much from now on.