RE: AT&T network recovery preparations

Mailman List Mirror (Read Only)
1

Along those lines:

I don't know if all of the colos are reacting this way, but at AT&T's
datacenter @ 811 10th Avenue in Manhattan, the security there has gone into
crisis mode. The steel doors are down to the main entrance - everyone must
enter through a side door, where your bags are thoroughly searched and IDs
checked to the Nth degree.

Something that's always been interesting about that facility as well are the
"Duress lights" at many of the corner junctions near security outposts.
There's a sign right next to these lights that reads "If the adjacent light
is lit, a security condition may exist around this corner. Entering may
constitute a serious risk to your person", or something like that.

Nice to know that they take security seriously.

Matt

2

Some providers put out press releases, others haven't. But I
believe colo providers which normally have a lot of security and
those which normally don't, took extra care with their security
last week.

But it does bring up another issue. As far as I know, Exodus is
the only colocation designated a "national infrastructure asset.
http://www.thestreet.com/tech/internet/1090327.html

I have no idea what that means in practical terms. But I did want
to raise the question. As an industry, we aren't vertically integrated.
Instead its an inter-linked set of dependencies. Its not like the old
days when the government could just call up Ma Bell, and find out what is
happening.

Carriers are tenents in facilities operated by others. All the colo
operators work very hard to maintain service, and have contigency
plans for foreseeable disasters. But when the unforseen does happen,
should we have pre-planed responses with federal authorities? Do
we need to include ISPs and the Internet in existing civil defense
plans? And finally, should additional facilities be designated as
national infrastructure assets?

3

But it does bring up another issue. As far as I know, Exodus is
the only colocation designated a "national infrastructure asset.
http://www.thestreet.com/tech/internet/1090327.html

I have no idea what that means in practical terms. But I did want
to raise the question. As an industry, we aren't vertically integrated.
Instead its an inter-linked set of dependencies. Its not like the old
days when the government could just call up Ma Bell, and find out what is
happening.

It is my understanding that the US Government has "national infrastructure"
inside Exodus facilities. As for what that means precisely, it's anyone's
guess. Security at the Exodus facility in Atlanta has always been pretty
good, but I wouldn't call it stellar.

I know that the USG at one point had significant infrastructure inside the
Atlanta facility. I recall seeing some serious looking cages around routers
and Sun gear, but I couldn't say for sure; it was a long time ago.

Tim

4

USG has assets inside a number of co-lo providers. One of the companies
I work with happens to provide some under sub-contract to DOE. No
details available as to where it is however. :-}

iii

Timothy Brown wrote:

5

What it means is that the US Treasury, and specifically the IRS, depend
heavily on those facilities to process electronic transactions
including electronic filing of tax returns. Obviously "national
infrastructure" would include the means of financiing the Government.

-Jim P.

6

Various departments and agencies of the US, state and local government
have assets in lots and lots of co-lo providers. I had several circuits
designated for "continuity of government."

However, no other provider has been designated by the White House as a
"National Infrastructure Asset." Sprint and Worldcom provide FTS2001,
but they haven't been designated "national infrastructure asset" by
the White House. Other than an Exodus press release, I can't find any
record of the phrase in the Code of Federal Regulations, the Federal
Register, or any official White House public record in the govdocs
database. I don't know if being designated a "national infrastructure
asset" is the same as the 50th wedding anniversary greeting the White
House sent my grandparents (which looks very impressive framed on
the mantel), or if it actually has some practical effect.

But my question wasn't really a debate over the phrase "national
infrastructure asset," but whether there are any other assets the
Internet community believes should be included in pre-planned responses.

7

I think the effect on the Internet & Telecoms infrastructure
(as opposed more important things such as human life)
would have been far greater had the 2 NY planes hit 60 Hudson and
111 8th Avenue. These buildings are significant PoF in NY, and
NY itself is pretty much an SPoF as far as transatlantic communication
is concerned. A preplanned response would be useful here. Not having
the PoF's would be more useful.

8

111 8th Avenue. These buildings are significant PoF in NY, and
NY itself is pretty much an SPoF as far as transatlantic communication
is concerned.

Well.. I would declare the SPoF rather in the whole area. Fibre trails are
coming in from the seaside mostly near Mineola.

A preplanned response would be useful here. Not having
the PoF's would be more useful.

The question has to be answered by the Fibre Owners. They can solve the
issue. If they do is another question

--jan

9

On Mon, Sep 24, 2001 at 01:56:46AM +0200, Jan-Ahrent Czmok typed:

Well.. I would declare the SPoF rather in the whole area. Fibre trails
are coming in from the seaside mostly near Mineola.

Last I checked, Mineola was right in the middle of the Island :slight_smile:

Most fiber comes via Shirley in the Town of Brookhaven.

10

When refering to points of failure, its important to note that not
just economic centers are vulerable. I mean, ever notice how much
fiber goes through kansas city?

11

Therefore i would declare all important exchange points, cities and anything
which has more than 1 telco there as volunerable.

--jan

12

I only visited NY once yet (would one it more often - nice city) when visiting
Lightning Internet Services.

But i guess, You're right...

--jan

13

Mostly? I donut think so; my recollection is a vast majority of
trans-atlantic stuff coming in ocean and monmouth county, NJ (manasquan,
etc).

-- Alex Rubenstein, AR97, K2AHR, alex@nac.net, latency, Al Reuben --
-- Net Access Corporation, 800-NET-ME-36, http://www.nac.net --

14

There are, and always will be points of failure. The mistake people
make is thinking they can build a bunker strong enough. You can't
build a building, vault, bunker, missle silo which can withstand
everything.

The question is really how do you manage your network diversity.

The loss of something like 60 Hudson shouldn't cause more than
an annoying route flap in your network. Exchange points like MAE-East
have completely failed in the past. There have been multiple fiber
cuts in the same day. Well designed networks continued to work.
Although I'l admit, it is hard work. You can't rely on a carrier
to do it for you. Grooming happens.

The biggest risk in most networks aren't the national exchange points,
although they get the press. All national providers, and most regional
providers are interconnected to multiple geographically diverse points.

The point in the network with limited diversity is the LEC end-offices.
And I use "LEC" deliberately, because even if you use a CLEC, most of the
time you are using the LEC for the last mile. You may have great path
diversity for 3,000 miles across the continent, but then you go through
140 West Street, or Rochelle Park or some other LEC office. Even if you
thought you went through Broad Street, a lot of folks found out they
were in fact routed through West Street.

Ok, so I just said you can't build a bunker strong enough. Are
carrier hotels, like 60 Hudson, history? I don't think so. They
have better diversity, better backup systems, and better security
than normal offices. The concentration of bandwidth and carriers
allowed very fast restoration and re-routing between locations still
standing. Carriers were using other carrier's circuits to restore
facilities. We may see some movement away from downtown areas, where
the danger is a near miss instead of being a direct target.

If you can't afford to build your own colo, are you better off hanging
off a spoke from a LEC central office. Or putting your equipment
in a building with built in diversity. In general, it is best to
put your equipment as close as possible to the point of diversity.
You can do this by either moving the point of diversity close to you,
or moving your equipment closer to the diversity. Being at the
end of a 6,000 foot T1 circuit to a CO is the worst of both worlds.
You will go down if an airplane hits either your office, or the CO,
or any point along that 6,000 feet of T1 circuit.

15

My point being that building a network which doesn't have more
than an annoying route flap, if /both/ 60 Hudson and 111 8th
avenue are lost, is extremely hard (*) (especially if it has
a transatlantic component). And that's true even if you
have your own fiber.

(*) hard means that it isn't compatible with existing topologies,
and building new ones is expensive.

Alex Bligh
Personal Capacity.

16

IIRC, a good deal of transatlantic fibre lands at the National Guard beach in Sea Girt (no public access).

-Bill

17

Which brings me back to my original question. Are there specific
locations which are more important to the functioning of the Internet
than others? You can't simply say everything is important. The FAA
breaks airports down into several catagories, large airports, medium
airports and small airports. A large airport has 1% or more of the
passenger traffic. Are there specific locations which handle 1% or
more of the Internet's traffic (assuming we had figures for the total
amount of traffic).

18

Sean Donelan wrote:

> My point being that building a network which doesn't have more
> than an annoying route flap, if /both/ 60 Hudson and 111 8th
> avenue are lost, is extremely hard (*) (especially if it has
> a transatlantic component). And that's true even if you
> have your own fiber.
>
> (*) hard means that it isn't compatible with existing topologies,
> and building new ones is expensive.

Which brings me back to my original question. Are there specific
locations which are more important to the functioning of the Internet
than others? You can't simply say everything is important. The FAA
breaks airports down into several catagories, large airports, medium
airports and small airports. A large airport has 1% or more of the
passenger traffic. Are there specific locations which handle 1% or
more of the Internet's traffic (assuming we had figures for the total
amount of traffic).

The national air traffic system makes a poor analogy to the Internet in
this case, IMHO. If O'Hare got nuked tomorrow, we'd have some serious
disruption in passenger traffic. If PAIX fell into the ocean, OTOH,
traffic would simply route around it. Isn't that how we try to engineer
the Internet?

So in other words, yes, everything is important, and yes, nothing is
particularly important.

Grant

19

"Grant A. Kirkwood" wrote:

Sean Donelan wrote:
>

The national air traffic system makes a poor analogy to the Internet in
this case, IMHO. If O'Hare got nuked tomorrow, we'd have some serious
disruption in passenger traffic. If PAIX fell into the ocean, OTOH,
traffic would simply route around it. Isn't that how we try to engineer
the Internet?

So in other words, yes, everything is important, and yes, nothing is
particularly important.

But there was a point in time when taking out a certain parking garage
in Va could have caused us a very great deal of difficulty. But I'd say
we are past that, for the most part.

Bob

20

Are we?

When 25 Broadway failed, approximately 1% of the global Internet
routing table also disappeared. Which I would guess qualifies it
as a "major" hub.

Verizon still has 100,000 lines out of service, and only now
begun to restore service to "small" businesses.

A couple of years ago a fiber cut in Ohio disrupted about 20% of
the Internet routing table.