Vulnerbilities of Interconnection

Thanks for all the answers on the previous question about Equinix.
The reason for the question is that we are working on a study here at
George Mason on assesing the vulnerbilities of the Internet and
telecommunications infrastructure to physical attack. One part that
we are looking at are the vulnerbilites of interconnection facilites.

Doomsday scenarios like 60 Hudson street being a target instead of the
WTC or the loss of MAE east, etc. The earlier question about Equinix
and the Big 7 was really a vulnerbility question. Are interonnection
locations aggreating or dispersing. How real of a threat is the
physical loss of a major interconnection point. Incidents like the
Balitmore train tunnel wreck point in one direction.

We are working on putting several models together to try and simulate
some of these scenarios with varying degrees of success, but I would
very much like to avoid doing the research in a vaccuum. I was hoping
a discussion on NANOG wold be a good first step. The project is quite
hot with the politicos and I very much want to make sure to best
recommendations are made. Formal industrsy cooperation is one side of
this, but I think a lot of information can be gained from an informal
approach as well. Any and all feedback is greatly appreciated

Thanks,

sean

http://www.infosecuritymag.com/2002/sep/2002survey/voices/verylarge.shtml

On security reporting...
"Since Sept. 11, state, local and federal authorities have tried to get
their arms around the potential threats to the nation's
infrastructure--including the telecommunications infrastructure. They have
asked us questions like, 'What are your 100 most vulnerable places in the
network?'"

"As much as we would like to help the government in its attempt to help
us, we believe it would be counterproductive to share such information
widely because if it were released, it would provide a terrorist with a
roadmap to our key locations. Unless the government agrees that it can
protect our information, we will continue to respectfully decline such
blanket requests."

Bill Smith
CTO and President of Interconnection Services, BellSouth

The crux of the issue are FOIA requests. The government won't make these
types of vulnerability reports immmune to FOIA requests - thus a foreign
terrorist or home-grown "farmbelt fuhrer" could simply order up a list of
the most vulnerable sites, and select some to attack.

Due to the distributed nature of the internet, and the routing protocols
that regulate it's traffic flow, there is no single point of failure.
However, we have seen how concerted attacks can be made at multiple
locations, almost simultaneously.

If the government could agree to allow this information to remain
confidential, it would greatly expedite the process of hardening appropriate
facilities, and identifying weaknesses.

- Daniel Golding

A quick point...Several folks have postulated that the internal (non-physical) threat dwarfs that of the physical threat, due to the lack of visibility, the difficulty of tracking and coordinating a response, and the millions of vulnerable systems world-wide capable of launching an internal attack. A physical attack (a hole in a wall for example) can typically be detected and corrected in a matter of hours or days, while an effective internal attack could be varied in time and scope causing at least as much damage invisibly for a much longer period of time.

That said, a few years back I wrote the "Interconnection Strategies for ISPs" white paper, which speaks to the economics of peering using exchange points vs. using pt-to-pt circuits. It documents a clear break even point where large capacity circuits (or dark fiber loops) into an IX with fiber cross connects within a building are a better fit (financially) than pt-to-pt circuits.

A couple physical security considerations came out of that research:
1) Consider that man holes are not always secured, providing access to metro fiber runs, while there is generally greater security within colocation environments

2) It is faster to repair physical disruptions at fewer points, leveraging cutovers to alternative providers present in the collocation IX model, as opposed to the Direct Circuit model where provisioning additional capacities to many end points may take days or months.

Finally, I have seen a balancing act between how much it costs to protect against a disruption versus the cost of the disruption. In today's economy (unlike say a few years ago) more folks seem to be focused on doing this mathematically calculation rather than just picking full mesh interconnect topologies.

Bill

That said, a few years back I wrote the "Interconnection Strategies for
ISPs" white paper, which speaks to the economics of peering using exchange
points vs. using pt-to-pt circuits. It documents a clear break even point
where large capacity circuits (or dark fiber loops) into an IX with fiber
cross connects within a building are a better fit (financially) than
pt-to-pt circuits.

This obviously would be a thesis of Equinix and other collo space providers,
since this is exactly the service that they provide. It won't, hower, be a
thesis of any major network that either already has a lot of infrastructure
in place or has to be a network that is supposed to survive a physical
attack.

A couple physical security considerations came out of that research:
1) Consider that man holes are not always secured, providing access to
metro fiber runs, while there is generally greater security within
colocation environments

This is all great, except that the same metro fiber runs are used to get
carriers into the super-secure facility, and, since neither those who
originate information, nor those who ultimately consume the information are
located completely within facility, you still have the same problem. If we
add to it that the diverse fibers tend to aggregate in the basement of the
building that houses the facility, multiple carriers use the same manholes
for their diverse fiber and so on.

2) It is faster to repair physical disruptions at fewer points, leveraging
cutovers to alternative providers present in the collocation IX model, as
opposed to the Direct Circuit model where provisioning additional
capacities to many end points may take days or months.

This again is great in theory, unless you are talking about someone who
is planning on taking out the IX not accidently, but deliberately. To
illustrate this, one just needs to recall the infamous fiber cut in McLean
in 1999 when a backhoe not just cut Worldcom and Level(3) circuits, but
somehow let a cement truck to pour cement into Verizon's manhole that was
used by Level(3) and Worldcom.

Alex

This obviously would be a thesis of Equinix and other collo space providers,
since this is exactly the service that they provide. It won't, hower, be a
thesis of any major network that either already has a lot of infrastructure
in place or has to be a network that is supposed to survive a physical
attack.

Actually, the underlying assumption of this paper is that major networks already have a large global backbone that need to interconnect in n-regions. The choice between Direct Circuits and Colo-based cross connects is discussed and documented with costs and tradeoffs. Surviving a major attack was not the focus of the paper...but...

When I did this research I asked ISPs how many Exchange Points they felt were needed in a region. Many said one was sufficient, that they were resilient across multiple exchange points and transit relationships, and preferred to engineer their own diversity separate from regional exchanges. A bunch said that two was the right number, each with different operating procedures, geographic locations, providers of fiber, etc. , as different as possible. Folks seemed unanimous about there not being more than two IXes in a region, that to do so would splinter the peering population.

Bill Woodcock was the exception to this last claim, positing (paraphrasing) that peering is an local routing optimization and that many inexpensive (relatively insecured) IXes are acceptable. The loss of any one simply removes the local routing optimization and that transit is always an alternative for that traffic.

> A couple physical security considerations came out of that research:
> 1) Consider that man holes are not always secured, providing access to
> metro fiber runs, while there is generally greater security within
> colocation environments

This is all great, except that the same metro fiber runs are used to get
carriers into the super-secure facility, and, since neither those who
originate information, nor those who ultimately consume the information are
located completely within facility, you still have the same problem. If we
add to it that the diverse fibers tend to aggregate in the basement of the
building that houses the facility, multiple carriers use the same manholes
for their diverse fiber and so on.

Fine - we both agree that no transport provider is entirely protected from physical tampering if its fiber travels through insecure passageways. Note that some transport capacity into an IX doesn't necessarily travel along the same path as the metro providers, particularly those IXes located outside a metro region. There are also a multitude of paths, proportional to the # of providers still around in the metro area, that provide alternative paths into the IX. Within an IX therefore is a concentration of alternative providers, and these alternative providers can be used as needed in the event of a path cut.

> 2) It is faster to repair physical disruptions at fewer points, leveraging
> cutovers to alternative providers present in the collocation IX model, as
> opposed to the Direct Circuit model where provisioning additional
> capacities to many end points may take days or months.

This again is great in theory, unless you are talking about someone who
is planning on taking out the IX not accidently, but deliberately. To
illustrate this, one just needs to recall the infamous fiber cut in McLean
in 1999 when a backhoe not just cut Worldcom and Level(3) circuits, but
somehow let a cement truck to pour cement into Verizon's manhole that was
used by Level(3) and Worldcom.

Terrorists in cement trucks?

Again, it seems more likely and more technically effective to attack internally than physically. Focus again here on the cost/benefit analysis from both the provider and disrupter perspective and you will see what I mean.

Actually, the underlying assumption of this paper is that major networks
already have a large global backbone that need to interconnect in
n-regions. The choice between Direct Circuits and Colo-based cross connects
is discussed and documented with costs and tradeoffs. Surviving a major
attack was not the focus of the paper...but...

If the major networks in questions are long-distance companies and local
phone companies, then they are already interconnected in N places. For one
reason or another, at the present time they are simply not running IP at
those points. It is equivalent to having networks in the common facilities
that choose not to interconnect.

When I did this research I asked ISPs how many Exchange Points they felt
were needed in a region. Many said one was sufficient, that they were
resilient across multiple exchange points and transit relationships, and
preferred to engineer their own diversity separate from regional exchanges.

Very few ISPs in reality have any physical divercity.

A bunch said that two was the right number, each with different operating
procedures, geographic locations, providers of fiber, etc. , as different
as possible. Folks seemed unanimous about there not being more than two
IXes in a region, that to do so would splinter the peering population.

Security is always considered a waste of money. It is nothing new. The
reason for that is that it is impossible to see the benefits when there is
no problem.

Fine - we both agree that no transport provider is entirely protected from
physical tampering if its fiber travels through insecure passageways. Note
that some transport capacity into an IX doesn't necessarily travel along
the same path as the metro providers, particularly those IXes located
outside a metro region. There are also a multitude of paths, proportional
to the # of providers still around in the metro area, that provide
alternative paths into the IX. Within an IX therefore is a concentration of
alternative providers, and these alternative providers can be used as
needed in the event of a path cut.

They are using the same paths to get into the buildings. If they are not
using the same paths exactly, their paths are close enough to each other
within N meters from the building.

> > 2) It is faster to repair physical disruptions at fewer points, leveraging
> > cutovers to alternative providers present in the collocation IX model, as
> > opposed to the Direct Circuit model where provisioning additional
> > capacities to many end points may take days or months.
>
>This again is great in theory, unless you are talking about someone who
>is planning on taking out the IX not accidently, but deliberately. To
>illustrate this, one just needs to recall the infamous fiber cut in McLean
>in 1999 when a backhoe not just cut Worldcom and Level(3) circuits, but
>somehow let a cement truck to pour cement into Verizon's manhole that was
>used by Level(3) and Worldcom.

Terrorists in cement trucks?

No, but since that caused a multi-day outages for certain customers due to a
single point of failure, I am sure someone can appreciate the outage that
can be caused by detonating a hundred killograms of high explosives inside a
collo facility.

Again, it seems more likely and more technically effective to attack
internally than physically. Focus again here on the cost/benefit analysis
from both the provider and disrupter perspective and you will see what I
mean.

Easily accessible brute-force *always* wins.

Any chain is not stronger than its weakest link and concentrated
infrastructure was, is and always will be, the weakest link if one can mount
an attack using bruce-force.

Neither the data centers, nor COs nor exchange points that are vital so far
had been designed in a way that they could withstand a direct physical
attack even by an individual with a handgun, not to mention anyone carrying
explosives. When that problem gets solved, we can concentrate on attracks
against IP infrastructure.

Alex

reflecting on my experiences in such facilities...

usually all i've ever needed to do at the door is sign in after proving
that i work for a company that has colo space. my boxes of equipment
have never been inspected.

therefore, to attack many colo facilities, it is sufficient to sign
contracts that i never intend to honor and then carry boxes of "stuff"
up that has nothing to do with colo.

richard

How many banks know what their customers have put in the safe deposit
boxes stored in the bank's vaults? Do you want guard rummaging through
your equipment? Even if they opened the boxes how would a guard know
what's inside a 12000 router?

Rent the movie Infinity (1996) or read Richard Feynman's books describing
the security around The Manhattan Project at Los Alamos.