Seriously though I want to start some discussion around outside plant protection. This isn't the middle of the ocean or desert after all.
There were multiple fiber cuts in a major metropolitan area, resulting in the loss of critical infrastructure necessary to many peoples daily lives (though twitter stayed up so it's all good). It would appear that this was a deliberate act by one or more individuals, who seemed to have a very good idea of where to strike which resulted in a low cost, low effort attack that yielded significant results.
So allow me to think out loud for a minute....
1) Why wasn't the fiber protected by some sort of hardened/locked conduit? Is this possible? Does it add extensive cost or hamper normal operation?
2) Why didn't an alarm go off that someone had entered the area? It was after business hours, presumably not in response to a trouble ticket, and as such a highly suspicious action. Does it make sense for these access portals to have some sort of alarm? I mean there is fiber running through and as such it could carry the signaling. Would this be a massive cost addition during construction?
3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry?
No amount of money can deter a determined entity. If there is a will,
there is a way, etc. Want to protect your "outside" plant, then make
it resilient network-wise. There use to be a time when dual paths
was acceptable, I (personally) think that quad paths should be the
norm.
These incidents happen all the time without sabotage being involved. A ship sank off the coast of Pakistan and took out both international cables serving the country ...
We had the undersea earthquake that seven seven cables in the Taiwan straits.
The truth is that physical diversity is an ideal, not a reality.
I have seen lots of accidents that took multiple operators and seriously disrupted in a given locality.
The only difference here is that in the Heart of Geek Territory. Hence the Natives are restless ...
Roderick S. Beck
Director of European Sales
Hibernia Atlantic
I tried to be very careful to say that it appears to have been sabatoage, but that it's not confirmed. Also this isn't the middle of the ocean, but cable underground. That usually doesn't get cut unless it's by a back hoe. And speaking of unions.... construction crews charge lots of money to work in the middle of the night, so it's usually avoided.
I think we'd only be speculating with no actual data surrounding the
vaults the bundles traversed. That said one would *hope* vault access
is not trivial and there are mechanisms in place to alert of
unauthorized, unlawful entry. I would also love it if bacon was
healthy for me and didn't make my cholesterol 280.
The bay area is also particularly unique in the sense that there aren't
many available paths to run fiber. There are mountains on one side and
the bay on the other. Your available diverse paths are "the left and
right side of the tracks," and as a coworker pointed out the left has
been full since 1996.
Seriously though I want to start some discussion around outside plant protection. This isn't the middle of the ocean or desert after all.
There were multiple fiber cuts in a major metropolitan area, resulting in the loss of critical infrastructure necessary to many peoples daily lives (though twitter stayed up so it's all good). It would appear that this was a deliberate act by one or more individuals, who seemed to have a very good idea of where to strike which resulted in a low cost, low effort attack that yielded significant results.
So allow me to think out loud for a minute....
1) Why wasn't the fiber protected by some sort of hardened/locked conduit? Is this possible? Does it add extensive cost or hamper normal operation?
This was supposedly an inside job, and I even heard the cabinets were locked. How do you stop an employee with the key from opening a lock? (See #2.)
2) Why didn't an alarm go off that someone had entered the area? It was after business hours, presumably not in response to a trouble ticket, and as such a highly suspicious action. Does it make sense for these access portals to have some sort of alarm? I mean there is fiber running through and as such it could carry the signaling. Would this be a massive cost addition during construction?
Possibly, and yes.
3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry?
Probably, and who knows?
How much did this cost the telcos involved? Probably nearly nothing. How much would it cost them to do what you suggest in #2? Probably 1,000,000 times nearly nothing, _at_least_. Guess what the telcos involved will choose? Hell, you would too in their place.
By the time the second plane hit WTC, intent was apparent. I think in this case intent is also apparent based on proximity (and the previously mentioned reward AT&T has posted for the capture of "vandals").
1) Why wasn't the fiber protected by some sort of hardened/locked conduit? Is this possible? Does it add extensive cost or hamper normal operation?
Some people do lock their vaults/pits/manholes. But, to be honest, I'm not sure it helps a lot. How many passersby would stop someone appearing to be in a phone company/telco high-vis vest using bolt cutters - telling them the lock had seized?
(I can also think of quite a few options which don't require opening a lid, but here's not the place to discuss!)
2) Why didn't an alarm go off that someone had entered the area? It was after business hours, presumably not in response to a trouble ticket, and as such a highly suspicious action. Does it make sense for these access portals to have some sort of alarm? I mean there is fiber running through and as such it could carry the signaling. Would this be a massive cost addition during construction?
Alarms mean power. Adding power to hundreds of km of a route to every pit/manhole would cost a lot - it's underground and often quite wet. Better to provide diverse route protection for the same cost - then you protect against accidental external aggression. Maybe you could do something neat with fibre and some of the active monitoring stuff to detect pit openning passively, but you'd want it to be pretty good and reliable. Lots of false alarms lead to NOCs not caring.
3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry?
Obscurity and that most people are blissfully unaware of manholes and other street furniture. Locking is certainly possible but I'm not convinced it adds a LOT (see above).
Accidental external aggression is far more likely. Backhoe fade and equipment failure is a bigger problem than vandalism.
That said one would *hope* vault access
is not trivial and there are mechanisms in place to alert of
unauthorized, unlawful entry.
I regularly drove on these roads when these lines were being put in up-and-down the SF Peninsula. There are 4 manhole covers every 1/4 mile or so that provide access to this fiber. Do the math. Multiply by the number of miles of fiber runs across the world, and the number of access points per mile on each run. Exactly how do you plan to make "vault access non-trivial" and yet make the access as easy as it needs to be for routine maintenance and repair?
My guess is that it is probably less expensive in the long run to leave them unprotected and just fix the problems when they occur than to try to "secure" the vaults and deal with the costs and extended outage delays when access it "secured" and it takes longer to get into a vault to fix things.
Ravi Pina wrote:
>
>That said one would *hope* vault access
>is not trivial and there are mechanisms in place to alert of
>unauthorized, unlawful entry.
I regularly drove on these roads when these lines were being put in
up-and-down the SF Peninsula. There are 4 manhole covers every 1/4 mile
or so that provide access to this fiber. Do the math. Multiply by the
number of miles of fiber runs across the world, and the number of access
points per mile on each run. Exactly how do you plan to make "vault
access non-trivial" and yet make the access as easy as it needs to be
for routine maintenance and repair?
Having never been in a vault or know how to get in one other than
apparently lifting a manhole cover I can't possible answer that
with anything more than guessing.
My guess is that it is probably less expensive in the long run to leave
them unprotected and just fix the problems when they occur than to try
to "secure" the vaults and deal with the costs and extended outage
delays when access it "secured" and it takes longer to get into a vault
to fix things.
I wasn't thinking Exodus/C&W/SAVVIS/Whoever level security, but
considering communications cables traverse such sites it is hardly
unreasonable to think they could implement some alarm that is
centrally monitored by a NOC. I'm guessing *anything* is better
than what appears to be the *nothing* that is in place now.
Also not to get sensationalist, but less expensive than a life that
could be lost if an emergency call can't be put through?
Not to turn this into an ethical typ discussion but this arguement would have to assume you could sue the telco not the 'vandal' due to a loss of life if it occured, and that, that dollar amt would be greater then 'securing' all cables.
The cost to fix all pintos' gas tanks was only $11 per car unit and it was gambled, though they lost it was cheeper then the lawsuits, I'm betting the while fewer units, its order of magnatitudes more then 11$ per unit to 'secure' access points with a lot less certain negative lawsuit outcomes.
I've really got ask if this thread has run it's course.
Given the nature of earlier discussions of off-topic issues, I think we've
pretty much jumped the shark with people's personal anecdotes of how to
disable fiber connectivity.
Not to turn this into an ethical typ discussion but this arguement
would have to assume you could sue the telco not the 'vandal' due to
a loss of life if it occured, and that, that dollar amt would be
greater then 'securing' all cables.
The alarm that goes off saying the lid got opened is only 2 minutes before the
big red alarm that says you just lost 5 OC-768s. So the link is *still* going
to drop even as you're on the 911 call to try to explain to them where your
manhole is, the cops *still* won't catch anybody (the perps may be gone before
you hang up on the 911 call), and you're taking 2 minutes off a 10-hour outage.
Its all risk and cost. You possibly couldn't have spent enough to stop
this event. The outside plant wasn't at fault, highly motivated and
informed individuals were. Pretty much a non issue, IMHO.
Your understanding is incorrect. I'm an average sized guy and I can pull a manhole cover with one hand on the right tool. It might take 2 hands if it hasn't been opened recently and has lots of pebbles and dirt jammed in around it. It's like everything else: if you know how to do it, and you have the right tool, it's simple.
And, yes, you can get lockable manhole covers. They aren't cheap. McGuard make a popular one.
(Yes, yes...why would I possibly know any of this.....I'm a fire marshal in a small town as a part time gig, so I have to deal with this kind of thing on a reasonably regular basis)
Agreed. Manhole covers are very simple to remove. I don't even need any tools. I've removed countless manhole covers to retrieve balls, frisbees, etc., with nothing more than my bare hands. It's a pretty trivial task.
Think about it. All anyone would need to do is pull up to the manhole, set a few orange cones around it, put on an orange vest and a hard hat, and crawl on in with your wire cutters and bolt cutter. Guaranteed NO ONE will even question it.
It would appear that this was a deliberate act by one or more individuals, who seemed to have a very good idea of where to strike which resulted in a low cost, low effort attack that yielded significant results.