RE: Fiber cut in SF area

Mailman List Mirror (Read Only)
1

Jo? wrote:

I'm confussed, but please pardon the ignorance.
All the data centers we have are at minimum keys to access
data areas. Not that every area of fiber should have such, but
at least should they? Manhole covers "can" be keyed. For those of
you arguing that this is not enough, I would say at least it?s a start.

That is an option, but it doesn't address the real problem.

The real problem is route redundancy. This is what the original contract
from DARPA to BBM, to create the Internet, was about! "The net" was
created to enable communications bttn point A and point B in this exact
scenario.

No one should be surprised that ATT would cut-corners on critical
infrastructure. The good news is that this incident will likely result in
increased Federal scrutiny if not regulation. We know how spectacularly
energy and banking deregulation failed. Is that mistake being repeated
with telecommunications?

The bad news is that some of the $16M/yr ATT spends lobbying Congress (for
things like fighting number portability and getting a free pass on illegal
domestic surveillance) will likely be redirected to ask for money to "fix"
the problem they created. This assumes ATT is as badly managed, and the US
FCC and DHS are better managed, than has been the case for the last 8
years. Time will tell.

For a good "man in the street" perspective of how the outage effected
things like a pharmacy's ability to fill subscriptions and a university
computer's ability to boot check out a couple of shows broadcast on KUSP
(Santa Cruz Public Radio) this morning:

   Ask Doctor Dawn

   http://geekspeak.org/

Roger Marquis

2

The real problem is route redundancy. This is what the original contract
from DARPA to BBM, to create the Internet, was about!

s/DARPA/ARPA/; s/BBM/BBN/; s/Internet/ARPAnet/.

BBN won the contract to build the first four IMPs.

Theory and research about it is older, look at:

But you are right, redundancy is the issue, cost is the factor.

Jorge.

3

The real problem is route redundancy. This is what the original contract
from DARPA to BBM, to create the Internet, was about! "The net" was
created to enable communications bttn point A and point B in this exact
scenario.

Uh, not exactly. There was diversity in this case, but there was also N+1 breaks. Outside of a few counties in the Bay Area, the rest of the country's telecommunication system was unaffected. So in that sense the system worked as designed.

Read the original DARPA papers, they were not about making sure grandma could still make a phone call.

For a good "man in the street" perspective of how the outage effected
things like a pharmacy's ability to fill subscriptions and a university
computer's ability to boot check out a couple of shows broadcast on KUSP
(Santa Cruz Public Radio) this morning:

Why didn't the "man in the street" pharmacy have its own backup plans?

Why didn't the pharmacy also have a COMCAST or RCN broadband connection for alternative Internet access besides AT&T or Verizon, a Citizens Band radio channel 9 for alternative emergency communications besides 9-1-1,
a satellite phone for alternative communications besides local cell phones, and a Hughes VSAT dish for yet even more diversity? Why was the pharmacy relying on a single provider? Or do it the old-fashion way before computers and telecommunications; keep a backup paper file of their records so they could continue to fill prescriptions?

Why didn't the pharmacy have more self-diversity? Probably the usual reason, more diversity costs more. That may be the reason why hospitals have more diversity than neighborhood pharmacies; and emergency rooms have other ways to get medicine. Maintaining diversity and backups is probably also part of the reason why filling a prescription at a hospital is much more expensive than filling a prescription at your neighborhood pharmacy.

Likewise, why didn't grandma have her own pharmacy backup plan. Don't wait until the last minute to refill a critical presciption, have backup copies of prescriptions with her doctor, have an account with an alternative pharmacist in case her primary pharmacist isn't reachable, etc.

Readiness works better if everyone does their part, including grandma.

Next time it won't be AT&T, it will be Cox or Comcast or Qwest or Level 3 or Global Crossing or .... or .... or .... . It won't be vandalism, it will be an earthquake, backhoe, gas main explosion, operator error, ....

Everything fails sometimes. What's your plan?

http://www.ready.gov/

personal opinion only

4

Anyone know how banks in the Bay Area did through this? I wonder how many
banks went dark and whether they had any backup plans/connectivity. Me
thinks its doubtful.

I also wonder if the bigger pharmacies such as Longs, Walgreens, Rite-Aid,
Etc had thought about these kinds of issues? I personally doubt it. I bet
you they went dark along with everyone else. Unfortunate.

The funny thing is that the California lottery would be somewhat immuned to
this kind of disaster as they actually use Hughes VSAT at every single
retailer.

Sorry for the random thoughts...

-Mike

5

Read the original DARPA papers, they were not about making sure grandma
could still make a phone call.

That's a great explanation in few words.

Everything fails sometimes. What's your plan?

Even the failover plans ...

Cheers
Jorge

6

While OT the news reports indicated ATMs were offline and many credit card
processing machines were down. This is no big shock because many ATM
networks are on frame relay and POS credit card machines use POTS lines.

The outage also impacted mobile service too if it hadn't been said.

I hope we can put this thread to rest soon.

-r

7

Mike Lyon wrote:

Anyone know how banks in the Bay Area did through this? I wonder how many
banks went dark and whether they had any backup plans/connectivity. Me
thinks its doubtful.

...

Because of the loss of the alarm systems, many banks went to a method
where only one or two people were let in at a time. Extra security was
also posted because of the inability to call 911.

8

Don't really care so much about the bank's security, especially if it was
one that received some the bailout money :slight_smile:

I was more worried about if people could make withdraws from their bank
accounts. Deposits they could do as they could enter them in later but
withdraws I think would be different.

9

Sean Donelan wrote:

,
Uh, not exactly. There was diversity in this case, but there was also
N+1 breaks. Outside of a few counties in the Bay Area, the rest of
the country's telecommunication system was unaffected. So in that
sense the system worked as designed.
....

About eight or ten years ago I went to PacBell (or whatever it was
called at the time) and requested that two large facilities get a sonet
ring between them. I was told I couldn't have it because they were both
fed through a single set of conduits and one backhoe could cut both
sides of the ring. It wouldn't be diverse so they wouldn't provison it
unless I paid for the digging of new paths.

So much for their theory of diverse. Sounds like the rules are
different for them.

There are one thing to also point out. That train track next to the
manholes in South San Jose is the major line between the Bay Area and
Southern CA. There are at least three or four fiber paths for different
companies buried along those tracks. There are also connections from
Gilroy to the Hollister/San Juan Bautista area and thence to Salinas.

It would have been very simple for the telcos to provision a backup path
southward.