VZ in the South Bay (San Jose) is out. As per news reports I watched at 6am PDT.
Yup. Abovenet fiber between 200 Paul SFO and 11 Great Oaks SJC is currently
out of commission.
jason
200 Paul Ave is seeing several carriers down. I am also in Santa Cruz and cannot make or receive long distance calls on my land lines. Unconfirmed reports of Caltrain cut.
Cheers,
Aaron
Hello,
Mercurynews.com is reporting telephone outages in Santa Clara and Santa Cruz counties that started around 2:00 am local time. I observed numerous carrier outages starting around 4:00 am local time. Does anyone know if this is due to the same fiber cut, or are these separate issues?
David
Hi,
I saw my Sonic.net-over-AT&T ADSL go dark at 02:30 local and it is still down, served on a fiber remote out of SNCZCA01. (I'm guessing the 200 Paul outages are associated with where this ATM terminates and that's the cause, rather than the service in/out of Santa Cruz County, but I have no way of telling which from here)
My own Gatespeed.net microwave to Equinix SV-3 is working fine (no surprise there), and I'm not seeing significant routing problems in/out of there with transit or peering. (Not even any down peers, so no inter-Equinix-site outage apparently).
Matthew Kaufman
matthew@eeph.com
Service to South Santa Clara county is completely down: Internet,
landline, and cellphones. Both Verizon and AT&T are affected. 911 is
also down.
My cellphones show one or no bars. Normally they are all four bars.
The idea that all of that is lumped in one fiber bundle is mind boggling.
My cellphones show one or no bars. Normally they are all four bars.
hmmm, probably not related but could be that some cellphone operators are
restricting coverage to give priority to emergency svcs communications.
That AT&T has stopped provisioning protection fiber for automatic restoral is mind boggling.
That our crack (or on crack) govt contracting/emergency-preparedness staff didn't demand protected facilities for 911 is another mind boggling issue.
That there is no over-under wide-area back-up coverage for the cellular canopy ...
We posture and orate about being prepared for terrorist attacks and natural disasters, and then events like these reveal the reality:
The emperor has no clothes.
Roy wrote:
Robert M. Enger wrote:
We posture and orate about being prepared for terrorist attacks and natural disasters, and then events like these reveal the reality:
The emperor has no clothes.
You wouldn't have clothes either if you could double your profit by not wearing any.
Matthew Kaufman
A sobering touché.
Jay Murphy
IP Network Specialist
NM Department of Health
ITSD - IP Network Operations
Santa Fe, New Mexico 87502
Bus. Ph.: 505.827.2851
"We move the information that moves your world."
That AT&T has stopped provisioning protection fiber for automatic restoral is mind boggling.
That our crack (or on crack) govt contracting/emergency-preparedness staff didn't demand protected facilities for 911 is another mind boggling issue.
This costs $$$ and usually isn't a problem as there are other ways to communicate. The law-enforcement folks qualify for GETS so get priority on wired/PSTN. They can also get radio priority w/ WPS.
That there is no over-under wide-area back-up coverage for the cellular canopy ...
The problem is how do you back up such a large area. WPS can get you priority.
We posture and orate about being prepared for terrorist attacks and natural disasters, and then events like these reveal the reality:
The emperor has no clothes.
I think the problem is there are clothes, some people/areas have none, some have an abundance. If people don't plan for going out in public, there is a chance you'll walk outside naked 
Roy wrote:
Service to South Santa Clara county is completely down: Internet,
landline, and cellphones. Both Verizon and AT&T are affected. 911 is
also down.My cellphones show one or no bars. Normally they are all four bars.
If you're an ISP, you may be able to obtain GETS or WPS for your engineers. This would allow you a better chance of getting a channel to respond to issues. This is a good test to see how your backup plans might work for communication in the case of a larger distaster (earthquake, or other).
Make sure you test the tools you have. The people I know with GETS cards are encouraged to test them regularly and verify they work. If someone has one, I'd be interested to know if it proved to be of value today.
- Jared
Yep.... it leads to:
Activity Type Code Desc: PROGRESS COMMENTS
Activity Type Code: PROG
OTDR readings were taken by AT&T West and a cut was located 1600 ft from
the San Jose, CA central office. AT&T West technicians are onsite
working to isolate the exact location of the cut. There are 4 cables
impacted. AT&T Mobility has 61 GSM and 45 co-located UMTS sites out of
service off of Santa Clara Base Station Controllers 15 & 23, and Santa
Clara Radio Network Controller 4. E911 has 52 Location Measuring Units
down. The AT&T West Santa Cruz 11 central office (41,803 ATNs) is
experiencing an SS7 isolation and the San Martin central office (11,904
ATNs) lost it's umbilical and is isolated at this time. The Bailey
remote site (4,973 ATNs) is also isolated. Scott's Valley has 3 out of 4
SS7 links down. The Santa Cruz 01, Aptos, Scott's Valley, Felton,
Boulder Creek, Ben Lomand, San Jose 11, San Jose 13, San Jose 21 central
offices have trunks impacted such that all lines are busy and incoming
calls are receiving trouble messages. The Santa Cruz County SO (178,040
ATNs), Scott's Valley PD (12,007 ATNs) and the UC Santa Cruz PD (14,909
ATNs) are all without ALI at this time. The Gilroy PD PSAP and the
Morgan Hill PD and CDF have been rerouted with ALI/ANI. The Felton CDF
has not been rerouted. There are 17 DSLAMS and 4 ATMS out of service
impacting DSL service. There are 3 SMDI Links down impacting voicemail
service. Verizon's Morgan Hill and Gilroy central offices are currently
isolated. There have been 224,865 blocked calls.
Robert M. Enger wrote:
Robert M. Enger wrote:
That AT&T has stopped provisioning protection fiber for automatic restoral is mind boggling.
That our crack (or on crack) govt contracting/emergency-preparedness staff didn't demand protected facilities for 911 is another mind boggling issue.
911 centers can work just fine without phones. They use radio for intra-agency communications with police, fire, etc. There is also a large ham radio community who jumps in to help with communications when needed (testing, simulations, and real disasters).
The primary problem a 911 center has if the phones go out is that people can't *reach* the 911 center if *their* phone lines don't work.
jc
Jared Mauch wrote:
That AT&T has stopped provisioning protection fiber for automatic restoral is mind boggling.
That our crack (or on crack) govt contracting/emergency-preparedness staff didn't demand protected facilities for 911 is another mind boggling issue.
This costs $$$ and usually isn't a problem as there are other ways to communicate. The law-enforcement folks qualify for GETS so get priority on wired/PSTN. They can also get radio priority w/ WPS.
I didn't know about WPS.
Interesting stuff.
Which is fine if you're a law-enforcement folk. Kinda sucks if
you're an ordinary private citizen who tries to dial 911 and gets a
reorder tone. Which I presume is what is happening, since everybody
is saying 911 is down. What's the point of having all the emergency
service personnel communicating with each other if they can't get 911
calls in the first place? (Rhetorical question, I know there are
other ways they can find out about emergencies, but 911 is the big
one.)
Maybe nostalgia just ain't what it used to be, but I thought the
PSTN used to be more reliable than this.
#ifdef CONSPIRACY_THEORIST
What if this isn't simple vandalism?
#endif
-- Ben
That AT&T has stopped provisioning protection fiber for automatic restoral is mind boggling.
Only helps with N-1 breaks. Unfortunately, sometimes there are N+1 breaks. Check the NANOG archives, I believe there were 5 breaks in one day in the 1990's; and even in the last year there have been 2-4 breaks
on some transoceanic cables at the same time.
On the other hand, I've never heard a carrier complain about digging more fiber as long as someone is willing to pay for it. How much more is someone willing to pay to get more diversity? Not willing to pay for it? I guess that's an answer too.
That our crack (or on crack) govt contracting/emergency-preparedness staff didn't demand protected facilities for 911 is another mind boggling issue.
This costs $$$ and usually isn't a problem as there are other ways to communicate. The law-enforcement folks qualify for GETS so get priority on wired/PSTN. They can also get radio priority w/ WPS.
If you don't know the acronyms, see www.ncs.gov.
GETS and WPS are good as long as the system is still connected. TSP and SHARES helps when the system becomes disconnected. Some carriers also have mutual aid pacts, and work with members of the mutual aid pact with spare facilities. Its better to sign up ahead time, rather than waiting until after the problem happens.
Even though those tools are useful, also work on how to maintain your own self-sufficiency until help arrives. There will always be some prioritization of repair efforts.
Although it had a big impact on some of the largest carriers in the region, especially for local services; its always interesting to see other stuff kept working. Not everything broke.
If you're an ISP, you may be able to obtain GETS or WPS for your engineers. This would allow you a better chance of getting a channel to respond to issues. This is a good test to see how your backup plans might work for communication in the case of a larger distaster (earthquake, or other).
Make sure you test the tools you have. The people I know with GETS cards are encouraged to test them regularly and verify they work. If someone has one, I'd be interested to know if it proved to be of value today.
It sucked, but its also an opportunity for ISPs to figure out better ways to do things.
personal opinions only
If my read is correct, this is multiple cuts in multiple locations.
To answer the what-if ("What if this isn't simple vandalism?") : It's not.
-jamie
No RF, no WPS.
If all the base stations are knocked out in a region, and there is no "over" coverage from towers out of the affected region then there are no channels to which priority access can be allotted.
A potential remedy (at least for conventional cell phones) would be to scatter back-up cell sites on high-vantage-point locations. Each would need to be equipped with multiple narrow sectors using high gain antennas. These lofty sites would form a secondary canopy over the region (hence the "over/under" naming). Assuming the secondary sites are hardened, provided with back-up power and trunked with physical diversity (perhaps one links using 70/80Ghz), they should provide some additional protection in situations like this. This would provide some service when primary towers in an entire sub-area are all knocked out. Who knows, in day to day routine usage they might even fill-in a few coverage holes that have been lingering in some systems. From the reports of "zero bars" on cell phones, we can presume no "over" canopy is in operation in that region.
There are other radio systems, but their scope is limited. Cellular provides wider availability. Granny can use her Jitterbug to call for help. Similarly, many business burglar/fire alarm systems use cellular to transmit alarms to the central station. With terrestrial and radio alarm reporting knocked out, many businesses will be sitting ducks.
But why waste the money on system improvements. Best to conserve the funds to pay bonuses to the corporate executives. No matter how egregious the error or omission, they always walk away with big checks, and the rest of us waddle away looking for Preparation-H.
Charles Wyble wrote: