Bee attack, fiber cut, 7-hour outage

http://www.chron.com/disp/story.mpl/ap/tx/5148125.html

(AUSTIN) Telephone service was out for seven hours in rural Central Texas after bees attacked a construction worker, causing him to jump off his tractor and hit a lever that lowered an auger that sliced a fiber-optic line.

Sean Donelan wrote:

http://www.chron.com/disp/story.mpl/ap/tx/5148125.html

(AUSTIN) Telephone service was out for seven hours in rural Central Texas after bees attacked a construction worker, causing him to jump off his tractor and hit a lever that lowered an auger that sliced a fiber-optic line.

Is this a 7 hour outage a comment on rural Central Texas availability of fiber splicers or novel ways fiber gets cut?

Anytime you talk about "rural" I'm impressed with 7 hours, however -- isn't SONET supposed to make this better?

Deepak

We had a customer hit by this, and actually saw services restored for a few minutes in just four hours, but then they went back down.

--Chris

I'm not in Texas, but I am rural - there's plenty of places around here
where it's just not economically feasible to run 2 diverse fiber paths
to a town. Heck, a lot of these places didn't get their *first* fiber
until fairly recently.

http://maps.google.com/maps?f=q&hl=en&geocode=&q=grundy,+virgina&ie=UTF8&ll=37.279107,-82.099457&spn=0.414686,0.782776&t=h&z=11&om=1

Not a place you'll find a redundant SONET ring. :wink:

Deepak Jain wrote:

Sean Donelan wrote:

http://www.chron.com/disp/story.mpl/ap/tx/5148125.html

(AUSTIN) Telephone service was out for seven hours in rural Central Texas after bees attacked a construction worker, causing him to jump off his tractor and hit a lever that lowered an auger that sliced a fiber-optic line.

Is this a 7 hour outage a comment on rural Central Texas availability of fiber splicers or novel ways fiber gets cut?

I'm thinking that getting hit by an auger might put the fiber more into the "mangled" category rather than simply cut.

Anytime you talk about "rural" I'm impressed with 7 hours, however -- isn't SONET supposed to make this better?

SONET... yeah, right. We had a fiber-seeking backhoe take out some fiber in the mid-Willamette Valley recently - it took out long distance for several smaller local phone companies for about that long as well. 911 service for at least some people too. I'm interestedly awaiting the final word on what happened there...

Jeff

Sure, if:
1. the protect path is configured and enabled
2. both the working and protect paths don't run through the same conduit/duct/buffer

jms

I'm forking this thread to complain about vendor L's international long haul network. Protected Sonet service (T3). DC to UK. I see more outage notifications than you'd *believe* since the service was established for a customer a few weeks ago. Whether its mandatory fiber relocation or some work in France... or all of the above.

Now, getting a notification about a 50ms "switch hit" for protected service is great. No worries or concerns -- even superlative.

However, when I see "Location of Maintenance: France" and a 5 minute outage for a protected SONET service on a supposedly redundant, high quality International voice/data network... well, let's just say I'm not impressed -- on 36 hrs notice, no less.

I can't do anything with respect to an SLA since there is advanced notice, but isn't it reasonable to assume that in this day-and-age running a properly protected T3 isn't *that hard* anymore??? Especially in advance – you know, shunt the traffic to one your other circuits because, you know, you are supposed to have this massive network.

I think I just put my naive hat on today and I need to go drink a little clue. I'm sure I'll be applying some clue with a manual re-route over another Vendor's network shortly. :wink:

Deepak

I am continually amazed at how often this is the case.

I realize that it's expensive to run these lines but when you put your
working and protect in the same cable or different cables in the same
trench (not even a trench a few feet apart, but the same trench and
same innerduct), you have to EXPECT that you're gonna have angry
customers. And yet when telco folks learn that this has occured, they
often fein being as surprised as the customers.

Truely amazing.

I realize that it's expensive to run these lines but when you put your
working and protect in the same cable or different cables in the same
trench (not even a trench a few feet apart, but the same trench and
same innerduct), you have to EXPECT that you're gonna have angry
customers. And yet when telco folks learn that this has occured, they
often fein being as surprised as the customers.

Has anyone calculated what the cost of doing this correctly once vs the ongoing support/SLA/etc issues of repairing it when it goes boom is? I've gotta believe that for >90% of the situations where diverse routes exist, just being used as dual linear paths, its cheaper in the long term to do it "right" and cut the size of your outside plant crew (assigned to break/fix) by 90%. :slight_smile:

Deepak

I realize that it's expensive to run these lines but when you put your
working and protect in the same cable or different cables in the same
trench (not even a trench a few feet apart, but the same trench and
same innerduct), you have to EXPECT that you're gonna have angry
customers. And yet when telco folks learn that this has occured, they
often fein being as surprised as the customers.

.. and as long as they are the only telco with copper in the area,
they could care less, I guess?

jump off his tractor and hit a lever that lowered an auger that sliced
a fiber-optic line.

ps: That story had a kind of "... that killed the rat that ate the
malt that made the house that jack built" feel to it.

srs

There's a difference between folding a ring or pushing out a spoke to feed a
few customers and providing connectivity to a town.

I think building a SONET ring, or any kind of redundancy, has more to do
with a rural telco's commitment to it's customers than the bottom line.
Remember, the building of plant contributes to the cost study, so it may end
up having zero cost in the end.

Frank

I'm forking this thread to complain about vendor L's
international long haul network. Protected Sonet service
(T3). DC to UK.

I wonder if anyone is using PWE3 for this kind of service,
perhaps in an academic/research environment? It would be
interesting to compare notes on outages, latency/jitter, etc.

--Michael Dillon

Has anyone calculated what the cost of doing this correctly once vs the ongoing support/SLA/etc issues of repairing it when it goes boom is? I've gotta believe that for >90% of the situations where diverse routes exist, just being used as dual linear paths, its cheaper in the long term to do it "right" and cut the size of your outside plant crew (assigned to break/fix) by 90%. :slight_smile:

It is definitely cheaper to "do it right" the first time, but to
"keep it right", the operator needs to pay very close attention
to each and every circuit groom that they perform, less they
end up with a degenerate loop somewhere. Depending on the
state of their circuit routing database(s), the exercise of checking
for overlap against "the other half" of the ring can anywhere
from trivial to impossible.

I don't think operators intentionally foul this up, but it's real
easy to get wrong, particular in the fallout after accumulating
a bunch of different companies fiber plants and circuit route
systems and trying to consolidate everything for savings. I'm
just providing this in answer to "why can't telcos get this right",
as there's no reason to think that grooming activity was at all
involved in why this particular carrier got stung...

:wink:
/John

Typically for a service like that, the carrier would have one or more
long-haul rings,
either SONET or DWDM with the SONET on top of it or something,
interconnected to access rings from a local access carrier at each end,
either close to the customer endpoint or possibly back at regional
interconnects.
The long-haul carrier might not control the local access carrier
except through an SLA,
so it could be that the local carrier messed up. One hopes that the
interconnections
between the networks are also at diverse POPs (at least for big
countries like France),
but it is possible for the interconnections to fail clumsily.

The "shunt the traffic to other circuits" approach is naively correct,
but I've seen more than one case of "discover that one side of an
access ring has failed
by shunting the traffic from the other side to do maintenance and
having a customer call you
a couple of minutes later to report an outage at their headquarters",
which is absolutely never supposed to happen, of course :slight_smile:

What a 5-minute outage sounds like to me is "The early-90s T3-based
restoration system
recovered the circuit on an end-to-end basis using DACSs instead of
SONET restoring it",
but I'd be surprised to see than happening at a newer carrier like
Vendor L who built most of their network after SONET technology became
affordable.