NYT: Attacks Expose Telephone's Soft Underbelly

The New York Times is reporting about some of the issues Verizon
is facing with its New York recovery.


Although most people are focusing on Verizon, things are much
more interesting when you look at the interaction of all the
different carriers in lower Manhattan. And what I think is
more important, what happened outside of Manhattan. Carriers
restored service be re-routing circuits through Cleveland and
London (Yes, England).

One issue that did come up was government planning was limited
to Verizon. So alternate capacity which could have been used
during the first few days went unused, and some services were
disrupted because employees of non-Verizon carriers weren't
allowed into the area.

I'm confused as to this comment:

"Assuming they are willing to spend the money, business customers can
achieve redundancy, or surplus and backup capacity, by running cables to
several different central offices or, in some cases, by using several
different communications carriers. Several of Verizon's competitors, in
fact, have benefited from the disruptions by signing up new customers in
Lower Manhattan. "

Are they referring to voice CLECs (or data CLECs for that matter)?

I don't see how this situation could have helped them, but only hurt them.
I mean, if you have a physical facilities issue (severed copper/fiber optic,
damaged CO), then you are gonna have problems with telephone or data service
no matter who you use. Plus, some of us know how long opening and
resolution of a trouble ticket with an ILEC can take when coming from a
CLEC, but now you could only imagine when Verizon has a serious issue, like

Even if it was a CLEC who ran/leased their own fiber/copper outside of
Verizon's network, chances are they are in the same tunnels as Verizon.

Are they maybe talking about telephone/data over satellites/microwave or
something else wireless?

- james

My employer is one example of an alternative, and I know of several
other companies that could fit the bill as well.

MFN (www.mmfn.com) can sell a private fiber optic ring to a customer.
In many cases (and Manhattan is a good one) that can be a truly
physically diverse ring. We have a number of customers who insist
on truly diverse rings, and things like two building entrances at
least 100 yards apart. For the most part (because nothing is 100%
sure) it is possible to protect against any single cable cut. The
largest problem with our service right now is that the barrier to
entry for the customer is higher than it should be -- if you need
a single T1 we can't compete.

For what it's worth, I believe it to be the case (*) that we had fiber
rings with a node in one of the trade center towers that were fully
operational (minus that node) after the disaster.

In the end, it's often a matter of what you're willing to pay for,
or if you leave it up to your carrier what risk they are willing
to assume. We've all fought to drive the price of T1's into the
ground, and at $200-$300 for a local circuit they aren't going to
build redundant fiber rings. You're going to get a single run
collapsed ring every time, simple economics. Copper services which
are still used in so many places are virtually impossible to make
path redundant due to the technologies and distance limitations

Verizon offers products that connect you to two CO's. I've ordered
them before. Companies like MFN can bypass CO's completely with
redundant fiber rings. Companies like Yipes and Cogent often build
their IP services in redundant metro configurations. Companies
like Level 3 and MFS have upscale redundant access products. There
are alternatives that could have protected the vast majority of
the service that went down. The problem with all of them is that
they cost more $$$'s than the non-redundant service. I think this
event is making a lot of companies look hard at their cost control
strategies and reconsider their disaster recovery plans. Can you
be 100% sure you'll always be up? No. Can you add several 9's to
the services currently available, absolutely.

* I don't work for the fiber side of the business and only know a
  little bit about the impact of this specific disaster to our

The majority of the CLECs pick up your copper in the 'remote', each Central
Office is serviced by a large number of remotes that concentrate the copper
and usually implement a protocol known as GR.303 to oversubscribe the
remote, normally at about a 4:1 ratio. The resulting lines are then routed
back to the CO via high cap circuits, usually on a SONET ring. The remotes
may be just fine, but if the CO is damaged/destroyed your calls have nowhere
to terminate.

CLECs and DLECs pick up the copper at the remotes just like the incumbent
telco, but they jump off of the SONET ring in different locations, not at
the CO. If you had service via the incumbent AND via one of the xLECs it
would generally require widespread devastation in multiple locations (or at
the remote itself) to take you completely out of service for any extended
time period.

FYI: We don't service the NorthEast US and I don't have explicit topology
information for NYC, but this description covers every telephone office I
have ever seen.

Tim McKee