NANOG36-NOTES 2006.02.15 talk 2 Katrina--telecom infrastructure impacts

2006.02.15 Hurricane Katrina: Telecom Infrastructure Impacts,
Solutions, and Opportunities, Paula Rhea, Verizon

A more interactive presentation from her, in the

Verizon Business group–combined MCI/Verizon team.

Hurricane Katrina Recap
Telecom infrastructure impacts
telecom provider successes
business continuity planning
appendices: case examples.

Many of the people in this room would be considered
part of the critical infrastructure for the nation
by the department of homeland security

After world trade center 9/11 issue, there was
a lessons learned; hopefully there will be a
similar report post Katrina.

New Orleans is still very much like a war zone
right now; it’s definitely a disaster recovery
training session for many industries.
Neighborhoods are wiped out; no capital
investments, infrastructure in holding pattern.
Many with no power, 20% of houses condemned.
Neighborhoods that are entirely silent–eerie.

Aim is not to diss anyone specifically, certainly
not in this room; aimed to be an assessment in a
neutral fashion.

Critical infrastructure:
food and water supply

Oddly enough, much of critical infrastructure is
privately owned, rather than government owned.

The domino model says that any one piece will
cause the rest to start to fall.

35th largest city in US
port of new orleans is #1 in US by tonnage
50% of total US grain exports shipped via gulf
10.8% of total US refining from new orleans
5th largest port
Key space shuttle facility in Michoud supported
fuel tanks for international space station

Storm recap
hirricane hit aug 29 2005
135MPH winds, 20foot storm surge sent inland
55foot surges logged in gul pior to landfall

levee failures create secondary crisis
2.3M homes without power
spawned 33 reported tornadoes in NA
1090 fatalities in LA recorded to date

people dancing about cat 5 dropping to cat 4,
thought they were spared, then levees broke;
had been predicted the year before. :frowning:

Still 2500 people missing/unaccounted for.

Map of eastern LA parishes
st bernards/plaque mines parishes between the lake
and the gulf, hardest hit when levees broke as water
headed back towards gulf.
Lack of interoperability between parish govt systems.

New orleans telecom impact (multi-carrier)
1.75M lines down immediately following kat.
38 911 centers out (1/3) initially
1,000 cellular towers out
two class 4 toll switches initially out of service
no power/unable to secure extended diesel fuel

Traffic out of lata logjammed with toll switches out.
LECs had backup power systems, but no fuel.

Took 4 days to inspect causeway to allow emergency
crews into the city with main bridge out. Most
nurses and doctors were in suburbs, not in city.

Central offices post katrina
new orlenas lake co
Venice LA CA
Buras CO

19 COs are totally destroyed, and will have to be

These slides are public domain info, no inside info.

I2/Abilene link from Houston to Atlanta initially out,
restored on sept 8 2005

fiber optic path on lake pchatrain bridge
offline following hurricane katrina

wifi, wimax and voip play key role in area communications

public internet was actually very resilient

Telecom provider successes: alphabetic
1,000 amateur radio operators helped
iridium added 10,000+ new phones to first responders
Sprint/Nextel donated up to 10M
Verizon donated 8M and 200 workers

Carriers have mutual aid agreements; Verizon sent 200
people who volunteered to spend 8 weeks living in a tent
to help rebuild–had to work with armed guards.

The CO rebuilds wasn’t any type of upgrade, it was
bulldozing damaged/destroyed facilities, digging new
vaults, and starting over to restore just what was
in place before hand.

Bill Norton–COs underwater, can you imagine some type
of preventative design that could have been put in
place to help avoid impacts like this?
Most of the area is reclaimed land, 2 miles below
sea level (some dispute about that number).
Bill wonders if they could be built above sea level
somehow. Even if they were, Paula notes that they
wouldn’t have power, wouldn’t have 2 weeks of diesel
fuel to run them, etc. Really, it comes back to the
Randy Bush noted that early on, community based wifi
was one of the early-on means of communication to
daisy-chain packets along.
Roland, from Cisco; did some logistical work with relief;
Verizon donated eVDO boxes to make eVDO to wifi bridges,
did VoIP over wifi to eVDO boxes to juryrig connectivity.
But doesn’t work so well with towers down, and no power.
With the cell phone infrastructure down, that really hurt

Thanks to Todd Underwood/Renesys for their graphs; did
a pre-and-post analysis routingwise.
Top red is LA;
about 170 networks totally out during the bulk period.
teal/TX not impacted,
MS also hit, in tierms of percentage more so than LA
AL somewhat hit.

There’s likely to be some significant number of businesses
that never recover from this.

6months later,
port of NO at 50% capacity
130,000 lines remain out in LO; awaiting local govt
repopulation lans
cellular service fully operational in occupied areas
outside of those with restritcted access
US congtressional budget office estimates damage
at $60B in insured and uninsured losses; insurance is
tagging it at $75B. It would have been far better to
have spent money helping fix the levees up front.

will your organization be able to survive a disaster?
1 what staf, materials, proceedures, and equipment is vital
to the firm
2 what suppliers and resources do I need daily
3 what will you do if your building is not accessible
4 have you planned for payroll continuity
5 who should yo invo

BCP compontents
NIST 800-34 document
develop contingency plan
intro/activation stage/recovery phase/reconstitution stage/
plan backup.

People are still in emergency mode six months later

disaster recovery journal (DRJ)

IAEM, international association of emergency managers
GAP in business continuity
% stating bc plan is important 62%
60% have a plan
58% tested in last 12 months
18% never tested plan
55% implmented security measures
only 49% have bckup servers

Telecom conclusinos
interdepenedt infrsatrurctures incrase risk
business continuity management should include plans
for “worst case” scenrios
factor in data recovery and storage into plans
be flexible

June 1st, hurricane season starts again

DNA records lost from courts, many records that
might have been mandated also lost.

Vision of public internet realized
ARPANet launched predecessor in 1969
Goals: use disparate computers to communicate in order
to provide high reliability and survivablility
Katrina proved it is possible for Internet to meet that