NANOG36-NOTES 2006.02.15 talk 3 Katrina Panel

2006.02.15 Katrina Recover Panel
moderator: Sean Donelan, Cisco

Members: Paula Rhea, Verizon
Josh Snowhorn, Terremark
Bobby Cates, NASA

Sean Donelan was with SBC when Katrina hit,
now with Cisco. Dave couldn’t be here, but
Sean will do his Bellsouth slides.

Lessons Learned
Industry has to be able to function as a first
responder to provide critical infrastructure in
support of state/local response.
certain sectors may need heightened support,
including power and voice/data communications
Providing security in times of crisis may fall back
to the private sector
Need to understand how the Government works in a crisis
National Response Plan, FEMA system, etc.

Bellsouth lost COs for first time in 100+ years
of business. When you get a direct hit, you
will be impacted, period. More important is how
your recover!

Most national disasters are pretty quick; we know
how to deal with short term, but as the issue drags
on, security for personnel becomes more and more vital,
and is turned over to private sector, public security
is engaged on more important issues.

We need to help shape up the government to avoid
issues like Katrina from happening again.

Bellsouth, lsessons learned
partnerships with other carriers, state and local
government, the power companies, and the federal
government made the difference
experience and trust are key in a crisis
Get involved–know how to reach teh communications
ISAC and national coordinationg center in a crisis
NCS at or NCC: telecom-isac at
operational 24/7/365
Konw what programs are available to you and your customers

Bobby Cates from NASA up next.

Supported first responders right after Katrina; they
were providing video coverage, supported voice over
IP, sat phones, etc. in the first days. The commercial
facilities were better than gov’t lines, actually.
When president came in, military took over all
satellite frequencies, so VoIP over commercial
internet was what was left. Phones from Bill Woodcock
from PCH, servers from some bay area folks; got gear
loaded onto a C5 that was warming up and flew it out,
the costs was less than one set of satellite phones.

TSP was interesting for Katrina; for higher bandwidth,
higher pricing, not much diff for TSP and non-TSP
restorral. Wimax and voip pretty much saved the day,
easy to implement.

Josh Snowhorn, NOTA, didn’t take Katrina too bad,
but Norma? hit him hard.

Only 3 cat 5 hurricanes (andrew, 92, camille 89,
and 1935). cat 4 and cat 3 hit more often, hundreds
a year of the smaller ones.

Saffir-simpson hurricane scale.
cat 1; winds 74-95, wind, water
cat 2: 96-110, storm surge 6-8ft
cat 3: 111-130, surge 9-12ft much structural damage
cat 4: 131-155mph, surge 13-18ft at landfall (katrina at coast)
cat 5: 155mph, surge 18ft

Wilma was cat 5 before landfall, as was katrina;
wilma was lowest barometric pressure every recovered.

27 named storms last year, lots of warm water heading into
the gulf.
formed off the bahamas, very little warning before it
hit into south florida. did a bunch of power line damage.

NOTA faced many issues during the storm.
2005 most storms in recorded history
2005 hurricane season went 27 named storms representing
first time in history that the naming scheme went into
the greek alphabet

NOTA–pre wilma, 3 happy balls
100mph winds on the curtain should be able to withstand it.
Lost one of their roof balls during wilma

NOTA lost commercial power, went on gensets for 31 hours
during katrina in July

NOTA lost commercial power for 10 hours with wilma, but
had to stay off for 30 hours it was so dirty

majority of enterprises and business in s florida
without power for 10 days

the day after wilma, had no less than 20 truckloads of
servers and infrastructure arrive at NAP loading docks
with sales people and contracts

within 2 days of the passing of wilma, we began to recieve
phone calls askign for fuel truck help from undersea
cable operators and large enterprises; everyone
pitched in to help all of the other operators in the
12 undersea cables coming in, you cut them off, s.america
largely goes away.

only 1 carrier fully lost a CO in north miami, bringing
down their circuits that came out of the NAP; water came
in, shorted things out.

Many companies did not plan properly for power failures
and staff recovery and access to systems after the storms
have passed.

large portion didn’t have DR plans or backups

staff who loose their homes need food/water, won’t go to

thow who want to work cannot go to devestated offices
so they need to work from phone

getting employees acces to systems is the singular issue
that IT diretors face post katrina and wilma

KEEP a dialup access poitn; it’s often the only thing
left in a disaster like that.

Sean: to NASA; for packet traffic, what traffic did you
see–lots of traffic you didn’t plan for, or business
as usual. The emergency response was for NCS, FEMA,
DoD, as well as their 2 centers that were hit. They
had a Ku band sattellite, could go to 45mb max. but
no QoS, so everyone has to be good neighbors on it.
NASA was using less traffic on the band than everyone
else. voip, other traffic was not too big, biggest
use was the football video stream.

Bill Norton asks if there was any impact due to the

C-band dishes on the 2 big ones; Ken was up there
in the small Ku-dome, 11meter dome, thinner pannels
so Ku band can get through it. It’s a protective
surround for the fully mobile dish, one panel got
pentrated, then the whole ball went to pieces.

What are the lessons learned that will change
how we behave in the future.

Paula notes that 100 mile used to be a sufficient
rule for disaster site; now, with a 250mph swath,
expand distance, and make it inland!

NASA found they actually accomplished more than
they expected to;

Josh notes they host 911 systems, DoD, etc, worked
well with them; GC lost their offices, they hopped
into their building. Biggest challenge is staffing,
making sure everyone has food, water, making sure
they’re safe and their families are safe so they
can focus on the work that needs to be done.
Bill meant more about the sensitive web of interconnectedness
that could affect multiple pieces. Power, physical bridge
outages, water, etc. What physical lessons did we learn
from it?
Internet design is a semi-mesh, not a full mesh; there
were some sites that stayed up the whole time, not all
paths out of NO were impacted. IP can re-route, but
you have to have a physical path, and a policy that
lets it do so.

Chris Morrow, how do organizations that have BCP for
simple things like server backups, etc. deal with the
bigger cases, like “I lost my whole office/central
office”–how do you make use of them.
MPLS/private IP service; product sold for backup
using vstat access. five large firms called, were able
to give them satellite access to stay on line.
MCI blue truck, an empty truck with PCs in it, FEMA
rented 3 of them, turned into a mobile office.
Josh notes that Mobile worker recovery units, mobile
switches, etc had already been deployed to NO, when
wilma hit, everything was already deployed to NO so
nothing left for south florida.
Again, Josh emphasises KEEP A POTS LINE!!!

someone comes to mike, notes that with such a huge
disaster, did it allow for shedding old legacy gear
infrastructure, practices, etc?
Paula notes there’s a huge capital plant infrastructure;
even though parts are destroyed, you want to get dial
tone back as quickly as possible; string cables across
trees, etc to get dial tone as soon as possible. You
don’t have time to do a structured upgrade to latest
gear, you go back to lowest common denominator recovery
gear to get them back up.

Again, many thanks to all the panelists, the moderators,
and everyone else for participating. Fill out your surveys
and be back from break a little before 11am for a short
movie from Bill Fenner from PARC.