Computer systems blamed for feeble hurricane response?

This is the first I've heard of this...

Via The Inquirer:

[snip]

REPORTERS at the Wall Street Journal said they have seen documents which show that a swift response by the US federal government to Hurricane Katrina was hampered because FEMA computer servers crashed.

Michael Brown, FEMA's head, resigned yesterday after being recalled by the Department of Homeland Security to Washington DC.

Attempts by agencies to spur the Federal Emergency Management Agency into urgent action were met with bouncing emails, the Journal said.

It quoted a Department of Health official as saying every email it had sent to FEMA staff bounced. "They need a better internet provider during disasters," the Journal quoted her or him as saying.

A number of US agencies made desperate calls to the Department of Homeland Security and to Congresswomen and men, the article claimed. [Subscription required.]

The newspaper did not say which computer systems FEMA uses.

[snip]

http://www.theinquirer.net/?article=26125

- ferg

Does anyone know what their mail infrastructure looks like? From what I
can see, they don't even have an MX record for fema.gov...

No MX record, and the A record for fema.gov does not accept smtp traffic.

# telnet fema.gov smtp
Trying 205.128.1.44...
telnet: connect to address 205.128.1.44: Operation timed out
telnet: Unable to connect to remote host

$ dig mx fema.gov
;; ANSWER SECTION:
fima.org. 3600 IN MX 0 smtp.secureserver.net.
fima.org. 3600 IN MX 10 mailstore1.secureserver.net.

;; AUTHORITY SECTION:
fima.org. 3600 IN NS PARK5.secureserver.net.
fima.org. 3600 IN NS PARK6.secureserver.net.

[This is Godaddy and their datacenter is obviously in Arizona]

$ dig fima.org
[snip]
$ ;; ANSWER SECTION:
fema.gov. 1800 IN A 205.128.1.44

;; AUTHORITY SECTION:
fema.gov. 1800 IN NS ns.fema.gov.
fema.gov. 1800 IN NS ns2.fema.gov.

$ whois -h completewhois.com 205.128.1.44
[snip]
Level 3 Communications, Inc. LVLT-ORG-205-128 (NET-205-128-0-0-1)
                                   205.128.0.0 - 205.131.255.255
Federal Emergency Management Agency FEDEMERGENCY-1-18 (NET-205-128-1-0-1)
                                   205.128.1.0 - 205.128.1.127

Note: They also have 192.206.40.0/24 (not routed), 205.142.100.0/22
(not routed), 64.119.224.0/20 (not in bgp) and 166.112.0.0/16
(announced by 2828 - XO).

While its possible that L3 or XO could have been down with one of
their southern links, I really dont think it would effect their
Washington, DC customers.

>Does anyone know what their mail infrastructure looks like? From what I
>can see, they don't even have an MX record for fema.gov...

No MX record, and the A record for fema.gov does not accept smtp traffic.

# telnet fema.gov smtp
Trying 205.128.1.44...
telnet: connect to address 205.128.1.44: Operation timed out
telnet: Unable to connect to remote host
#
Then again, it might be that they use different email addresses ? @dhs.gov ?

Their "contact us" page on fema.gov lists several @fema.gov addresses, so
I doubt it.

fema.gov nameserver = ns.fema.gov
fema.gov nameserver = ns2.fema.gov
ns.fema.gov internet address = 166.112.200.142
ns2.fema.gov internet address = 162.83.67.144

Looks Solaris'ish

# telnet ns2.fema.gov smtp
Trying 162.83.67.144...
Connected to ns2.fema.gov.
Escape character is '^]'.
220 ns2.fema.gov ESMTP Sendmail 8.11.7p1+Sun/8.11.7; Tue, 13 Sep 2005
09:49:36 -0400 (EDT)

Well, how is any automated system supposed to find it? Sheesh.
Apparently, that host accepts mail to postmaster; we'll see if it is
actually delivered/read/responded to.

>
> Looks Solaris'ish
>
> # telnet ns2.fema.gov smtp
> Trying 162.83.67.144...
> Connected to ns2.fema.gov.
> Escape character is '^]'.
> 220 ns2.fema.gov ESMTP Sendmail 8.11.7p1+Sun/8.11.7; Tue, 13 Sep 2005
> 09:49:36 -0400 (EDT)

Well, how is any automated system supposed to find it? Sheesh.

Apparently, that host accepts mail to postmaster; we'll see if it is
actually delivered/read/responded to.

SOA said root.ns2.fema.gov. It might be someone actually read's roots mail ?

I will cc that addr so if its read, they can see the thread at

http://www.merit.edu/mail.archives/nanog/msg11505.html

and perhaps comment.

         ---Mike

while the lot of you can debate proper DNS records and what OS their
mail server might be running, does anyone else find it highly odd and
worrisome that they're sending emails to alert FEMA of a crisis,
instead of, I don't know - phone calls? if I'm a federal agency and I
require FEMA's resources immediately, I'm going to pick up the phone
and call them; not fire off an email marked "urgent".

aaron.glenn

http://www.fema.gov/staff/extended.jsp

Lists an "IT Services Division" that has ~250 possible points of contact.

Surely one of them has some clue... :-/ I think this sort of problem shows the endemic disease currently in place at FEMA. It's not just an "IT gaffe" or firewall mistake. It's a failure much more serious, sadly.

-David

does anyone else find it highly odd and
worrisome that they're sending emails to alert FEMA of a crisis,
instead of, I don't know - phone calls? if I'm a federal agency and I
require FEMA's resources immediately, I'm going to pick up the phone
and call them; not fire off an email marked "urgent".

Imagine the following email:

   I have just received a phone call from one of my constituents
   who was visiting friends in New Orleans. She is trapped along
   with 50 other people on the second floor of the Baptist Church
   at the corner of ABC Street and XYZ Avenue approximately a mile
   west of the Superdome. The nearest building with any part of
   it above water is the Odeon Theatre 3 blocks north of the church.
   They have had no water to drink for over 24 hours and they fear
   that some of the children and elderly are literally dying of thirst.
   Is there a fax number I can send this information to?

What part of the above email message is it preferably to
communicate by telephone instead of email?

Many people choose the medium of communication based on
whether or not they want a record of the information communicated.
If they want the content kept secret, they use the phone.
If they want the content recorded, they use email. In my
hypothetical example, a politican from another state is trying
to help a constituent. Naturally, being a politician, they
prefer to have a record of the content.

Also, the sender of the email recognizes that some of the
information is important to have in written form, such as the
address, distance, direction, number of blocks. Things like
that can get wrongly transcribed on a voice call. This is a
life or death situation so it can be argued that it is TOO
IMPORTANT to risk a voice call.

If only FEMA's email infrastructure was geared for emergencies...
Or their web page. Or the web page of the American Red Cross.
Fact is that a lot of organizations got caught with their pants
down because they were not prepared to respond to an emergency
and they were not prepared to use modern communications methods.
Anyone who was searching for friends and relatives during the
aftermath knows how chaotic it was to find information about
the whereabouts of the refugees.

This is a real wake-up call for all kinds of organizations,
not just FEMA and not just government agencies. Could your diesel
gensets cope with an extended running period like the one that
DirectNIC has experienced? Do you have enough bottled water in
your data center to keep critical staff ALIVE in the case of
an extended emergency like this? Anyone who runs any type of
critical infrastructure will have dozens of lessons to learn
after analyzing the outcome of the New Orleans disaster, even
moreso than the 911 commission or the Columbia accident inquiry.

--Michael Dillon