RE: Computer systems blamed for feeble hurricane response?

http://www.fema.gov/staff/extended.jsp

Lists an "IT Services Division" that has ~250 possible points of
contact.

Surely one of them has some clue... :-/ I think this sort of
problem
shows the endemic disease currently in place at FEMA. It's not just
an "IT gaffe" or firewall mistake. It's a failure much more
serious,
sadly.

ObOp: Email is NOT a reliable form of communication.

      DHS shouldn't start to think so either. NANOG
      shouldn't worry about if someones email is working
      as a byproduct, but sure worry if the store and forward
      function of an ISP is. '

  Anything below that is the individual SP's problem, IMO.
      Perhaps there are reasons some corporate or volunteer
      mail service is not working i.e. blocked, disallowed on port,
      etc.

ObNotOp:

Anyone who needs to contact FEMA, already knows how. If they
are using a web page address, they probably shouldn't be contacting
FEMA directly, but working through their own government hierarchy.

ObOp: Email is NOT a reliable form of communication.

         ^^^ unrelated and I disagree...

      DHS shouldn't start to think so either. NANOG
      shouldn't worry about if someones email is working
      as a byproduct, but sure worry if the store and forward
      function of an ISP is. '

        ^^^ There exist networks and operators who do not run ISPs. People often forget.

      Perhaps there are reasons some corporate or volunteer
      mail service is not working i.e. blocked, disallowed on port,
      etc.

        ^^^ I'm sure there is a reason. My first guess is that it's broken. My second is that it was never intended to be a domain used for email and the website techs never got the memo.

ObNotOp:

Anyone who needs to contact FEMA, already knows how. If they
are using a web page address, they probably shouldn't be contacting
FEMA directly, but working through their own government hierarchy.

In dealing with incidents it is possible to cover many areas of failure. There are many cases where the chain of command, the hierarchy process and many other elements fail. In those times, sometimes getting to a website and finding a contact address serve as a real means of communication and should be regarded as such. History proves the point that out of band comms and other forms of handling are often used during an emergency that were not expected.

Right now if I go to http://www.fema.gov and click on "How to get help" and then "Contact us" I get a 404 forbidden. That's a failure. It's narrow-sighted to underestimate the importance of things like FEMAs website in dealing with national disaster and incident response.

-david

...

Which indeed means they have no MX servers listed and that MAY be a
problem for some mail servers (though normally mail servers are supposed
to send email based on A record then).

Obviously not having MX record is not considered to be good email
service setup in this century and it also means if they receive
too many messages and their mail server can not handle all the
connections, the mail will bounce (since there is no secondary
mail server to go to).

...

Wrong ...

...

Actually it is worse than that. fema.gov has an IP (205.128.1.44) which does
not respond for mail so most MTA will try the IP first, meaning that most
mail will fail even is ns.fema.gov or ns2.fema.gov do answer for mail.

...

Wrong ... in detail, anyway ...

...

Uh, which mainstream mail server out there is ignorant enough not to
send to A record?

...

None, one may hope, although MS keeps amazing me ...

...

SOA said root.ns2.fema.gov. It might be someone actually read's roots mail ?

...

This [deliberate human intervention] is the ONLY WAY that mail might be
delivered to ns2.fema.gov ...

...

So having no MX server is really not such a good idea nowdays...

Obviously FEMA's problems are a lot worth since ip address 205.128.1.44
is behind firewall and does not accept port 25 connections.

...

*sigh*

Making a network connection using the application "telnet" vs the application "sendmail" (or whatever MTA one uses) seems to be the same when doing a tcpdump on the data. I am not sure how a firewall would know -- purely at the network layer -- what the other side's application was/is that initiated the connection. Yes, the other end could try and connect back to the host, but there is no 2 way traffic as the 3way handshake is not completing and I dont see any other traffic coming back from that host attempting to discern any info.

         ---Mike

I don't know, myself. I said they try. Perhaps they succeed. Perhaps
they check the speed of incoming queries. Perhaps they try to use a
Telnet OPTION. I don't know. Perhaps it's a sales gag. [I think it
was a telnet OPTION, actually.]

OK, I'll bite. A long time ago, I saw code that would trap the fact that many
telnet binaries would send option negotiation on ports other than 21. What
are they keying off now? Since the host in question gave a 'Connection Refused',
it obviously made its decision based on the initial SYN packet. So what are
they looking at? TCP options? initial window? other?

16:25:37.240700 IP h80ad2467.async.vt.edu.43404 > listserv.vt.edu.smtp: S 1026334142:1026334142(0) win 5840 <mss 1460,sackOK,timestamp 3672334 0,nop,wscale 2>
16:25:57.420455 IP h80ad2467.async.vt.edu.45093 > listserv.vt.edu.smtp: S 1074086420:1074086420(0) win 5840 <mss 1460,sackOK,timestamp 3677379 0,nop,wscale 2>

One was a telnet connection, one was Sendmail. Damned if I can tell.. :wink:

Of course, a busticated firewall trying to tell the difference *would* explain why
they aren't accepting mail. :slight_smile:

Not everyone implements that. You would make a large part of the internet unreachable via email

vinyl# telnet mx2.mail.yahoo.com 587
Trying 67.28.114.36...
telnet: connect to address 67.28.114.36: Connection refused
Trying 4.79.181.13...

         ---Mike

There is no requirement - even in this century - for MX records. It is
a Good Idea(tm). But not a requirement. Lack of MX records does NOT
mean that you lose the store-and-forward capability of SMTP. Lack of a
secondary server, while equally not a Good Idea(tm), does NOT mean that
you lose the store-and-forward capability, only that you exercise it
more often.

I don't disagree but it so happens not all mail software is fully RFC2821
compliant - that maybe either by choice or by ignorance of the authors
or simply not reading RFC closely enough. If you ever wonder how bad it
is - try looking at your Received header lines and compare to what RFC2821
says about them. So yes, I'll say it again - there are mail servers that don't respond appropriately when there is no MX record.

Besides what RFC2821 says, it is also well-known that use of 'A' if
there is no 'MX' is feature to support legacy [pre-1990] systems/domains and for individual hosts that don't usually used to receive email (but still have working postmaster address, etc). And every recent manual, book, etc for mail server software says that when setting up *domain*
to receive email MX record must be setup.

Oh, and also ... please consider that some firewalls try to discern
whether the connection on port 25 is from a mail server or from Telnet.

Could you elaborate on how firewall will determine if the connection is
from mail server or from telnet on port 25?

They both will have the same destination TCP port, both will use random source TCP port number, etc. I really don't see how L4 device (like most firewalls are) can do this unless they keep list of known mail servers ip addresses - and with millions of them I don't think anyone
is crazy enough to compile that into their firewall.

william(at)elan> Could you elaborate on how firewall will
    william(at)elan> determine if the connection is from mail server
    william(at)elan> or from telnet on port 25?

Perhaps because most telnet clients will attempt telnet option
negotiation? If so one could avoid this by using a client such as
netcat...

  -roy

Telnet option negotiation is at Layer 7 after TCP connection has been
established. Firewalls typically don't operate at this level (TCP session
is Layer 4 if I remember right) and would refuse or reject (difference
type of ICMP response) based solely on attempt to connect to certain
ip or certain TCP/UDP port.

Application layer firewalls have existed for at least 6 years.

--Adam

Adam McKenna wrote:

Telnet option negotiation is at Layer 7 after TCP connection has been
established. Firewalls typically don't operate at this level (TCP session
is Layer 4 if I remember right) and would refuse or reject (difference
type of ICMP response) based solely on attempt to connect to certain
ip or certain TCP/UDP port.

Application layer firewalls have existed for at least 6 years.

AAAAAAAAAAAGGGGGGGGGGHHHHHH!

But the point is that you would still establish a TCP connection
before a MTA, firewall, IPS, or whatever could know it was telnet!
The FEMA address that started this whole thing was timing out. You
can tell the difference between a telnet filter and something
completely, silently blocking 25/tcp.

CAN THIS DIE NOW? Pulllleeeeeese...

You're talking about the packet filters that marketeers sell as
"firewalls". The best firewalls operate at the application layer. And,
yes, that's an OPINION, no need to rave.

No they won't. I don't have any copies of BSD to hand from before 1987,
but even then Berkeley Telnet would not do unsolicited option negotiation
if you specified a port number.

Tony.

Wrong host. 587 / msa is for outbound email

suresh@frodo 16:57:22 [~]$ telnet smtp.mail.yahoo.com 587
Trying 216.136.173.18...
Connected to smtp.mail.yahoo.com (216.136.173.18).
Escape character is '^]'.
220 smtp017.mail.yahoo.com ESMTP
quit
221 smtp017.mail.yahoo.com
Connection closed by foreign host.

Sorry, my point was that you could not just assume the sending host was a mail server by connecting back to the host on the submission port as it might not be listening on that port or that because the host sending is not in the MX list, that its not a mail server.

         ---Mike