Criminals, The Network, and You [Was: Something Else]

Mailman List Mirror (Read Only)
1

Re-sending due to Merit's minor outage.

- ferg

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

The fact that they're rejecting on a 5xx error based on no DNS PTR is a=

bit harsh. While I'm all for requiring all hosts to have valid PTR
records, there are times when transient or problem servers can cause a
DNS lookup failure or miss, etc. If anything they should be returning a=

4xx to have the remote host"try again later".

Oh, wait till you realize that some of the HTTP returns are bogus
altogether -- and actually still serve malware.

It's pretty rampant right now. :-/

- - ferg

2

My mail servers return 5xx on NXDOMAIN. If my little shop can spend not too much money for three-9s reliability in the DNS servers, other shops can as well. When I first deployed the system, the overwhelming majority of the rejects were from otherwise known spam locations (looking at Spamhaus, Spamcop, and a couple of other well-known DNSBLs). The number of false positives were so small that whitelisting was easy and simple to maintain.

If a shop is not multihomed, they can contract with one or more DNS hosts to provide high-availability DNS, particularly for their in-addr.arpa zones.

It's not hard. Nor expensive.

Paul Ferguson wrote:

3

Hi All,

It seems to me reverse DNS just isn't an acceptable anti-spam measure.
Too many broken reverses exist with smaller companies (try getting a 3rd
party to fix it). It's not that hard for a bot to figure out a DSL's
reverse entry and use that for its HELO. And there are a lot more
effective pre-processing anti-spam measures, including greylisting (with
its own problems) and reputation-based systems.

Best Regards,
Jason

4

You get NXDOMAIN when an authoratitive servers says there is no such domain,
it doesn't occur if the DNS servers aren't available. So I fail to see the
connection to reliability of DNS servers.

All well engineers mail services provide 4xx (or accept the email) on SERVFAIL
(or other lookup failure), if they insist on checking DNS information as part
of accepting email. One has to allow for the case where the mail servers
can't speak to the DNS servers, which may include cases where the DNS servers
are available, but say routing, or other parts of the DNS are fubar.

Serious programmer(s?) spent a lot of time making sure the MTA we use does the
right thing under all error conditions so far encountered, I'd consider
altering that behaviour vandalism. I feel like some sort of clumsy cave man
compared to the authors every time I configure it as it is.

5

On this topic, I again solicit remarks from interested parties on the
IETF dnsop reverse-mapping-considerations draft, which the editors
currently think is baked. If you have remarks about it, now would be
an excellent time to make them:

http://tools.ietf.org/wg/dnsop/draft-ietf-dnsop-reverse-mapping-considerations/

Thanks!

A

6

Your first sentence and your third are in direct conflict, as are your
first and your fourth - please understand that the use of rDNS
(especially generic - as distinct from known dynamic or static) is an
extremely effective tool against the botnets, and can itself be an
extremely useful input into "reputation-based systems".

As for your second sentence, well, you're right in saying that blocking
solely on the perceived absence of, or generic nature of, rDNS naming is
probably prone to false positives, but nonetheless it's not really my
responsibility to ensure that you choose a decent service provider with
the ability to provide proper and current and specific identification
for your IP. If more ISPs dealt with abuse issues on their own networks,
this wouldn't be such a big deal - but it's difficult for me to accept
mail from, say, a host named 'dsl-static-pool.1.2.3.4.bigisp.example.net'
when I've seen hundreds of thousands of abusive messages from hosts with
that same naming convention, all bots. YMM, of course, V.

As for the third, well, now you know why I use generic rDNS detection to
defeat bots. As you say, "It's not that hard for a bot to figure out
[any infected host]'s reverse entry and use that for its HELO". In fact,
that's exactly what many of them do, when they're not forging well known
services or sending unqualified/unresolvable strings in HELO/EHLO. And
that, in itself, is responsible for over a fifth of our SMTP-time spam
detections (and rejections, so there's no outscatter, unlike with a wide
variety of "antispam" appliances, such as Barracudas). It's a sensible
and sane perimeter defense tactic, far better than what I see most doing.

If you're running a mail source, make damn sure it's got non-generic
rDNS /and/ that it's configured to HELO with something that doesn't make
it look like a "bot", and you'll stand a much better chance of
delivering mail to me and my service's users. If you're not, well, the
time is running short for you to fix that brokenness.

Steve

7

Hi Steve,

In my opinion, the first and fourth statements are not necessarily in
conflict. A reputation system based purely on reverses is pretty broken.
Also, it is not necessary to use it as a factor in calculating a very
reliable reputation. I'm having trouble seeing how the first and third
are in conflict as well, but I may be indexing statements differently.

Regarding the second, you're absolutely right. It's not your
responsibility if a 3rd party doesn't have a rDNS entry (at all or
non-generic), however the reality is you're going to have to deal with
it anyway. If your customers allow you to tell the senders to buzz off
and fix it, that's terrific. However, you're in a more authoritarian
(IT-wise) environment than most I would suspect. Also, you risk hurting
your customers. As an example, it's not a suitable answer to our law
firm customers who are critically-dependent on receiving e-mail from
hopelessly broken senders.

As for the third, well, now you know why I use generic rDNS detection

to

defeat bots. As you say, "It's not that hard for a bot to figure out
[any infected host]'s reverse entry and use that for its HELO". In

fact,

that's exactly what many of them do, when they're not forging well

known

services or sending unqualified/unresolvable strings in HELO/EHLO. And
that, in itself, is responsible for over a fifth of our SMTP-time spam
detections (and rejections, so there's no outscatter, unlike with a

wide

variety of "antispam" appliances, such as Barracudas). It's a sensible
and sane perimeter defense tactic, far better than what I see most

doing.

It's not disputable that rejecting generic rDNS hits (or failures
depending on your point of view) will gain you benefits. What I think is
disputable, is the benefit to false positive ratio. About 60% of our
botnet analyses show unqualified, or outright out-of-spec HELOs. One can
catch the remaining 40% through correlation of certain SMTP factors with
the results of content-analysis. Near real-time data mining of both
informational inputs shouldn't be underplayed. Lastly, I fully agree one
should reject as much as possible before the SMTP session ends.

Whether or not rDNS is a good anti-spam measure for you entails a lot of
factors. I posit from our own statistical analyses the benefit to pain
ratio issue is not high enough. Particularly, when there so many other
correlations you can run that have lower false positive rates.

Best Regards,
Jason

8

I've always wondered why the bad guys can't wrap postal packages correctly or spell.

http://www.usps.com/postalinspectors/pos84.pdf

On the other hand, if you get a package that looks like that, is it really worth taking the chance? Appearences can be important.

Hopeless broken senders seem to figure out how to fix things when their
critically important (sent via an unreliable e-mail) messages keep getting
returned to sender.

9

In my opinion, the first and fourth statements are not necessarily in
conflict. A reputation system based purely on reverses is pretty broken.

Actually, it's amazingly reliable, when the patterns (literal string
matches and regular expressions) are carefully compiled and cross-checked.

One example of the thousands of patterns I block outright is *.fios.verizon.net.
I do this (a) because I'm seeing lots of spam from generically-named hosts
in that subdomain, e.g., within the last hour attempts have been noted from:

  relay=pool-71-164-205-139.dllstx.fios.verizon.net [71.164.205.139]
  relay=pool-71-170-104-162.dllstx.fios.verizon.net [71.170.104.162]
  relay=pool-71-170-70-195.dllstx.fios.verizon.net [71.170.70.195]
  relay=pool-71-172-239-30.nwrknj.fios.verizon.net [71.172.239.30]
  relay=pool-71-189-201-76.lsanca.fios.verizon.net [71.189.201.76]
  relay=pool-71-190-246-40.nycmny.fios.verizon.net [71.190.246.40]
  relay=pool-72-73-220-84.cmdnnj.fios.verizon.net [72.73.220.84]
  relay=pool-72-76-240-99.nwrknj.fios.verizon.net [72.76.240.99]
  relay=pool-72-84-85-199.nrflva.fios.verizon.net [72.84.85.199]
  relay=pool-72-89-97-187.bflony.fios.verizon.net [72.89.97.187]
  relay=static-71-183-52-18.nycmny.fios.verizon.net [71.183.52.18]

(b) because I've yet to receive a single report of a false positive from
this pattern after using it for a considerable period of time and (c) because
Verizon seems disinclined to do anything about the problem other than
have their paid professional spokesliars repeat the usual B.S., viz.:

  http://www.theregister.co.uk/2007/09/10/isps_ignore_strorm_worm_and_other_malware/
  ISPs turn blind eye to million-machine malware monster
  By Dan Goodin in San Francisco
  Published Monday 10th September 2007 06:02 GMT

  [...]
  Verizon turned down requests for an interview with a security
  engineer, but a spokeswoman said officials are aware of the
  rankings and are working to put new measures in place by the end
  of the year to curb the spam flowing out of its network. "We are
  concerned about it," the spokeswoman, Bobbi Hensen, said. "We
  don't like spam. We are aggressively working on that."
  [...]

Uh-huh. This problem was reported to Verizon roughly FIVE YEARS ago,
and should have, of course, been competently addressed withing a matter
of days. It hasn't been. There is no sign that it will be. It's therefore
been necessary for those of us enduring torrential quantities of
Verizon-originated abuse to take appropriate defensive actions.

Like rejecting all SMTP traffic from *.fios.verizon.net.

And Verizon is merely one of the offenders -- the article cited above lists
several others. I just happened to single them out for use as an example
here because I found the contrast between their years-long history of utter
negligence and their officially-stated position to be particularly striking.
Comcast, Charter, SBCGlobal, Ameritech, Level3, SWBell, Nextgentel, Pacbell,
and Qwest, just to name a few off the cuff, are equally culpable.

Anyway: the use of generic rDNS patterns for outright rejection turns out
to be quite effective with a very low FP rate.

Regarding the second, you're absolutely right. It's not your
responsibility if a 3rd party doesn't have a rDNS entry (at all or
non-generic), however the reality is you're going to have to deal with
it anyway. If your customers allow you to tell the senders to buzz off
and fix it, that's terrific. However, you're in a more authoritarian
(IT-wise) environment than most I would suspect. Also, you risk hurting
your customers. As an example, it's not a suitable answer to our law
firm customers who are critically-dependent on receiving e-mail from
hopelessly broken senders.

Any firm that is critically dependent on email (beyond an intranet
environment) is being naive and foolish by relying on a known-unreliable
communications medium. Connections fail. DNS breaks. Servers croak.
Disks fill. Poor software is deployed. And the entire Internet-wide
infrastructure for mail is under constant stress from spam, DoS attacks,
misconfiguration, and outright stupidity (e.g. SAV).

As to hopelessly-broken senders, we do not do them any favors by
accomodating their brokeness. It is better in the long term for all
of us to educate them about the not just the de jure, but the de facto
minimum requirements for mail servers -- which in 2007 include not
only functional rDNS, but rDNS pointing to non-generic hostnames.

---Rsk

10

here because I found the contrast between their years-long history of utter
negligence and their officially-stated position to be particularly striking.
Comcast, Charter, SBCGlobal, Ameritech, Level3, SWBell, Nextgentel, Pacbell,
and Qwest, just to name a few off the cuff, are equally culpable.

They all suck isn't very useful information. Although collectively they've probably fixed hundreds of thousands of customer computers over
the years, like bad Boston drivers, there is always more.

Instead of they suck, it might be more useful to highlight providers of
similar scale which you think do a good job which others could emulate.

Anyway: the use of generic rDNS patterns for outright rejection turns out
to be quite effective with a very low FP rate.

Some people think that users on dynamic addresses should be read-only, and
not allowed to operate servers or send messages. Like most forms of red-lining, it tends to become self-fulling. Websites that only support Internet Explorer probably get very few false positives because people affected are used to working around that or just ignore them. Networks
that don't update their Bogon lists probably get very few false positives
because people learn to work around them or ignore them. And so on.

Unlesss accept all those messages from those addresses and check them, you really don't know the false positive rate. You only know the self-reported complaint rate; which is usually a fraction of the actual false positive rate.

11

Instead of they suck, it might be more useful to highlight
providers of similar scale which you think do a good job
which others could emulate.

How about some smarter statistics. Instead of counting the spam emails
from Network X, count the spam sources and divide that by the number of
end user customers (or hosts) in Network X. By doing this you get a
clearer picture of who is cleaning their house, and who is letting it
slide.

Think of a messy house. You say that there were 8 dirty plates in the
living room, on the floor, the sofa the coffee table. Horrible, right?
Not if there are 8 people living in the house. In that case it
represents one evening of laziness, going to bed without cleaning up
first. But if only one person lives in the house and there are no
guests, then the 8 dirty plates represent a big mess.

Whenever you scale up anything, small nits also grow in absolute
magnitude. The small scale operator who ignores the nits is following
the same practices as the large scale operator who ignores the nits. If
there are lots of nits, I want to know if the large scale operator
should be criticised for not adjusting their processes to deal with
scaling up, or whether somebody really is being incompetent. There are
different remedies to the two situations. Scaling issues can be solved
by paying attention, education, installing tools/products/services. But
incompetence generally requires replacing people, especially management
who allow the incompetence to thrive.

Unlesss accept all those messages from those addresses and
check them, you really don't know the false positive rate.
You only know the self-reported complaint rate; which is
usually a fraction of the actual false positive rate.

Yes. It is tempting to take numbers at their face value, but I find that
whenever somebody has an axe to grind, their numbers are based on flawed
reasoning or measuring the wrong things.

--Michael Dillon

12

Instead of they suck, it might be more useful to highlight providers of
similar scale which you think do a good job which others could emulate.

A first-order examination of the data suggests that Cox may be
doing something pro-active. The number of IP addresses on their
network engaged in recent spam delivery attempts is considerably
less than that from others, e.g.:

  Comcast: 1240
  Verizon: 1594
  Cox: 25

Of course, that could be due to the relative sizes of the networks
involved, but a second-order examination of the same data suggests
something more: the abuse-emitting IP addresses on Cox's network do not
appear to show up repeatedly over long periods of time. Contrast with
those on Comcast and Verizon, where it's quite common to see the same ones
in the logs for days/weeks/months. This suggests to me that Cox
is actually paying attention to abuse outbound from their network
and is either disconnecting or quarantining hosts which emit it.

Some people think that users on dynamic addresses should be read-only, and
not allowed to operate servers or send messages. Like most forms of
red-lining, it tends to become self-fulling.

Of course, I've never suggested any such thing. Users of such ISPs
should normally be using their own ISPs outbound mail server(s),
a solution which is perfectly adequate for the overwhelming majority
of users. And it should be no problem for them to use other mail
servers, using SMTP AUTH (with TLS or SSL) or via HTTP. But
the days when direct-to-MX traffic is acceptable from all addresses
are as gone as those when operation of an open SMTP relay was acceptable.

Moreover, those who wish to operate servers (and whose ISPs permit
that operation per their TOS) should have no difficulty in having
the ISP assign them non-generic DNS/rDNS, and/or assign them static
address space which is reserved for such users.

Unlesss accept all those messages from those addresses and check them, you
really don't know the false positive rate. You only know the
self-reported complaint rate; which is usually a fraction of the actual
false positive rate.

Actually, I know quite a bit more than that. But since this is NANOG
and not an anti-spam discussion list, I elected not to delve into the
rather lengthy details of the methodology used to ascertain the FP rate.
But *briefly* and glossing heavily, when a particular IP address (with
generic rDNS, let's say):

  - fails to wait for the SMTP greeting
  - fails to send a QUIT
  - fails to retry when given a deferral response
  - retries immediately when given a deferral response
  - attempts delivery to an MX that hasn't been an MX for years
  - attempts delivery to a domain whose MX record hasn't
    been pointed to this MX for years
  - HELOs as ebay.com (or similar)
  - HELOs as a non-existent domain
  - HELOs as a very-recently-registered domain
  - HELOs as something different each time it connects
  - HELOs as *my* MX or a domain handle by it
  - attempts to send mail with a putative sender @paypal.com (or similar)
  - attempts delivery repeatedly with different putative senders/different
    putative sender domains
  - attempts to send mail with putative senders whose domain does
    not exist or does not resolve
  - attempts to send mail with a putative senders whose domain has
    A or MX records which resolve to invalid network space
  - attempts to send mail with a putative sender whose domain
    has very suspicious DNS records [1]
  - attempts to send mail from a putative sender using
    a known-spammer-owned domain
  - attempts to send mail from a putative sender using
    a domain which resolves to DROP-listed space. [2]
  - attempts to send mail from a putative sender using
    a very-recently-registered domain
  - attempts to send mail from a putative sender using
    *my* domain
  - attempts delivery to spamtraps
  - attempts delivery to never-existed addresses
  - attempts delivery to one-off addresses available only in SMTP
    reject notices
  - attempts delivery of messages whose headers contain known
    spamware signatures
  - attempts to send mail whose body-part contains URIs which
    match well-maintained lists of spammer-controlled URIs
  - has already been noted by other independent observers as
    attempting spam delivery to their MX's
  - passive-OS-fingerprints as running Windows
  - is listed in the A or NS records of a known spammer domain
    (see [1] again)
  - has been noticed carrying out other attacks, e.g., attempting
    exploitation via HTTP, SSH, etc.
  - and so on
  
or multiple of the above (as is the case most of the time), then it's
very, very unlikely that refusal of the traffic constitutes a FP.

---Rsk

[1] A recent example: segron.com, queried yesterday (a query
today will likely return different values, BTW):

  segron.com a 62.214.228.21
  segron.com a 75.47.248.183
  segron.com a 79.113.34.148
  segron.com a 79.120.16.19
  segron.com a 80.133.250.133
  segron.com a 81.198.35.132
  segron.com a 85.179.60.176
  segron.com a 89.110.23.181
  segron.com a 89.178.60.48
  segron.com a 89.212.0.195
  segron.com a 121.137.164.24
  segron.com a 121.200.140.244
  segron.com a 218.254.186.186
  segron.com a 221.126.244.121
  segron.com a 221.127.158.128

which resolve to:

  i3ED6E415.versanet.de
  adsl-75-47-248-183.dsl.lsan03.sbcglobal.net
  79-113-34-148.rdsnet.ro
  p5085FA85.dip.t-dialin.net
  e179060176.adsl.alicedsl.de
  pppoe-181.23.110.89-adsl.spbnit.ru
  89-178-60-48.broadband.corbina.ru
  89-212-0-195.dynamic.dsl.t-2.net
  244.140.200.121.megaegg.ne.jp
  cm218-254-186-186.hkcable.com.hk
  (5 addresses yield NXDOMAIN)

Clearly, segron.com's "web site" is being hosted on hijacked systems.

[2] See http://www.spamhaus.org/DROP/ for details, but the gist is
that this short and very carefully maintained list enumerates network
space which is either hijacked or 100% spammer-controlled or both.

13

in the logs for days/weeks/months. This suggests to me that Cox
is actually paying attention to abuse outbound from their network
and is either disconnecting or quarantining hosts which emit it.

Its nice to see Cox getting some praise for a change. Last month people were castigating it for being too agressive at trying to block Bots.
It seems like half the net is always criticizing ISPs for doing
too little and half the net is always criticizing ISPs for doing
too much.

Cox blocks a lot of ports on its network (25, 80, 135-139, 445, 1900,
1433, 1434, 1900, subseven ports); blackholes networks and DNS names;
firewall software that blocked sites with bad TCP software stacks such
as Craigslist; and so on. Some people think Cox is being pro-active
on the security front; other people think Cox is violating a sacred
trust. ISPs are pretty much just damned.

Why should an network user have to petition his or her ISP to authorize
their use of a valid network protocol? Shouldn't application
authorization occur at the application level instead of relying on
the equivalent of .rlogin network-level checks.

Companies like DynDNS show there is user demand to operate their own
servers (including P2P servers, mail servers, web servers, dns servers, etc) on dynamic IP addresses without needing a special "static" IP address or different in-addr.arpa name.

With Fast-Flux, it looks like the next network port that should be blocked on broadband/dialup connections is DNS tcp/udp 53.

or multiple of the above (as is the case most of the time), then it's
very, very unlikely that refusal of the traffic constitutes a FP.

Until a false positive happens. I see 1-2 false positives a year
using checks for "generic-looking" in-addr.arpa names; and a few more false positives for IP addresses without in-addr.arpa names. Nevertheless I still continue to use those checks because the false positive rate is below my pain threshold. But I don't pretend it never happens or may not be a concern to someone else.

I also almost never get a valid e-mail to my postmaster account, just
spam; but some people still think every mail server should accept mail
to the postmaster account anyway no matter how rarely it gets legitimate
email. They even set up RBLs of mail servers without postmaster accounts. Maybe we need a RBL of mail servers that don't accept mail from generic in-addr.arpa or dynamic IP addresses.

14

Why should an network user have to petition his or her ISP to authorize
their use of a valid network protocol?

Because many (most?) ISPs have done such a poor job of controlling SMTP
abuse outbound from their networks over the past decade that it's now
a best practice to consider all mail from generic hostnames/dynamic
IP space highly suspect -- at best.

Those ISPs have repeatedly proven over many years that they aren't capable
of detecting and squelching SMTP abuse sources on their own networks; [1] this
leaves everyone else with a choice: either (a) put up with it or (b) devise
measures to stuff a sock in it. And (a) simply isn't tenable for mail
servers receiving abuse in torrential quantities.

If any of those ISPs are unhappy with the choice of tactics encompassed
by (b) then perhaps they should have anticipated that unhappiness years
ago when they were first alerted to this problem. Had they taken even
rudimentary steps to solve it (instead of merely having their spokesdroids
repeat the bare-faced lie that they "take the spam problem seriously")
then perhaps it would not have been necessary for others to devise
methods to deal with their failures.

If any network user is unhappy (and I can easily see why they would be),
then he or she should take that up with their ISP, since it's quite
likely that their own ISP has been a contributor to the problem.

Companies like DynDNS show there is user demand to operate their own
servers (including P2P servers, mail servers, web servers, dns servers,
etc) on dynamic IP addresses without needing a special "static" IP address
or different in-addr.arpa name.

That model is no longer viable, unfortunately. I wish that weren't the
case, but the combination of ISP and end-user negligence along with mass
hijacking of end-user systems has rendered it so.

They even set up RBLs of mail servers without postmaster accounts.
Maybe we need a RBL of mail servers that don't accept mail from generic
in-addr.arpa or dynamic IP addresses.

You are certainly free to set up a DNSBL or RHSBL using any listing
criteria you wish, but please be aware that if you set up one using
that particular criteria, anyone using it will likely be refusing a LOT
of valid mail, including that of some very large organizations, since
(as I said above) blocking such traffic has long since been established
as a best practice. There are multiple DNSBLs, RHSBLs, and static
lists which enumerate such hosts; for example, consider the Spamhaus PBL:

  http://www.spamhaus.org/pbl/index.lasso

which relies in part on input from the ISPs themselves, and is one
of the zones included in the comprehensive "zen" DNSBL zone published
by Spamhaus.

---Rsk

[1] I still adhere to the quaint/outdated/antique concept that everyone
is responsible for making sure that their networks are not an operational
hazard to everyone else's networks, and that they should plan, budget,
staff, build, operate and train accordingly.