[Feel free to respond with: take it to list XYZZY]
There's been an ongoing DDoS here at world.std.com (The World) tho
it's not quite DoS'ing (you got this, right?) it's getting very tiring
and obviously is affecting many systems "out there".
The MO: (easy to understand but pretty nasty):
What I presume is a zombie army sending out gazillions of emails to
thousands of hosts out there (not ours) with a randomly generated
(usually) return/source address @ our domain(s). The target addresses
are usually also unknown so it just bounces back at us.
Besides the obvious SMTP traffic this also generates a lot of DNS
traffic. At this point the DNS traffic seems to be more of a nuisance
probably because so many target hosts are retrying. At one point we
were doing around 10K pkts/second in DNS traffic, very unusual.
This has been going on for about a week.
I'd hoped some little mitigation tricks here and there and a few days'
patience and the excess mouths would get tired of this and go back to
stuffing neighbors' pets down their garbage disposals for yucks, etc.
So where does one start. It seems a mother ship needs to be shut down
somewhere, etc. Obviously ID'ing a miscreant would be a nice result.
P.S. If you think "get a firewall": The problem traffic is coming from
legitimate hosts in the form of DNS+SMTP, not the bots (not to us
anyhow.) So not so simple, what's the filter?
What I presume is a zombie army sending out gazillions of emails to
thousands of hosts out there (not ours) with a randomly generated
(usually) return/source address @ our domain(s). The target addresses
are usually also unknown so it just bounces back at us.
Some sort of a user check should mitigate most of this.. ie, drop at
the smtp level, don't bounce.
Besides the obvious SMTP traffic this also generates a lot of DNS
traffic. At this point the DNS traffic seems to be more of a nuisance
probably because so many target hosts are retrying. At one point we
were doing around 10K pkts/second in DNS traffic, very unusual.
10K/s is a lot.. I would expect a lot less.. Presumably the source
of the DNS requests would be another DNS server who should be caching
the result.
Try increasing the TTL for the "offending" records... I see it's at
24 hours at the moment though.
Can you do some sniffing to determine the source of the lookups?
Perhaps a broken dns server or two out there?
P.S. If you think "get a firewall": The problem traffic is coming from
legitimate hosts in the form of DNS+SMTP, not the bots (not to us
anyhow.) So not so simple, what's the filter?
Throttle on the gateway? Specifically, throttle DNS traffic to start
if that's doing the most damage, and then throttle smtp if necessary..
Depend on the remote retry to handle any timeouts..
Besides the obvious SMTP traffic this also generates a lot of DNS
traffic. At this point the DNS traffic seems to be more of a nuisance
probably because so many target hosts are retrying. At one point we
were doing around 10K pkts/second in DNS traffic, very unusual.
This has been going on for about a week.
At least some broken resolvers will keep re-querying you so see if you
cant throttle or rate limit dns queries from problem IPs for a while.
That, and increase TTLs a bit.
As for the smtp -
* Dont accept email for catchall aliases - try to reject all you can
at the gateway
* Bounces and backscatter - RFC violation or not, accepting bounces
takes a backseat to keeping your mail system up and running.
TEMPORARILY turn off accepting mail from:<>, especially if you're
seeing far, far more bounce traffic to nonexistent addresses on your
site than valid bounces.
Long term - see if you can't use http://www.mipassoc.org/batv/
especially if all your users send email through your smtp server from
outside (say using AUTH) or ssh in and use pine / elm or whatever on
your shell servers.
So where does one start. It seems a mother ship needs to be shut down
somewhere, etc. Obviously ID'ing a miscreant would be a nice result.
You sure its just one botnet hitting you? Shutting off a mothership
often means that the zombies become even more zombied and keep
pounding on your server long after the mothership is dead.
--srs