If so, would you mind hitting me up offlist? I have a few questions that i
am unable to get answered through normal channels.
Cheers,
Mike
You guys might want to be aware that isprime.com (I am not affiliated or
representing them, just passing on info since friends and I noticed this)
is actively under a DOS where lots of people's dns servers around the world
are being queried with bogus sourced dns requests not from port 53 for
'NS? .'. This then bounces back to their authoritative nameservers which
are getting traffic overload. They've asked that those of us that can
should block all but port 53 from the following two IP's (their dns
servers as seen on whois) so as not to block legitimate dns info:
66.230.128.15
66.230.160.1
Here is the response from their abuse department:
On Tue, 2009-01-20 at 14:55 -0600, Todd T. Fries forwarded:
From: ISPrime Support <support@isprime.com>
These are the result of a spoofed dns recursion attack against our servers. The actual packets in question (the ones reaching your servers) do NOT originate from our network as such there is no way for us to filter things from our end.
If you are receiving queries from 76.9.31.42/76.9.16.171 neither of these machines make legitimate outbound dns requests so an inbound filter of packets to udp/53 from either of these two sources is perfect.
If you are receiving queries from 66.230.128.15/66.230.160.1 these servers are authoritative nameservers. Please do not blackhole either of these IPs as they host many domains. However, these IPs do not make outbound DNS requests so filtering requests to your IPs from these ips with a destination port of 53 should block any illegitimate requests.
I've been seeing a lot of noise from the latter two addresses after
switching on query logging (and finishing an application of Team Cymru's
excellent template) so I decided to DROP traffic from the addresses
(with source port != 53) at the hosts in question.
Well, blow me down if they didn't completely stop talking to me. Four
dropped packets each, and they've gone away.
Something smells "not quite right" here - if the traffic is spoofed, and
my "Refused" responses have been flying right back to the *real* IP
addresses, how are the spoofing hosts to know that I'm dropping the
traffic?
Even if I used a REJECT policy, I'd expect the ICMP messages to go back
to the appropriate - as in real - hosts, rather than the spoofing
sources.
Something here is very odd, very odd indeed... or I'm being dumb. It's
happened before.
Graeme
Hello,
Representing ISPrime here.
This attack has been ongoing on 66.230.128.15/66.230.160.1 for about 24 hours now, and we are receiving roughly 5Gbit of attack packets from roughly 750,000 hosts.
It's somewhat absurd to suggest that we are attacking our own nameservers, I assure you, we didn't spend many hours looking for your specific nameserver to start sending 10 requests per second for the root zone, and our nameservers serve many popular domains.
Given the attack is still in progress, I can't really say much more publicly, but suffice to say, we're working on the situation.
-Phil
AS23393
In looking at my query logs I am seeing only requests from 66.230.160.1 and
66.230.128.15 so I've done the same thing with iptables and the rules are
resulting in an ever growing number of packets being dropped.
# iptables -nvL | grep -F -B 1 -A 1 66.230.160.1 | awk '{ print
$1,$2,$3,$8,$10,$11,$12 }'
pkts bytes target source
49517 2228K DROP 66.230.160.1 udp spt:!53 dpt:53
35905 1616K DROP 66.230.128.15 udp spt:!53 dpt:53
Can't some upstream provider find a source of the DNS NS . questions
that is other than isprime?
This attack has been ongoing on 66.230.128.15/66.230.160.1 for about 24 hours now, and we are receiving roughly 5Gbit of attack packets from roughly 750,000 hosts.
I'm only receiving NS queries for "." from spoofed 66.230.128.15 and
66.230.160.1 via above.net (of my three transit providers) and none from
peering. This usually indicates a single source, such as one rooted machine
on non-BCP38 net spewing most of a gigabit.
Given the attack is still in progress, I can't really say much more publicly, but suffice to say, we're working on the situation.
Have you had any luck tracking back the source of the spoofed packets? If
me talking to above.net sounds useful, let me know.
-- Aaron
Graeme Fowler wrote:
On Tue, 2009-01-20 at 14:55 -0600, Todd T. Fries forwarded:
I've been seeing a lot of noise from the latter two addresses after
switching on query logging (and finishing an application of Team Cymru's
excellent template) so I decided to DROP traffic from the addresses
(with source port != 53) at the hosts in question.Well, blow me down if they didn't completely stop talking to me. Four
dropped packets each, and they've gone away.
I've seen that behaviour in the past, but not this time?
I've seen a few of these attacks bouncing off my nameservers recently, and when I add "DROP" rules to my firewall, the incoming traffic disappears soon after. But the most recent set (66.230.160.1 and 66.230.128.15) are still hammering away...
Representing ISPrime here.
Well... representing myself and nobody else, so if that stretches my
credibility thin so be it.
It's somewhat absurd to suggest that we are attacking our own
nameservers, I assure you, we didn't spend many hours looking for your
specific nameserver to start sending 10 requests per second for the
root zone, and our nameservers serve many popular domains.
I just checked to make sure I did not make that assertion. I did not.
I observed something odd, and stated as much to see if anyone else did.
I apologise if you read my message as insinuating what you stated, but I
assure you that wasn't the intention.
I did say "maybe I'm being dumb", and that is indeed the answer - I
applied a temporary netfilter ruleset, then made it permanent - and it
switched the DROP and LOG statements round so that... the packet got
dropped first and the log statements never got hit. Schoolboy error (and
interesting that someone else has observed this behaviour before!)...
Normal service has been resumed. I should write a haiku here (sorry,
MLC, poor joke).
Given the attack is still in progress, I can't really say much more
publicly, but suffice to say, we're working on the situation.
In a previous job I've been on the receiving end of similar attacks so I
have a large degree of understanding of the pressure you're under at the
moment. I wish you the best of luck sorting it out.
Graeme
Graeme Fowler <graeme@graemef.net> writes:
I've been seeing a lot of noise from the latter two addresses after
switching on query logging (and finishing an application of Team Cymru's
excellent template) so I decided to DROP traffic from the addresses
(with source port != 53) at the hosts in question.Well, blow me down if they didn't completely stop talking to me. Four
dropped packets each, and they've gone away.Something smells "not quite right" here - if the traffic is spoofed, and
my "Refused" responses have been flying right back to the *real* IP
addresses, how are the spoofing hosts to know that I'm dropping the
traffic?
Did you filter *only* 66.230.128.15/66.230.160.1, or are you dropping
traffic from other sources too? Looks like some of the other source
addresses are controlled by the DOSers. Possibly used to detect filters?
These clients may look similar to the DOS attack, but there are subtle
differences:
Jan 18 05:08:33 canardo named[32046]: client 211.72.249.201#29656: view external: query (cache) './NS/IN' denied
Jan 18 05:08:33 canardo named[32046]: client 211.72.249.201#29656: view external: query (cache) './NS/IN' denied
Jan 18 05:08:34 canardo named[32046]: client 211.72.249.201#29656: view external: query (cache) './NS/IN' denied
Jan 18 05:47:00 canardo named[32046]: client 211.72.249.201#29662: view external: query (cache) './NS/IN' denied
Jan 18 05:47:01 canardo named[32046]: client 211.72.249.201#29662: view external: query (cache) './NS/IN' denied
Jan 18 05:47:01 canardo named[32046]: client 211.72.249.201#29662: view external: query (cache) './NS/IN' denied
Jan 18 06:25:22 canardo named[32046]: client 211.72.249.201#29664: view external: query (cache) './NS/IN' denied
Jan 18 06:25:22 canardo named[32046]: client 211.72.249.201#29664: view external: query (cache) './NS/IN' denied
Jan 18 06:25:23 canardo named[32046]: client 211.72.249.201#29664: view external: query (cache) './NS/IN' denied
Jan 18 07:03:41 canardo named[32046]: client 211.72.249.201#29667: view external: query (cache) './NS/IN' denied
Jan 18 07:03:41 canardo named[32046]: client 211.72.249.201#29667: view external: query (cache) './NS/IN' denied
Jan 18 07:03:42 canardo named[32046]: client 211.72.249.201#29667: view external: query (cache) './NS/IN' denied
Jan 18 07:42:08 canardo named[32046]: client 211.72.249.201#29670: view external: query (cache) './NS/IN' denied
Jan 18 07:42:09 canardo named[32046]: client 211.72.249.201#29670: view external: query (cache) './NS/IN' denied
Jan 18 07:42:09 canardo named[32046]: client 211.72.249.201#29670: view external: query (cache) './NS/IN' denied
Jan 18 08:20:29 canardo named[32046]: client 211.72.249.201#29673: view external: query (cache) './NS/IN' denied
Jan 18 08:20:29 canardo named[32046]: client 211.72.249.201#29673: view external: query (cache) './NS/IN' denied
Jan 18 08:20:30 canardo named[32046]: client 211.72.249.201#29673: view external: query (cache) './NS/IN' denied
Jan 18 08:58:50 canardo named[32046]: client 211.72.249.201#29678: view external: query (cache) './NS/IN' denied
Jan 18 08:58:51 canardo named[32046]: client 211.72.249.201#29678: view external: query (cache) './NS/IN' denied
Jan 18 08:58:51 canardo named[32046]: client 211.72.249.201#29678: view external: query (cache) './NS/IN' denied
Jan 18 09:37:12 canardo named[32046]: client 211.72.249.201#29679: view external: query (cache) './NS/IN' denied
Jan 18 09:37:12 canardo named[32046]: client 211.72.249.201#29679: view external: query (cache) './NS/IN' denied
Jan 18 09:37:13 canardo named[32046]: client 211.72.249.201#29679: view external: query (cache) './NS/IN' denied
Jan 20 07:02:51 canardo named[32046]: client 213.61.92.192#46716: view external: query (cache) './NS/IN' denied
Jan 20 07:02:51 canardo named[32046]: client 213.61.92.192#46716: view external: query (cache) './NS/IN' denied
Jan 20 07:02:51 canardo named[32046]: client 213.61.92.192#46716: view external: query (cache) './NS/IN' denied
Jan 20 07:41:21 canardo named[32046]: client 213.61.92.192#46752: view external: query (cache) './NS/IN' denied
Jan 20 07:41:21 canardo named[32046]: client 213.61.92.192#46752: view external: query (cache) './NS/IN' denied
Jan 20 07:41:21 canardo named[32046]: client 213.61.92.192#46752: view external: query (cache) './NS/IN' denied
Jan 20 08:19:46 canardo named[32046]: client 213.61.92.192#46785: view external: query (cache) './NS/IN' denied
Jan 20 08:19:46 canardo named[32046]: client 213.61.92.192#46785: view external: query (cache) './NS/IN' denied
Jan 20 08:19:46 canardo named[32046]: client 213.61.92.192#46785: view external: query (cache) './NS/IN' denied
Jan 20 08:58:12 canardo named[32046]: client 213.61.92.192#46808: view external: query (cache) './NS/IN' denied
Jan 20 08:58:12 canardo named[32046]: client 213.61.92.192#46808: view external: query (cache) './NS/IN' denied
Jan 20 08:58:12 canardo named[32046]: client 213.61.92.192#46808: view external: query (cache) './NS/IN' denied
Jan 20 09:36:34 canardo named[32046]: client 213.61.92.192#46833: view external: query (cache) './NS/IN' denied
Jan 20 09:36:34 canardo named[32046]: client 213.61.92.192#46833: view external: query (cache) './NS/IN' denied
Jan 20 09:36:34 canardo named[32046]: client 213.61.92.192#46833: view external: query (cache) './NS/IN' denied
Jan 20 10:14:58 canardo named[32046]: client 213.61.92.192#46858: view external: query (cache) './NS/IN' denied
Jan 20 10:14:58 canardo named[32046]: client 213.61.92.192#46858: view external: query (cache) './NS/IN' denied
Jan 20 10:14:58 canardo named[32046]: client 213.61.92.192#46858: view external: query (cache) './NS/IN' denied
Jan 22 06:27:28 canardo named[32046]: client 66.238.93.161#34373: view external: query (cache) './NS/IN' denied
Jan 22 06:27:28 canardo named[32046]: client 66.238.93.161#34373: view external: query (cache) './NS/IN' denied
Jan 22 06:27:28 canardo named[32046]: client 66.238.93.161#34373: view external: query (cache) './NS/IN' denied
Jan 22 07:05:55 canardo named[32046]: client 66.238.93.161#34420: view external: query (cache) './NS/IN' denied
Jan 22 07:05:55 canardo named[32046]: client 66.238.93.161#34420: view external: query (cache) './NS/IN' denied
Jan 22 07:05:55 canardo named[32046]: client 66.238.93.161#34420: view external: query (cache) './NS/IN' denied
Jan 22 07:44:20 canardo named[32046]: client 66.238.93.161#34473: view external: query (cache) './NS/IN' denied
Jan 22 07:44:20 canardo named[32046]: client 66.238.93.161#34473: view external: query (cache) './NS/IN' denied
Jan 22 07:44:21 canardo named[32046]: client 66.238.93.161#34473: view external: query (cache) './NS/IN' denied
Jan 22 08:22:38 canardo named[32046]: client 66.238.93.161#34503: view external: query (cache) './NS/IN' denied
Jan 22 08:22:38 canardo named[32046]: client 66.238.93.161#34503: view external: query (cache) './NS/IN' denied
Jan 22 08:22:38 canardo named[32046]: client 66.238.93.161#34503: view external: query (cache) './NS/IN' denied
Jan 22 09:00:56 canardo named[32046]: client 66.238.93.161#34540: view external: query (cache) './NS/IN' denied
Jan 22 09:00:56 canardo named[32046]: client 66.238.93.161#34540: view external: query (cache) './NS/IN' denied
Jan 22 09:00:56 canardo named[32046]: client 66.238.93.161#34540: view external: query (cache) './NS/IN' denied
Jan 22 09:39:20 canardo named[32046]: client 66.238.93.161#34574: view external: query (cache) './NS/IN' denied
Jan 22 09:39:21 canardo named[32046]: client 66.238.93.161#34574: view external: query (cache) './NS/IN' denied
Jan 22 09:39:21 canardo named[32046]: client 66.238.93.161#34574: view external: query (cache) './NS/IN' denied
Notice the pattern:
3 probes every 38 minutes
Each probe from the same source port
Source port increases slowly and steadily
This looks like some application actually waiting for a response. The
slow source port change is probably an indication that this client only
tests a small number of DNS servers. I guess that this client is either
one of the many bots used to send the spoofed requests, or maybe a bot
not allowed to spoof its source and therefore used for other
purposes. In any case, I assume that other DNS servers may see such
control sessions coming from other addresses.
These 3 clients started probing my DNS server almost simultaneously on January 8th:
Jan 8 19:33:52 canardo named[26496]: client 213.61.92.192#31195: view external: query (cache) './NS/IN' denied
Jan 8 19:33:52 canardo named[26496]: client 213.61.92.192#31195: view external: query (cache) './NS/IN' denied
Jan 8 19:33:52 canardo named[26496]: client 213.61.92.192#31195: view external: query (cache) './NS/IN' denied
Jan 8 19:36:29 canardo named[26496]: client 66.238.93.161#11299: view external: query (cache) './NS/IN' denied
Jan 8 19:36:29 canardo named[26496]: client 66.238.93.161#11299: view external: query (cache) './NS/IN' denied
Jan 8 19:36:30 canardo named[26496]: client 66.238.93.161#11299: view external: query (cache) './NS/IN' denied
Jan 8 19:37:47 canardo named[26496]: client 211.72.249.201#29112: view external: query (cache) './NS/IN' denied
Jan 8 19:37:47 canardo named[26496]: client 211.72.249.201#29112: view external: query (cache) './NS/IN' denied
Jan 8 19:37:47 canardo named[26496]: client 211.72.249.201#29112: view external: query (cache) './NS/IN' denied
Maybe preparing for the attack on ISPrime? I didn't start receiving
spoofed requests from 66.230.128.15/66.230.160.1 before January 20th
I just tried filtering the probing addresses. This made the probing
stop immediately after dropping a set of 3 probes. But the spoofed
requests continuted at the same rate as before, so this does not support
my theory.
However, I believe it would be too much of a coincidence if there isn't
some connection between the probing and the DOS attack. It would be
interesting to hear if others see similar probing.
Bjørn
Just a friendly notice, the attack against 66.230.128.15/66.230.160.1 seems to have stopped for now.
-Phil
Hi,
I agree with seeing no traffic to/from 66.230.128.15 but am still seeing flows 'from' 66.230.160.1
Regards,
Steve
Looks to me like the target has moved, anyone else seeing similar?
Jan 23 20:19:08 LND02 named[9611]: client 63.217.28.226#39489: view
external: query (cache) './NS/IN' denied
Jan 23 20:19:09 LND02 named[9611]: client 63.217.28.226#20558: view
external: query (cache) './NS/IN' denied
Jan 23 20:19:11 LND02 named[9611]: client 63.217.28.226#38525: view
external: query (cache) './NS/IN' denied
Jan 23 20:19:12 LND02 named[9611]: client 63.217.28.226#41535: view
external: query (cache) './NS/IN' denied
Jan 23 20:19:12 LND02 named[9611]: client 63.217.28.226#51220: view
external: query (cache) './NS/IN' denied
Jan 23 20:19:13 LND02 named[9611]: client 63.217.28.226#28869: view
external: query (cache) './NS/IN' denied
Jan 23 20:19:14 LND02 named[9611]: client 63.217.28.226#12337: view
external: query (cache) './NS/IN' denied
Jan 23 20:19:15 LND02 named[9611]: client 63.217.28.226#41346: view
external: query (cache) './NS/IN' denied
Jan 23 20:19:15 LND02 named[9611]: client 63.217.28.226#56831: view
external: query (cache) './NS/IN' denied
Jan 23 20:19:17 LND02 named[9611]: client 63.217.28.226#13352: view
external: query (cache) './NS/IN' denied
Jan 23 20:19:18 LND02 named[9611]: client 63.217.28.226#55466: view
external: query (cache) './NS/IN' denied
Jan 23 20:19:18 LND02 named[9611]: client 63.217.28.226#24586: view
external: query (cache) './NS/IN' denied
Jan 23 20:19:19 LND02 named[9611]: client 63.217.28.226#43105: view
external: query (cache) './NS/IN' denied
Are they responses to queries? Or are they queries directed at servers in your network? The latter are to be expected, I think.
Joe
We [AS3491] null0'd the IP earlier. Rest-of-world encouraged to do the same 
Seeing the same here, it's 1 query per second per nameserver--time to work some magic with PF.
Hi,
I agree with seeing no traffic to/from 66.230.128.15 but am still seeing flows 'from' 66.230.160.1
Regards,
Steve
Hi Steve,
There is at least an iptables rule you can use to drop this specific query, assuming your nameservers run linux.
http://www.stupendous.net/archives/2009/01/24/dropping-spurious-nsin-recursive-queries/
The bind-users mailing list suggested having the ISPs trace back the flows and find the networks emitting the spoofed packets, and have those networks implement BCP 38. While that's the 'right' solution (everyone should be doing ingress filtering, sure, impossible to argue against it), not every network out there is operated by people who give a damn.
This will work at least until the kiddies improve their scripts to query for names that actually exist.
We [AS3491] null0'd the IP earlier. Rest-of-world encouraged to do the same
Good luck with that. Right now they're targetting ISPrime, and you've just made the DoS even more effective for them. With any luck, the rest of the world will follow suit and the bad guys win! yay! 
Short of getting the rest of the world to properly implement ingress filtering (ha, ha), I think dropping the specific packets that generate the reflected traffic is good enough for now. The load on the reflectors is minimal.
Nathan.
Wrong approach, they are *innocent* in this as are the new targets.
insert into your favourite acl:
deny udp host 66.230.160.1 neq 53 any eq 53
deny udp host 66.230.128.15 neq 53 any eq 53
But it's much less work to add a filter on the name server as others
have mentioned.
Noel Butler wrote:
I respectfully disagree. Network engineers have to keep up with many
tasks and preventing DoS/DDoS should be the responsibility of
everyone. I see more folks worried about spam than they are actual
security.
My two cents.