DNS deluge for x.p.ctrc.cc

Mailman List Mirror (Read Only)
1

Hi, NANOGers.

] other cctld servers have seen what are effectively ddos. rob thomas
] seems to have the most clue on this, so i hope this troll will entice
] him to speak.

Did someone say "troll?" :slight_smile:

Yes, this is a real problem. These attacks have exceeded several
gigabits per second in size, and during one attack 122K DNS name
servers were abused as amplifiers. Ouch!

This abuse can be mitigated. Here are a few tips.

Limit recursion to trusted netblocks and customers. Do not permit
your name servers to provide recursion for the world. If you do,
you will contribute to one of these attacks.

Watch for queries to your name servers that ask for "ANY" related
to a DNS RR outside of the zones for which you are authoritative.
This DNS RR will be LARGE.

Limit UDP queries to 512 bytes. This greatly decreases the
amplification affect, though it doesn't stop it.

Scan your IP space for name servers that permit recursive queries.
It's amazing just how many of these name servers exist.

Refer to the following guides for some excellent insight and
suggestions.

   <http://www.us-cert.gov/reading_room/DNS-recursion121605.pdf>
   <http://cc.uoregon.edu/cnews/winter2006/recursive.htm>
   <http://dns.measurement-factory.com/surveys/sum1.html>

Note we have our own Secure BIND Template which will help on the
BIND side of life.

   <http://www.cymru.com/Documents/secure-bind-template.html>

If you need assistance with any of this, have endured one of these
attacks, or have any other questions, please don't hesitate to ping
on us at team-cymru@cymru.com. We're here to assist!

Thanks!
Rob.

2

Once upon a time, Rob Thomas <robt@cymru.com> said:

Limit recursion to trusted netblocks and customers. Do not permit
your name servers to provide recursion for the world. If you do,
you will contribute to one of these attacks.

One thing to note: we've discovered that on some common DSL routers, the
internal DNS caching server is on by default and answers requests on the
outside IP address. IIRC some even do it when configured for NAT.

So, even when you disable outside recursion, things you may not think of
on the inside of your network may still allow outside DNS recursion.

3

] other cctld servers have seen what are effectively ddos. rob thomas
] seems to have the most clue on this, so i hope this troll will entice
] him to speak.

Did someone say "troll?" :slight_smile:

Yes, this is a real problem. These attacks have exceeded several
gigabits per second in size, and during one attack 122K DNS name
servers were abused as amplifiers. Ouch!

This abuse can be mitigated. Here are a few tips.

  <there has -GOT- to be a better name for this>

Limit recursion to trusted netblocks and customers. Do not permit
your name servers to provide recursion for the world. If you do,
you will contribute to one of these attacks.

  <recursion is a fundamental DNS design feature,
   restricting it to "walled gardens" cripples its usefullness>

Watch for queries to your name servers that ask for "ANY" related
to a DNS RR outside of the zones for which you are authoritative.
This DNS RR will be LARGE.

  <a valid concern, w/ the following caveat: LARGE, relative
   to current traffic>

Limit UDP queries to 512 bytes. This greatly decreases the
amplification affect, though it doesn't stop it.

  <limiting UDP to 512 has other, unwanted effects,
   edns0 for one... crippling ENUM, DNSSEC, IPv6, etc...
   is this really what is wanted?>

Scan your IP space for name servers that permit recursive queries.
It's amazing just how many of these name servers exist.

  <yup... again, a feature that has made the DNS as useful as
  it has become>

Refer to the following guides for some excellent insight and
suggestions.

   <http://www.us-cert.gov/reading_room/DNS-recursion121605.pdf&gt;
   <http://cc.uoregon.edu/cnews/winter2006/recursive.htm&gt;
   <http://dns.measurement-factory.com/surveys/sum1.html&gt;

Note we have our own Secure BIND Template which will help on the
BIND side of life.

   <http://www.cymru.com/Documents/secure-bind-template.html&gt;

If you need assistance with any of this, have endured one of these
attacks, or have any other questions, please don't hesitate to ping
on us at team-cymru@cymru.com. We're here to assist!

Thanks!
Rob.
--
Rob Thomas
Team Cymru
http://www.cymru.com/
ASSERT(coffee != empty);

  ok, so i'm being a bit of a curmudgion here but just how,
  if we throttle DNS to the minimum suite for todays services,
  can we be expected to add new features/services? grump grump grump...

-- (grumpy) bill

4

bmanning@vacation.karoshi.com wrote:

Limit recursion to trusted netblocks and customers. Do not permit
your name servers to provide recursion for the world. If you do,
you will contribute to one of these attacks.
    
  <recursion is a fundamental DNS design feature,
   restricting it to "walled gardens" cripples its usefullness>

I don't really think that preventing every Tom, Dick, and Harry from using my nameserver to look up domains I'm not authoritative for is going to cripple DNS. They really should have their own severs that do that for them, or they should use the ones provided to them by their ISP.

5

In message <Pine.GSO.4.62.0602241629470.21514@qentba.nf23028.arg>, Rob Thomas w
rites:

Limit UDP queries to 512 bytes. This greatly decreases the
amplification affect, though it doesn't stop it.

Unfortunately, the intention of the DNS developers is just the
opposite. Things like DNSSEC require larger packet sizes; in fact,
there's a DNS extension (EDNS0) whose purpose, among others, it to
permit this.

    --Steven M. Bellovin, Steven M. Bellovin

6

Hey, Bill.

As many say, you own your network, and are free to run it as you see
fit. :slight_smile: That said, please be aware that if you leave your name
servers open to recursive query requests from any source, you WILL
unwittingly help to amplify these attacks. It's the same as ICMP
directed broadcast and the like.

I don't like it. You don't like it. The miscreants love it. It's
always a balancing act.

Thanks,
Rob.

7

robt wrote:

[snip]

> Limit recursion to trusted netblocks and customers. Do not permit
> your name servers to provide recursion for the world. If you do,
> you will contribute to one of these attacks.

  <recursion is a fundamental DNS design feature,
   restricting it to "walled gardens" cripples its usefullness>

The bad guys abused open SMTP relaying and we couldn't use it anymore.*
They've moved to the next thing that is widely open and will be abusable
for a long time while some folks clamp down quickly, others argue against
it, etc. Until we can factor out the bad guys, the diminishing returns
on playing whack-a-mole will force us all to install more functional
equivalent of signs saying "restrooms are for customers only". And no
I don't like it either.

Cheers,

Joe

* well, except those who wish to be marginalized.

8

Efficient Networks DSL routers suffer from this problem if DNS servers are defined in the DHCP server config on the router. It's more of a DNS proxy though. It doesn't do any caching.

9

As many say, you own your network, and are free to run it as you see
fit. :slight_smile: That said, please be aware that if you leave your name
servers open to recursive query requests from any source, you WILL
unwittingly help to amplify these attacks.

and there is discussion on treating this similarly to how open
smtp relays are treated today.

randy

10

I've taken the liberty of following up on this thread on a different
mailing list (dns-operations@lists.oarci.net), since I'd like to explore
it at a depth and breadth that would seem overly obsessive on a general
purpose ops list like nanog. <http://lists.oarci.net/mailman/listinfo/>
is the entry point if you aren't subscribed and want to be. (There are
about 50 folks on that list, which I'm calling "critical mass" for the
purpose of starting the first real discussion over there.)

11

vixie@vix.com (Paul Vixie) (hey, that's me!) writes:

<http://lists.oarci.net/mailman/listinfo/&gt; ... (There are
about 50 folks on that list, which I'm calling "critical mass" for the
purpose of starting the first real discussion over there.)

oops. 154 as of this morning, i guess i wasn't paying attention. note
that the list is open, and the subscriber list is open to those who are
subscribed. this isn't a vetted insider-only kind of mailing list like
OARC would normally run, it's a public service side-effect kind of thing.

12

This has been an issue for years. Before the DDoSers started using open recursive DNS servers as a modern way to "smurf", spammers were abusing them by registering a domain, setting up DNS, loading the data into open recursive servers (by sending them queries), and then pointing the domains at those recursive servers...getting free DNS service and misdirecting complaints.

The argument that DNS servers have always been open to recursion (so we shouldn't change it) sounds a lot like the open SMTP relay issue 5-10 years ago. It took years, but all but a few wingnuts seem to have finally caught on to the idea that open SMTP relays are a bad idea...enough so that spammers had to move on and adapt to open proxies, and then to botted systems / trojan proxies.

Besides, don't the DNS specs dictate that a proper DNS resolver will try again with TCP if the server tells it the UDP reply was truncated?

13

Expanding on this slightly, since I think this merits more discussion -- if there was widespread filtering of 53/udp packets to 512 bytes, I can see two consequences:

1. A temporary decrease in attack traffic: the malware authors will adapt, and the floods will continue except with smaller packets. For attacks launched from drones, the amplification arguably isn't as important to the attacker as it might be for attacks which have single sources.

2. In future, increased use of things like DNSSEC, dynamic updates and IPv6 (with attendant AAAA records) are going to make legitimate, EDNS0, large 53/udp packets more common, and crippling the transport for EDNS0 is going to cause migraines for helpdesks across the world.

As a temporary mitigation tool today, when the volume of legitimate, large-packet EDNS0 traffic is near-zero, blocking big 53/udp packets might *sound* reasonable. However, we all know how permanent temporary filters can be. Crippling EDNS0 transport in the future seems like a very high price to pay for what might be a very temporary, short-term reduction in attack traffic.

Joe

14

As a temporary mitigation tool today, when the volume of legitimate,
large-packet EDNS0 traffic is near-zero, blocking big 53/udp packets
might *sound* reasonable. However, we all know how permanent

how are you certain that the udp/53 1500 byte packet is 'dns'? and not
kazaa/gnutella/bittorrent/vpn-in-udp-53 ? It seems that filtering the
TRAFFIC is short sighted on several fronts :frowning: deciding if you will/won't
be part of the global-recursive-dns-server 'problem' is entirely different
though.

temporary filters can be. Crippling EDNS0 transport in the future
seems like a very high price to pay for what might be a very
temporary, short-term reduction in attack traffic.

seems like global tcp/139|tcp/445 filters, or bogon filters... bits put
into configs 'now' and completely forgotten about 'tomorrow' :frowning:

15

christopher.morrow@verizonbusiness.com ("Christopher L. Morrow") writes:

seems like global tcp/139|tcp/445 filters, or bogon filters... bits put
into configs 'now' and completely forgotten about 'tomorrow' :frowning:

speaking of which, f-root has about 35 nodes world wide, and about a third
to a half of them aren't reachable by udp/161, and the blockage is not in
our immediate neighbors but rather on transit paths. this is due to the
cisco snmp vulnerability five years or so ago. filtering in the core to
protect vulnerable edges has to be done a LOT more carefully than that.
(BCP38 is an example of how to do it well, but apparently impractically?)

i'm not following up on the dns related parts of this, since dns-operations@
seems to be pulling some of the dns related load today and i don't want to
say the same thing in both places. see this URL for details:

http://lists.oarci.net/pipermail/dns-operations/2006-February/author.html

16

i'm not following up on the dns related parts of this, since dns-operations@
seems to be pulling some of the dns related load today and i don't want to
say the same thing in both places. see this URL for details:

http://lists.oarci.net/pipermail/dns-operations/2006-February/author.html
--
Paul Vixie

  hum... i subscribed to this dns-operations@ list some days back and have
  yet to see any postings. i guess i'm not worthy.

--bill

17

# hum... i subscribed to this dns-operations@ list some days back

as what?

#in2.oarc:amd64# bin/list_members dns-operations | grep -i manning
#in2.oarc:amd64# bin/list_members dns-operations | grep -i ep.net
#in2.oarc:amd64#

# and have yet to see any postings. i guess i'm not worthy.

i apologize to the gallery for cc'ing nanog, but bill did (by mistake, one
hopes) and i want to make it clear that bill is completely worthy, and the
list is completely open, and if a subscription doesn't work, you don't have
to send mail to nanog to get it fixed, admin@oarc.isc.org can help fix things.

and fyi:

#in2.oarc:amd64# bin/list_members dns-operations | wc -l
     190

18

I thought I would chime in quickly, one of my customers has been one of the targets of this attack. The x.p.ctrc.cc DNS server was shut down on the 15th, the response itself had a 360000 TTL so that should be expired by now.

On this end of it, the largest traffic spike we received was around 8 Gbps. The last time we saw this traffic was on the 21st around 2 GMT with traffic at about 2 Gbps, it has lost a lot of steam. If you see unusual DNS traffic to AS32787 or 72.52.0.0/18, chances are it is part of this attack or the attacker setup a new RR to query against.

I've yet to see a copy of the malware that is doing the spoofed queries itself. If anyone has it, I would like to take a look.

Thanks and I am really impressed with everyone's reaction to this attack. Especially Rob Thomas, he really has a grip on it.

Cheers,

-Barrett

19

] Thanks and I am really impressed with everyone's reaction to this attack.
] Especially Rob Thomas, he really has a grip on it.

Thanks muchly, Barrett, but the credit goes to Steve Gill. :slight_smile: