DNS DoS ???

my DNS servers were getting slow so I blocked recursive queries for
all but my own network.

Then I was getting so many of these:

ns2 named[5056]: client 78.159.111.190#25345: query (cache)
'isc.org/ANY/IN' denied

that is was still slowing things down. I've since written a script to
watch the log and throw these into the box local firewall. If I
expire the entries after 24 hours then I accumulate about 10200 unique
IPs. If I expire after 48 hours, then it's just over 20000 unique
IPs.

Is anyone else seeing this?

Elliot

Ping me offline, there are a few other folks who have seen this as well. The isc.org record is commonly used in reflection attacks because the size of the record is so large, so the amplification factor is greatly increased. Can you check to see if +edns=0 was set in the query? That would be a sure sign this is related to what others have seen...

Sorry for the top post, I'm on my iPad.

Stefan Fouant
JNCIE-M, JNCIE-ER, JNCIE-SEC, JNCI
Technical Trainer, Juniper Networks
http://www.shortestpathfirst.net
http://www.twitter.com/sfouant

I see this all the time on my personal servers. I finally just told bind to stop logging it.

We've been seeing this for several years on and off.

thanks,
-Drew

I've seen this for the same on about 3 sets of nameservers I operate. fail2ban doing a 72 hour iptables drop rule.

This should be the standard practice. By operating an open recursor, you lend your DNS server to abuse as a contributor to DNS reflection/amplification attacks.

This should be the standard practice. By operating an open recursor, you lend your DNS server to abuse as a contributor to DNS reflection/amplification attacks.

An awful lot of older/smaller deployments have single servers doing both authoratative and recursive DNS. These should be setup with either an allow-recursion { ACL;} statement or separate authoratative and recursive views limiting recursion to just those networks that should be sending recursive queries.

Another option is to run separate services bound to different individual IPs on the server. i.e. bind9 or powerdns for authoratative DNS and unbound for recursion.

Dns anycast can in addition to acl help distribute load.

I don't think anycast works the way you think it does. It'll distribute load
for single dns servers, but not the case that he is describing.

-j

With these types of attacks, usually anycast will cause rolling
outages. Anycast gives you failover, which makes sure the attack (and
good) traffic makes it to the next available server to be impaired or
taken offline.

A problem for providers of DNS recursive servers as a hosted service, is
the client sender IP address may be dynamic and off-net.
And the DNS protocol does not provide a method of authentication, or
passing credentials from the client
to the server to authorize the use of recursive DNS.

This differs from SMTP. There really is no such thing as a "closed
recursive resolver", except where
unwanted queries are blocked by IP.

All we really have is TSIG for such scenarios, and most client resolvers do
not support loading the resolver
with a secret key, in order to authorize recursive access.

So it follows, that in a number cases, "closing recursive access" is not
a good option.

A good example, would be services such as OpenDNS.

Regards,

One can argue a) that services like OpenDNS aren't necessarily a Good Thing when run by those who don't take the proper precautions and b) that OpenDNS in particular is run by smart, responsible folks who take extra measures to ensure they don't become a party to DNS reflection/amplification attacks, even though they operate an open recursive lookup service.

;>

More good stuff here: http://www.team-cymru.org/Services/Resolvers/

Frank

> A good example, would be services such as OpenDNS.
One can argue a) that services like OpenDNS aren't necessarily a Good Thing
when run by those who don't take the proper precautions

Is there an RFC specifying precisely what are considered the proper
precautions?
"precautions" should ideally be enabled in BIND by default.

and b) that OpenDNS in particular is run by smart, responsible folks who
take extra measures to ensure they don't become a party to DNS
reflection/amplification attacks, even though they operate an open recursive
lookup service.

My point here is that... splitting recursive service and cordoning off
recursive service by
using a firewall or ACL based on IP address, is only a partially effective
operational
workaround to a serious defect in the protocol.
Authoritative service is as subject to DoS as ever, especially with IPv6 and
DNSSEC
deployments.

It doesn't fall under "best common practices"; it falls under "necessary
evils", that everyone pretty much has to do.

Right now we have to say "separate and restrict access to your recursive
service", because it is the only option
we can implement without changes to the protocol, and for now it should be
done wherever possible.
But this is not pallatable at all. The DNS protocol should be fixed so we
don't need this workaround.

Standardization effort should concentrate on changing the protocol.

Restricting recursive service is a good short term solution, but it is not
for every case.
The workaround is not a best practice, it is evidence that the DNS
protocol should be changed.

How can it be changed?
Add a proof-of-work for every DNS query and an additional round-trip
for the first recursion-desired DNS query a client makes to any
restricted DNS server, to obtain
some "client token" or other unique hash to be sent along with
queries.

Is there an RFC specifying precisely what are considered the proper precautions?
"precautions" should ideally be enabled in BIND by default.

Not of which I'm aware. I'm happy to contribute to any efforts you or anyone else are volunteering to coordinate on this topic, as it's important work which ought to be done sooner & not later.

;>

My point here is that... splitting recursive service and cordoning off recursive service by
using a firewall

Using a stateful firewall is contraindicated.

or ACL based on IP address, is only a partially effective operational workaround to a serious defect in the protocol.

It's actually a combination of serious defects in a number of protocols, starting with IPv4 itself.

IPv4 went into service ~27 years ago as testbed protocol. The first formal security assessment of the IPv4 core protocols was completed last month.

IPv6 inherits all the security problems inherent in IPv4, & introduces new ones - the ND mess & the untold billions of dollars in opex costs that the consonance of the letters 'B', 'C', 'D', & 'E' will entail are just two glaring examples.

So, if we're looking for protocol architecture dragons to slay, there're far more basic issues in need of a strong sword-arm before we even get to outliers like the DNS, heh.

Authoritative service is as subject to DoS as ever, especially with IPv6 and DNSSEC
deployments.

And there are deployment & operational BCPs which can mitigate DDoS of authoritative DNS services, from logical separation of functions (see <https://files.me.com/roland.dobbins/m4g34u&gt; for an example) to S/RTBH to flowspec to commercial IDMS solutions.

[Full disclosure: my employer is a vendor of such solutions.]

The DNS protocol should be fixed so we don't need this workaround.

By all means.

In the meantime, it's important to realize that many of the resource constraints which were extant at the time the DNS was designed no longer hold sway. DNS reflection/amplification attacks can be eliminated by the simple (and, nowadays, practical, IMHO) expedient of re-configuring resolver-facing DNS servers to force the use TCP by default.

The best part is that DNS operators can start doing this today, without recourse to standards bodies, working groups, et. al. No need for any inter-organizational coordination at all - each can move at his own pace, within his own span of administrative control.

As an aside, it should be noted that switching the DNS over to TCP by default would accomplish a great deal of what DNSSEC is intended to provide, with far less in the way of architectural, operational, & administrative complexity.

Most folks who still misguidedly filter TCP/53 are unlikely to support EDNS0, either, so they're already hurting - a widespread switch to TCP would likely finally force them out of their jungle fastnesses and out blinking into the sunlight.

;>

Standardization effort should concentrate on changing the protocol.

It's important to keep in mind how long the last substantive changes to the DNS have required, as opposed to the near-term expedient of implementing BCPs & switching to TCP.

Restricting recursive service is a good short term solution, but it is not for every case.

Concur 100%. Meanwhile, we continue to resolve with the DNS we have until the new, improved version is designed, tested, & generally deployed.

;>

The workaround is not a best practice, it is evidence that the DNS protocol should be changed.

There's a reason proof-of-work solutions have never been widely deployed - in the final analysis, they all too often end up simply a form of cost-shifting which leaves the system in question just as vulnerable (if not more so) to DoS as it was in the first place, just within a different transaction path/layer.

So, while I agree with you that changes are called for, implementing proofs-of-work into the DNS transactional model isn't one I'd personally recommend, FWIW.

> Is there an RFC specifying precisely what are considered the proper prec=
autions?
> "precautions" should ideally be enabled in BIND by default.

Not of which I'm aware. I'm happy to contribute to any efforts you or anyon=
e else are volunteering to coordinate on this topic, as it's important work=
which ought to be done sooner & not later.

Named already takes proper precautions by default. Recursive service
is limited to directly connected networks by default. The default
was first changed in 9.4 (2007) which is about to go end-of-life
once the final wrap up release is done.

> My point here is that... splitting recursive service and cordoning off =
recursive service by
> using a firewall

Using a stateful firewall is contraindicated.=20

> or ACL based on IP address, is only a partially effective operational wo=
rkaround to a serious defect in the protocol.

It's actually a combination of serious defects in a number of protocols, st=
arting with IPv4 itself.=20

The real problem is that many ISP's don't do effective ingress/egress
filtering. This prevents compromised machines impersonating other
machines.

There is absolutely no reason that DSL and Cable connection shouldn't
be tied down to provider source addresess and perhaps a specfic set
of exception source addresses on request. There is absolutely no
reason that NOC's and other service networks in the ISPs shouldn't
be tied down. If you to spoof addresses for testing do it from
networks explictly setup to support this.

Named already takes proper precautions by default. Recursive service is limited to directly connected networks by default. The default
was first changed in 9.4 (2007) which is about to go end-of-life once the final wrap up release is done.

This alone isn't enough. There are quite a few other things folks must do from an architectural and operational standpoint which aren't found in named.conf.

The real problem is that many ISP's don't do effective ingress/egress filtering.

Well, no. The real problem is a protocol set/implementation which lends itself so readily to spoofing in the first place, followed (as you say) by ISP/endpoint network inattention to anti-spoofing, followed by protocols which make use of the eminently-spoofable UDP for a critical service.

This prevents compromised machines impersonating other machines.

Concur, but see above - spoofing is the symptom, not the disease.

> Named already takes proper precautions by default. Recursive service is =
limited to directly connected networks by default. The default
> was first changed in 9.4 (2007) which is about to go end-of-life once the=
final wrap up release is done.

This alone isn't enough. There are quite a few other things folks must do =
from an architectural and operational standpoint which aren't found in name=
d.conf.

> The real problem is that many ISP's don't do effective ingress/egress fil=
tering.

Well, no. The real problem is a protocol set/implementation which lends it=
self so readily to spoofing in the first place, followed (as you say) by IS=
P/endpoint network inattention to anti-spoofing, followed by protocols whic=
h make use of the eminently-spoofable UDP for a critical service.

And even if DNS/TCP was use by default machines can still get DoS'd
because IP is spoofable.

This one looks like a direct attack on the machine as there are
multiple source addresses rather than a reflector attack unless they
are attempting to attack thousands of sites simultaniously.

They can be DDoSed with spoofed or non-spoofed packets, and there are defenses against such attacks.

Apologies if I was unclear - my point was that huge, crushing, multi-gigabit-per-second DNS reflection/amplification attacks would no longer be possible with a TCP-only DNS, and that there would be other benefits, as well. Large-scale testing of TCP-only DNS would be quite informative, IMHO.