Running [unauthenticated UDP-based service du jour] will continue to
be a major problem as a DDoS platform on the Internet until everyone
implements BCP38.
That [unauthenticated UDP-based service du jour] should thus be
disallowed is an untenable position. We depend on [unauthenticated
UDP-based service du jour] for the correct operation of the Internet,
including such examples as authoritative DNS servers.
We've been down this path before where we try to tighten the belt on
everything we don't absolutely critically need for the sake of
allowing the root problem to keep eking by. It ain't pretty and
ultimately it isn't successful either: we merely create an arms race
where the bad actors converge on the services we -can't- shut down.
Folks should be deploying unicast-rpf facing their statically routed infrastructure. This includes server lans, PPPoE Pools, etc. Place the filtering at the edge where feasible. This would also include things like your firewall and other devices that shouldn't leak/emit spoofed packets.
If you don't know how to do this, or check on it, please ask around, either here or on cisco-nsp or juniper-nap for your platforms.
There have been some moderate size attacks recently that I won't go into detail here about. The IPs that are on the website are certainly being used/abused. A recent attack saw a 90% match rate against the "master list" here. This means your open resolver is likely being used.
Anything to raise the bar here will minimize the impact to those networks under attack. Turn on RPF facing your colocation and high-speed server lans. We all know hosts become compromised. Help minimize the impact of these attacks by
a) doing BCP-38
b) locking down your recursive servers to networks you control
c) locking down your authority servers to not provide the same answer 15x in a second to the same querying IP. If it's asking that same question 15x, then it's not you that's broken, it's that client. (Or it's being abused).
DNS servers (recursive and authoritative-only) are the low-hanging fruit du jour. I agree that there are many other effective amplifiers, and that even maximum DNS hygiene will not make the wider problem go away.
A quick note on your final comment, though: whilst adaptive response rate limiting (so-called RRL) is fast developing into an effective mitigation for reflection attacks against authority-only servers, there is far less experience with traffic patterns or the effects of rate-limiting (using RRL or anything else) on recursive servers.
The best advice for operation of recursive servers remains "restrict access to legitimate clients", not "apply rate-limiting".
Twice agree. I try to have ::1 as resolver on my server machines that
are in a position to be used, and only accept queries on ::1. Takes care
of access control nicely.
For auth servers, those serving DNSSEC records are especially attractive
as amplifiers. At the moment, I'd have a hard time defending unrestricted
query rates on auth servers if they serve DNSSEC.
I've successfully applied the Redbarn patches to my BIND, and I expect
the NSD rate-control to be of similar quality, or better.
We've formed the opinion at ICANN that the observed reaction to reflection attacks by BIND9 + Schryver/Vixie RRL is definitely different from NSD + NSD-RRL, but we don't yet know whether either one is better.
Dave Knight is busy building a test lab at DNS-OARC so he can replay identical attack traffic against BIND9, NSD and knot with equivalent RRL configurations to observe their behaviour. The source data he's using initially is from a reflection attack against L-Root that landed in Hamburg; if others here have full pcaps of similar events and are interested in comparing the reactions to it from those three nameservers, let me know and I can put you in touch.
Dave plans to talk about his methodology and findings at the DNS-OARC workshop in Dublin in May (assuming his presentation proposal is accepted).
(The DNS-OARC workshop is cojoined with the RIPE meeting, for those who are DNS-curious and haven't already considered a couple of extra days of DNS fun alongside the RIPE meeting they were already planning to attend.)
>
>>> From: "Jared Mauch" <jared@puck.nether.net>
>>> Open resolvers pose a security threat.
>> Could you clarify, here, Jared?
>>
>> Do "open DNS customer-resolver/recursive servers" *per se* cause a problem?
>>
>> Or is it merely "customer zone servers which are misconfigured to recurse",
>> as has always been problematic?
>>
>> That is: is this just a reminder we never closed the old hole, or
>> notification of some new and much nastier hole?
> There have been some moderate size attacks recently that I won't go into detail here about. The IPs that a
re on the website are certainly being used/abused. A recent attack saw a 90% match rate against the "master
list" here. This means your open resolver is likely being used.
>
> Anything to raise the bar here will minimize the impact to those networks under attack. Turn on RPF facing
your colocation and high-speed server lans. We all know hosts become compromised. Help minimize the impact
of these attacks by
>
> a) doing BCP-38
> b) locking down your recursive servers to networks you control
> c) locking down your authority servers to not provide the same answer 15x in a second to the same querying
IP. If it's asking that same question 15x, then it's not you that's broken, it's that client. (Or it's bein
g abused).
>
> - Jared
I think most of the audience here knows and are sensitive about it.
The problems come from from those who don't give a *shit*... And
they've been not giving a *shit* it for years.
The magic is in "how" to make them care.
There is only one way sure way to make them care which is to cut
them off for a period and repeat the punishment if they fail to
clean up their act. You give them notice. You publicise that you
are going to do it unless they address their issue by date X. On
date X you stop accepting routes through them or to them unless
they have cleaned up their act. At the end of the period you start
accepting traffic again.
You leave the open recursive servers open. They are your canaries.
BCP 38 was published in May 2000. There is no excuse for any ISP
to not have the requisite equipement to do this.
From: "Jared Mauch" <jared@puck.nether.net>
Open resolvers pose a security threat.
Could you clarify, here, Jared?
Do "open DNS customer-resolver/recursive servers" *per se* cause a problem?
Or is it merely "customer zone servers which are misconfigured to recurse",
as has always been problematic?
That is: is this just a reminder we never closed the old hole, or
notification of some new and much nastier hole?
There have been some moderate size attacks recently that I won't go into detail here about. The IPs that are on the website are certainly being used/abused. A recent attack saw a 90% match rate against the "master list" here. This means your open resolver is likely being used.
Anything to raise the bar here will minimize the impact to those networks under attack. Turn on RPF facing your colocation and high-speed server lans. We all know hosts become compromised. Help minimize the impact of these attacks by
a) doing BCP-38
b) locking down your recursive servers to networks you control
c) locking down your authority servers to not provide the same answer 15x in a second to the same querying IP. If it's asking that same question 15x, then it's not you that's broken, it's that client. (Or it's being abused).
- Jared
I think most of the audience here knows and are sensitive about it.
The problems come from from those who don't give a *shit*... And
they've been not giving a *shit* it for years.
The magic is in "how" to make them care
If this started to move into an AUP violation direction (e.g.: ala spammers, etc) would that motivate people?
Do the industry need to go "a la PCI-DSS" for Peers?
I think that any effort we can take here to help educate people to the right standards is helpful. I'd like to see people fix hosts, routers and a number of other things.
PS: My pico ISP is soooo on your list Jared =D Not for long hopefully.
Appreciated. And many thanks for others that have emailed me saying their hosts have been fixed as well, and those that have emailed me updated text for the webpage.
> Some of us have both publicly-facing authoritative DNS, and inward
> facing recursive servers that may be open resolvers but can't be
> found via NS entries (so the IP addresses of those aren't exactly
> publicly available info).
Just proving my point - you didn't find the webpage that also lists
their IPv6 addresses.
Now explain how you find a recursive nameserver that isn't listed in an NS
entry and *hasn't* been publicized someplace that Google can find it.
also
; <<>> DiG 9.7.0-P1 <<>> @198.82.247.34 www.google.com
That might, must *might* mind you, be somewhat tangentially related
to why I asked Jared what the BCP is for dealing with mobile devices
with hardcoded DNS lists.
(Otherwise read as "we'll be glad to fix it if somebody has a brilliant
idea on how to do so without generating more calls to the help desk than
the near-zero rate we currently get about DNS amplification issues"....)
Then wait for a while while it churns through the ~224*2^24 packets it
needs to scan the entire ipv4 internet. Of course, you could write your
own code, but that would take at least 1/2 an hour.
Then you have every open resolver on the internet.
Now, can you tell me how this is beyond the computing skill of someone who
controls a bigass botnet?
(Otherwise read as "we'll be glad to fix it if somebody has a brilliant
idea on how to do so without generating more calls to the help desk than
the near-zero rate we currently get about DNS amplification issues"....)
The whole point of this thread is that dns amplification hurts other
people, not the resolver which is being abused. Just like in the old days,
abusing open mail relays hurt other people more than the relay operator.
Actually, it often hurts the resolver(s) being abused, as well, leading to availability problems for those who legitimately need the recursive service in question.
From: Jared Mauch [mailto:jared@puck.nether.net]
>> From: "Jared Mauch" <jared@puck.nether.net>
>
>> Open resolvers pose a security threat.
>
> Could you clarify, here, Jared?
>
> Do "open DNS customer-resolver/recursive servers" *per se* cause a
problem?
>
> Or is it merely "customer zone servers which are misconfigured to recurse",
> as has always been problematic?
>
> That is: is this just a reminder we never closed the old hole, or
> notification of some new and much nastier hole?
There have been some moderate size attacks recently that I won't go into
detail here about. The IPs that are on the website are certainly being
used/abused. A recent attack saw a 90% match rate against the "master list"
here. This means your open resolver is likely being used.
I'm just going to jump in here and ask what is probably a silly question, but let's suppose I just happen to have, or have access to, a botnet comprised of (tens of) millions of random hosts all over the internet, and I feel like destroying your DNS servers via DDoS; what's stopping me from just directly querying your servers continuously from said botnet until you melt? Those machines send you traffic indirectly through open resolvers, or hit you directly, but either way, it's the same number of machines issuing the same number of queries, and you're no less inundated. If your own servers rate limit to protect themselves, you're losing valid traffic, and if they don't, once you melt down, you're losing valid traffic...
Is there an officially sounding document with "open resolver considered harmful" or alike?
It's not trivial to deal with a corporate client with an open resolver. You can't really shut them off, can't filter them etc. Googling for <open resolver considered harmful> yields nothing I can point customers to.
DNS reflection/amplification attacks aren't intended as attacks against the DNS, per se; they're intended to crush any/all targeted servers and/or fill transit pipes.
let's suppose I just happen to have, or have access to, a botnet comprised of (tens of) millions of random hosts all over the internet, and I feel like destroying your DNS servers via DDoS;
DNS reflection/amplification attacks aren't intended as attacks against the DNS, per se; they're intended to crush any/all targeted servers and/or fill transit pipes.
To be more clear, the point of DNS reflection attacks is to amplify the amount of bandwidth the botnet can muster (and perhaps hide the true source).
If you have 10s of millions of bots, you don't need to amplify. You can crush any single IP address on the 'Net.
To be more clear, the point of DNS reflection attacks is to amplify the amount of bandwidth the botnet can muster (and perhaps hide the true source).
Yes, hence the 'amplification' part.
;>
More than hiding the actual sources, I think it's more about making it difficult (at first blush) for folks to seine out and filter the attack traffic from the normal 'background radiation' of legitimate traffic.
