Hi!
I was wondering if anyone had any experience with dealing with open resolvers as a web hoster? We currently have some 40,000 ip's that respond to DNS in our AS, the majority of which are not "open" but do reply with a referral to the root zones. We've been sending emails to our clients but as the servers are not managed by us, there's not much we can do at that level.
Recently we've seen a large increase in the number and volume of DNS amplification DDOS's that are being reflected off of our AS. Just today we've seen at least 6 different attacks with between 4 and 10gbps leaving our AS each time. It's not really causing us issues at the moment because we have the capacity, but I'd hate to be on the receiving side. (and indeed, have been on the receiving side in the past, so I know how much it can suck)
Has anyone ever tried mitigating/rate-limiting/etc these attacks in the network before? (vs at the server/application level)
We have an Arbor peakflow device, but it's not really geared for this scenario I find. It will detect the outgoing attack via the flows, but all we can really do is null-route the victims ip in our AS. Ideally we would need a way to rate-limit DNS packets based on source ip. Maybe a linux box that handles dropping packets from the same source-ip over 1000/sec with some policy-based routing sending the DNS traffic to it? Does such a box exist already?
If anyone has any ideas or suggestions, then by all means! There must be a better way to do this, and I'd really like to avoid re-inventing the wheel if it's been invented already. 
Thanks!
Thomas
We've been sending emails to our clients but as the servers are not managed by us, there's not much we can do at that level.
Sure, there is - shut them down if they don't comply. Most ISPs have AUP verbiage which would apply to a situation of this type.
Has anyone ever tried mitigating/rate-limiting/etc these attacks in the network before? (vs at the server/application level)
QoS doesn't work, as the programmatically-generated attack traffic 'crowds out' legitimate requests.
We have an Arbor peakflow device, but it's not really geared for this scenario I find.
Peakflow SP is a NetFlow-based anomaly-detection system which performs attack detection/classification/traceback. Please feel free to ping me offlist about additional system elements which perform attack mitigation.
Hi!
We've been sending emails to our clients but as the servers are not
managed by us, there's not much we can do at that level.
Sure, there is - shut them down if they don't comply. Most ISPs have AUP
verbiage which would apply to a situation of this type.
Unfortunately I somehow doubt management is going to look favourably on a
request to shut down so many clients. 
The large majority of the servers
being used in the attacks are not open resolvers. Just DNS servers that
are authoritative for a few domains, and the default config of the dns
application does referrals to root for anything else.
Yes there are ways of protecting against this on the server itself, but I
don't see it happening here given the complexity of many of the solutions.
I hate to say it, but if it's not "next -> next -> next -> finish", or
integrated as an option in one of the common web hosting panels (cPanel,
Plesk, etc) people won't do it. We still struggle just getting people to
close actual open resolvers, and that is easy to configure.
Has anyone ever tried mitigating/rate-limiting/etc these attacks in the
network before? (vs at the server/application level)
QoS doesn't work, as the programmatically-generated attack traffic
'crowds out' legitimate requests.
We have an Arbor peakflow device, but it's not really geared for this
scenario I find.
Peakflow SP is a NetFlow-based anomaly-detection system which performs
attack detection/classification/traceback. Please feel free to ping me
offlist about additional system elements which perform attack mitigation.
Pinged off-list!
Thanks!
Thomas
Offering a DNS service to your customers may allow you to provide a good
alternative to push those customers onto. You can then manage it properly.
But I think DNS isn't the real issue here, it's the fact you're receiving
spoofed traffic. I'd start by tracking the attacks backwards through your
upstreams, as obviously someone in the path isn't enforcing BCP 38. Stop
the spoof capability and the attacks will stop. It requires less effort
overall (vs your counterparts at every hosting provider needing to solve
the problem for their networks) and provides the best benefit to the
victims.
Damian
Please look at something like rate limiting.
Please look at preventing these spoofed packets from entering your network and report the issue.
Please provide advice and insights as well as directing customers to the openresolverproject.org website. We want to close these down, if you need an accurate list of IPs in your ASN, please email me and I can give you very accurate data.
Thanks!
Hi Damian!
We offer a DNS hosted solution, most people still use their own servers though. (especially those with control panels such as cPanel or plesk, where it's built-in).
As for BCP38, I would love to stop the spoofed packets, however with them coming from our upstreams, (Level3, Cogent, Tata, etc) I don't see how we can.
Thanks!,
Thomas
Contact them on a case-by-case basis to report the spoofed traffic used to stimulate the servers into responding, including the layer-4 classification criteria, traffic rates, and timestamps available via flow telemetry.
I think that a public list of open-resolvers is probably overdue, and
the only way to get them fixed.
It is trivial to scan the entire IPv4 address space for DNS servers
that do no throttling even without the resources of a malicious
botnet.
Smurf was only "fixed" because, as there were fewer networks not
running `no ip directed-broadcast,` the remaining amplification
sources were flooded with huge amounts of malicious traffic. The
public list of smurf amplifiers turned out to be the only way to
really deal with it. I predict the same will be true with DNS.
It certainly helped; but the real solution was to get Cisco, et. al. to disable directed broadcasts by default.
Well,
I was going more for a public list of ISP that refuse to BCP38 their
networks.
But that's just me =D
On point: (If your corporation is massive enough)
Basically:
. Mirror DST Port 53;
. Write some software to stats who's spamming the same DST IP with
the same query;
. Dynamic ACL them;
then
. Give a talk to your customers =D
It sounds like you're already aware that this is the default behavior for an authoritative-only server, and while the referral to the roots is a largeish response and has been used for amplification attacks, it's also rather difficult to mitigate against.
A BIND server can be configured to not do that, but contacting each of your customers about it might not have a good ROI. See https://www.dns-oarc.net/oarc/articles/upward-referrals-considered-harmful for more information.
Meanwhile, thank you very much for being proactive in this regard. Would that more SPs were as net.responsible as you.
Doug