Hi,
as a comsequence of a virus diffused in my customer-base, I often receive big bursts of traffic on my DNS servers.
Unluckly, a lot of clients start to bomb my DNSs at a certain hour, so I have a distributed tentative of denial of service.
I can’t blacklist them on my DNSs, because the infected clients are too much.
For this reason, I would like that a DNS could response maximum to 10 queries per second given by every single Ip address.
Anybody knows a solution, just using iptables/netfilter/kernel tuning/BIND tuning, without using any hardware traffic shaper?
I know this is kind of a crazy idea but how about making cleaning up all these infected machines the priority as a solution instead of defending your dns from your infected clients. They not only affect you, they affect the rest of us so why should we give you a solution to your problem when you don’t appear to care about causing problems for the rest of us?
Hi,
as a comsequence of a virus diffused in my customer-base, I often receive
big bursts of traffic on my DNS servers.
Unluckly, a lot of clients start to bomb my DNSs at a certain hour, so I
have a distributed tentative of denial of service.
I can't blacklist them on my DNSs, because the infected clients are too
much.
For this reason, I would like that a DNS could response maximum to 10
queries per second given by every single Ip address.
Anybody knows a solution, just using iptables/netfilter/kernel tuning/BIND
tuning, without using any hardware traffic shaper?
"I have a bots infested network, they really task my services! How can I
make my services ignore them so that the clients start calling me and
spending my tech support budget?"
I know this is kind of a crazy idea but how about making cleaning up all
these infected machines the priority as a solution instead of defending your
dns from your infected clients. They not only affect you, they affect the
rest of us so why should we give you a solution to your problem when you
don't appear to care about causing problems for the rest of us?
George Roettger
Atually, reading your reply (which is the same as my own, pretty much), I
figure the guy asked a question and he has a real problem. Assuming he
doesn't want to clean them up is not nice of us.
Luke:
It is possible the DNS queries made are for non existent domains, fake
replies, perhaps even making them something in 1918 space, and they MAY
stop being not nice netizens.
Actually, reading your reply (which is the same as my own, pretty much), I
figure the guy asked a question and he has a real problem. Assuming he
doesn't want to clean them up is not nice of us.
Infected machines (bots) will cause a lot more than just DNS issues. Issues
like this have a way of getting worse all by themselves if not addressed.
Anyway, to play nice.. how about using a router to dampen traffic much like
icmp dampening? Would it be possible to do DNS dampening?
I know this is kind of a crazy idea but how about making cleaning up all these infected machines the priority as a solution instead of defending your dns from your infected clients. They not only affect you, they affect the rest of us so why should we give you a solution to your problem when you don't appear to care about causing problems for the rest of us?
Has anyone figured out a remote but lawful way to repair zombie machines?
I think the trouble comes when you want to limit the request rate *per client source address*, rather than limiting the request rate across the board. That implies the retention of state, and since DNS transactions are brief (and since the client population is often large) that can add up to a lot of state to keep at an aggregation point like a router.
There some appliances which are designed to hold large amounts of state (e.g. f5's big-ip) but you're talking non-trivial dollars for that. Beware enterprise-scale stateful firewall devices which might seem like sensible solutions to this problem. They are often not suitable for use in front of busy DNS servers (even a few hundred new flows per second is a lot for some vendors, despite the apparent marketing headroom based on the number of kbps you need to handle).
You may find that you can install ipfw (or similar) rules on your nameservers themselves to do this kind of thing. Take careful note of what happens when the client population becomes large, though -- the garbage collection ought to be smooth and painless, or you'll just wind up swapping one worm proliferation failure mode for another.
Host-based per-client rate limits scale better if there are many hosts providing service, e.g. behind a load balancer or using something like <http://www.isc.org/pubs/tn/isc-tn-2004-1.html>.
As to the wider question, cleaning up the infected hosts is an excellent goal, but it'd certainly be nice if your DNS servers continued to function while you were doing so. Having every non-infected customer phone up screaming at once can be an unwelcome distraction when you already have more man hours of work to do per day than you have (staff * 24).
"I have a bots infested network, they really task my services! How can I
make my services ignore them so that the clients start calling me and
spending my tech support budget?"
Or:
"I have bots on my network and as part of a multi-pronged approach to
cleaning my network while keeping the services available to those who
aren't infected, I'd like to research ways that I can minimize the
effect these bots have on the rest of my customers"
Very interesting question. I personally believe that OS EULAs and ISP
ToS guidelines provide for an ISP or an OS mfg (i.e. Microsoft) to force
updates and fixes via any means. That is: if I am your customer and my
PC/router/USB-Camera/whatever is throwing crap your way, crap that
violates your ToS or indicates that I am out of compliance with an EULA,
then I believe others have the right (and IMHO the obligation) to step
in and correct things (it's what parents do for their kids everyday).
So, according to me, any corrective action is lawful when dealing with
customers and equipment that have violated an EULA or ToS guidelines.
Indeed. It is generally accepted that it is easier to simply scale your service to provide adequate headroom than implement per-client traffic policies.
of course, you could also work on cleaning up the mess, but I will charitably assume you are working the problem from both directions simultaneously.
matto
--matt@snark.net------------------------------------------<darwin><
Moral indignation is a technique to endow the idiot with dignity.
- Marshall McLuhan
Configuring your nameservers to randomly give bad answers isn't considered being a "nice netizen" either, the last time I checked.
--matt@snark.net------------------------------------------<darwin><
Moral indignation is a technique to endow the idiot with dignity.
- Marshall McLuhan
Has anyone figured out a remote but lawful way to repair zombie machines?
Very interesting question. I personally believe that OS EULAs and ISP
ToS guidelines provide for an ISP or an OS mfg (i.e. Microsoft) to force
updates and fixes via any means. That is: if I am your customer and my
PC/router/USB-Camera/whatever is throwing crap your way, crap that
violates your ToS or indicates that I am out of compliance with an EULA,
then I believe others have the right (and IMHO the obligation) to step
in and correct things (it's what parents do for their kids everyday).
So, according to me, any corrective action is lawful when dealing with
customers and equipment that have violated an EULA or ToS guidelines.
Sending updates in automated way or forcing updates is only ok if
person previously authorized such action, i.e. enabled automated
updates. This is in fact dangerous in itself since it also presents
single point of potential failure if system providing updates is
itself compromised - that is why many choose not to do it
and enterprises setup their own updates distribution systems.
As far as your question, in my opinion it would be legal for you to
check if somebody did or did not do an update but only using tools
that check publicly available data reported from the system (i.e.
what you can gather by sending it packets to open ports). As an
ISP it would be legal for you to warn customer that if they fail
to install an update you reserve the right to disconnect their
system or limit access to certain ports or only to certain sites
(i.e. your own for them to check email but nothing else). And
obviously once issue is reported to you (i.e. their machine is
spewing and compromised), that is exactly what you should do.
Just my $.02.
Due to inflation with US currency I'll make it a nickel $.05
I think the trouble comes when you want to limit the request rate *per client source address*, rather than limiting the request rate across the board. That implies the retention of state, and since DNS transactions are brief (and since the client population is often large) that can add up to a lot of state to keep at an aggregation point like a router.
There some appliances which are designed to hold large amounts of state (e.g. f5's big-ip) but you're talking non-trivial dollars for that. Beware enterprise-scale stateful firewall devices which might seem like sensible solutions to this problem. They are often not suitable for use in front of busy DNS servers (even a few hundred new flows per second is a lot for some vendors, despite the apparent marketing headroom based on the number of kbps you need to handle).
Folks should also look at some of the DNS appliances (I know, this is "extra hardware"). Although the usually run BIND, they tend to be fairly optimized and have extra management functionality that may help with the rate limiting (if not, its probably a feature request that the vendors would entertain rapidly, as there's some pretty intense competition). Some folks to talk to - Infoblox and Bluecat. If you have really large DNS rate requirements, I'd consider talking to Nominum.
I'm curious as to just how bursty things are - how large of a departure from normality are we talking about? An order of magnitude? Two?
