Today, we experienced (3) separate DDoS attacks from Eastern Asia, all generating > 2Gbps towards a single IP address in our network. All 3 attacks targeted different IP addresses with dst UDP 19, and the attacks lasted for about 5 minutes and stopped as fast as they started.
Does anyone have any suggestions for mitigating these type of attacks?
A couple of things that we've done already...
We set up BGP communities with our upstreams, and tested that RTBH can be set and it does work. However, by the time that we are able to trigger the black hole, the attack is almost always over.
For now, we've blocked UDP 19 incoming at our edge, so that if future, similar attacks occur, it doesn't affect our internal links.
What I think that I need is an IDS that can watch our edge traffic and automatically trigger a block hole advertisement for any internal IP beginning to receive > 100Mbps of traffic. A few searches are initially coming up dry...
Eric Miller, CCNP
Network Engineering Consultant
(407) 257-5115
Today, we experienced (3) separate DDoS attacks from Eastern Asia, all generating > 2Gbps towards a single IP address in our network. All 3 attacks targeted different IP addresses with dst UDP 19, and the attacks lasted for about 5 minutes and stopped as fast as they started.
Does anyone have any suggestions for mitigating these type of attacks?
The phrase automated offensive cyber counter-attack has been coming to mind rather frequently, of late. I wonder if DARPA might fund some work in this area.
Here's a thought-provoking video on what Brocade has done with its SDN
software load on the MLX: http://vimeo.com/87476840 (demo at ~15 minute mark)
I've written it before: if there was a software feature in routers where I
could specify the maximum rate any prefix size (up to /32) could receive,
that would be very helpful. If my fastest speed (residential) customer was
100 Mbps and I specified that 200 Mbps was the highest, I would never see
high-rate attacks enter our network.
QoS generally isn't a suitable mechanism for DDoS mitigation, as the programmatically-generated attack traffic ends up 'crowding out' legitimate traffic.
S/RTBH, flowspec, and other methods tend to produce better results.
There's no doubt, rate-limiting is a poor-man's way of getting the job done,
but for small operators who aren't as well instrumented (whether that due to
staff or resources), a simple rule such as:
access-list 100 ip host 0.0.0.0 0.0.0.0 rate-limit 200000
access-list 100 ip host 0.0.0.0 0.0.0.255 rate-limit 5000000
int vlan 10
description Internet uplink
ip access-group 100 in
!
would be great.
Yes, the /32 under attack would essentially be out of service, but at least
the downstream network doesn't get congested and more customers affected.
When you're being hit with one of the UDP reflection DDoS's, attacking the world in response isn't likely to work too well.
In theory, you could write something that takes flow data from your transit routers, and in either near or real time, looks at that data and triggers an RTBH route for any IP that is responsible for more than a certain defined threshold of inbound traffic. In practice, it gets a little more complicated than that, as you'll likely want to whitelist some IPs and/or maybe be able to set different thresholds for different IPs. But it's not that complicated a problem to solve. Have a default threshold, and a table of networks and thresholds. Once a minute, look at the top X local destinations over the past minute. For each one, check to see if it has a custom threshold. If it doesn't, it gets the default. Then see if it's over its threshold. If it is, generate an RTBH route and email your NOC.
The tricky part is when to remove the route...since you can't tell if the attack has ended while the target is black holed by your upstreams.
You can with NetFlow, if you've D/RTBHed the IP in question on your own infrastructure. NetFlow reports statistics on dropped traffic (except on a few platforms with implementation deficiencies).
But this kind of thing punishes the victim. It's far better to do everything possible to *protect* the target(s) of an attack, and only use D/RTBH as a last resort.
A quick and dirty win is to get your upstream(s) to kill anything UDP 19 to
your prefixes at their ingress points if it becomes a common thing. You
lose visibility as to when you're getting targeted by that type of attack
again though, which could matter depending on your network.
The tricky part is when to remove the route...since you can't tell if the attack has ended while the target is black holed by your upstreams.
You can with NetFlow, if you've D/RTBHed the IP in question on your own infrastructure. NetFlow reports statistics on dropped traffic (except on a few platforms with implementation deficiencies).
I'm assuming from the OP's comment:
"We set up BGP communities with our upstreams, and tested that RTBH can
be set and it does work."
that they have their upstreams null routing the traffic, so they no longer see the attack traffic.
But this kind of thing punishes the victim. It's far better to do everything possible to *protect* the target(s) of an attack, and only use D/RTBH as a last resort.
I'm sure it's not always the case, but in my experience as a SP, the victim virtually always did something to instigate the attack, and is usually someone you don't want as a customer. When I worked for a cloud hosting provider, the DDoS "victims" tended to be fraudulent signups who were doing malicious or anti-social things on the net and were not paying customers anyway.
As someone else mentioned, it's better to sacrifice the one target and end the impact quickly than to piss off all or even some subset of your customers.
How many holes are you going to stick fingers in to stop the flows? Good luck getting your provider to put in such a filter and make it anything more than temporary...and then there's still DNS, NTP, SNMP, and other protocols an attacker can easily utilize when they find that chargen isn't getting the job done.
I'm sure it's not always the case, but in my experience as a SP, the victim virtually always did something to instigate the attack, and is usually someone you don't want as a customer.
This may be a reflection of your experience and customer base, but it isn't a valid generalization. Legitimate customers are attacked all the time, for various reasons - including unknowingly having their servers compromised and used as C&Cs by miscreants, who're then attacked by other miscreants.
But to say that attacks are 'virtually always' provoked by customers themselves simply isn't true. DDoS extortion, ideologically-motivated DDoS attacks, maskirovkas intended as a distraction away from other activities, simple nihilism, et. al. are, unfortunately, quite common.
When I worked for a cloud hosting provider, the DDoS "victims" tended to be fraudulent signups who were doing malicious or anti-social things on the net and were not paying customers anyway.
Many DDoS attacks are miscreant-vs.-miscreant, that's certainly true. Compromised machines are 'attractive nuisances', which is yet another reason it's important to have visibility into your network traffic (it's easy to get started with NetFlow and open-source tools).
I wouldn't have suggested it if I hadn't successfully made these requests
myself. Most attacks don't last long enough to put a dent on billing so
it's in everyone best interest to cull it quickly.
Providing the upstream network is big enough and your attacks aren't huge
pipefills, they will usually place the acl on your customer port first,
which in those cases should be enough.
For smaller attacks you can try to drop at your router/absorb
it/ scrub it inside your border if you have the kit - but anything
significant like the NTP reflection attacks earlier in the year, you come
up against the "bandwidth arms race" problem.
There are services out there like Prolexic/Black Lotus offer where they can
scrub traffic for you, but I've never used one first hand so can't comment.
Like have the temerity to have a successful online store. Or be featured in
the mainstream media for providing information during a natural disaster.
The bastards. I've dealt with far more DDoS attacks that were for the
purposes of extortion or lulz than were due to the recipient "instigating
the attack". Perhaps that's a function of not attempting to cater to the
lowest common denominator.
I've written it before: if there was a software feature in routers
where I
could specify the maximum rate any prefix size (up to /32) could receive,
that would be very helpful.
QoS generally isn't a suitable mechanism for DDoS mitigation, as the
programmatically-generated attack traffic ends up 'crowding out'
legitimate traffic.
if you can identify attack traffic well enough to police it reliably
then you can also drop it on the floor.
S/RTBH, flowspec, and other methods tend to produce better results.
But that's my point: many small operators don't have tools and/or staff to
identify flows in order to police and/or drop the traffic, and definitely
not a NOC that can intervene in under 5 minutes. How much simpler if there
was a generic rule that said "no one IP can receive more than 200 Mbps", log
on that, and then if it takes 30 or 90 minutes for someone to react, that's
fine, but in the meantime other customers weren't affected.
