We've had an increasing rate of DoS attacks that spew tens-of-thousands of
small UDP packets to a destination on our network. We are getting roughly
2x our entire normal pps across all providers through one interface, or
about 4x normal through the individual interface. The Cisco
7206VXR/NPE-G1 CPU melts (>95% load vs 15% average, 20% normal peak) when
this hits.
I'm using CEF and ip-route-cache flow on the outside interface. Unicast
RPF is also enabled on the interface. Unicast RPF in conjunction with a
BGP black-hole generator handles TCP attacks fairly well.
Two questions:
- Are there any knobs I should be turning in the Cisco config to help with
mitigate this?
- Are there any platforms that deal with high PPS/small packet more
gracefully?
We are looking at a network refresh and aren't locked into Cisco as a
vendor (although our current IP network consists entirely of Cisco gear).
Our current aggregate (all providers, in- plus out-bound) bandwidth is
~500Mbs, but projected growth is 1Gbs within the year.
Thanks,
Rick
S/RTBH can deal with any type of packet-flooding DDoS at layer-3, up to the capacity of the platform in question. It sounds as if a) you should investigate getting DDoS mitigation assistance from your upstreams and/or b) moving from your currently software-based platform to a hardware-based platform at your edge to provide increased performance (this holds true irrespective of which vendor you select for your edge platform).
If you move to a hardware-based edge platform, be sure to first investigate all the particulars of its uRPF implementation so as to ensure that you can use it for S/RTBH, and if at all possible, test it before buying.
Couple of things come to mind:
1. Take a packet capture to see some UDP traffic characteristics, based
on which traffic rate-limiting may be configured by your upstream
providers, so that this traffic doesn't saturate your pipes, and maybe
the ISP can even drop it. That is if they're willing to help you.
2. As far as hardware is concerned, we're in the same boat as far as
various UDP/ICMP floods, and our Juniper M10i's handle it with no issues
(running multiple BGP sessions, OSPF, firewall sets/access lists).
Sincerely,
David Kotlerewsky,
Sr. Network Engineer
Right - a hardware-based platform is required to deal with high pps rates (the Cisco equivalent is the ASR1000; I'm not familiar with boxes from other vendors, but I'm pretty sure there are others in this same class).
Although the problem we had wasn't DoS, but rather high packet rates for market data, we saw a huge improvement by moving from a 7204VRX to a 7600 platform. Going from a software switched environment to a hardware one help deal with large number of packet drops during peaks of burst activity.
We looked at the ASR1000, but found the price too high. Although cisco doesn't promote it, the 7604 with the Sup32 engine (WS-SUP32-GE-3B) with 8 x GE interfaces is a very cost effective hardware router.
Replying to my own since there are currently about a dozen responses.
- Hardware/ASIC routers are a consistent response. We are currently
evaluating Juniper for other reasons, but I'll add DoS mitigation to
mix.
- Upstream involvement: We get transit from 701, 1239, etc. I've had
mixed results getting timely responses from our upstreams. It's useful
for long-term issues, but I need as much local and timely control as I
can get.
- I'm not having a problem with pipe bandwidth, but high pps.
- uRPF and RTBH helped internally, but anything passing through that
upstream connection was impacted.
- This instance was a DoS, not DDoS. Single source and destination, but
the source (assuming no spoofing) was in Italy. Turning off netflow
seemed to help, but the attack itself stopped at about the same time.
Also, thanks for the offers of individual help in mitigation, although I'd
be concerned that "Hey, can somebody block traffic {from} or {to}?" would
be an interesting experiment in a socially-engineered DoS.
Finally, there were some suggestions "S/RTBH". RTBH I get, but my
Google-fu is weak on S/RTBH. Details?
Thanks,
Rick
Before moving to hardware based platforms, we used a lot of G1s on sticks. One of the advantages of this is the ability to filter DOS traffic on the switch in front of the router - anything 2950 or higher (with L3 snooping capabilities) can do this with an access list.
Router1 Gi0/1 ----- Gi0/1 Switch1 Gi0/2 ----- Upstream
On Switch1 configure something like:
access-list 100 deny ip host x.x.x.x
access-list 100 permit ip any any
interface GigabitEthernet0/2
ip access-group 100 in
So if your topology allows for it, this is a great short term fix. Note that this means you lose high speed convergence due to immediate link state notifications, and should use aggressive timers to compensate.
* Rick Ernst:
We've had an increasing rate of DoS attacks that spew tens-of-thousands of
small UDP packets to a destination on our network. We are getting roughly
2x our entire normal pps across all providers through one interface, or
about 4x normal through the individual interface. The Cisco
7206VXR/NPE-G1 CPU melts (>95% load vs 15% average, 20% normal peak) when
this hits.
I'm using CEF and ip-route-cache flow on the outside interface.
Is the UDP stream a single flow, or does it consist of lots of
different flows?