UDP port 80 DDoS attack

We just saw a huge flux of traffic occur this morning that spiked one of our upstream ISPs gear and killed the layer 2 link on another becuase of a DDoS attack on UDP port 80.

Wireshark shows this appears to be from a compromised game server (call of duty) with source IPs in a variety of different prefixes.

Only solution thus far was to dump the victim IP address in our block into the BGP Black hole community with one of our 2 providers and completely stop advertising to the other.

Anybody see this recently and have any tips on mitigation, reply on or off list.

Thank You,

Ray Gasnick III
CISSP, Technology Specialist: Network Security & Infrastructure
Miles Technologies
www.milestechnologies.com<http://www.milestechnologies.com/>

Phone: (856) 439-0999 x127
Direct: (856) 793-3821
How am I doing? Email my manager at itmanager@milestechnologies.com<mailto:itmanager@milestechnologies.com>

Computer Networking – IT Support – Business Software – Website Design – Online Marketing & PR

Hi.

We had a customer that was attacked by the same "game server feature".
We received aprox 10 Gbit of traffic against the customer.

The attacker sends spoofed packets to the game server with the target
IP as "source", the gameserver sends replies back via UDP to the target
host. The attacker sends a couple of hundred packets per second and thus
generating a 10 Mbit UDP flood.

There is fixes/workarounds for the game servers, just a matter of the
admin taking care of it.
See: http://rankgamehosting.ru/index.php?showtopic=1320

The "attacking" IPs aren't spoofed, so just compile a list and send
e-mails to each provider.

We had 1000+ IPs gathered and sent 100+ abuse e-mails, only received
reply from less than 20%.
Sad that people care so little about mitigating DDoS/UDP/ICMP floods.

There aren't very many ways to combat DDOS. That's why it's so popular.
Some ISP's partner with a company that offers a tunnel based scrubbing
service where they DPI all your traffic before they send it to you. If you
only have a few upstreams it may be helpful to you. I spoke to them last
year but we have too many links and too many blocks to use it. I think the
name of the company was prolexic. They're also a L3 VAR if you have L3
links. There isn't alot of BGP (AFAIK) magic that doesn't involve cutting
someone off to save the rest of your customers.

Yep, we've got a customer who's been hit with it a couple of times (5Gbps
the first time, 3Gbps the second). For hysterical raisins, we don't
actually control the network for this particular customer, but the network
provider did pretty much what you did -- blackholed the victim IP. We've
mitigated the problem by using a full-time traffic-scrubbing service -- the
hope is that the scrubbing service will pay for all the traffic and only the
good stuff will get through. Only time will tell if it works. We also had
to renumber the customer, as the attacks were obviously remembering the old
IP and still knocking it off the network even after the DNS was repointed at
the scrubbing service.

- Matt

Start with the various infrastructure/host/service BCPs, and S/RTBH, as outlined in this preso:

<https://files.me.com/roland.dobbins/dweagy>

An entire power point just to recommend ACL's, uRPF, CPP, DHCP snooping,
and RTBH? The first four will not work against a DDOS attack and the last
one just kills the patient so he does not infect other patients. As I said
earlier beyond traffic scrubbing offsite there isn't much defense against
DDOS.

An entire power point just to recommend ACL's, uRPF, CPP, DHCP snooping, and RTBH?

Actually, no, that isn't the focus of the preso.

The first four will not work against a DDOS attack

This is incorrect - suggest you read the preso.

and the last one just kills the patient so he does not infect other patients.

S/RTBH - as opposed to D/RTBH - doesn't kill the patient. Again, suggest you read the preso.

There's been a lot of discussion on this topic on NANOG, suggest you take a look through the archives.

More info here:

<https://files.me.com/roland.dobbins/l53gjr>

> An entire power point just to recommend ACL's, uRPF, CPP, DHCP snooping,
and RTBH?

Actually, no, that isn't the focus of the preso.

> The first four will not work against a DDOS attack

This is incorrect - suggest you read the preso.

The ACL's are configured on the routers belonging to the victim AS which
will not save their access pipe if it's overrun unless I'm missing
something. uRPF may help with spoofed traffic, but sometimes causes
problems with multi-homing and is often more harmful than helpful depending
on the network design.

> and the last one just kills the patient so he does not infect other
patients.

S/RTBH - as opposed to D/RTBH - doesn't kill the patient. Again, suggest
you read the preso.

Source RTBH often falls victim to rapidly changing or spoofed source IP"s.
It also isn't as widely supported as it should be. I never said DDOS was
hopeless, there just aren't a wealth of defenses against it.

S/RTBH can be rapidly shifted in order to deal with changing purported source IPs, and it isn't limited to /32s. It's widely supported on Cisco and Juniper gear (flowspec is a better choice on Juniper gear).

If folks don't want to read the presos or search through the archives, that's fine, of course. The fact is that there are quite a few things that operators can and should do in order to mitigate DDoS attacks; and making the perfect the enemy of the merely good only helps the attackers, doesn't it?

;>

> Source RTBH often falls victim to rapidly changing or spoofed source
IP"s.

S/RTBH can be rapidly shifted in order to deal with changing purported
source IPs, and it isn't limited to /32s. It's widely supported on Cisco
and Juniper gear (flowspec is a better choice on Juniper gear).

I was referring to support from ISP's not from hardware vendors.

If folks don't want to read the presos or search through the archives,

that's fine, of course. The fact is that there are quite a few things that
operators can and should do in order to mitigate DDoS attacks; and making
the perfect the enemy of the merely good only helps the attackers, doesn't
it?

Yes but assuming everything discussed at a conference is instantly adopted

by the entire industry gives one false hope no?

I'm certainly not making that assumption - hence the presos.

;>

This is so very easily automated. Even if you don't actually want to trigger the routes automatically, finding the sources you want to blackhole is as simple as a monitor port, tcpdump and some basic Perl.

...and as far as this not having been deployed in many ISPs (per your next message)... their mitigation strategies should be asked up front, and if they don't have any (or don't know what you speak of), find a new ISP.

Steve

S/RTBH - as opposed to D/RTBH - doesn't kill the patient. Again, suggest

you read the preso.

Source RTBH often falls victim to rapidly changing or spoofed source IP"s.
It also isn't as widely supported as it should be. I never said DDOS was
hopeless, there just aren't a wealth of defenses against it.

This is so very easily automated. Even if you don't actually want to
trigger the routes automatically, finding the sources you want to blackhole
is as simple as a monitor port, tcpdump and some basic Perl.

This is still vulnerable to spoofing which could cause you to filter
legitimate traffic and make the problem worse. Not saying that S/RTBH is a
bad idea. RTBH is effective and a great idea just not very elegant.

...and as far as this not having been deployed in many ISPs (per your next
message)... their mitigation strategies should be asked up front, and if
they don't have any (or don't know what you speak of), find a new ISP.

You sometimes have to weigh the pro's and cons. You can't always pick the
guys with the coolest knobs.

2012/2/5 Steve Bertrand <steve.bertrand@gmail.com
        Source RTBH often falls victim to rapidly changing or spoofed
        source IP"s.
        It also isn't as widely supported as it should be. I never said
        DDOS was
        hopeless, there just aren't a wealth of defenses against it.

    This is so very easily automated. Even if you don't actually want to
    trigger the routes automatically, finding the sources you want to
    blackhole is as simple as a monitor port, tcpdump and some basic Perl.

This is still vulnerable to spoofing which could cause you to filter
legitimate traffic and make the problem worse. Not saying that S/RTBH
is a bad idea. RTBH is effective and a great idea just not very elegant.

Agreed. Diligence does play a role. However, the times I have implemented and used (s/)RTBH, I thought it was most elegant. I love its simplicity and effectiveness.

    ...and as far as this not having been deployed in many ISPs (per
    your next message)... their mitigation strategies should be asked up
    front, and if they don't have any (or don't know what you speak of),
    find a new ISP.

You sometimes have to weigh the pro's and cons. You can't always pick
the guys with the coolest knobs.

Agreed. But to me, DDOS mitigation is not just a cool knob. If my ISP can help mitigate a 1Gb onslaught so my 100Mb pipe isn't overwhelmed, that's more functional than cool. Ranks right up there with IPv6 :wink:

Steve

What transit providers are doing flow-spec, or otherwise, to allow
their downstreams to block malicious traffic by SOURCE address?

The point is well taken that cloud scrubbing can be an essential component of mitigating a volumetric flood. However, it is important to note that DDOS attacks do not only consist of volumetric floods. Current attacks often incorporate a multi-vectored attack campaign including a combination of low and slow and application layer attacks on upper layer protocols, ie. DNS & HTTP(s). These campaigns are designed to fly under the triggers of other flow based analysis (cloud scrubbing) protections in place today. As with any security protection a layered approach is required in order to protect the perimeter from DDOS. In addition to the previous recommendations of ACL, uRPF, RTBH, CoPP, inspection of the full stack is required. The best protection today includes a detector capable of inspecting the full stack and signaling back to the cloud scrubbing station to swing the route if the attack becomes volumetric. The premise device should have technique in order to challenge the source and counter the attack with intelligence. I'm aware of two vendors offering some of these capabilities today, Radware and Arbor.

It also isn't as widely supported as it should be. I never said DDOS was
hopeless, there just aren't a wealth of defenses against it.

there is a fix for it, it's called "putting a fuckton of ram in -most- routers on the internet" and keeping statistics for each destination ip:destination port:outgoing interface so that none of them individually can (entirely/procentually compared to other traffic) flood the outgoing interface on that router... end result, if enough routers are structured
like that, is that ddos attacks will be come completely useless.

keyword here, is terabytes of ram.

statistic tables on very basic ipv4 stuff alone would already take multiple 100's of gigabytes...

(keep in mind, this won't be "cpu work", its just using the table entry size * dest ip as an offset and reading it out :wink:

the good news is, ram doesn't cost a flying fuck anymore...

it just requires a complete re-think of router design, but then again, with the prices that cisco and juniper charge we expect a bit more than a 1960's telephone exchange look and feel device, or we might as well use 1 linux box/blade per 20gbit/s throughput and consider that whole thing a "hotpluggable interface". cisco and juniper, at the moment, sell faster versions of their crap from the 1990s, but not much effort went into completely re-designing the stuff to be suitable for the internet as it is today, they still forward all packets they can get their hands on, and besides rather simple stuff like QoS, not much effort went into inteligently spreading the capacity available on the outgoing interface (at least for that individual router) over all the possible targets.

now, if an outgoing interface is 10ge, and 10mbit traffic should go to 1.2.3.4 and 20ge (ddos) traffic should go to 4.3.2.1, i'd say, a router should give 1.2.3.4 the full 10ge, as it is available, and lower volume should basically just get a higher priority.

we have not quite worked out the formula yet... but it should be something along those lines (simply to say, any destination ip can never fill more than half of the outgoing interface, doesn't quite cover it, it needs some "intelligent adjustment of the percentage")..

basically...
table:

destip
destport
packetcounter

every so and so many packets/timeslices,whatever that packetcounter is decreased by 1

every packet, the packet counter is incremented

after inactivity for that ip for timeperiod x, the packet counter is cleared.

with ipv4, this "destip" entry is such a small (32 bit) integer that its better to just not store the ip itself but rather just throw more ram at it and use the ip address number as the offset in the array

(for faster lookups, preferably in hardware logic).

the same thing could be applied to pretty much every other concept still done with slow lookups nowadays (arp, routes, etc)... throw more ram at it and use the destination as the offset, who cares if 99.9% of the ram is not actually used. for the price of a juniper, you can buy a truck full of ram chips ;).. it's faster that way, and it allows us to do a lot of things, like not passing potential ddos floods in the first place, and intelligently allowing everyone, not an equal share of the traffic capacity, but the part they need.

if you don't mind wasting 50% of the available capacity the formula to determine wether to forward a packet or not is quite simple, if you also want to give a destip 100% of the traffic in situations where there is no other traffic, it becomes a bit more complicated..

as stated before, we haven't quite worked it out in full yet, but in any case... this would require for at least 30% of the routers that currently are on the internet and are 110 kg heavy "1960s telephone exchange models" to be replaced with 2012 style hardware, not "just faster cisco 7200 like things".

There are two obvious problems with your approach.

First, adding the policers you suggest, at the scale needed, is a
little harder than you imagine. It's not a simple matter of the cost
of RAM but also power/heat density per port.

Second, if you re-engineer every router on the Internet to prevent an
interface from being congested by malicious flow(s) destined for one
particular destination IP:port, then DDoS attacks will simply target
multiple ports or multiple destination IP addresses that are likely to
traverse a link they are able to congest.

If you want to dramatically increase the cost of routers in order to
solve the problem of DDoS with one deft (and expensive) move, you have
to imagine that the people behind DDoS attacks aren't complete idiots,
and will actually spend some time thinking about how to defeat your
system.

> there is a fix for it, it's called "putting a fuckton of ram in -most-
> routers on the internet" and keeping statistics for each destination
> ip:destination port:outgoing interface so that none of them individually
can
> (entirely/procentually compared to other traffic) flood the outgoing
> interface on that router... end result, if enough routers are structured
> like that, is that ddos attacks will be come completely useless.

There are two obvious problems with your approach.

First, adding the policers you suggest, at the scale needed, is a
little harder than you imagine. It's not a simple matter of the cost
of RAM but also power/heat density per port.

Since when are policers implemented in ram? You're talking FPGA if you
want to be able to make forwarding/filtering decisions assuming it's
possible which it isn't you're 1 million dollar boxes suddenly become
hundred million dollar boxes. Then there's v6 info..

Second, if you re-engineer every router on the Internet to prevent an
interface from being congested by malicious flow(s) destined for one
particular destination IP:port, then DDoS attacks will simply target
multiple ports or multiple destination IP addresses that are likely to
traverse a link they are able to congest.

Not to mention that the routers themselves become an attack vector. This
cache will have a finite limit because there's no such thing as an infinite
amount of cache among other flaws. When that limit is reached it will do
something no one want's it to do and that will become the new DDOS attack.

If you want to dramatically increase the cost of routers in order to
solve the problem of DDoS with one deft (and expensive) move, you have
to imagine that the people behind DDoS attacks aren't complete idiots,
and will actually spend some time thinking about how to defeat your
system.

Not to mention cost? You're not going to get a tier I ISP to upgrade to

this new super router (assuming it's possible to build such a things)
without an act of congress or at least the FCC. They won't even spend
enough on fiber to bring broadband into rural areas.