Effective ways to deal with DDoS attacks?

Mailman List Mirror (Read Only)
1

There's been plenty of discussion about DDoS attacks, and my
IDS system is darn good at identifying them. But what are
effective methods for large service-provider networks (ie
ones where a firewall at the front would not be possible) to
deal with DDoS attacks?

Current method of updating ACLs with the source and/or
destination are slow and error-prone and hard to maintain
(especially when the target of the attack is a site that
users would like to access).

A rather extensive survey of DDoS papers has not resulted in
much on this topic.

What processes and/or tools are large networks using to
identify and limit the impact of DDoS attacks?

Thanks.
Pete.

2

http://www.secsup.org/Tracking/

UUNet uses that...others might as well, Shrug.

Quick, simple, effective tracking of DDoS attacks.

As for identifying attacks, quite honestly large ISP's are typically still
relying on customer notification. I know that's how we do it.

3

Hazaa.. something I know a little about.

DDoS attacks by their very nature, are distributed.
The primary purpose of more DDoS attacks is to flood the target's upstream
connection to the point of saturation.

As time goes by, tools are being developed (in fact they're used now) that
completely randomize the TCP or UDP ports attacked, or use a variety of
icmp types in the attack.
So cuurrently the only way you can 'block' such attacks is to block all
packets for the offending protocol as far upstream as you possibly can,
but this is not ideal.

If you're being attacked by a SYN flood, you can ask try to rate-limit the
flood at your border (possible on Cisco IOS 12.0 and higher, and probably
other routers too?)

If you're being smurfed, you can block ICMP Echo Reply's inbound to the
target IP.

It all depends on the TYPE of attack.

Having said that, it's only a matter of time before somebody releases a
tool that saturates a line by spooofing the source, randomizing the
protocol, and ports, and maybe even atacking other hosts on the same
subnet, etc etc.

The only thing you can try and do is work with your upstream provider and
try to trace the source of the attacks back, but that's incredibly
difficult.

As a side note, does anyone know the status of the ICMP Traceback
proposal? The ieft draft expired yesterday:
http://www.ietf.org/internet-drafts/draft-ietf-itrace-01.txt

4

"DDoS attacks" is such a generic term. There are a wide variety of attacks
which each need to be handled in their own way, the extra "D" is just one
possible twist. Can you explain what kind of attack you're interested in?

I've tried to compile a list of the *practical* things everyone needs to
know (but usually doesn't) to handle DoS effectively, try reading:

http://www.e-gerbil.net/ras/projects/dos/dos.txt

5

Actually the original goal (and probably still the primary benefit) of
DDoS attacks was to evade detection by using so many hosts that any
specific one either went unnoticed or unreported.

Having multiple sources to bypass any potential congestion (since a DoS is
only as effective as the weakest link along the path) and filtering is
still what I would rank as a secondary effect.

6

There's been plenty of discussion about DDoS attacks,

and then again, there has been much discussion on simple DoS attacks, where
the term DDoS is erroneously used... I am very much not trying to imply that
this is the case here, but it's important that the two be thoroughly
distinguished from each other - they are totally different things to deal
with.

and my
IDS system is darn good at identifying them.

Chances are your IDS is detecting simple DoS, or maybe tiny scale DDoS. Full
DDoS attacks do not require and IDS to detect :wink: In fact, if your IDS
doesn't tip over under the load of a full blown DDoS, I'd sure like to know
what it's using for an engine...

But what are
effective methods for large service-provider networks (ie
ones where a firewall at the front would not be possible) to
deal with DDoS attacks?

True DDoS attacks, fortunately, are rarer than most people believe. If they
were not, the Internet as we know it would look a lot more like a telephone
system in USSR-at-it's-worst-days. For example, of the two recent DDoS's I
have been on the receiving end of, the first was generating a little over
300mbit/sec (steady for a prolonged time), and the second went over that by a
fair bit. In both cases, we had core equipment (M20's and BSN5000's) fall
over and die trying to "work" the events. Additionally, our upstream peers
also had core equipment fall over, and we all came the [now obvious]
conclusion that the only way to stop these attacks was to completely null
route ourselves at our upstreams (they tried filter-fishing for specific data
which may have helped our investigation, but when their routers started
wheezing, we gave them the OK to just send us straight into the bit bucket
till it was over...

Current method of updating ACLs with the source and/or
destination are slow and error-prone and hard to maintain
(especially when the target of the attack is a site that
users would like to access).

We captured several seconds of the last DDoS and came up with over 700
participating hosts...

A rather extensive survey of DDoS papers has not resulted in
much on this topic.

What processes and/or tools are large networks using to
identify and limit the impact of DDoS attacks?

A great deal of thought is being expended on this question, I am certain,
however, how many of these thought campaings have born significant fruit yet,
I do not know.

7

> What processes and/or tools are large networks using to
> identify and limit the impact of DDoS attacks?

A great deal of thought is being expended on this question, I am certain,
however, how many of these thought campaings have born significant fruit

yet,

I do not know.

How about the following :

We develop a new community , being fully transitive (666 would be
appropriate ) and either build into router code or create a route map to
null route anything that contains this community. The effect of this being
the distribution of the force of the attack.

This aside, how effective would be using a no export community with ones
peers (being non transitive, it would still distribute the force of the
attack).

8

Then you are pushing out /32's and peers would need to accept them. Then
someone will want to blackhole /30's, /29's, etc. Route bloat. Yum!

Additionally you are creating a way to basically destroy the Internet as a
whole. One kiddie gets ahold of a router, say of a large backbone
provider, takes one of their aggregate blocks (/16? /10? /8?) and splits
it into /32 announcements.

Anyways, some providers already allow you to set a community on a route,
and they will inturn "blackhole" it for you. I believe Teleglobe does
this for some customers and I know UUNet does this for all customers.

9

Then you are pushing out /32's and peers would need to accept them. Then
someone will want to blackhole /30's, /29's, etc. Route bloat. Yum!

I am in no way proposing discounting current filtering rules. There are
alway two
different intersts one must consider, one that of the customer and two that
of the service provider. If a large block must be filtered so be it.

Where are providers drawing the line ? Anyone have somewhat detailed
published policies as to what a provider can do in order to protect their
nework as a whole.
At what point (strength of the attack) does a customers netblock (assuming a
/24 for
example) get null routed by whichever party.

Anyways, some providers already allow you to set a community on a route,
and they will inturn "blackhole" it for you. I believe Teleglobe does
this for some customers and I know UUNet does this for all customers.

When the attack is distributed, having one or two providers (even if they
are UUNET
or Teleglobe) is just not enough. Must private routing policy be developed
in order to make my suggestion work. The reason that so many methods likely
fail are the difficulty of implementation and low implementation.

10

In a message written on Wed, May 01, 2002 at 08:17:04PM -0500, dies wrote:

Then you are pushing out /32's and peers would need to accept them. Then
someone will want to blackhole /30's, /29's, etc. Route bloat. Yum!

I'm not sure what form this would take, but I have long wished
route processing could be sent into a "programming language". For
this specific example it would be nice to set a maximum number of
route limit for the total number of routes on the session, as well
as /per community/.

That is, community xxxx:666 == blackhole me, and I could limit each
peer to say, 6 of these at a time. More would not take down the
session, but simply be ignored.

I can carry 6 /32's for every peer I have, and if they only have
6, they will probably use them for the most abusive target.

There are, of course, approximately an infinitate number more
applications for a more flexible mechanism. Of course, it would
require more human smarts, which might be why vendors don't do it.

11

How about the following :

We develop a new community , being fully transitive (666 would be
appropriate ) and either build into router code or create a route map to
null route anything that contains this community. The effect of this being
the distribution of the force of the attack.

This has been proposed a dozen times over, and I agree that there should
be a well known community for discarding packets. Go try and get the IETF
to add it, let me know how it goes. :slight_smile:

This aside, how effective would be using a no export community with ones
peers (being non transitive, it would still distribute the force of the
attack).

Many people do this already. If you're looking to purchase transit and you
think this is something you'll care about, ask for it or vote with your
wallet.

12

In a message written on Wed, May 01, 2002 at 08:17:04PM -0500, dies wrote:
> Then you are pushing out /32's and peers would need to accept them. Then
> someone will want to blackhole /30's, /29's, etc. Route bloat. Yum!

I'm not sure what form this would take, but I have long wished
route processing could be sent into a "programming language". For
this specific example it would be nice to set a maximum number of
route limit for the total number of routes on the session, as well
as /per community/.

Agreed wholeheartedly. But then you'd have to have network engineers who
could program (and no perl doesn't count). :slight_smile:

That is, community xxxx:666 == blackhole me, and I could limit each
peer to say, 6 of these at a time. More would not take down the
session, but simply be ignored.

I can carry 6 /32's for every peer I have, and if they only have
6, they will probably use them for the most abusive target.

I give it 2 months, then they'll start hitting random dst IPs in a target
prefix (say a common /24 going through the same path).

13

We experience a lot of types of attacks ("education/research
network" = "easy hacker target"). With DDoS incidents, it
seems we are more often an unknowing/unwilling participant
than the target, partly due to owning big chunks of IP
address space.

We most frequently are the zombie/reflector participants in
an attack that originates outside our network, to a target
outside our network. As many as 8,000 hosts on our network
are reflecting SYN floods in the current attacks.

Identification doesn't seem to be a problem. Snort is doing
far too well notifying us. Responding and managing all of
the defenses is becoming a lot of pain-staking work, and
error-prone (why can't Cisco make ACLs easier to manage).

Our approach so far has been temporary blocks (via ACL) of
the target address. Blocking 8,000 internal addresses, many
legitimate (secured) Web servers, generates more complaints.

I'm thinking about a scripted Zebra feed where route
injections are triggered by Snort. Routes for the target
and/or SYN flood reflector hosts could be injected
temporarily during the attack to border routers, which would
route-map those routes to Null0. Script periodically
withdraws routes to see if the attack is over (some of these
last weeks, some only last a few seconds), to minimize the
impact on those otherwise legitimate hosts.

Has anyone tried this kind of an approach or any other type
of automated/efficient approach to dampen the "zombie" side
of the DDoS attack?

Pete.

14

We experience a lot of types of attacks ("education/research
network" = "easy hacker target"). With DDoS incidents, it
seems we are more often an unknowing/unwilling participant
than the target, partly due to owning big chunks of IP
address space.

Universities are hacker training grounds, but also have much
better network security response than most corporate networks.
Whatever problems you have, the rest of us will have soon enough
it may just take us longer to notice it.

Has anyone tried this kind of an approach or any other type
of automated/efficient approach to dampen the "zombie" side
of the DDoS attack?

Has anyone implemented Bellovin's Pushback in a production
network yet?

15

and then again, there has been much discussion on simple
DoS attacks, where the term DDoS is erroneously used...
I am very much not trying to imply that this is the case
here, but it's important that the two be thoroughly
distinguished from each other - they are totally
different things to deal with.

Sorry, I should have been more clear.

My issue (currently) is not being the target of the DDoS
attack, but being a (unwilling) participant. People outside
our network are launching DDoS attacks (distributed SYN
floods) against destinations outside our network, using
about 8,000 Web server hosts on our network as reflectors.

These are not zombies. They are secured, uncompromised Web
servers. The attack spoofs the target address as the source,
and one of our machines as a destination, port 80. Getting
everyone to implement defenses (SYN cookies) on their Web
servers is nearly impossible (most don't even have a
defense--printers and routers with Web interfaces).

SYN packet comes in, one of these machines responses with a
RST to the "source", which is actually the target of the
attack. Unfortunately, the target is often a site that
people would like to get to, as is the reflector, so
permanent filters on the target or reflector create lots of
complaints.

We captured several seconds of the last DDoS and came up
with over 700 participating hosts...

Some of them probably appear to be from our network...

Pete.

16

What we use and we're a 'largeish' network:

http://www.secsup.org/Tracking/
(shameless plug #1)

Among other things this is a tool we use... there was a great set of
slides and presentation given at NANOG23:

http://www.nanog.org/mtg-0110/greene.html
(shameless plug #2)

There is also a set of papers Barry Greene from Cisco has available on the
Cisco website... I'm positive he'll respond to this with the link, if he
doesn't search the NANOG mailing list archive for the link it should be
obvious in posts from Barry.

If you want more pointers I'd be glad to chat on the phone with you,
numbers included below.

--Chris
(chris@uu.net)

17

> A rather extensive survey of DDoS papers has not resulted in
> much on this topic.
>
> What processes and/or tools are large networks using to
> identify and limit the impact of DDoS attacks?

Hazaa.. something I know a little about.

DDoS attacks by their very nature, are distributed.
The primary purpose of more DDoS attacks is to flood the target's upstream
connection to the point of saturation.

As time goes by, tools are being developed (in fact they're used now) that
completely randomize the TCP or UDP ports attacked, or use a variety of
icmp types in the attack.
So cuurrently the only way you can 'block' such attacks is to block all
packets for the offending protocol as far upstream as you possibly can,
but this is not ideal.

If you're being attacked by a SYN flood, you can ask try to rate-limit the
flood at your border (possible on Cisco IOS 12.0 and higher, and probably
other routers too?)

Let me say this one more time... "RATE LIMITS DON'T DO SHIT TO STOP
ATTACKS" for the victim atleast, all they do is make the job of the
attacker that much easier. For instance:

1) I synflood www.avleen.org
2) you rate-limit syns to 1MB
3) I now only flood 1MB and I still win

So, don't rely on a rate-limit as its not going to help.

If you're being smurfed, you can block ICMP Echo Reply's inbound to the
target IP.

It all depends on the TYPE of attack.

This point is very good, depending on the attack your ISP (or you) might
have to take different counter measures to stop the attack... You (or your
ISP) will need to know as much as possible about the attack, the defenses
available on the hardware/software in the network, and actually be
available when there is a problem.

Having said that, it's only a matter of time before somebody releases a
tool that saturates a line by spooofing the source, randomizing the
protocol, and ports, and maybe even atacking other hosts on the same
subnet, etc etc.

The only thing you can try and do is work with your upstream provider and
try to trace the source of the attacks back, but that's incredibly
difficult.

This depends :slight_smile: Call us, if you are our customer, and I guarantee that
someone will be there to resolve your issue, most times in 5 minutes or
less. Perhaps other ISP's should start having some folks on staff and
available for these tasks??? (hint, Hint, HINT!!!)

As a side note, does anyone know the status of the ICMP Traceback
proposal? The ieft draft expired yesterday:
http://www.ietf.org/internet-drafts/draft-ietf-itrace-01.txt

This is a wasted effort. It's not feasible, nor is it reasonable. There
are tools in place today that perform this function without the required
changes to protocols/functionality. Why would you want to make things
incrementally more difficult when the technology exists today to do this?

-Chris

18

True DDoS attacks, fortunately, are rarer than most people believe. If they
were not, the Internet as we know it would look a lot more like a telephone
system in USSR-at-it's-worst-days. For example, of the two recent DDoS's I
have been on the receiving end of, the first was generating a little over
300mbit/sec (steady for a prolonged time), and the second went over that by a
fair bit. In both cases, we had core equipment (M20's and BSN5000's) fall
over and die trying to "work" the events. Additionally, our upstream peers

Your M20 tipped over?? What were you doing? We regularly stop large
(+100Mb->800Mb) attacks with less horsepower than this. Truthfully, a
cisco is even capable of filtering (done right) at +200kpps...

also had core equipment fall over, and we all came the [now obvious]
conclusion that the only way to stop these attacks was to completely null
route ourselves at our upstreams (they tried filter-fishing for specific data
which may have helped our investigation, but when their routers started
wheezing, we gave them the OK to just send us straight into the bit bucket
till it was over...

Hmm, this highlights the need to learn how to use the equipment, learn its
boundaries and learn defenses inside these boundaries...

-Chris

19

> > What processes and/or tools are large networks using to
> > identify and limit the impact of DDoS attacks?
>
> A great deal of thought is being expended on this question, I am certain,
> however, how many of these thought campaings have born significant fruit
yet,
> I do not know.

How about the following :

We develop a new community , being fully transitive (666 would be
appropriate ) and either build into router code or create a route map to
null route anything that contains this community. The effect of this being
the distribution of the force of the attack.

How about no. How about you do this inside YOUR network, perhaps get an
agreement with your peers to accept a /32 route from you and you can do it
with your peers also in times of need... There is something ominous about
'automagically propogating' a blackhole route.

1) I hack connected ISP X
2) I inject www.ebay.com /32 blackhole route
3) no more ebay

I use ebay as an example of course, I wouldn't want them harmed cause how
would I be able to buy all that nice routing gear at bargain basement
prices without them? :slight_smile:

This aside, how effective would be using a no export community with ones
peers (being non transitive, it would still distribute the force of the
attack).

For YOUR PEERS this is a fine idea, provided this fits with your peer's
edge policies and doesn't step on his already-used community.

20

Then you are pushing out /32's and peers would need to accept them. Then
someone will want to blackhole /30's, /29's, etc. Route bloat. Yum!

Yes.

Additionally you are creating a way to basically destroy the Internet as a
whole. One kiddie gets ahold of a router, say of a large backbone
provider, takes one of their aggregate blocks (/16? /10? /8?) and splits
it into /32 announcements.

Or, blackhole the /16 :slight_smile: more fun! (assuming no other smaller
announcements inside that /16 of course)

Anyways, some providers already allow you to set a community on a route,
and they will inturn "blackhole" it for you. I believe Teleglobe does
this for some customers and I know UUNet does this for all customers.

Hmm, Mr. 'dies' is almost correct... if you are a UUNET customer and you
would like to do this please call the customer service center and they
will help you to configure this 'service'.

Thanks though Mr. 'dies' :slight_smile: