Effective ways to deal with DDoS attacks?

Mailman List Mirror (Read Only)
1

In the referenced message, Iljitsch van Beijnum said:

> for single-homed customers, simple uRPF
> for multihomed customers, acl exceptions based upon their registered
> IRR-policy, since the customer should already be registering in the IRR
> you have a list of all networks reachable via the customer, regardless of
> the actual real-time announcements or policy applications (prepending,
> localpref tweaks, etc)

This doesn't make any sense. If you use uRPF on a customer interface, it
will check the packets coming in from the customer to see if they match
the prefixes you route to that customer. So as long as what you route to
them and what they use as source addresses are identical, you don't have a
problem.

think MEDs and localpref twiddles., identical prefixes do not necessarily
relate to best paths.

For multihomed customers, these sets of prefixes should be identical, just
like with single homed customers. The only time when those sets of
prefixes is NOT the same is for a backup connection. But if a connection
is a pure backup for incoming traffic, it's reasonable to assume it's a
pure backup for outgoing traffic as well, so as long as the backup is
dormant, you don't see any traffic so no uRPF problems.

Not always the case, customer behaviour can not be accurately modeled.

Using an exception access list is pretty silly: if you're going through
the trouble of listing all a cutomer's prefixes in a list, you might as
well just apply this acl to the interface rather than uRPF with the acl as
exceptions.

the acl will only be used on packets failing the rpf check. interface acls
get applied to all traffic.

There is another way to handle backups: you can also set the weight to a
higher value for customer routes. This way, the router connecting to the
customer will always use the routes announced by the customer, even if the
rest of the network deems them inferior to another route to this customer
because of a lower local pref, a higher MED or a longer AS path.

This mechanism is also useful for peering: because of the higher weight,
the border router will always prefer the exchange (or private peering
link) for all traffic to the customer, so uRPF works. The rest of the
network can still decide to send the traffic to another exchange point.

I'm quite leery of having islands of widely divergent policy, and I wouldn't
think it would help if you have multiple non-equal-cost-paths on the
same router with which to accept traffic on.

> for non-clued peers, accept based upon any available forwarding path
> [hopefully, by the 100th time you beat on the peer about spoofed crud
> coming from them, they'll get religion and register, since you'll have
> less overall spoofing to track down, you can devote it to slapping
> them around more]

If people stop peering with those network polluters they'll clean up their
act soon enough.

This is unlikely to occur, unfortunately, so merely making it as annoying
as possible for polluters and less annoying as possible for non-polluters,
is probably the way to go.

> you should also in/egress filter known bogons at all borders, like
> src/dst in rfc1918
> src/dst in class e

Why? That doesn't buy you much except job security because someone
spending so much time on those impressive filters can't be missed of
course.

Because it is polite to not send crap to your neighbors, and advantageous
to not have crap coming into your network.

> src in class d (not dest, however)

Some multicast apps set the source to the group address as well.

How... odd...

2

I was hoping someone else might mention this, BUT what about the case of
customers providing transit for outbound but not inbound traffic for their
customers? We have many, many cases of customers that are 'default
routing' for their customers that get inbound traffic down alternate
customers or peers or wherever... uRPF seems like a not so good solution
for these instances :frowning: especially since some of these are our worst
abusers :frowning:

-Chris

3

> > like with single homed customers. The only time when those sets of
> > prefixes is NOT the same is for a backup connection. But if a connection

> Not always the case, customer behaviour can not be accurately modeled.

I was hoping someone else might mention this, BUT what about the case of
customers providing transit for outbound but not inbound traffic for their
customers? We have many, many cases of customers that are 'default
routing' for their customers that get inbound traffic down alternate
customers or peers or wherever...

Is there a compelling reason you should allow this? If yes, you can't use
uRPF and you have to install an acl to do sanity checking on the
customer's source addresses. If no, they'll have to announce those routes
to you. If they set the no export community they still won't get any
inbound traffic to speak of.

uRPF seems like a not so good solution
for these instances :frowning: especially since some of these are our worst
abusers :frowning:

Well if these are your worst abusers, it seems to me uRPF is exactly what
those customers need. :wink:

4

two methods:
  [1] if your customer has their own AS, have them route the (valid) networks
      to you with the no-export bgp attribute set.

  [2] if they're not BGP connected, then surely you have some idea of what subnet(s)
      they're sending traffic out from? (i hope so).
      if so, then you'd have static-routes for those subnets pointing at their interface.
      you don't necessarily have to include those static-routes in announcements to
     your peers.

both of [1] & [2] may mean that more traffic may 'prefer' the link from you to the customer. (probably doubly so given you're uunet and the amount of transit that goes thru you). in that case, perhaps using the no-advertise community so that the route stays 'local' to a router (or local to a city) will prove sufficient.

cheers,

lincoln.

5

This dilemma has far reaching repercussions:

If _you_ allow this and forego the unicast RPF check for these customers,
this means your peers can't do uRPF towards you without breaking
connectivity for these customers.

In a perfect world, there would be no need to do uRPF on peering
interfaces, because peers make sure they don't send packets with spoofed
source addresses. Unfortunately, in the real world many networks STILL
don't bother and thereby allow hard to trace and filter DDoS attacks to be
launched from inside their networks.

So what is the collective wisdom on the NANOG list? Is uRPF on peering
interfaces a viable option and if it breaks esoteric customer
configurations, too bad; or is it something that should be discouraged
because it breaks legitimate customer needs?

If you feel strongly one way or the other, but don't want to join the
discussion, please reply with a "yes to peering uRPF" or "no to peering
uRPF" in private email, and I'll summerize to the list.

Iljitsch van Beijnum

6

This is not an all or nothing proposition. Some people argue that because
they can't filter everyone perfectly, they shouldn't filter anyone at all.
This is nonsense, if you can take the *30 SECONDS* of your time to take
care of the 95% of your customers who aren't doing anything complex and
who aren't going to have problems, both you and the entire internet are
better off than if you spent the next 3 years trying to figure out how to
do it for everyone.

I personally would NOT try to filter peers this way, there are just too
many legitimate chances for asymmetric routing at that level. Consider the
simplest one of all: You buy transit from provider A and peer with
provider B, destination C buys transit from provider B and peers with
provider A. Obviously by localpref dest C is going to be sending traffic
to you via provider A and you are going to be sending to them via provider
B. Asymmetric routing is not evil, it is simply how the internet works.

What we should be doing is encouraging the people with the 95% easy
customers (or maybe even 100% depending on thier business) to start
filtering. I've had a transit provider *coughinternapcough* refuse to
apply RPF to my single homed port when I ASKED for it (how many of your
customers will do this? :P), because they thought it would place too much
load on their routers. And then we have Foundry, who in their infinite
wisdom decided that an access-list should be sufficient for statically
routed customers, so the RPF they finally came up with is only accessable
as a BGP neighbor statement. Lots of people have their boxes for customer
aggregation (because it'll be a cold day in hell before they're doing L3
in a core :P), and they decided to make it apply only to BPF neighbors,
the one place you probably DON'T need it. These people need to be beaten
with the clue stick on a regular basis until something finally sinks in,
because obviously the first round didn't take.

This is a situation similar to smurf, a mass education of people who don't
read nanog (though I doubt most of the people here do) is required. The
problem is, you can't make it a default configuration because of
asymmetric routing (so someone actually has to think, not just type), and
you can't run probes and tell if someone has RPF setup on a remote network
(the only people with the resources to tell are the people with DDoS nets,
and I don't think they're going to help) so you can publicly shame them.
Until someone solves one of the above two, I don't think much work is
going to be done. Making RPF where reasonable a requirement for peering is
a place to start, but I don't see that as being enforcable.

7

Be mindful that uRPF Strict Mode was created to help scale BCP 38 filtering.
If you have 1000 lease line customers and can use uRFP Strict Mode on 80% of
those customers, that is 80% fewer BCP38 ACLs that you need to manage.

For the other 20% you have uRFP + BGP tweaks or plain old ACLs. But as Chris
inferred, that 20% where you cannot use simple uRPF is also the 20% most
difficult customers.

8

As one of the key people pushing uRPF .... uRPF 'Strict Mode' was never ever
designed to put on the ISP peering links. It was created to help ISPs scale
BCP 38 filtering on the ISP-Customer edge. Knock out the easy 80% of the
customers who are simple single homed customers - leaving the other 20% for
special BCP38 uRPF/BGP or ACL configs.

uRPF 'Loose Check' was designed for any part of the edge - ISP to ISP (could
be customer or peer); the ISP to Customer edge; or the ISP to Multihomed
edge. The objective was to provide a quick way to trigger a network wide
source address based black hole. It also provides an effective way to filter
source addresses with martian and bogons (i.e. addresses not in the FIB). So
consider uRPF Loose Check as a source address based "noise reduction" filter
and a network wide DDOS counter measure.

So people need to get the two straight. The are completely different:

uRPF Strict Mode for BCP 38 (not for ISP-ISP Peering links)

  Cisco:
    ip verify unicast source reachable-via rx

  Juniper (as of 5.3):
    unicast-reverse-path active-paths

uRPF Loose Check Mode (suitable for ISP-ISP Peering Links)

  Cisco:
    ip verify unicast source reachable-via any

  Juniper (as of 5.3):
    unicast-reverse-path feasible-paths

So you need to frame you response as to the appropriate uRPF mode.

One key reminder - uRPF is just another security tool for an ISP's security
tool kit. There is no such thing as a perfect security tool. A craftsman is
known not for his/her tools, but how well they use the tools they got to
perform their art.

Barry

9

First of all, this list is great! I feel this is one of the best lists I have ever seen... I thank you all for the great technical depth and openness.

We have a product that offers DDoS mitigation with ACLs but I am interested in learning more about uRPF Loose Check Mode to see if we can integrate this idea into our product.

In particular, I am interested in the ability of eliminating specific routes from the FIB under uRPF Loose Check Mode to effectively filter specific source addresses that are flooding.

As I understand the concept, eliminating an address from the FIB such as
x.y.0.0/24 would have the equivalent effect of installing a network-wide
ACL rule:

deny ip x.y.0.0/24 any

Is this right?

If this is right, is there any way to use uRPF Loose Check Mode to have
an equivalent network-wide ACL rule of the form:

deny <proto> x.y.0.0/24 <destination>

where: <proto> is NOT ip and <destination> is NOT any?

The reason why I ask is that we would like to keep control of these
two important aspects of the traffic to avoid filtering out too much
and therefore possibly affecting legitimate traffic. Think of the case where
a flood targets one of multiple downstream customers and the spoofed
addresses correspond to a popular address range (such as Yahoo). Doing
a "deny ip x.y.0.0/24 any" would effectively shut down Yahoo's traffic
for all downstream customers thus amplifying the attacker's effect.

Livio.

10

In particular, I am interested in the ability of eliminating specific
routes from the FIB under uRPF Loose Check Mode to effectively filter
specific source addresses that are flooding.

As I understand the concept, eliminating an address from the FIB such as
x.y.0.0/24 would have the equivalent effect of installing a network-wide
ACL rule:

deny ip x.y.0.0/24 any

Not quite.

First, lets be specific by what you mean by "remove from the FIB", as
there are a number of different methods you could use. You could simply
block it from the RIB when generating the FIB, you could go back after FIB
generation and try to make it unresolved, or you could change the nexthop
to "discard". If you're trying to replicate traditional firewall behavior
(filter no matter what) you would have to do it post FIB generation, but if
you are trying to replicate normal routing behavior (ex: a null route) you
would have to do it during FIB generation, so that you can potentially
have more specific routes which escape the "filter".

Secondly, when you remove something from your FIB, you also block
destination routing as well as source.

The reason why I ask is that we would like to keep control of these
two important aspects of the traffic to avoid filtering out too much
and therefore possibly affecting legitimate traffic. Think of the case where
a flood targets one of multiple downstream customers and the spoofed
addresses correspond to a popular address range (such as Yahoo). Doing
a "deny ip x.y.0.0/24 any" would effectively shut down Yahoo's traffic
for all downstream customers thus amplifying the attacker's effect.

It sounds like what you are looking for has nothing to do with the RPF or
the FIB, but rather simply manual source address filtering.

However, if the reason you're interested in RPF is because you want to
source match filtering more efficiently, you may be interested in the data
structure. Rather then walking a straight access-list rule set doing a
comparison for every rule, you can make a "Filtering" Information Base
mtrie for source address rules. This is the entire point of standard
access-lists, and more recently compiled access-lists.

11

Richard A Steenbergen wrote:

In particular, I am interested in the ability of eliminating specific routes from the FIB under uRPF Loose Check Mode to effectively filter specific source addresses that are flooding.

As I understand the concept, eliminating an address from the FIB such as
x.y.0.0/24 would have the equivalent effect of installing a network-wide
ACL rule:

deny ip x.y.0.0/24 any

Not quite.

First, lets be specific by what you mean by "remove from the FIB", as
there are a number of different methods you could use. You could simply
block it from the RIB when generating the FIB, you could go back after FIB generation and try to make it unresolved, or you could change the nexthop to "discard". If you're trying to replicate traditional firewall behavior (filter no matter what) you would have to do it post FIB generation, but if
you are trying to replicate normal routing behavior (ex: a null route) you would have to do it during FIB generation, so that you can potentially have more specific routes which escape the "filter".

escaping the "filter" with more specific routes would be absolutely necessary.

Secondly, when you remove something from your FIB, you also block destination routing as well as source.

Good point; so in ACL equivalent language you are saying that taking out a FIB entry in uRPF Loose Check Mode is equivalent to a network-wide insertion of:

deny ip x.y.0.0/24 any (from the uRPF Loose Check Mode)
deny ip any x.y.0.0/24 (from the absence of a route)

(modulo the more specific routes to escape the "filter" which could be expressed as prepended permits in the ACL equivalent world)

The reason why I ask is that we would like to keep control of these
two important aspects of the traffic to avoid filtering out too much
and therefore possibly affecting legitimate traffic. Think of the case where
a flood targets one of multiple downstream customers and the spoofed
addresses correspond to a popular address range (such as Yahoo). Doing
a "deny ip x.y.0.0/24 any" would effectively shut down Yahoo's traffic
for all downstream customers thus amplifying the attacker's effect.

It sounds like what you are looking for has nothing to do with the RPF or
the FIB, but rather simply manual source address filtering.

Well, I am investigating if it is possible today to use uRPF Loose Check Mode to achieve network-wide source/destination address filtering functionality (it seems not from what you write). I immagine that it would be useful to use route advertisements to enforce network-wide access control policies. These policies, however, to be generally interesting for DDoS would have to be at least as expressive as "<deny|permit> <proto> <source> <destination>" (hence my questions).

Livio.

12

Network wide? Ok that was my first guess actually, that you were trying to
create an "acl protocol".

Well, like I said, there isn't really such a thing as "removing" something
from a FIB. A FIB is generated from a RIB, so something special would
have to happen to make it "unresolved" (as if there was no route at all).
You can make a route not installed in a RIB, but not remove others of the
same or longer length from the FIB. :slight_smile:

Now on the other hand, it's interesting to consider what would (or could)
happen if you had a special route for example to null0. Obviously with
strict mode, you're not going to match because the nexthop is different.
In loose mode on the other hand, there is some room to consider a route to
null0 as not a valid route at all, and make it not match. I have no idea
if that is the current behavior of anyone's implementation, but it
potentially could be. Of course then you'd need protocol extensions to
carry around actual null0 routes instead of a nexthop just reserved for
null routes... So this entire conversation is pretty pointless. :slight_smile:

What we all really need is a protocol which can distribute filtering
information network-wide. Go make one. :slight_smile:

13

No, what we need is a protocol that can do *secured* distribution of
filtering info net-wide. Otherwise, some bozo is going to accidentally
inject a flter for 127/8, causing as much fun as the announcement of same
a few years ago. And I'm *sure* there's at least a few people on this
list that would be *very* tempted to inject filters for RFC1918 space
for the benefit of those providers that don't egress filter it currently :wink:

14

Nononono, by network-wide I ment *MY* network not the Internet. :slight_smile: Though
I really wouldn't mind seeing a well known community for "nexthop null0".
How can people sit around pontificating on useless features for useless
protocols all day long, and yet not do this?

BTW, I don't know what announcing 127/8 would break since that should
never leave or enter any systems, and I still take issue with the need to
filter 1918 packets.