D/DoS mitigation hardware/software needed.

Looking for D/DoS mitigation solutions. I've seen Arbor Networks mentioned
several times but they haven't been responsive to literature requests (hint,
if anybody from Arbor is looking...). Our current upstream is 3x GigE from
3 different providers, each landing on their own BGP endpoint feeding a
route-reflector core.

I see two possible solutions:
- Netflow/sFlow/***Flow feeding a BGP RTBH
- Inline device

Netflow can lag a bit in detection. I'd be concerned that inline devices
add an additional point of failure. I'm worried about both failing-open
(e.g. network outage) and false-positives.

My current system is a home-grown NetFlow parser that spits out syslog to
our NOC to investigate potential attacks and manually enter them into our
RTBH.

Any suggestions other than Arbor? Any other mechanisms being used? My idea
is to quash the immediate problem and work additional mitigation with
upstreams if needed.

I could probably add some automation to my NetFlow/RTBH setup, but I still
need to worry about false-positives. I'd rather somebody else do the hard
work of finding the various edge-cases.

Thanks,
Rick

We have substantial direct experience with both RioRey and IntruGuard.
RR is more plug and play while IG is more robust but both are great.
Use a robust firewall such as a Netscreen in front of your mitigation
tool.

Best regards, Jeff

Rick,

If you pass me your contact info I can forward it to our Arbor Sales guy who can get in touch with you. I been pretty impressed by Arbor so far.

Thanks,
Raj Singh

Several responses already, and Arbor has poked their head up.

I'm going to start there and keep the other suggestions at-hand.

Thanks,

Ask them if they'd come down to $10 - 20k for a full featured model
and they might make two sales, although I doubt it unfortunately.

Best regards, Jeff

Absolutely not - the firewall will fall over due to state-table exhaustion before the mitigation system will. Firewalls (which have no place in front of servers in the first place), load-balancers, and any other stateful devices should be southbound of the mitigation system.

Kinda funny you state that Roland. I know of at least two very large
carriers that uses Netscreens (and soon SRX's) for their DoS/DDoS
mitigation.

State table exhaustion on the netscreen platform is something that is very
easy to protect against with some protections and hasn't been a problem for
years. If you can fill up a session table on a higher end SRX then I would
be very very impressed. I would argue that firewalls place is in fact
directly infront of servers and load balancers to protect them.

What Roland said, I've seen people do this, no rules in place, still
was able to kill the box (firewall) with a single CPU server.

-jim

If you want to recreate D/DoS from captures (for testing purposes) you
might want to check out:

http://www.pcapr.net/dos

This lets you validate how your mitigation solutions are holding up.

K.

The very idea of placing a stateful firewall in front of a Web/DNS/email/etc. server, in which *every single incoming packet is unsolicited, and therefore, leaves no state to be inspected in the first place*, is absurd.

There is simply no valid argument for doing so. Hardening the OS/apps/services, combined with stateless ACLs in hardware which can handle mpps, is the way to enforce policy.

In fact, the idea is such a poor one that one of the major firewall vendors came out with a 'stateful inspection bypass' feature - the idea being that, you buy their 10gb/sec, $100K-plus stateful firewall, stick it in front of servers, and then . . . disable the stateful inspection, heh.

;>

None of the large, well-known Web properties on the Internet today - at least, the ones which stay up and running, heh - have stateful firewalls in front of them. Including prominent vendors of said stateful firewall solutions.

But as you said, they're willing to sell them to you. Then claim
that the traffic you're receiving is out of profile. :slight_smile:

(I'm not jaded about this, oh no..)

Adrian

We have such a configuration in progress, it works great without any of the
issues you're proposing.

Jeff

So .. this is interesting.

The firewall would have to frontend your mail / web / whatever
application .. and if something goes beyond the firewall's rated
capacity (100k ++ - maybe nearly 150..175k connections per second for
a high end firewall), the firewall falls over.

And even before that, there's the risk of whatever application you're
protecting getting pounded flat if your firewall passes even a small
percentage of this traffic.

Do you -

1. Have (say) two firewalls in HA config?

2. Back your firewall with routing based measures, S/RTBH, blackhole
communities your upstream offers, etc [the standard nspsec bootcamp
stuff]

3. Simply back the firewall with a netflow based device?

4. Estimate that the risk of a DDoS that exceeds your firewall's rated
capacity is extremely low? [and yes, 150k ++ connections per second
ddos is going to be massive, and relatively rare for most people]

--srs

Then you aren't testing it to destruction, heh.

;>

If it's a stateful firewall, and state-tracking is turned on, it's quite possible to craft sufficient pathological traffic which conforms to the firewall policies and yet which leads to state-table inspection.

And the stateful firewall serves no purpose in front of servers, in which *every incoming packet* is unsolicited. Far more sensible to enforce policy in stateless ACLs in ASIC-based router/switch hardware.

That should read 'state-table exhaustion', apologies for the typo.

Two more options. And for Netflow device - read that to mean Arbor or
its competitors.

5 Ditch the stateful firewall and exclusively use a netflow device

6. Outsource to a hosted DDoS mitigation service (Prolexic etc)

NetFlow analysis is very useful for network visibility, and detection/classification/traceback. There are both open-source and commercial NetFlow collection and analysis systems available (full disclosure: I work for a vendor of both NetFlow analysis and DDoS mitigation solutions); however, they don't provide mitigation, which is where S/RTBH, flow-spec, and/or IDMS come into play.

PCI DSS iatrogenically *requires* that a 'Web application firewall' be placed in front of Web servers which process credit card information (PCI DSS completely ignores availability, and contains a number of recommendations which are actually harmful from an opsec standpoint). Running mod_security or its equivalent on the front-end Web servers themselves fulfills this requirement without putting a stateful DDoS chokepoint in front of the servers.

It's also a really good idea to front Web servers with a tier of caching-only transparent reverse proxies; Squid is a good choice for this, as well as various commercial offerings. WCCPv2 clustering (supported by Squid and several commercial caching/proxying solutions) allows this tier to be scaled horizontally in order to meet capacity demands.

What Roland said, I've seen people do this, no rules in place, still
was able to kill the box (firewall) with a single CPU server.

not to pile on, but... +1 to roland here as well. I've seen more than
enough folks put in a 'firewall' in front of their 'server' (say a
mail server) and then watch that die long before the rest of the
system did.

Now, if you have equipment capable today of doing a few million
session creates/second and you feel comfortable that you can keep
track of how attacks grow vs your capacity stays the same and move
ahead of the curve well enough, then... by all means do as you want :slight_smile:

There's a cost analysis which Roland sidestepped here as well,
state-tracking at the rates required is expensive, as compared to
relatively simple acls in hardware with no state on the upstream
router.

Spend where it matters, and make sure you understand where the failure
points are that you place into your network.

-chris

1. We have multiple nodes conducting DDoS scrubbing, one failing would not
be catastrophic.

2. Indeed.

3. Sort of, such devices are downstream for extremely valid reasons I won't
get into now.

4. Indeed, were equipped to handle substantially higher than 150kpps.

I'm sure Arbor is really neat but I disagree that any DDoS appliance is a
standalone solution. I don't expect an employee of the vendor themselves to
attest to this though.

Best regards, Jeff

Best regards, Jeff

A lot of this has to do with scaling the environment. I've had plenty of
asa's and even netscreens fall over from state-table and session
limitations. I've also seen a hosts fill up the connection table prior to a
firewall being affected. I'm not familiar with the specs and anyone can
chime in, but the newer variety of SRX's, I believe implement more in
hardware as line-rate routers do. A layered approach is useful as well. If
the source can be identified via netflow and null routed before it gets to
the firewall and content layer, then all the better. This is much more
tricky with DDOS so having robust firewall that can eat traffic is helpful.

My 3 cents

-b