D/DoS mitigation hardware/software needed.

Mailman List Mirror (Read Only)
1

Firewalls do a good job of protecting servers, when properly configured,
because they are designed exclusively for the task. Their CAM tables,
realtime ASICs and low latencies are very much unlike the CPU-driven,
interrupt-bound hardware and kernel-locking, multi-tasking software on a
typical web server. IME it is a rare firewall that doesn't fail long,
long after (that's after, not before) the hosts behind them would have
otherwise gone belly-up.

Then you need to get rid of that '90's antique web server and get
something modern. When you say "interrupt-bound hardware," all you
are doing is showing that you're not familiar with modern servers
and quality operating systems that are designed to mitigate things
like DDoS attacks.

"Stateful filtering" is to firewalls what "interrupt-based packet
processing" is to web servers. Both are recipes for disaster.

... JG

2

Then you need to get rid of that '90's antique web server and get
something modern. When you say "interrupt-bound hardware," all you
are doing is showing that you're not familiar with modern servers
and quality operating systems that are designed to mitigate things
like DDoS attacks.

"Modern" servers? IP is processed in the kernel on web servers,
regardless of OS. Have you configured a kernel lately? Noticed there
are ~3,000 lines in the Linux config file alone? _Lots_ of device
drivers in there, which are interrupt driven and have to be timeshared.
No servers I know do realtime processing (RT kernels don't) or process IP
in ASICs.

What configurations of Linux / BSD / Solaris / etc does web / email / ntp
/ sip / iptables / ipfw / ... and doesn't have issues with kernel
locking? Test it on your own servers by mounting a damaged DVD on the
root directory, and dd'ing it to /dev/null. Notice how the ATA/SATA/SCSI
driver impacts the latency of everything on the system. How would you
replicate that on a firmware and ASIC drive appliance?

Roger Marquis

3

Dobbins, Roland wrote:

My employer's products don't compete with firewalls, they *protect* them;
if anything, it's in my pecuniary interest to *encourage* firewall
deployments, so said firewalls will fall down and need protection, heh.

Nobody's disputing that Roland, or the fact that different specialized
appliances will protect against different perimeter attacks. The only
thing you've said that is being disputed is the the claim that a firewall
under a DDoS type of attack will fail before a server under the same type
of attack.

I question this claim for several reasons.

  * because it doesn't correlate with my 22 years of experience in systems
  administration and 14 years in netops (including Yahoo netsecops where I
  did use IXIAs to compile stats on FreeBSD and Linux packet filtering),

  * it doesn't correlate with experience in large networks with multiple
  geographically disperse data centers where we did use Arbor, Cisco and
  Juniper equipment,

  * it doesn't correlate with server and firewall hardware and software
  designs, and last but not least,

  * because you have shown no objective evidence to support the claim.

I did this kind of testing when I worked for the largest
manufacturer of firewalls in the world

Where then, can we find the results of your testing?

Here's the thing; you're simply mistaken, and you hurl insults
instead of listening to the multiple people on this
thread who have vastly more large-scale Internet experience than
you do and who concur with these prescriptions.

Nobody has "hurled insults" in this thread other than yourself Roland.
Shame on you for such disreputable tactics. To make the case you need
more than repeated dismissal of requests for evidence and repeated
unsupported claims of "vast experience" with failing servers and
firewalls. We just need some actual statistics.

Roger Marquis

4

From someone who mostly lerks but has been in network engineering operations biz for 17 years, the only OS that seems to always keel over under a ddos and need a firewall is windows. Linux in its current incarnation can handle a substantially larger attack before needing mitigation by firewall type device.

   So in the end I believe its the environment dictates the use of products unless you have aformentioned windows os which for me has always necessitated a firewall.

Manolo
Sent from my BlackBerry

5

> Then you need to get rid of that '90's antique web server and get
> something modern. When you say "interrupt-bound hardware," all you
> are doing is showing that you're not familiar with modern servers
> and quality operating systems that are designed to mitigate things
> like DDoS attacks.

"Modern" servers? IP is processed in the kernel on web servers,
regardless of OS. Have you configured a kernel lately? Noticed there
are ~3,000 lines in the Linux config file alone? _Lots_ of device
drivers in there, which are interrupt driven and have to be timeshared.

Yes, but all the fast network adapters are able to do a lot of stuff like
interrupt coalescing so you don't need to take an interrupt on every packet.

And "have you configured a kernel lately" is another red herring - yes, there
are indeed be 4,533 lines in the current Fedora .config. But that's because
that config turns on everything under the sun. I just checked, and my current
kernel config has only 960 '=y' lines, and another 220 '=m' lines - and a large
portion of those could easily be turned off. I have a minimal config file that
comes in under 730 non-comment lines.

No servers I know do realtime processing (RT kernels don't) or process IP
in ASICs.

That's because in general, processing the IP in an ASIC simply Does Not Work
as well as you might hope. Alan Cox did a nice discussion of some of the
issues here:

http://lkml.indiana.edu/hypermail/linux/kernel/0307.1/2116.html

One should read his last paragraph carefully, and note that what he
wrote back in 2003 is still true today:

http://www.internet2.edu/lsr/history.html

What configurations of Linux / BSD / Solaris / etc does web / email / ntp
/ sip / iptables / ipfw / ... and doesn't have issues with kernel
locking?

So let me get this straight - you perceive a problem with locking inside
the kernel, where if you're lucky the lock is in an already-hot cache line
and your biggest worry is cache line ping-ponging, and if you're unlucky
you actually have to go out the southbridge and hit main memory, at main
memory access speeds.

And to fix this, you're going to move one of the things contending for the
lock off the CPU, so now every time the lock is contended, it has to go out
through the PCI bridge to an external card?

How the heck is that supposed to help? You're suggesting the same "go talk
to another card" solution that the router vendors learned is the *last* thing
you want to do - calling out to the supervisor card rather than handling it
onboard the line card is guaranteed performance death.

Test it on your own servers by mounting a damaged DVD on the
root directory, and dd'ing it to /dev/null. Notice how the ATA/SATA/SCSI
driver impacts the latency of everything on the system. How would you
replicate that on a firmware and ASIC drive appliance?

There's two little things you went astray on here:

1) I've in fact had to do this while doing data recovery. It doesn't do
squat to the latency of anything that doesn't have to go through the same
controller as the DVD. Everything else works just fine. Heck, it isn't even
enough to cause audio playback skips (and those are noticeable even at
the millisecond level).

2) Your latency hit is because the controller is *busy* while trying to
re-read and error-correct a bad block.

So yeah - trying to do I/O through a controller that's taking a several-second
time-out dealing with bad media will cause a latency hit *for that I/O*. What's
your point?

6

The only thing you've said that is being disputed is the the claim that a firewall
under a DDoS type of attack will fail before a server under the same type
of attack.

It's so obvious that well-crafted programmatically-generated attack traffic, if nothing else, will crowd out the good traffic that I'm just dumbfounded anyone thinks 'proof' of this is needed. Same thing for the fact that horizontally-scaled Web farm (with or without reverse caching proxies) will of necessity handle a great deal more TCP state than the biggest, firewall made to date.

* because it doesn't correlate with my 22 years of experience in systems
administration and 14 years in netops (including Yahoo netsecops where I
did use IXIAs to compile stats on FreeBSD and Linux packet filtering),

It doesn't correlate with my 25 years in the industry, a good portion of the last 15 years spent handling DDoS after DDoS after DDoS, during which the biggest, baddest firewalls choked and died over and over again, through multiple generations of said firewalls.

Again, I was able to take down a hardware-based (for whatever value of 'hardware-based' is possible) firewall rated at 2gb/sec with 80kpps of traffic.

* it doesn't correlate with experience in large networks with multiple geographically disperse data centers where we did use Arbor, Cisco and Juniper equipment,

It correlates with my experience in large networks with geographically-dispersed IDCs with heterogeneous gear.

* it doesn't correlate with server and firewall hardware and software designs, and last but not least,

Which is a non-sequitur.

* because you have shown no objective evidence to support the claim.

I've my own broad subjective experience, and that of several other people who've commented on this thread have similar experiences. Since you haven't yet acquired this subjective experience, you can cause it to happen in a controlled test environment, should you so choose.

Where then, can we find the results of your testing?

The testing I did when I worked for the vendor in question is proprietary, as you can well surmise. You're free to do your own testing and confirm these assertions for yourself.

Nobody has "hurled insults" in this thread other than yourself Roland.

You accused me of acting in my own pecuniary interest, of trying to 'sell' things, *for no reason at all*.

We just need some actual statistics.

If you actually care about the truth of the matter, you're free to generate your own. If you read the RoK/USA DDoS preso to which I linked, you see the attack throughput and bandwidth metrics/host, and you also see where I noted multiple 'Web Application Firewalls', load-balancers, and so-called 'IPS' falling over as a result of those attacks. That gives you a range right there, along with some attack traffic characteristics, including average packet size.

It makes no sense to put a stateful inspection device in front of servers, where *every single packet* is unsolicited, and therefore no state tracking is even possible in the first place. Stateless filters in hardware capable of mpps do a much better job, without the risk of falling over due to state-table exhaustion.

Folks who've been unlucky enough to be subjected to significant DDoS attacks have run into this issue again and again and again. Perhaps you've simply been lucky; but one can't count on one's luck holding forever.