Linux Router distro's with dual stack capability

Date: Thu, 11 Feb 2010 18:20:13 -0500
From: Chuck Anderson <cra@WPI.EDU>

> > Lots of people roll FreeBSD with Quagga/pf/ipfw for dual stack. See
> > the freebsd-isp list.
> FreeBSD's network stack chokes up in DDoS attacks due to interrupt
> flooding. We used to use FreeBSD for firewalling and basic routing, but
> when noticing that we had horizontal scalability (e.g. a Celeron 667mhz
> performed nearly as well as a dual dual-core Xeon system when DDoS
> attacks happened), we switched to Vyatta, and generally have not looked
> back.

Have you tried using FreeBSD's polling mode instead of interrupt mode?

No experience with it myself, but it sounds cool:

Device Polling support for FreeBSD

Polling is excellent for low speed lines, but for Gig and faster, most
newer interfaces support interrupt coalescing. This easily resolves the
issue in hardware as interrupts are only issued when needed but limited
to a reasonable rate, Polling does not use interrupts, but consumes
system resources regardless of traffic.

FreeBSD has supported polling for a long time (V6?) and interrupt
coalescing since some release of V7. (Latest release is V8.)

I'm pretty sure it's been around for a lot longer than that. I seem to
recall playing with both back in 4.x. Of course interrupt coalescing is
mostly a function of the NIC (though some driver involvement is required
to take advantage of it), so the quality of the implementations have
varied significantly over the years. The first generation GE NICs which
offered it didn't do a particularly good job with it though, so for
example it was still possible to cripple a box with high interrupt
rates while the same box would be perfectly fine with polling.

That said, I think your use case for polling is backwards. As you say,
"normally" the NIC fires off an interrupt every time a packet is
received, and the kernel stops what it is doing to process the new
packet. On a low speed (or at least low traffic) interface this isn't a
problem, but as the packet/sec rate increases the amount of time wasted
as interrupt processing "overhead" becomes significant. For example,
even a GE interface is capable of doing 1.488 million packets/sec.

By switching to a polling based model, you switch off the interrupt
generation completely and simply check the NIC for new packets a set
rate (for example, 1000 times/sec). This gives you a predictable and
consistent CPU use, so even if you had 1.488M/s interrupts coming in you
would still only be checking 1000 times/sec. If you did less than
1000pps it would be a net increase in CPU, but if you do more (or ever
risk doing more, such as during a DoS attack) it could be a net benefit.
This is makes the most sense for people doing a lot of traffic

Of course the downside is higher latency, since you're delaying the
processing of the packet by some amount of time after it comes in. In
the 1000 times/sec example above, you could be delaying processing of
your packet by up to 1ms. For most applications this isn't enough to
cause any harm, but it's something to keep in mind. Interrupt coalescing
works around the problem of large interrupt rates by simply having the
NIC limit the number of interrupts it generates under load, giving you
the benefits of low-latency processing and low-interrupt rate under high
load. I haven't played with this stuff in many many years, so I'm sure
modern interrupt coalescing is much better than it used to be, and the
extra work of configuring polling and dealing with the potential
latency/jitter implications isn't worth the benefits for most people. :slight_smile:

Also IIRC you can tune the hash cache / tree algorithm - ie if your
traffic is mostly a few addresses then the default prefix search is
fine (with the caching) but for more sparse traffic as you'd see at an
edge, disabling the cache and using the other algo proved a lot
faster. There's a paper on this I saw a few years ago, will forward if
I find it.

-Jack Carrozzo

FreeBSD has supported polling for a long time (V6?) and interrupt
coalescing since some release of V7. (Latest release is V8.)

exactly. and they kick ass