Let's talk about Distance Sniffing/Remote Visibility

carl.p.hirsch · March 28, 2002, 3:23pm

Yeah, the device I've got in my head is a 1U server with 4 (or more?)
interfaces... not so much to simultaneously pull 400Mbps of bandwidth for
analysis but rather to just have a interface going to each switch I might
want to monitor and then span traffic to the Ethereal box. Given that I'm
trying to attain remote visibility, it might be nice not to need remote
hands to be swapping patch cords back and forth.

I'm imagining that even with a relatively speedy box, if you were trying to
do analysis from multiple interfaces you'd at least choke the disk I/O.
There's always stringent filters, I guess.

thanks for the input,
-carl

                    "E.B. Dreger"
                    <eddy+public+spam@noc.ever To: CARL.P.HIRSCH@sargentlundy.com
                    quick.net> cc: nanog@merit.edu
                    Sent by: Subject: Re: Let's talk about Distance Sniffing/Remote
                    owner-nanog@merit.edu Visibility

                    03/28/02 09:02 AM

"C" is close. Not sure what you mean by "a ton of interfaces".
Most (all?) good managed switches have a "monitor port" or
"mirror port" where they can blind copy traffic from other ports
to the one that's set aside for snooping.

Four-port ethernet cards are readily available. How many
switches do you wish to monitor simultaneously? Even with only
four ports (more in one box is certainly possible), you can have
a fair amount of traffic to digest.

Richard_A_Steenbegen · March 28, 2002, 5:19pm

Disk I/O on a sniffer box? Sounds like you've been sniffing something
other than packets my friend.

You can build your own box like that easily enough. If you're going for
FastE sniffing I highly recommend the Adaptec Quartet 4-port cards. If
you're going for GigE sniffing, I STILL highly recommend anything Alteon
Tigon 2 based (NetGear GA620's were the cheapest if you can still find
them, not the 621/622).

I've had great luck making the Tigon 2's into dedicated sniffers. You
don't even have to do anything fancy with the card firmware, there is a
native command for receiving only part of the frame. Check out the
programming manuals at http://people.freebsd.org/~wpaul/Alteon/, and I
recommend you use FreeBSD for this of course. Just add in a PARTIAL_RX_CNT
command, and the card will only DMA part of the packet (say 64 bytes for
full headers) across the PCI bus. Combined with interrupt coalescing (or
luigi's device polling and tuning the card to allocate all memory to RX
and remove the TX functionality completely), you can sniff quite a few
"gigabits" of traffic on a single cheap PC server. You can dump it through
the BPF mechanism and still maintain support for all your favorite sniffer
programs. Or if you're comfortable writing kernel code, I recommend you
make a character device for sniffer device control, and use it to pass
page-aligned malloc'd memory pointers from userland into the nic driver,
which you then pass to the card as the RX ring buffers. This will let you
DMA your packets directly into userland. If not, at least unhook
ether_input().

Or you can buy these things commercially. My favorite was from a company
called Tekelec, who sold a VERY expensive box which turned out to be a
pentium 200ish box running solaris x86 and completely useless sniffing
software, with a bunch of ISA ethernet cards hooked up by proprietary (and
VERY expensive) cables, all in a box made out of what I swear was some
kind of lead/neutron star material alloy. Of course that was a couple
years ago, maybe they've upgraded to the current market's $50 processor.

E.B_Dreger · March 28, 2002, 5:59pm

Date: Thu, 28 Mar 2002 12:19:55 -0500
From: Richard A Steenbergen <ras@e-gerbil.net>

(snipping throughout)

Disk I/O on a sniffer box? Sounds like you've been sniffing
something other than packets my friend.

I like to log interesting packets; I agree with Carl.

You can build your own box like that easily enough. If you're going for
FastE sniffing I highly recommend the Adaptec Quartet 4-port cards. If

D-Link DFE-570TX are _very_ cheap if you're happy with 32-bit /
33 MHz PCI.

[ snip FreeBSD + Alteon ]

I did not know about the partial-packet DMA transfers. Mmmmm....

Or if you're comfortable writing kernel code, I recommend you
make a character device for sniffer device control, and use it
to pass page-aligned malloc'd memory pointers from userland
into the nic driver, which you then pass to the card as the RX
ring buffers. This will let you DMA your packets directly into
userland. If not, at least unhook ether_input().

Never done this. About how much "capacity" does the zero-copy
approach add?

Richard_A_Steenbegen · March 28, 2002, 6:14pm

I like to log interesting packets; I agree with Carl.

Logging interesting packets is easy enough, its logging ALL packets that
would be a problem. At any rate, you'd run out of harddrive space pretty
quick if you were pushing max performance at any length of time. I can
write a linerate FastE's worth of data to a $100 IDE disk on a $100
processor easily enough, so as long as you're buffering it intelligently
it shouldn't be an issue.

> Or if you're comfortable writing kernel code, I recommend you
> make a character device for sniffer device control, and use it
> to pass page-aligned malloc'd memory pointers from userland
> into the nic driver, which you then pass to the card as the RX
> ring buffers. This will let you DMA your packets directly into
> userland. If not, at least unhook ether_input().

Never done this. About how much "capacity" does the zero-copy
approach add?

Eliminating the bulk "data" being DMA's across the PCI bus is what adds
most of your "capacity". Doing zero copy just lets you spend all your CPU
time doing actual analysis instead of copying stuff around unnecessarily.
I never did get the opportunity to benchmark it at 1.4million packets/sec,
(I spent more time trying to get the 20ft of fiber to reach the lab at the
previous employeer than I did writing the code to do this in the first
place) but I don't see any reason it shouldn't work, with proper interrupt
coalescing of course.

E.B_Dreger · March 28, 2002, 6:38pm

Date: Thu, 28 Mar 2002 13:14:17 -0500
From: Richard A Steenbergen <ras@e-gerbil.net>

Logging interesting packets is easy enough, its logging ALL packets that
would be a problem. At any rate, you'd run out of harddrive space pretty
quick if you were pushing max performance at any length of time. I can
write a linerate FastE's worth of data to a $100 IDE disk on a $100
processor easily enough, so as long as you're buffering it intelligently
it shouldn't be an issue.

This is true. Logging interesting packets, efficient buffering,
and selective parsing make the big difference.

I guess it also depends on log format: Raw packet( fragment)s
work great. Human-readable -- a la, say, Linux kernel verbose
IP logs -- make things get ugly in a hurry.

With fixed-size packet captures, it would be trivial to use a
disk slice as one big scratchpad, much like a swap partition. No
real need for fs overhead, and one could reserve blocks for
indices or other conveniences.

Eliminating the bulk "data" being DMA's across the PCI bus is what adds
most of your "capacity". Doing zero copy just lets you spend all your CPU
time doing actual analysis instead of copying stuff around unnecessarily.

Hmmmm. Looking back at Agner Fog's Pentium optimization guide,
it does appear that the memblk cp would be less of an issue than
the DMA transfers.

I never did get the opportunity to benchmark it at 1.4million packets/sec,
(I spent more time trying to get the 20ft of fiber to reach the lab at the
previous employeer than I did writing the code to do this in the first
place) but I don't see any reason it shouldn't work, with proper interrupt
coalescing of course.

It would be an interesting test. But ~100 MB/sec of traffic
would choke most any single spool drive... and, assuming that all
the data were of interest, it would probably take people awhile
to review all the data.