does anyone have expirience using a sniffer on a hi-capacity network
segment, that might know if there are limitations I need to worry about?
example: customers doing EMC database replication across a mpls link, and
when the capacity reaches aprox. 250 Mbp/s packets are arriving out of
sequence etc. So we need to put sniffers on both sides to capture some
data to see whats happeneing when the capacity reaches 250mbps.
what kind of system requirements would be needed to be able to be able to
capture that amount of data. For some reason, I dont think that the Dolch
Pac 65 sniffers we have (running nt4 and sniffer pro2) would be able to
handle that kind of data? If they cant, we can probbaly use a sun box.
what kind of specs would the box need?
tia,
tr
It probably depends more on pps than bandwidth.
At a prior job, I used FreeBSD 4.x machines to capture over 400,000 pps, I think, on gigabit links.
You need a nic that is supported with one of the device polling drivers to keep CPU manageable. (Intel, not yet broadcom.)
FreeBSD far surpassed Solaris in packet capture performance.
Linux 2.6 machines may do OK, using NAPI - but I've no experience with that.
todd romero wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Steve Francis wrote:
It probably depends more on pps than bandwidth.
At a prior job, I used FreeBSD 4.x machines to capture over 400,000 pps,
I think, on gigabit links.
You need a nic that is supported with one of the device polling drivers
to keep CPU manageable. (Intel, not yet broadcom.)
FreeBSD far surpassed Solaris in packet capture performance.
Linux 2.6 machines may do OK, using NAPI - but I've no experience with
that.
Eric Weigle and Wu-Chen Feng presented a paper at PAM2002 entitled
"TICKETing High-Speed Traffic with Commodity Hardware and Software"
where they showed collecting traffic at greater than 600Mbps and to 1Gbps
in some configurations. See RADIANT Publications and Presentations
- --
Todd - first thought I have is to get a linux box with a gigE port and
anything pentium III based or faster. Depending on the amount of analysis
you want to do, just running tcpdump to a file and then playback after
the fact. Etherman would make for a good UI to review capture in.
Should be able to write 250mbps out to a fast drive...wouldn't build a
box with that spec with parts from compusa, though.
John
It probably depends more on pps than bandwidth.
Although if you have very high bandwidth you may run into trouble with the PCI bus. 33 MHz 32 bit PCI can barely manage 1 Gbps, and that's withough taking overhead into account.
At a prior job, I used FreeBSD 4.x machines to capture over 400,000 pps, I think, on gigabit links.
I managed to do 600k with 32% CPU on a non-too-high-end machine two years ago. (Just taking the packets off the wire and running them through BPF, no processing, though.)
If you use BPF or pcap, don't forget to increase the capture buffer or you'll have overruns, and don't capture more of the packet than you need.
Date: Fri, 3 Dec 2004 10:47:08 -0500 (EST)
From: todd romero <todd@routeflap.net>
To: nanog@nanog.org
Subject: using sniffer on high-bandwidth pipes
does anyone have expirience using a sniffer on a hi-capacity network
segment, that might know if there are limitations I need to worry about?
example: customers doing EMC database replication across a mpls link, and
when the capacity reaches aprox. 250 Mbp/s packets are arriving out of
sequence etc. So we need to put sniffers on both sides to capture some
data to see whats happeneing when the capacity reaches 250mbps.
Well, there was a nice presentation at SANE 2004 about using Linux
with some tweaks... It also compared it model and performance wise
with the features available under FreeBSD (4.x IIRC):
http://www.nluug.nl/events/sane2004/abstracts/ab.html?id=100
Luca is the man behind NTOP:
Luca showed that moderate hardware is capable of handling Gb/s speeds
at above 90% capture rate if you use the right combination of logic
and tools (PF_Ring). In his case a moderate P3 and I believe somewhere
upwards of 600Mbps... The goal was mainly to reduce the load of the
CPU to allow the machine to actually process the packets it has
captured 
The ntop website has some papers:
http://www.ntop.org/documentation.html
tia,
tr
Kind Regards,
JP Velders
If you want to get serious, check out endace cards... www.endace.com
Their cards offload much of the pcap processing to the specialized nic... It is only for sniffing. They manage to do a zero copy directly to memory... You can capture near line rate at gigabit speeds.
They are expensive, but worth it for serious monitoring.. Tie this with a fast dell tower that you can put on a cart or a something like a Dell 2950 with lots of disk space and run argus http://www.qosient.com/argus/index.htm
Peter
todd romero wrote:
We are using FreeBSD 4.x on 1Gbit Ethernet (for snifferring). Never had a
problems (but I should not garantee 100% snifferring on 400,000pps).
In reality, correct, pps is important, bandwidth is not important. If
traffic is VoIP, it's a problem; if it is 90% WEB, it's an easy task.