How is this being done? I've looked at looked at PF_RING and TNAPI... is
there anything better out there?
--Kyle
http://events.ccc.de/congress/2006/Fahrplan/attachments/1225-23c3-slides-av.pdf
That should give you some answers. 
-- Attilla
How is this being done? I've looked at looked at PF_RING and TNAPI... is
there anything better out there?
Those two (thanks to Luca) can get you most of the way there, but to
really hit the target you need dedicated kit like Endace (and a few
others) make. They basically do what was represented in the CCC slides
somebody else posted (FPGA with own logic), but on a PCIe card.
Once you've got the ethernet -> interface problem addressed, you need to
examine bottlenecks in interface->bus and particularly bus->disk.
Regards,
Michael Holstein
Cleveland State Unversity
Those two (thanks to Luca) can get you most of the way there, but to
really hit the target you need dedicated kit like Endace (and a few
others) make. They basically do what was represented in the CCC slides
somebody else posted (FPGA with own logic), but on a PCIe card.Once you've got the ethernet -> interface problem addressed, you need to
examine bottlenecks in interface->bus and particularly bus->disk.
One good open source solution on the disk side is Gluster with 10 gig infiniband on the back end. Gluster allows you to build a distributed storage over many servers. You can find 10 gig infiniband cards on ebay for around $50 and a good 24 port topspin/cisco switch will cost you about $1K.
<>
Nathan Stratton CTO, BlinkMind, Inc.
nathan at robotics.net nathan at blinkmind.com
http://www.robotics.net http://www.blinkmind.com
Might also take a look at Gigamon, Anue Systems, and similar vendors. It's possible to use these switches to "slice and dice" traffic from a 10g input to a farm of 1g tools for packet capture, ids, waf, content filtering etc. Although there is a cost, it's usually cheaper than having to upgrade multiple existing tools to 10g speeds. It also solves the issues with the number of source span's allowed on many Cisco switches, and avoids the bus/disk issues tools run into when dealing with 10g linerates. (For now at least)
~jdh
The paper that I wrote for this talk might give you a bit more information than the just the slides:
http://events.ccc.de/congress/2006/Fahrplan/attachments/1153-23C3_ArienVijn.pdf
This solution filters at full line rate. I am happy to tell more if you are interested.
-- Arien