We've deployed a bunch taps in our network and now we need a platform on which to capture the data. Our bandwidth is currently pretty low but I've got 8 links to tap, which means I need 16 ports. Has anyone done any research on doing accurate packet capture with commodity hardware?
Check out packet forensics depending on what your ultimate requirements are.
Jared Mauch
solera makes some nice boxes also
Check out packet forensics depending on what your ultimate requirements are.
I would also add a 'see packet forensics'...
Richard's blog @ http://taosecurity.blogspot.com/search?q=taps and
especially his books (Tao of Network Security Monitoring and Extrusion
Detection) are the best sources I have ever found, concerning [not only]
taps and[/but] so much more on the subject - proper usage and best
methodologies and practices for network monitoring (and not only for
security!!!)
Stefan
There are several things that you can do with open source solutions,
however looking at the data may be a bit more difficult than something
like Network Generals or Solera Networks capture appliances. It is
still doable and is definitely much much cheaper...
Something you might want to look into is traffic aggregation with a
switch or hub. You can buy an Allied Telesyn switch and basically turn
it into a hub by disabling switchport learning. Just an idea.
You can use regular old tcpdump with the -C option to rotate logs
tcpdump -i blah -s0 -C <filesize to rotate>, etc.
or you can use Daemonlogger which does pretty much the same thing...
http://www.snort.org/users/roesch/Site/Daemonlogger/Daemonlogger.html
Hubs sure are fun...
I would trunk the ports you are monitoring, and run the port monitor on
the trunk port instead (one trunk port, one port per VLAN, plus one
span) which will help with your density. This is assuming the analysis
software you have can read the dot1q tags, but means you do not need to
burn two ports per monitor.
Something you might want to look into is traffic aggregation with a
switch or hub. You can buy an Allied Telesyn switch and basically turn
it into a hub by disabling switchport learning. Just an idea.
Never try to aggregate multiple TAPs with a hub.
You will just create a bucket load of collisions and end up with a useless data feed presented to your monitoring tool. If you want to aggregate multiple TAP feeds into a smaller number of devices(s), most of the TAP vendors make some form of link aggregation device.
Or, depending on the OS and sniffer you use, you may be able to bond the interfaces on the capture device.
-Leon
This might be a stupid question, but where can one get small hubs these days? All of the common commodity (eg: 4 port Netgear) "hubs" these days are actually switches.
What I am looking for is:
Small enough to live in my notebook bag (e.g.: 4 port with a wall wart.)
Cheap
Simple
10/100/1000Mbps
While a tap would work, I'd prefer a hub because I can then use it to connect machines together in a pinch.
W
Warren Kumari wrote:
Hubs sure are fun...
This might be a stupid question, but where can one get small hubs these
days? All of the common commodity (eg: 4 port Netgear) "hubs" these
days are actually switches.What I am looking for is:
Small enough to live in my notebook bag (e.g.: 4 port with a wall wart.)
Cheap
Simple
10/100/1000MbpsWhile a tap would work, I'd prefer a hub because I can then use it to
connect machines together in a pinch.
Hubs are still available that are REAL hubs. I got 4 netgears about a
year ago and they are still available.
However, there is a problem with your specification: No hub (that I am
aware of) can do 1Gbps. All hubs are 10/100 AFAIK.
Jon Kibler
- --
Jon R. Kibler
Chief Technical Officer
Advanced Systems Engineering Technology, Inc.
Charleston, SC USA
o: 843-849-8214
c: 843-224-2494
s: 843-564-4224
My PGP Fingerprint is:
BAA2 1F2C 5543 5D25 4636 A392 515C 5045 CF39 4253
Warren Kumari wrote:
Hubs sure are fun...
This might be a stupid question, but where can one get small hubs these days? All of the common commodity (eg: 4 port Netgear) "hubs" these days are actually switches.
True enough. For those of us who need and want something non-switched, eBay and other used hardware places are the only real option.
What I am looking for is: Small enough to live in my notebook bag
(e.g.: 4 port with a wall wart.) Cheap Simple 10/100/1000Mbps
I don't believe that such a thing ever existed. Hubs that did 10/100, certainly, but I've never ever seen a hub that did gig speeds. When I realized hubs were about to be an endangered species, I started purchasing new and used. I have at least two that (other than testing) have never been used.
While a tap would work, I'd prefer a hub because I can then use it to connect machines together in a pinch.
The original poster needed to deploy a tap, and a hub (for him) would defeat the purpose entirely. If you really really need a hub (or two), your best bet is to start looking at various resellers. Pity you're not closer; I'm retired, and no longer really need the six or eight that I still have.
The Cisco 8 port 10/100/1000 switch (WS-C2960G-8TC-L) supports RSPAN which would allow you to tap all the ports even though it's a switch. It's about $750, so it's not a cheap option, but it's not outrageous either. It's the right size also.
We've deployed a bunch taps in our network and now we need a platform on which to capture the data. Our bandwidth is currently pretty low but I've got 8 links to tap, which means I need 16 ports. Has anyone done any research on doing accurate packet capture with commodity hardware?
A hardware based capture card is the only way to get to any real throughput. Check out Endace cards, that will let you do line rate gig e or better and has native libpcap interface. You also may want to check out WildPackets cards.
<>
Nathan Stratton CTO, BlinkMind, Inc.
nathan at robotics.net nathan at blinkmind.com
http://www.robotics.net http://www.blinkmind.com
I believe Endace also have a productized box containing their capture cards (NinjaProbe); it can be used to capture packets, and can also export NetFlow telemetry based upon the captured traffic. Arbor, Narus, and Lancope have similar NetFlow-via-packet-capture capabilities.
Warren Kumari wrote:
Hubs sure are fun...
This might be a stupid question, but where can one get small hubs these days? All of the common commodity (eg: 4 port Netgear) "hubs" these days are actually switches.
What I am looking for is:
Small enough to live in my notebook bag (e.g.: 4 port with a wall wart.)
Cheap
Simple
10/100/1000MbpsWhile a tap would work, I'd prefer a hub because I can then use it to connect machines together in a pinch.
D-Link sells a smallish 8-port managed Gigabit switch that allows
you to disable learning on the ports -- DGS-3200-10 --
http://www.dlink.com/products/?sec=0&pid=674
I don't know where they hide the manuals on the D-Link
US site, but Google turned them up on their Russian ftp server ??
While not incredibly cheap, it seems reasonable at about $300.
As a bonus, it seems to have pretty complete IPv6 support.
We wanted to do something similar with a 10G switch (SMC8708L2).
It let's you set the size of the MAC table, but not to zero. However,
we found that setting the size of the table to 1 entry effectively disabled
learning.
W
---In the past I have bought some cheap 4 port commodity switches (form Circuit City or somewhere similar), found the datasheet for the chipset (it was a Broadcom something or other) and tied the pin to ground that disables the learning mode (actually, I think that the pin just set the size of the learning table to be 0 entries). While this works, doing it once was more than enough
Nice hack!
Lynda wrote:
Warren Kumari wrote:
What I am looking for is: Small enough to live in my notebook bag
(e.g.: 4 port with a wall wart.) Cheap Simple 10/100/1000MbpsI don't believe that such a thing ever existed. Hubs that did 10/100, certainly, but I've never ever seen a hub that did gig speeds.
Depends what you mean by 'hub' I guess. I thought the term referred to a device that was half-duplex only, and had no address learning. GE has never supported half-duplex.
Sam
Warren Kumari wrote:
Hubs sure are fun...
This might be a stupid question, but where can one get small hubs these days? All of the common commodity (eg: 4 port Netgear) "hubs" these days are actually switches.
What I am looking for is:
Small enough to live in my notebook bag (e.g.: 4 port with a wall wart.)
Cheap
Simple
10/100/1000Mbps
You won't find the gig-e hub out there for sale despite some ieee 802.3 participants staunch defense of 1/2 duplex gig-e support and the resulting complications that caused/s...
Perversely when traveling I actually use the Ethernet ports on my soekris configured as a bridge for this application. A device with 4 Ethernet ports plus a wifi radio which can be configured as bridges, routed, nated etc if that's what's desired. the soekris is not gig-e capable and it's forwarding capacity is a bit closer to the low hundreds of megs, but it travels in my bag, has disk, wifi etc.
MSI industrial makes a mini-itx mainboard that will take an intel core2 has 3 embedded gig-e ports and a 16x pci-e slot that you can put a multiport gig or 2 x 10Gbe interface in... I have a utility 10" deep rackmount that I drag around with that in it when I need more power than the soekris can deliver...
Second that.
Using hub to tap into a single link is also risky. I used to monitor single
FE link with 100M hub. After link had moderate utilization >20%, collision
led was lit all the time.
I've had good experience with VSS Monitoring Ethernet Aggregator taps. Also
Catalyst 2960 SPAN seems to work OK.
As for capture PC, we've been using regular PC with Wireshark. That's good
for single FE link, but has problem with GE and multiple links.
BR,
Juuso
Second that.
Using hub to tap into a single link is also risky. I used to monitor single FE link with 100M hub. After link had moderate utilization >20%, collision led was lit all the time.
I've had good experience with VSS Monitoring Ethernet Aggregator taps. Also Catalyst 2960 SPAN seems to work OK.
As for capture PC, we've been using regular PC with Wireshark. That's good for single FE link, but has problem with GE and multiple links.
If you need to increase the speed of your capture tool, maybe this [1] link may be of use.
It is an implementation of a libpcap that implements a shared memory ring buffer which can result in some capture performance gains.
[1] http://public.lanl.gov/cpw/
-Leon
And, note carefully: some "dual-speed hubs" are actually a 10BT hub and
a 100BT hub *with a switch between them*. I forget which brand I
caught this on, but it bit me a couple of years back.
Which speed cable you plug in determines which hub you're talking to.
Yes, it's weird.
Cheers,
-- jra