Techniques for passive traffic capturing

Mailman List Mirror (Read Only)
1

Hello everyone,

Over the past two years, there's been a trend toward doing more and
more analysis and reporting based on passive traffic analysis.

We started out using SPAN sessions to produce an extra copy of all of
our transit links for these purposes. But the Cisco limits of two
SPAN sessions per device (on our platforms) is a major limitation.

Does anyone have a better soultion for more flexible data collection?

I've been thinking about a move to a system based on optical taps of
each of the links. I'd aggregate these links into something like a
3750 and use remote-span VLANs to pass the traffic onto servers that
sniffing on their interface on that 3750. Do products like the
NetOptics Matrix Switches offer a substantial advantage?

Comments or suggestions?

2

<braindump>
I see little point in aggregating tapped traffic, unless you have only a small amount of it and you're doing it to save cost on monitoring network interfaces - but is that saved cost still a saving when you factor in the cost of the extra 3750s in the middle? I'd guess no.

Depending on how well saturated your circuits are, get double- or quad- GE network cards (Intel make some fixed ones, there are others that take SFPs and fat GBICs) and plug them directly in to the optical taps. If you need your monitoring equipment a distance from the optical taps, use netoptic's regeneration taps, which split 70/30 and then amplify the 30 before sending to your equipment on a different floor/whatever.

There are other vendors, I like netoptics because they have cute purple optical patch leads, provide per-tap specs as tested at the factory, and they all worked beautifully out of the box - another vendor had a 50% failure rate, I've forgotten who they were though.

A PC with 4 GE optical ports is much simpler and probably more cost effective than doing remote span complications.

Note that for a single GE link, you'd need 2GE of remote span backhaul (one GE in each direction).

Matrix switches aren't useful for your case, as you're talking about monitoring for trending etc. I think. Matrix switches are good when you have lots of links, and want to be able to switch between them. Is the cost of matrix switch ports worth the saving in GE interfaces on PCs?

Netoptics have taps that aggregate several links in to one monitoring feed. Not really cost effective when the cost of a single GE network interface for a PC is so low.

The above is based on the assumption you're using PCs for monitoring, the economics of aggregating tap traffic may make more sense if you're using some fancy monitoring platform.

If you find that you need lots of GE interfaces per PC or something, and are saturating the PCI bus, look at DAG cards from Endace. They're designed for passive monitoring, and will send you only headers and do BPF in hardware. I looked at these for a similar project, but didn't bother as it was cheaper to buy more PC chassis' and commodity GE cards. They can do 10GE monitoring, so if you need several 10GE's per chassis I'd recommend these.
</braindump>

3

We started out with SPAN ports, then moved on to Netoptics taps.

Lately we've been using a combination of Cisco Netflow (from remote routers),
and native Argus flows (from local taps) where we need more details.

Flows are useful to answer "What happened X minutes/hours/days ago?",
and where you do not need/want to capture full packet bodies
(though with Argus you can choose whether to include payload data).

http://qosient.com/argus/

4

Cool - good to know that the Netoptics gear is good. Seems like
there's a few resounding approvals of them.

Netflow would be lovely to export from our border routers.
Unfortunately, we are somewhat married to the 6500 platform which has
absolutely awful netflow support. Very small TCAM, export is CPU
expensive, and sampling makes both problems worse. So a mirrored copy
of the transit link is being sent to a pmacct box for flow generation.

5

I stumbled across these last night.

http://www.dovebid.com/assets/display.asp?ItemID=cne11811

I don't know anything about them and haven't done any research. The auction description would however lead me to believe that they might be useful in this case. There are many of them listed in the main auction catalog.

Justin

Ross Vandegrift wrote:

6

I see little point in aggregating tapped traffic, unless you have only
a small amount of it and you're doing it to save cost on monitoring
network interfaces - but is that saved cost still a saving when you
factor in the cost of the extra 3750s in the middle? I'd guess no.

Thanks for all the info Nathan - lots of good leads in your email.
Let me include some more information.

The problem is finding a way to multiplex that traffic from the
optical tap to multiple things that want to peek at it. The
remote-span trick solves that, as well as integrating media
converters. 3750 is nice since you can stack em up and mix/match the
SFP and copper ports.

For example - we have an FCP box from Internap. It wants to see
mirrored traffic so it can watch for TCP setup problems and try to
find blackholes. It takes 10G feeds of aggregated transit links.

Then, we want to do some passive IDS analysis. But snort can only
really only handle 600-800Mbps before it starts saturating CPU
(not multithreaded...) - so one collector per gigE transit seems
logical.

We'd like to generate flow data out of our forwarding plane since
we use 6500s to pull in border transit links. The Netflow on those
boxes is terrible. pmacct does a much better job, but it needs to see
all the traffic out of band.

Note that for a single GE link, you'd need 2GE of remote span backhaul
(one GE in each direction).

We're mostly a content network, very few eyeballs. Our ingress
traffic is negligable compared to egress, which makes the problem
easier.

Matrix switches aren't useful for your case, as you're talking about
monitoring for trending etc. I think. Matrix switches are good when
you have lots of links, and want to be able to switch between them. Is
the cost of matrix switch ports worth the saving in GE interfaces on
PCs?

I guess what made me look at them is their ability to multiplex the
stream of data. Take it from an optical tap, spit the same data out
of multiple ports.

The remote-span trick seems to do the same thing, so I'm wondering
where the gotcha is. If there's an advantage to using something like
the Matrix switches, I'd love to know that now.

The above is based on the assumption you're using PCs for monitoring,
the economics of aggregating tap traffic may make more sense if you're
using some fancy monitoring platform.

Yea - the fact that we have both makes the aggregation method look
good. The FCP takes 10G aggregated feeds. The PCs will want single
gig views of the transit links.

If you find that you need lots of GE interfaces per PC or something,
and are saturating the PCI bus, look at DAG cards from Endace. They're
designed for passive monitoring, and will send you only headers and do
BPF in hardware. I looked at these for a similar project, but didn't
bother as it was cheaper to buy more PC chassis' and commodity GE
cards. They can do 10GE monitoring, so if you need several 10GE's per
chassis I'd recommend these.

Ah the Endace gear looks really interesting. Thanks for the pointer!

7

Ross Vandegrift <ross <at> kallisti.us> writes:

> I see little point in aggregating tapped traffic, unless you have only
> a small amount of it and you're doing it to save cost on monitoring
> network interfaces - but is that saved cost still a saving when you
> factor in the cost of the extra 3750s in the middle? I'd guess no.

Thanks for all the info Nathan - lots of good leads in your email.
Let me include some more information.

The problem is finding a way to multiplex that traffic from the
optical tap to multiple things that want to peek at it. The
remote-span trick solves that, as well as integrating media
converters. 3750 is nice since you can stack em up and mix/match the
SFP and copper ports.

http://www.gigamon.com. Taps+MultiPlexing+Filtering+Clustering+10g. I've been
using them very successfully for exactly what you describe for the last 2 years.
If they are a bit too pricey, look at http://www.vssmonitoring.com. Similar
capabilities to Gigamon, slightly less flexibility (fixed hardware
configurations vs Gigamon's modular configuration) and possibly cheaper
depending on your needs.