Network Traffic Collection

Mailman List Mirror (Read Only)
1

Hello,

I am trying to collect traffic traffic from pcap file and store it in
a database but really confused how to organize it. Should I organize
it on connection basis/ flow basis or IP basis.

It might be an effort to write a customized traffic analysis tool like
wireshark with only required functionality. I would really appreciate
if someone can give me direction on write way of organizing the data
because right now I only see individual packets and no way of putting
them in some order.

Best,
Ali

2

Does this all not completely depend on what you actually want to do with
it? You might want to start there instead of the other way around.

Greets,
Jeroen

3

I want to be able to see information like how much traffic an ip send
over a period of time, what machines it talked to etc from this
perspective it should be IP based but I would really like to know how
other people do it.

Best,
Ali

4

Netflow + netflow collector.

Ken Matlock
Network Analyst
Systems and Technology Service Center
Sisters of Charity of Leavenworth Health System
12600 W. Colfax, Suite A-500
Lakewood, CO 80215

303-467-4671
matlockk@exempla.org

5

Run argus on a span port.

-Suresh

6

Random thought, anyone ever used Splunk for this kind of thing?

-mike

7

Splunk is an amazing tool and did an awesome thing and introduced a free license in 4.3.

I'm using it at two sites now and I'm loving it!

8

Various folks have, the problem of course comes down to processing
power, thus you'll need to throw a lot of hardware against it to be able
to process traffic in a decent network.

Check http://www.raffy.ch/ and http://pixlcloud.com/ etc for more
details about this.

Greets,
Jeroen

9

Run it with hadoop in EC2?

10

Truth is that most people probably don't do it, beyond temporary, ad-hoc deployments, to solve a specific problem at a specific point in time. Traffic capture and analysis doesn't scale too well into multi-Gb/s service provider environments.

Netflow tools are an option if 'reasonably accurate' is good enough for your needs.

jms

11

Netflow / Sflow with one of the fallowing software packages

http://www.plixer.com/products/netflow-sflow/scrutinizer-netflow-sflow.php
http://www.solarwinds.com/NetFlow

http://www.arbornetworks.com/
Or the hand full of other open source options out there.

Carlos Alcantar
Race Communications / Race Team Member
101 Haskins Way, So. San Francisco, CA. 94080
Phone: +1 415 376 3314 / carlos@race.com / http://www.race.com

12

For high speed switched Ethernet environments, consider using sFlow.

You can treat sFlow as remote packet capture and use Wireshark/tcpdump
for troubleshooting network traffic:

http://blog.sflow.com/2011/11/wireshark.html

Or use sFlow reporting tools to find IP sources, protocols etc.:

http://sflow.org/products/collectors.php

Which tool to choose depends on your requirements.

13

PCAP is not well suited to what you describe. Most people use Sflow/Cflow/...
instead.

Owen

14

+1 This guide should give you a good start.

http://techowto.files.wordpress.com/2008/09/ntop-guide.pdf

Regards

15

Thanks Mukom for the wonderful guide, this is really helpful. I have
few questions about ntop though.

How can I get access to the log files generated by ntop and do my own
parsing rather than looking for webbased results that are generated.
Are there any programs available that do parsing of ntops log files.
When I run ntop on pcap I don't get the throughput graphs as rrd
doesn't work on pcap is there any work around for that.

Thanks,
Ali

16

Hi Ali

Thanks Mukom for the wonderful guide, this is really helpful. I have
few questions about ntop though.

How can I get access to the log files generated by ntop and do my own
parsing rather than looking for webbased results that are generated.

It's been a while i looked under the hood of ntop. Remember that ntop
itself usually needs to be 'fed' traffic to analyse. I have never done
it myself but if I needed the raw data, I'd mirror a port and capture
it with tcpdump into a pcap file (watch disk space!!) the use whatever
analysis tool suits my needs to look at it.

Are there any programs available that do parsing of ntops log files.
When I run ntop on pcap I don't get the throughput graphs as rrd
doesn't work on pcap is there any work around for that.

Not to my knowledge no. I think there's a switch (-f) for reading data
from a pcap file as opposed to a live feed. I have never played with
that as well.

There are other (possible more feature laden) commercial flow
collectors and analysers out there). I also started following trisul
earlier on in the project, you might want to check it out.