Would anyone care to share their experience using collectd as an
alternative to rtg for high-resolution polling of interface traffic and
long term storage?
I am investigating the various options for large data set size, lossless
long term traffic charting (not RRAs which lose precision over time). One
possible use is precision 95th billing.
Be aware that collectd itself is a collection agent. It doesn't include
(last I checked) a grapher. There are however a number of graphers out
there to work with those RRD files, if you use that to store the data.
I personally have been using collectd across hundreds of Linux systems,
using rrdcached to a central collector and wrote my own grapher for that
stuff I am interested int.
Ulf.
Collectd is great, IMHO. I was using collectd+graphite to gather and display stats for a large collection of VMs, servers, routers, and switches. Collectd itself was pretty low overhead, easy to configure (I managed configs via puppet) and Just Worked.
Graphite and carbon cache were a little more tricky to set up - carbon by default aggregates/averages older data, so if not setup correctly, when you go back a few months and try to drill into a graph at a 5 minute interval, you get unexpected results.
I’d highly recommend looking at Graphite, as well. Once you get used to it, being able to apply functions[1] to aggregate, manipulate, and quickly find patterns in data is super useful (ex: look at all interfaces on this switch, only display graphs for the top 5 abnormal traffic). Jason Dixon has written some great blogs posts about it’s use on obfuscurity.com.
John
1: https://graphite.readthedocs.org/en/latest/functions.html
Correct me if I’m wrong, but I believe that collectd uses RRD files for the backend, which you said you don’t want.
You might check out Grafana (http://grafana.org/). Its based off graphite and uses something like opentsdb or influxdb for the backend. I think this is probably more what you’re looking for.
Collectd supports a large number “write” plugins[1] that can write out to various sources. I had been eyeing Grafana and OpenTSDB, they’re probably worth a look
John
1: https://collectd.org/wiki/index.php/Table_of_Plugins
Thus spake Eric Kuhnke (eric.kuhnke@gmail.com) on Wed, Mar 16, 2016 at 11:45:26AM -0700:
Would anyone care to share their experience using collectd as an
alternative to rtg for high-resolution polling of interface traffic and
long term storage?I am investigating the various options for large data set size, lossless
long term traffic charting (not RRAs which lose precision over time).
No, the storage interval is configurable and so are the (optional)
consolidation functions. You may want to look tuning at the defaults
your application is choosing.
Dale
Devices that support sFlow natively implement collectd type
functionality for streaming interface counters to a time series
database (InfluxDB, Graphite, OpenTSB, etc.) Tools like Grafana can be
used to query the database and build dashboards.
Host sFlow (http://sflow.net) is very similar to collectd in the
metrics it exports, but with the added ability to export flow data
from host adapters, bridges, vSwitches, firewalls, routing, VMs,
containers etc.
Prometheus is also worth taking a look at.
http://prometheus.io/docs/introduction/comparison/
*[image: userimage]Scott Larson[image: los angeles]
<https://www.google.com/maps/place/4216+Glencoe+Ave,+Marina+Del+Rey,+CA+90292/@33.9892151,-118.4421334,17z/data=!3m1!4b1!4m2!3m1!1s0x80c2ba88ffae914d:0x14e1d00084d4d09c>Lead
Systems Administrator[image: wdlogo] <https://www.wiredrive.com/> [image:
linkedin] <https://www.linkedin.com/company/wiredrive> [image: facebook]
<https://www.twitter.com/wiredrive> [image: twitter]
<https://www.facebook.com/wiredrive> [image: instagram]
<https://www.instagram.com/wiredrive>T 310 823 8238 x1106
<310%20823%208238%20x1106> | M 310 904 8818 <310%20904%208818>*