Graph Utils (Open-Source)

Hi List,

Anyone out there using something other than rrdtool for creating graphs?? I
have a project that will need a trend taken, and unfortunately rrdtool
doesn't fit the bill. All of the scripting, data collection,
database archival, etc will be custom written or is already done (with some
hacks of course :). So really what i'm looking for is something along the
lines of GNUplot. Has anyone used it before and would like to share
experiences?? Seems like it will be able to my plot data accordingly, but
wanted to see if there were any other popular tools I've yet to come across.

(Open-Source only please)

TIA,
M

If you're comfortable with Python, Graphite is gaining some serious traction http://graphite.wikidot.com/

Paul

Thomas Boutrell's 'GD'.

http://www.libgd.org/Main_Page

Joe

From:
Max Pierson <nmaxpierson@gmail.com>
To:
nanog group <nanog@nanog.org>
Date:
02/18/2011 02:15 PM
Subject:
Graph Utils (Open-Source)

Hi List,

Anyone out there using something other than rrdtool for creating graphs??
I
have a project that will need a trend taken, and unfortunately rrdtool
doesn't fit the bill. All of the scripting, data collection,
database archival, etc will be custom written or is already done (with
some
hacks of course :). So really what i'm looking for is something along the
lines of GNUplot. Has anyone used it before and would like to share
experiences?? Seems like it will be able to my plot data accordingly, but
wanted to see if there were any other popular tools I've yet to come
across.

(Open-Source only please)

TIA,
M

we use both gd (in php and in perl and in c++) and google's graphing magic in various places. http://code.google.com/apis/chart/

I've used gnuplot for several projects and found it very flexible. Gnuplot is also handy because it's easy to feed it commands over a pipe. I also recommend the "Gnuplot In Action" book - it saved me a ton of time. I have also used matplotlib within Python.

For more interactive graphs I've played with the Processing environment a bit, but not enough to provide a useful comparison with the other tools.

Peter

What's wrong with GNUplot? I used it to do graphing of dialup server port utilization in some CGI I did back in the mid 90s.

What's wrong with GNUplot?

Nothing at all :slight_smile: My problem is with rrdtool. It doesn't scale for this
project. I was looking into GNUplot, but wanted to see what else was out
there as well.

Thanks for all of the on and off list replies so far. I'll follow up after
test driving a few of the mentioned tools.

Mostly I've heard bad things about matplotlib under Python. Lots of good features, but buggy and a bit of a memory hog. How did you find it?

If you're searching something like Gnuplot, you could
also try GNU R:
http://www.r-project.org/

HTH,

Also qtiplot an SciDAVIS.

Not to mention others such as found on the exhaustive list:

http://en.wikipedia.org/wiki/List_of_graphing_software

Not to mention others such as found on the exhaustive list:

List of information graphics software - Wikipedia

wow, layer 8 issue on my part :slight_smile: should have checked wiki before I
posted.

Thanks Jim and thanks for all the feedback. I believe a combination of
Graph.pm and some Perl/PHP foo will do nicely.

M

Is scaling of rrdtool still a problem for you with rrdcached?
http://oss.oetiker.ch/rrdtool/doc/rrdcached.en.html

Cheers,
Ren´┐Ż

Even with rrdcached, the I/O from many RRD files being updated often
will hammer the I/O subsydtem of most hosts :slight_smile:

We have a host with around 50k RRD data files and rrdcached running,
most are updated every 5 mins, some every minute (Nagios + PNP) - with
RAID 10 and 10k rpm disks the io wait on the system hangs out at about
15-20 pct, while load hangs around 1-2.

I like rrd a lot but scaling it is hard.

Max

Twitter is releasing a high volume metrics collection store based on
cassandra as open source soon - if you will be scaling big, might be
worth looking into.

Is scaling of rrdtool still a problem for you with rrdcached?

This helps on some of my network/server related graphs, but this data is not
exactly time based (well timestamps are recorded, but not at cyclic
intervals). Plus the dataset is extremely large (100's of millions or rows
already in mySQL). This isn't really network or server related metrics i'm
trying to plot.

Regards,
Max

Anyone out there using something other than rrdtool for creating graphs?? I
have a project that will need a trend taken, and unfortunately rrdtool
doesn't fit the bill. All of the scripting, data collection,
database archival, etc will be custom written or is already done (with some
hacks of course :). So really what i'm looking for is something along the
lines of GNUplot. Has anyone used it before and would like to share

I haven't heard of gnuplot used often with other software as a framework
for graphing/visualizations. For simple visualizations, I think usually a
'native' framework/API is preferred, e.g. JGraph for java apps.

I suspect one reason gnuplot is not used as widely as it could be otherwise
is, its licensing is not as "friendly" as other graphics frameworks.
gnuplot license is GPL incompatible and does not seem to even fully meet
the open source definition,

Because redistributing complete modified source code of gnuplot itself
is not allowed by the license; a clear reading of gnuplot license suggests
only patches, unmodified source code, can be freely redistributed,
redistributed binaries based on modified source have special rules).

Aside from that caveat, which most likely does not normally impair private
use by a network operator: gnuplot is a really good tool.
If you need to paint a bunch of arbitrary X and Y values on a graph from
an input file or based on simple equations, gnuplot will happily
oblige; it can
handle chart types rrdtool cannot, and you have more direct control of output.

If you want some 3D / surface graphs, RRDTool won't do it, anyways.
Gnuplot's less expensive
than Matlab / Maple.

You can even set terminal type to "dumb" in gnuplot, and generate some fancy
ASCII art graphs on stdout.

In regards to scalability...

About the millions of rows... err..
Try plotting a test dataset with 500 million datapoints. Chances
are gnuplot won't
necessarily scale that well either, and you need some method to be
selective of which rows are
provided as input to the plotting framework, in that case.

If you have a million datapoints on your X axis, each X position is
smaller than 1/1000 of
a display pixel (on a graph that fits on a display at say 1920x1080);
displaying such high resolution of all datapoints at once on the
unzoomed graph is beyond
the display hardware capabilitiy.

there should normally be some form of averaging / smoothing /
"selection of points" contemplated,
if the dataset is huge

If you can use Java, JfreeChart is pretty nice. It has the ability to
create many different types of charts/grpahs. I've only used it a
little bit for a project I was looking into that uses it, but it seems
really capable. I think it's licensed under gpl or lgpl, but the
creator charges for documentation/examples if you need them.

http://www.jfree.org/jfreechart/

Hiya Jimmy!!

How has it been?

For simple visualizations, I think usually a 'native' framework/API is

preferred, e.g. JGraph for java apps.

Unfortunately, I'm not savvy with Java at all, so the really cool viz API's
wont work for me (there's just something about Java ... I simply can't get
into it and I see alot of Java based apps that are resource hogs). I was
looking at mostly using some simple Perl + PHP (or even Python) for the
graph generation. My own cacti if you will, just not as feature filled
but template driven.

If you need to paint a bunch of arbitrary X and Y values on a graph from
an input file or based on simple equations, gnuplot will happily
oblige; it can
handle chart types rrdtool cannot, and you have more direct control of

output.

If you want some 3D / surface graphs, RRDTool won't do it, anyways.
Gnuplot's less expensive
than Matlab / Maple.

This is why rrdtool won't work for me. I'll be inputting data sets for
charts that rrdtool doesn't understand and I would like to add a z-axis for
some data that would be better understood on a 3D plot. The Graph.pm has
hooks for xrt3d which looks pretty neat. I'm also looking at Grace as well
using that same module (xmgrace), and it also has gnuplot I can work with.
This way I can reuse code if I decide to use one over the other.

The other issue is that all of this data is already stored (at least most of
it) in mysql, and I don't need to waste cycles and chunks on riping data
from one DB and have to create another just to display a graph. Plus some of
the RRA's would be huge and therefore slow.

Try plotting a test dataset with 500 million datapoints.
there should normally be some form of averaging / smoothing / "selection of

points" contemplated, if the dataset is huge

The few hundred million rows are spread out across different servers and are
not all the same table types. I believe it's 280-300 million rows by
first estimates, but they will be represented in different manors. I won't
be injecting the volume of inserts aggregate onto one plot over time. I
haven't gone that mad yet :slight_smile: For measurements such as that, averaging will
be used for such trends.

Good talking to you again,
Max

Unfortunately, I'm not savvy with Java at all, so the really cool viz

API's

wont work for me (there's just something about Java ... I simply can't

get

into it and I see alot of Java based apps that are resource hogs). I was
looking at mostly using some simple Perl + PHP (or even Python) for the
graph generation. My own cacti if you will, just not as feature filled
but template driven.

The GD 'C' package has great Perl interfaces called GD, and GD:Graph. Easy
to work with ...

GD: GD - Interface to Gd Graphics Library - metacpan.org

GD:Graph: http://search.cpan.org/~bwarfield/GDGraph-1.44/

Joe

The GD 'C' package has great Perl interfaces called GD, and GD:Graph

I'm test driving GD as well. Trying out a few other tools that were
mentioned here also. Trying to get a feel for which I like best.

Thanks for all of the replies!