I promised to summarize responses to my query

So who actually measures their network performance and how?

As most responses were private, I have removed attribution. Thanks to all
constructive respondees.

I have proposed a survey panel for the next NANOG if we do not exhaust the
subject beforehand.


- - - - - - - - - - - - - - c u t h e r e - - - - - - - - - - - - - -

We do SNMP polling ever 15 minutes at SESQUINET on every line over which we
have administrative control and over every peering point. We produce a daily
reports on errors and usage.

We are getting ready to switch to Vulture or NetScarf (or some combo) to
give us more interactive information.

- - - - - - - - - - - - - - c u t h e r e - - - - - - - - - - - - - -

We perform measurement of certain basic network parameters, such as
usage (bandwidth used / total bandwidth) and line error rates on all of our
non-customer links. We perform CPU usage, memory usage, and eviron-
mental monitoring of all our routers. We also perform the line usage and
error rate on all customer lines. We monitor all of our customers' routers
unless they say otherwise, and notify them of any problems. Finally, we
monitor select points throughout the Internet (root name servers, etc.) on
a 4 times an hour basis using pings.

We accomplish this monitoring using the following items an in-house built
package that uses SNMP, traceroute, and ping to provide graphs and tabular
statistical information. We use cabletron's Spectrum for a quick network

- - - - - - - - - - - - - - c u t h e r e - - - - - - - - - - - - - -

We do. SNMP MIB-II stuff, plus the cflowd stuff and something we call
'mxd' (measures round trip times, packet loss and potential reason,
etc. from a whole bunch of different points in our network to a bunch of
other points in our network... we use it to create delay matrices,
packet loss reports and other reports). There are some other things,
but these are the biggies.

I was hoping to get our mxd developer to present at NANOG, but she was
unable to attend and is sort of the shy type (too bad, she's one of our
better people). Maybe I can throw together some information on what we
measure for a bullet or two at ISMA, and why. If there's any interest,
that is. The mxd thing was originally just sort of a toy for neat
reports, but in the last year it's become a critical tool for measuring
delay variance for one of our VPDN customers that does real-time video
stuff (and is to some extent helping us figure out where we've got delay
jitter and why; on the other hand it's also raising more questions :-)).

- - - - - - - - - - - - - - c u t h e r e - - - - - - - - - - - - - -

Since most of my professional career has been in the enterprise world, I
can offer you what we used to measure availability to our mail servers,
web servers, DNS servers, etc., at one of my previous employers.

We employed several application tests, along with network performance
tests. Our primary link was via UUnet, a burstable T1. We purchased an
ISDN account from another local provider, who wasn't directly connected to
UUnet. Probably a good example of a joe-average-user out there.

Every 5 minutes, we measured round-trip response times to each of the
servers and gateway router (via ping) and recorded it. We also had
application tests, such as DNS lookups on our servers, timing sendmail
test mails to a /dev/null account, and time to retrieve the whole home
page. We trended the results into graphs and used it

This wasn't meant to be a really great performance monitoring system; it
was actually meant to 1) check how our availability looked from a "joe
user" perspective on the net (granted, reachability/availability wasn't
perfect because it was only one point in the net) and 2) look at response
time trends / application trends to see if our hardware/software was
cutting it.

- - - - - - - - - - - - - - c u t h e r e - - - - - - - - - - - - - -

We use a traffic flow monitoring system from Kaspia Systems.
( The Kaspia product collects all sorts of data from router
ports and RMON probes, stores the data and performs various trend analysis.
We collect traffic flow, router CPU usage and router memory information plus
various errors. There is a data reduction process which runs once a day,
and a very nifty web interface. The product isn't cheap, but the system
definitely fills a void here.

Maybe I should organize a talk on what we're doing with it for an upcoming
NANOG? As an old instrumentation engineer, I think the basis of our use of
the tool is pretty solid. Plus, I actually developed a means for
calibration of the accuracy of the flow data. Haven't had time yet to work
out a validation for the trends, but I'll get to it one of these decades.

Also, the Kaspia people will give you a thirty day trial on their product at
no charge.

- - - - - - - - - - - - - - c u t h e r e - - - - - - - - - - - - - -

It's brain-dead simple, and probably not of real interest to you, but we
keep a few basic stats, going back about two years...

For non-intrusive stuff, we keep a log of all interface status changes on
our routers, and we pull five-minute byte-counts inbound and outbound on
each interface, which we graph against port speed. Watching the graphs for
any sort of clipping of peaks gives a pretty good indication of problems,
and watching for shifts of traffic between ports on parallel paths likewise.

As far as intrusive testing, we do a three-packet min-length ping to the
LAN-side port of each of our customers' routers once each five minutes, and
follow that up with additional attempts if those three are lost. We log
latency, and if we have to follow up with a burst, we log loss rate from the
burst. Pinging through to the LAN port obviously lets us know when CPE
routers konk out, as occasionally we see hung routers that still have
operational WAN ports talking to us, likewise, simply watching VC-state
isn't a reliable enough indicator of the status of the remote router. Plus
it tells you if the customer has kicked the Ethernet transceiver off their
equipment, for instance. Wouldn't matter to you, probably, but our demarc
is all the way out at their WAN port, since we own and operate our
customers' CPE.

I think a bit about what more we could be doing; flows-analysis and
whatnot... It's nice to think about, and eventually we'll get around to it,
but programmer-time is relatively precious, and other things have higher
priority, since the current system works and tends to tell us most of what
we seem to need to know to provide decent service.

- - - - - - - - - - - - - - c u t h e r e - - - - - - - - - - - - - -

we do. in fact, we place quite a bit of emphasis on network stats. currently
we have about 3 years of stats online, and are working on converting our
inhouse engine to an rdbms so we can more easily perform trend
analysis. besides kaspia, other commercial packages include trendsnmp
( and concord's packages ( our inhouse
stuff is located at if you are curious about
what we do.

- - - - - - - - - - - - - - c u t h e r e - - - - - - - - - - - - - -

We're neanderthals right now - we use a hacked rcisco to feed data to nocol.
We watch bandwidth (separately as well) on key links - and also watch input
errors and interface transitions (for nocol) - all done with perl and
expect-like routines, parsing 'sho int's every few minutes.

Emergency stuff goes through nocol; bandwidth summaries are mailed to
interested parties overnight.

- - - - - - - - - - - - - - c u t h e r e - - - - - - - - - - - - - -

We have running here now the MRTG package that generate some fancy graphics,
but in my opinion these graphics are useless and looking in detail to some
of the reports they are not accurate, several of our clients request the raw
data but this package only mantain few raw data just to generate the graphs,
mean also useless.

In the past we use to have also a kind of ascii reports (Vikas wrote some of
the scripts and programs) generated from information obtained using the old
snmp tool set developed by nysernet but I guess that nobody mantained the
config files and I believe that the snmp library routines used aren't
working fine.

So, I need to invest some time to provide a fast solution to this, I'll
apreciate your help to identify some useful package or directions about how
to generate some good looking and consistent reports.

- - - - - - - - - - - - - - c u t h e r e - - - - - - - - - - - - - -

We have been using the MRTG package which is basically a special SNMP agent
that queries the routers for stats and then does some nice graphing of the
data on the web.

- - - - - - - - - - - - - - c u t h e r e - - - - - - - - - - - - - -

SNMP queries with a heavily-modified version of MRTG from the nice guy in
Germany. Works very nicely. We have recently installed NetScarf 2.0, and
are contemplating merging NetScarf 3.0 with the MRTG front end.

- - - - - - - - - - - - - - c u t h e r e - - - - - - - - - - - - - -

I'm researching whether I can rewrite Steve Corbato's fastpoll program
using the fastsnmp library from the NetScarf people. I think this will
allow fastpoll to scale better. I've successfully written a quick C
program that uses the library to collect the required data for a router --
now I've just got to make it so we can manage it easily (i.e.
auto-generated config files from our databases).

My goal is to be able to collect 1-2 minute period data on all links that
are greater than 10 Mbps -- 15 minutes data for everything else. The 2
minute collection period will allow to scale up to 280Mbps before
experiencing two counter roll-overs within a polling interval. Hopefully
that will hold us over the interface counters are available as Counter64
objects via SNMPv2 (if that ever happens).

BTW -- what fastpoll collects now is ifInOctets, ifOutOctets, ifInUcastPkts,
ifOutUcastPkts, ifInErrors and ifOutDiscards. Rather than storing the raw
counters, it calculates the rate by taking the delta and dividing by the
period. Getting the accurate period is actually the hard part -- I'm having
SNMP send me the uptime of the router in each query and using that to
calculate the interval between polls and to detect counter resets due to
reboots. The other trick to handle is the fact that, while IOS updates the
SNMP counters for process-switched packets as they are routed, it looks like
the counter for SSE switched packets on C70X0 routers only get updated once
every 10 seconds.

I'll let you know how my tests come out.