Keynote/Boardwatch Internet Backbone Index

For shits and grins:


[as posted to the NANOG mailing list]

Digital Experience


To measure backbone performance, Keynote downloaded 10,000 bytes from one web
server on each
backbone network and recorded how long this took, from the time the URL was
requested until the bytes
were downloaded from the server across the Internet to the measurement
location. Downloads were
attempted every 15 minutes, 24 hours a day, between April 20 and May 20, 1997,
from each of the 27
Keynote Perspective software measurement agents located in major metropolitan
areas around the United
States. Each complete measurement was multiplied by 5 to produce a result more
representative of how
long it would actually take a user to download a full web page with all its

Sorry, you can't just multiply your measurements by 5 and expect to come up
with meaningful numbers. If you want to know the time needed to download a
full web page then you need to measure that specifically.

Keynote decided to measure a backbone provider's own public web
server on the assumption that the
provider would locate its own server in the best-performing hosting
location for that provider.

Whoa! Bad assumption. I know that in Priori's case, our web server will be
located in the WORST performing location because the performance of our web
server is not critical to our network's performance. Our office will be
connected to our local POP via a T1 whereas a colo customer would be in the
same room as a backbone router. I wouldn't be surprised to see similar
scenarios elsewhere, especially among the large providers who have built
out special web farm colo space with the best connectivity. As always, the
shoemaker's children are most likely to be in bare feet.

This is an interesting way to measure end-to-end performance but these two
flaws make this particular set of data completely meaningless.

If you want to measure the service that a backbone provides to its
customers then you really need to use a customer server, not the backbone
operator's own public server which may be located in-house at their
offices. And you need to use real-world example pages that are replicated
to each of the servers. And, quite frankly, I don't think that automated
data collection really can provide a clear picture of the relative
performance of the different backbones. There are too many different things
that can happen on the network and it doesn't make sense to average them
all together. The methodology doesn't mention whether or not outliers in
the data were discarded and what steps they took to identify those
outliers. The study as it stands does not accurately reflect the quality of
the user's experience and that seems to be the original intent of this

Keynote touts itself as the "premier supplier of software and services for
measuring and managing the responsiveness of web-based applications" and
yet they do not even appear to be aware of the tools and methodologies
available from the IPMA project or from CAIDA or some of the other end-to-end performance
measurement info from sites listed in the "Tools" section at
NLANR -- National Laboratory for Applied Network Research

Golan Ben-Oni wrote:

For shits and grins:

Digital Experience


Hmmm. A web server on a 56k line from my house would have faired
pretty good with their bogus "methodology". But in the advertising
world "junk sells". Yet another dubious source of Internet information.

Peter E. Giza || Technical Consultant || ADSmart Corporation
fone 508.684.3609 || phax 508.684.3618 || page 800.632.1746 ||