Hi,
Am having a debate on the results of speed tests sites.
Am interested in knowing the thoughts of different individuals in regards to this.
Regards,
Jacob
They are completely unreliable and not to be trusted except for an occasional general indication of speed.
Couldn't agree more, it's unfortunate that so many users take them as gospel!
They are very useful for like-for-like comparison, for an indication of where your minimum performance levels are probably at, for a quick check that things are working properly and as expected.
To determine the exact max effective speed? To test qos policies? To determine whether you are meeting SLA's?
Not so much.
Joe
Leigh Porter wrote:
In my opinion they are only "somewhat reliable" if they are on your network
or very close to your network -we operate one of the speedtest.net sites and
for our own eyeball traffic find it to be a "reasonable indicator" of what
kind of speeds the customer is getting.
To put it a different way, if a customer is getting 20X1 Internet service
and the speedtest shows 17 X 0.8 then case closed - if they are getting a
speedtest result of 5 X 0.5 then our helpdesk will take a further look -
this is really in rough terms...
Paul
Rolling your own with NDT (web100 tools), placing it on a common transit point within your network, and providing access to your customers will provide decent, consistent and pretty detailed results regarding TCP attributes. There are facilities within the software to perform packet captures for each test too. Works for our help desk to understand relative performance and respond accordingly.
Trust nothing off your managed net.
Tate
From the consumer viewpoint:
No single data point should be extrapolated to infinity, but comparing problematic behavior with "normal" behavior is a standard process across all fields.
Speed tests from several locations done regularly give a baseline for performance. Major departure from expected numbers from a set of speed test sites can be regarded as an indicator of local loop problems. Did you know that local loops suffer from backhoe fade? And, DSLAMS fail.
In my home office, speed tests are just another useful diagnostic helping to locate problem areas - just like in Paul's example. DSLReports line monitoring service is a similarly useful tool.
James R. Cutler
james.cutler@consultant.com
I love using speedtest. My FIOS at home is 25/25. And speedtest consistently hits that mark
so I know FIOS is giving me what I paid for.
When Verizon was having internet issues last week my numbers were bad.
Like someone else said, I would not use it much more for quick gauge. To get more granular info
you should be using other tools....
They are just a measurement, which need to be correctly used and
interpreted (that's the difficult part).
Reading bad numbers is not necessarily an indication of a link problem.
Reading "good enough" numbers is only meaningful for the duration of the
test.
To me, the big problem is that they don't state all the details of the
tests (for example, how exactly to they do the transfer). Geographical
location is good, but sometimes not enough. Do they use http, https, ftp
or their own JS implementation of whatever weird protocol they though of?
How do I know if I'm hitting my firewall, web cache or ALG?
I only use them to get a generic overview of the link.
Hello,
Am having a debate on the results of speed tests sites.
Am interested in knowing the thoughts of different individuals in regards to this.
They are just a measurement, which need to be correctly used and
interpreted (that's the difficult part).Reading bad numbers is not necessarily an indication of a link problem.
Reading "good enough" numbers is only meaningful for the duration of the
test.To me, the big problem is that they don't state all the details of the
tests (for example, how exactly to they do the transfer). Geographical
location is good, but sometimes not enough. Do they use http, https, ftp
or their own JS implementation of whatever weird protocol they though of?
How do I know if I'm hitting my firewall, web cache or ALG?
I agree. But one that is fairly clear in what (and how) it tests (but
to be fair isn't really a 'speed test') that I've come across is ICSI
Netalyzr. It's pretty useful to give a first impression to a tech of
what's going on with a link.
Take a look at an example report (from a dodgy connection) I dug up:
http://netalyzr.icsi.berkeley.edu/restore/id=43ca208a-28820-e88f1efc-a129-4c92-8968
More info and examples are at http://netalyzr.icsi.berkeley.edu/
I also think that sometimes having a 'speed test' or similar hosted on
a network you are trying to connect to can be useful to find out if a
link is congested, or other problems getting from you to that network.
An example of this is The BBC's iPlayer diagnostic at
http://www.bbc.co.uk/iplayer/diagnostics (think Hulu, but in the UK).
It tests to all their CDNs (Akami, Limelight etc) using different
streaming methods and gives the results. Only useful as an overview,
but a decent first guide nevertheless
.
I only use them to get a generic overview of the link.
Heck yes!
Alex
If you want to understand the issue in detail, check out the report from
MIT this year, written by Steve Bauer and available at
http://mitas.csail.mit.edu/papers/Bauer_Clark_Lehr_Broadband_Speed_Measurem
ents.pdf.
- Jason
It's one data point of many.
Depending on the speed test site, the protocols it uses, where the
test is located, any local networking gear (I've seen transparent
proxies get great speedtest ratings!), etc, they can be useful,
particularly in verifying that a provider's off-net interconnects and
partners are doing well.
However, they are susceptible to things like wireless network issues,
TCP limitations (one stream vs. many streams), and misconfiguration of
devices at the customer location. And the speed test box isn't
necessarily configured/speced correctly either.
I second the thoughts on NDT and I like the ICSI Netalyzer. But I
wouldn't necessarily put either tool in most end users' hands (I think
they are too complex for most end users to interpret the results
properly).
Am having a debate on the results of speed tests sites.
Am interested in knowing the thoughts of different individuals in regards to this.
They are excellent tools for generating user complaints.
(just like the "do traceroute and count the hops" advice from gamer mags
of old).
(my $0.02)
Michael Holstein
Cleveland State University
I have seen some surreal results reported by some of the speed test sites
if you have a sufficiently fat pipe. Near as I could tell, every single hop was
gigE or better all the way, the speedtest site then tried to apply a correction
for the bottleneck it knew about on its local gigE nterface, and basically decided
that the *rest* of the path must be near-infinite speed. 
Am having a debate on the results of speed tests sites.
Am interested in knowing the thoughts of different individuals in regards to this.
It's one data point of many.
Depending on the speed test site, the protocols it uses, where the
test is located, any local networking gear (I've seen transparent
proxies get great speedtest ratings!), etc, they can be useful,
particularly in verifying that a provider's off-net interconnects and
partners are doing well.However, they are susceptible to things like wireless network issues,
TCP limitations (one stream vs. many streams), and misconfiguration of
devices at the customer location. And the speed test box isn't
necessarily configured/speced correctly either.
I don't imagine it accounts for l3 emcp either... To be clear, what one
is I assume generally looking for from a speed test is usable throughput
from the vantage point of the end-user running it.
Just a note on this subject although not directly related to the original
question - There some interesting tests available here:
http://www.measurementlab.net/
I find that they are useful for filtering out some of the completely bogus complaints. We encourage customers to include some test results when they contact our NOC to avoid being ignored when they send an "its slow" complaint.
That said - people get fixated on the numbers. 80% of the purchased speed on non-CIR services is cause for a complaint.
Our biggest issue is people doing tests to destinations 300+ ms away that only last for a few seconds and then complaining about poor performance. As soon as you mention things like bandwidth delay product the eyes glaze over. Heavy use of lossy WISP access network providers doesn't help.
Or that most ADSL lines have about 20% ATM cell "tax" on them.
I did get caught up on a speed test today. I was turning up a GBLX 100Mb
circuit. I got the /30 and all the pings were good to the router. I then
pinged some known hosts in the Westin (about a block away where GBLX's
router was) and saw some not so nice ping times. I then ran a speedtest
and only got about 2Mb/s. Come to find out that this was going to be an
MPLS path to the company's California office. Since it hadn't been setup
fully the router had found some path through it's management network to
ping the world through the tester's DSL line on the other side.
So, know the path you are testing.
We host an Ookla Speedtest server onsite and find it a very reliable means
to identify throughput issues. The source of any performance issues may or
may not be ours, but if a customer says things are slow we can usually
identify whether it's their PC or network (browsing is slow but speed test
runs fine) or a local or regional network issue (speed test runs slow).
If a customer gets less than 90% of the advertised throughput, we follow up
on it.
Frank
The MIT article is good read, thanks for sharing that.
One thing to watch out for is if the last mile provider is the one hosting
the speedtest site, that's another variable removed from the equation. In
some cases that is a good thing, in others it's not, depending on what you
are trying to measure. It's also theoretically possible (and in my opinion
not only likely but probably fairly common) for some large residential ISP's
to not rate-limit these on-net test sites (either by design or as a side
result of at what point in the network they apply the rate limiting),
thereby showing much higher results than the end user could ever possibly
see in a real world scenario.
Also, when using some of the popular public Ookla/speedtest.net sites, their
FAQ clearly states that the tests are not suitable for certain connection
types like high speed services and non-residential services in general. One
good example is Speakeasy's site, which in my personal experience has been
the one most commonly used by end users (especially those contacting us
about "speed problems"):
http://www.speakeasy.net/speedtest/issues.php
"Our speed test is tuned to measure residential broadband services up to 20
Mbps over HTTP. It takes a very customized installation to be able to
accurately measure up to 100 Mbps over HTTP."
-Scott