Hello,
My company is small clec / broadband provider serving rural communities
in northern California, and we are the recipient of a small grant from
the state thru our public utilities commission. We went out to 'middle
of nowhere' and deployed adsl2+ in fact (chalk one up for the good
guys!), and now that we're done, our state puc wants to gather
performance data to evaluate the result of our project and ensure we
delivered what we said we were going to. Bigger picture, our state is
actively attempting to map broadband availability and service levels
available and this data will factor into this overall picture, to be
used for future grant/loan programs and other support mechanisms, so
this really is going to touch every provider who serves end users in the
state.
The rub is, that they want to legislate that web based 'speedtest.com'
is the ONLY and MOST AUTHORITATIVE metric that trumps all other
considerations and that the provider is %100 at fault and responsible
for making fraudulent claims if speedtest.com doesn't agree. No
discussion is allowed or permitted about sync rates, packet loss,
internet congestion, provider route diversity, end user computer
performance problems, far end congestion issues, far end server issues
or cpu loading, latency/rtt, or the like. They are going to decide that
the quality of any provider service, is solely and exclusively resting
on the numbers returned from 'speedtest.com' alone, period.
All of you in this audience, I think, probably immediately understand
the various problems with such an assertion. Its one of these situations
where - to the uninitiated - it SEEMS LIKE this is the right way to do
this, and it SEEMS LIKE there's some validity to whats going on - but in
practice, we engineering types know it's a far different animal and
should not be used for real live benchmarking of any kind where there is
a demand for statistical validity.
My feeling is that - if there is a need for the state to do
benchmarking, then it outta be using statistically significant
methodologies for same along the same lines as any other benchmark or
test done by other government agencies and national standards bodies
that are reproducible and dependable. The question is, as a hotbutton
issue, how do we go about getting 'the message' across, how do we go
about engineering something that could be considered statistically
relevant, and most importantly, how do we get this to be accepted by
non-technical legislators and regulators?
Mike,
For general tests of most things an ISP does, ICSI's netalyzr tests can't be beat.
http://netalyzr.icsi.berkeley.edu/
There are also tests at m-lab that may be useful: http://www.measurementlab.net/
As in all pieces of software, these may have bugs; netalyzr was under detecting bufferbloat on high bandwidth links until recently; this should be fixed now, I hope.
And SamKnows is doing the FCC broadband tests.
The speedtest.net tests (and pingtest.net) are good as far as they go (and you can host them someplace yourself; as others have noted, having and endpoint at someplace you control is wise); but they don't tell the whole story: they miss a vital issue that has been hidden.
Here's the rub:
Most tests have focussed on bandwidth (now misnamed "speed" by marketing, which it isn't).
Some tests have tested latency.
But there have been precious few that test latency under load, which is how we've gotten into a world of hurt on broadband over the last decade, where we now have a situation where a large fraction of broadband has latencies under load measured in *seconds*. (See: http://gettys.wordpress.com/ and bufferbloat.net). These both make for fuming retail customers, as well as lots of service calls (I know, I generated quite a few myself over the years). This is a killer for lots of applications, VOIP, teleconferencing, gaming, remote desktop hosting, etc.
Netalyzr tries to test for excessive buffering, as does at least one of the mlabs tests.
Dave Clark and I have been talking to SamKnows and Ookla to try to get latency under load tests added to the mix. I think we've been having some traction at getting such tests added, but it's slightly too soon to tell.
We also need tests to identify ISP's failing to run queue management internal to their networks, as there is both research and anecdotal data that shows that that is also much more common than it should be. Some ISP's do a wonderful job, and others don't; Van Jacobson believes this is because Sally Floyd and his classic RED algorithm is buggy, and tuning it has scared many operators off; I believe his explanation.
So far, so bad.
Then there is the home router/host disaster:
As soon as you move the bottleneck link from the broadband hop to the 802.11 link usually beyond it these days (by higher broadband bandwidth, or by having several chimneys in your house as I do, dropping the wireless bandwidth), you run into the fact that home routers and even our operating systems sometimes have even worse buffering than the broadband gear, sometimes measured in hundreds or even thousands of *packets*.
We're going to need to fix the home routers and user's operating systems. For the 802.11 case, this is hard; Van says RED won't hack it, and we need better algorithms, whether Van's unpublished nRED algorithm or Doug Leith's recent work.
So you need to ensure the regulators' understand that doing testing carefully enough to know what you are looking at is hard. Tests not done directly at the broadband gear may mix this problem with the broadband connection.
This is not to say tests should not be done: we're not going to get this swamp drained without the full light of day on the issue; just that current speedtest.net tests misses this entire issue right now (though may detect it in the future), and that the tests (today) aren't something you "just run" and get a simple answer, since the problem can be anywhere in a path.
Maybe there will be tests that "do the right thing" for regulators in a year or two; but not now: the tests today don't identify which link is at fault, and that the problem can easily be entirely inside the customer's house, if the test tests for bufferbloat at all.
I think it very important we get tests together that not only detect bufferbloat (which is very easy to detect, once you know how), but also point to where in the network the problem is occurring, to reduce the rate of complaints to something manageable, where everyone isn't having to field calls for problems they aren't responsible for (and unable to fix).
You can look at a talk about bufferbloat I gave recently at:
http://mirrors.bufferbloat.net/Talks/BellLabs01192011/
Let me know if I can be of help. People who want to help the bufferbloat problem please also note we recently opened a bufferbloat.net web site to help collaboration on this problem.
Best regards,
Jim Gettys
Bell Labs
