Ok, so I said I wasn't going to comment on the methodology, I lied.
I wouldn't say the Keynote study is the worst ever. There are some
really rotten studies in the fields of psychology and sociology.
Since the Boardwatch/Keynote study didn't 'test' DRA Net, I guess I'm
one of the few independent, disinterested parties to comment on the
A problem with the Keynote study is it seems very dependent on the
location, type and connections of the testing platforms. Keynote
mentions that connections from Dallas and Phoenix were slow to 'every'
backbone site. This would indicate some systematic problem with
the testing sites. Perhaps the results are even more dependent on
the testing systems than the systems under test. There are also
problems with outlier data points. For example, elsewhere on the
Keynote site, the MCI web site had very fast access from 28 test
sites (< 4secs), and very slow access from one test site in
philly (> 14secs). Mixing and matching data points, if you left
out the one outlier data point, MCI would have been faster than
Savvis. So I don't know if the rankings are very meaningful if a
single test site can have such a pronounced effect.
Unlike a scientific study, there doesn't seem to be enough information
to independently reproduce the results, so I'm just going from the bits
and pieces I can glean from the Keynote pages, press release, and article.