I hate to say I told you so. Actually, I don't, but I can be such a pain
in the ass anyway what difference does it make.
It would appear that everyone is pretty smugly satisfied by concensus that
the performance series we ran actually measures server performance and that
since all ISPs run weeny home servers, this was not "really" a test, flawed
methodology, etc. I corresponded with Doug the Hump at Digex about this.
I've liked this guy since I first met him largely because he's funny and
doesn't take himself too seriously. He's got a yen for black helicopters
that still has me in stitches.
In any event, he didn't appear to be emotionally involved, but noted that
he did think their web server was the problem in their case and that it was
a real issue with the other backbones as well. He said that if we had
measured one of their honking customer web servers that it would have all
been better.
Now I was clear with Doug, as I have been in this mailing list, that
servers DO have impact. When you are attempting to measure end to end, you
certainly hope that the results cummulatively represent everything that has
an effect. But I have been very clear that I didn't buy the "measuring
server" theory. It will have an effect, but not near the effect you all
have apparently dreamed up amongst you with no data at all.
Anyway, Doug coughed the names of a couple of "honkin" customer sites. One
on a UNIX machine running Apache. One on an NT server. As it so happens
they are the NIKE site and the FORBES site. I agreed to run them all for a
few days and publish the results rather openly. Here they are:
Metro Area Abbr Population Mean Std Dev Data Pts. Rating
Omaha OMA 640000 - - - -
Norfolk ORF 1443000 0.917 0.787 477 -
Milwaukee MKE 1607000 2.641 5.545 427 -
Cleveland CLE 2860000 2.893 4.155 432 -
Washington D.C WAS 6727000 3.326 10.274 436 -
Kansas City MKC 1583000 3.334 3.881 440 -
Detroit DTT 5187000 3.424 11.593 470 -
Atlanta ATL 2960000 3.434 5.774 447 -
Denver DEN 1980000 3.497 2.725 433 -
Tampa TPA 2068000 3.541 9.753 440 -
Minneapolis-St. Paul MSP 2539000 3.697 7.263 420 -
Pittsburgh PIT 2395000 4.039 9.484 441 -
Miami MIA 3193000 4.336 11.227 409 -
Chicago CHI 8240000 4.433 7.977 432 -
Philadelphia PHL 5893000 5.687 36.621 446 -
Columbus CMH 1345000 6.582 10.275 352 -
San Diego SAN 2498000 7.124 6.563 111 -
Houston HOU 3731000 7.156 19.197 426 -
Boston BOS 5455000 8.147 68.522 342 -
New York NYC 19550000 8.911 48.549 436 -
San Francisco SFO 6253000 9.462 70.834 367 -
Phoenix PHX 2238000 12.226 26.802 447 -
Seattle SEA 2970000 14.989 74.818 201 -
Dallas-Ft. Worth DFW 4037000 16.24 36.844 421 -
Los Angeles LAX 14532000 33.879 78.169 572 -
Salt Lake City SLC 1072000 48.719 80.656 383 -
Portland PDX 1793000 65.23 88.641 446 -
11.287 42.959 10654
www.nike.com
Metro Area Abbr Population Mean Std Dev Data Pts. Rating
Omaha OMA 640000 - - - -
Norfolk ORF 1443000 1.359 6.704 385 -
Washington D.C WAS 6727000 2.838 6.486 426 -
Cleveland CLE 2860000 2.892 5.958 424 -
Detroit DTT 5187000 3.032 8.818 380 -
Milwaukee MKE 1607000 3.262 10.087 420 -
Tampa TPA 2068000 3.263 5.876 427 -
Philadelphia PHL 5893000 3.697 7.305 433 -
Kansas City MKC 1583000 3.82 7.477 429 -
Los Angeles LAX 14532000 3.979 7.142 441 -
Denver DEN 1980000 4.22 8.725 421 -
Miami MIA 3193000 4.338 13.774 398 -
Pittsburgh PIT 2395000 4.392 10.68 431 -
Minneapolis-St. Paul MSP 2539000 4.545 8.688 409 -
Atlanta ATL 2960000 4.577 16.279 438 -
New York NYC 19550000 5.163 13.025 427 -
Boston BOS 5455000 5.725 16.02 347 -
Chicago CHI 8240000 5.95 11.926 425 -
San Francisco SFO 6253000 7.254 21.74 353 -
Houston HOU 3731000 7.557 14.675 421 -
Seattle SEA 2970000 9.359 22.033 184 -
Columbus CMH 1345000 9.991 21.586 358 -
Phoenix PHX 2238000 14.089 25.896 432 -
Dallas-Ft. Worth DFW 4037000 17.677 44.361 407 -
San Diego SAN 2498000 19.029 10.833 16 -
Salt Lake City SLC 1072000 37.156 92.765 302 -
Portland PDX 1793000 79.404 166.148 437 -
9.868 44.374 9971
www.forbes.com
Metro Area Abbr Population Mean Std Dev Data Pts. Rating
Omaha OMA 640000 - - - -
Kansas City MKC 1583000 0.0050 0.0 14 -
Norfolk ORF 1443000 2.165 11.616 380 -
Miami MIA 3193000 2.715 10.382 396 -
Washington D.C WAS 6727000 3.151 22.964 419 -
Philadelphia PHL 5893000 3.177 14.547 423 -
Milwaukee MKE 1607000 3.204 19.791 416 -
Atlanta ATL 2960000 3.364 17.068 434 -
Minneapolis-St. Paul MSP 2539000 3.847 10.974 405 -
Cleveland CLE 2860000 3.879 20.768 415 -
Denver DEN 1980000 3.974 22.303 417 -
Tampa TPA 2068000 4.001 19.877 421 -
Pittsburgh PIT 2395000 4.333 25.179 426 -
Detroit DTT 5187000 4.428 20.891 376 -
New York NYC 19550000 4.912 21.919 423 -
Boston BOS 5455000 5.347 30.106 340 -
Seattle SEA 2970000 6.324 12.105 179 -
Chicago CHI 8240000 6.378 26.431 421 -
San Francisco SFO 6253000 6.575 35.339 351 -
Houston HOU 3731000 7.49 20.885 417 -
Columbus CMH 1345000 7.726 19.924 351 -
Dallas-Ft. Worth DFW 4037000 13.068 22.705 404 -
Phoenix PHX 2238000 15.445 50.526 427 -
Los Angeles LAX 14532000 23.117 150.429 460 -
Salt Lake City SLC 1072000 47.801 227.832 297 -
San Diego SAN 2498000 85.378 153.109 14 -
Portland PDX 1793000 89.043 241.734 425 -
11.53 79.068 9451
The bottom line is that there is some slight variation, but as I predicted,
not much. And as it so happens, it was generally in the wrong direction.
Nike was a little better on the mean and a little worse on the standard
deviation. Forbes was a little worse (in fractions) from Digex on the
mean, and more so on the deviation. Digex's original figures for the April
20- May 20 period were 9.162 seconds on the mean and 31.752 on the standard
deviation - slightly better than average. Note that 30 days and five days
are apples and oranges if you comprendo fruit. The means for the five day
period were roughly 9.9, 11.3, and 11.5 with the "weeny" Digex server in
the middle.
So yeah, servers do have an impact, but not nearly what you had hoped and
believed. I would say miniscule in a universe where our results ran from
1.5 to 26.8 seconds. And while I'm not shy about "I told you so's" my real
reason for putting this out is that I have heard from several backbones
that are scrambling to upgrade and move their home page servers etc. I
personally would get them up to what you think they ought to be anyway, but
if you go to extraordinary measures, you're probably going to be
disappointed in how little the numbers move - as I predicted. It just
won't move the numbers much. A little perhaps, and if you're not careful -
potentially the wrong direction. Doug was pretty emphatic that these
customer servers were the "good" ones on the good part of the net, and the
home server was the weeny one. Maybe Doug Mohney can jump in and remind me
which was which as far as NT and UNIX goes if anyone is interested. I
would guess off hand that Forbes is taking a little more load than Nike,
but I may be reading messages from God in standard deviation cloud
formations. There just isn't that much difference - certainly not in the
mean.
The bottom line is that if we are actually measuring server performance, we
should be able to measure three different servers, two avowed muscle boxes
and one avowed weeny one, all on the same network (connected differently
I'm told) and get at least as wide a variation as we saw between networks.
We didn't by about a mile and a fortnight. At LEAST it ought to be in the
predicted direction. This clearly was not. Good theory - but not so -
even in the lab.
Again, I think you guys should take a look at this stuff a little more open
mindedly and professionally. It's certainly NOT to scientific laboratory
standards, but it is certainly interesting and I would claim very VALID
information. Better information than you have previously had at your
disposal. It's an attempt to look at the FOREST, not tree limb diameters,
leaf patterns and nutrient flows - all very interesting though those may be
I do grant you. A great deal of this network has operated on theories that
once scaled up, nobody really knows if they work that way or not. I can
tell you from personal experience that most of what I know is wrong, and I
find that out over, and over, and over again. I might also mention that a
lot of what I'm told turns out to be wrong as well. I'm only going to
SUGGEST that I may not be alone.
We'll continue to work on it. I discussed the universal test page
suggestion with Gene Shklar this afternoon and we will make it so. Again,
I don't think it will move any numbers around much, but certainly, as
Forest Gump says, ONE LESS THING.... And if you can make a case for a
different server ON YOUR OWN NETWORK, we will certainly entertain requests
to shoot at another machine. Nominally July 15-August 15th though I'm not
signing up to those precise dates at this time.
Regards
