RE: 923Mbits/s across the ocean

The jumbo frames effectively increase the congestion avoidance additive increase of the congestion avoidance phase of TCP by a factor of 6. Thus after a congestion event, that reduces the window by a factor of 2, one can recover 6 times as fast. This is very important on large RTT fast links where the recovery rate(for TCP/Reno) goes as the MTU/RTT^2. This can be seen in some of the graphs at:

http://www-iepm.slac.stanford.edu/monitoring/bulk/fast/stacks.png or more fully at:
http://www-iepm.slac.stanford.edu/monitoring/bulk/fast/

We saw little congestion related packet loss on the testbed. With big windows SACK becomes increasingly important so one does not have recover a large fraction of the window for a single packet.

Once one gets onto networks where one is really sharing the bandwidth with others performance drops off rapidly (see for example the measuremsnts at http://www-iepm.slac.stanford.edu/monitoring/bulk/fast/#Measurements%20from%20Sunnyvale%20to%20Amsterdam and compare them with those at http://www-iepm.slac.stanford.edu/monitoring/bulk/fast/#TCP%20Stack%20Comparisons%20with%20Single%20Streams

One of the next things we want to look at next is how the various new TCP stacks work on production Academic & Research Networks (e.g. from Internet2, ESnet, GEANT, ...) with lots of other competing traffic.