packet reordering at exchange points

Date: Tue, 09 Apr 2002 16:03:53 -0400
From: Richard A Steenbergen <>
Subject: Re: packet reordering at exchange points

To transfer 1Gb/s across 100ms I need to be prepared to buffer at least
25MB of data. According to pricewatch, I can pick up a high density 512MB
PC133 DIMM for $70, and use $3.50 of it to catch that TCP stream. Throw in
$36 for a GigE NIC, and we're ready to go for under $40. Yeah I know thats
cheapest garbage you can get, but this is just to prove a point. :slight_smile: I
might only be able to get 800Mbit across a 32bit/33mhz PCI bus, but

Of course, in reality, things like choice of NIC can matter tremendously
when it comes to going even moderately fast, which is why people continue
to pay a premium for high performance NICs such as those by Syskonnect. (When
you see vendors touting near-gigabit throughput for inexpensive gig NICs,
beware: you're probably looking at benchmarks consisting of multiple streams
rather than "real world" performance associated with a single wide area tcp
flow across a 1500 byte MTU link).

The problem isn't the lack of hardware, it's a lack of good software (both
on the receiving side and probably more importantly the sending side), a
lot of bad standards coming back to bite us (1500 byte packets is about as
far from efficient as you can get), a lack of people with enough know-how
to actually build a network that can transport it all (heck they can't
even build decent networks to deliver 10Mbit/s, @Home was the closest),
and just a general lack of things for end users to do with that much
bandwidth even if they got it.

In the university community, it is routine for students in residence halls
to have access to switched 10 (or even switched 100 Mbps) ethernet; of
course, at that point, the issue isn't a lack of things for end users to do
with that much (potential) bandwidth, it is the *cost* of provisioning wide
area commodity bandwidth to support the demand that that speedy local
infrastructure can generate that becomes the binding constraint.

And speaking of University users, if you look at Internet2's excellent weekly
reports (see, for example: ) you'll see that wide
area TCP non-measurement single flows *are* occuring at and above the 100Mbps
mark, at least across that (admittedly rather atypical) network infrastructure.
[Maybe not as commonly as we'd all like to hope, but they are happening.]



beware: you're probably looking at benchmarks

(I can't resist passing this along for posterity.)

One of our guys described it thus: Engineering wants to see how fast they can get the wheels to spin on a car. Operations wants to know how fast the car will go. These are different.