MTU of the Internet?

this was sent to me personally but seems generally relevant.

  There is at least one case where multiple connections will
be faster, and I suspect there are actually several.

tcp without congestion avoidance is always faster than tcp with
congestion avoidance. whether you take away congestion avoidance
by running lots of short lived connections in parallel, or whether
you actually dyke out the code that avoids congestion, the result
is the same: other tcp's will back off and give you more of the
channel.

this is another example of a cooperative protocol suite trying to
survive in a competitive world. that tcp only behaves well if all
other tcp's behave well, where "behaves well" means "shares paths
fairly" creates an automatic incentive on the part of some designers
to not "behave well" so they can get more than their fair share.

this feeds directly into arguments about flat rate, peering, open
smtp relays, checking of source addresses on egress gateways, and
the rest of the usual nanog chatter.

  [there is a] paper on TCP performance over satellite links
(really any long-fat-pipe), and [there is] a lot about TCP performace
I would have never expected. One of the simpler ideas is the
"TCP speed limit". You must have buffer space that is greater than the
bandwidth * delay product to keep the pipe full.

yes. this is why we have a window scale tcp option now. and this buffer
requirement is per end-host, whether there are multiple tcp sessions or
just one.

                                                    On very high delay
links (satellite) or very high bandwidth (OC-3/OC-12 to the desktop)
current workstations don't allocate enough buffers, either system
wide or per-connection.

at least one satellite ip provider is evaluating transparent caching as a
way to obviate these effects by aggregating dumb-client connections into
something smarter for the sky link.

  So, if you have a system with per-connection bufffering, and
that hits a "speed limit" of 512 kbits/sec over your satellite T1,
you can raise the speed up to the full T1 by opening 3 streams. Thus
more connections == faster downloads.

it's still the same amount of buffering per end host though.

  ... The same group ... did some satellite testing, trying
a parallel FTP client they had. They benchmarked 1, 2, 4, 8, and 16
streams between two SunOS 4 boxes over a T1 satellite (500ms RTT).
Maximum performance (shortest transfer time) was achieved with 4
TCP streams.

that's got more to say about the sunos tcp implementation than about tcp
as a protocol.

  I'll throw in one more subjective measure. When Netscape
first came out some of the CS people at Virginia Tech wondered about
the perception, and did some tests with willing college student
subjects using Mosaic (which at the time supported HTTP keepalives,
and used them to connect to the server once for all transfers)
and Netscape (multiple connections, no keepalives).
...
  On the more complex pages Mosaic downloaded and displayed the
page faster, in wall clock time, but the users always preferred
Netscape. The observation is that many people don't wait for
a full download, they click on picture links, for instance,
when they are half down....so having them all half down allows someone
to proceed with browsing, rather than having half fully there, and
half not visable.

which means the server gets a lot of RSTs and a lot of data which is
committed into the path ends up being thrown away. seems like if we
wanted things to go faster, we would waste less bandwidth by sending
only data which was useful and not creating a huge number of retrans-
missions by effectively turning off tcp congestion avoidance.

  ... - I would suspect that two connections
with keepalives would be optimum....you get multiple stream advantages,
minimize load on the server, and can drive those two connections
fast (past slow-start) as they are long lived.

i agree. too bad the browser manufacturers are driven by competition rather
than cooperation. of course, human nature has in this case created a product
opportunity in transparent caching.

two things which I wanted to comment on.

tcp without congestion avoidance is always faster than tcp with
congestion avoidance. whether you take away congestion avoidance
by running lots of short lived connections in parallel, or whether
you actually dyke out the code that avoids congestion, the result
is the same: other tcp's will back off and give you more of the
channel.

this is another example of a cooperative protocol suite trying to
survive in a competitive world. that tcp only behaves well if all
other tcp's behave well, where "behaves well" means "shares paths
fairly" creates an automatic incentive on the part of some designers
to not "behave well" so they can get more than their fair share.

Yup. _I phone_.

I used to run a 40 port ISP on a 64k uplink.

It worked _just_ fine... until VocalTec took over.

> I'll throw in one more subjective measure. When Netscape
> first came out some of the CS people at Virginia Tech wondered about
> the perception, and did some tests with willing college student
> subjects using Mosaic (which at the time supported HTTP keepalives,
> and used them to connect to the server once for all transfers)
> and Netscape (multiple connections, no keepalives).
> ...
> On the more complex pages Mosaic downloaded and displayed the
> page faster, in wall clock time, but the users always preferred
> Netscape. The observation is that many people don't wait for
> a full download, they click on picture links, for instance,
> when they are half down....so having them all half down allows someone
> to proceed with browsing, rather than having half fully there, and
> half not visable.

which means the server gets a lot of RSTs and a lot of data which is
committed into the path ends up being thrown away. seems like if we
wanted things to go faster, we would waste less bandwidth by sending
only data which was useful and not creating a huge number of retrans-
missions by effectively turning off tcp congestion avoidance.

Alas, the problem is that "faster" isn't what we really want here. In
the quest for "efficiency", we've forgotten that what is _really_
desired is "effectiveness" (not "do the thing fast", but "do the
_right_ thing"), and the problem is that what "the right thing" _is_
depends almost entirely on the design on the page.

If there aren't sufficient structural clues in the markup, the server
_cannot_ accurately guess what's most necessary to get out there
quickly, and will be right, at best, 50% of the time.

Cheers,
-- jra

Paul A Vixie <paul@vix.com> writes:

yes. this is why we have a window scale tcp option now. and this buffer
requirement is per end-host, whether there are multiple tcp sessions or
just one.

You need to support approximately a window worth of
buffering in each device along the path where there is the
possibility of transient congestion; this turns out to
mean you need approximately the line bandwidth * ca. 400ms
per link (or at least space for a decent delay * bandwidth
product on very very fast links). See the work by Guy
Almes et al. on the T3 NSFNET link to Hawaii, and endless
rants by me, Peter and Curtis.

So, big amounts of unacknowledged data == big amounts of buffer.

Big amounts of buffer not being used == big amounts of
space for TCP to fill probe into and fill up. Bad news
for a number of reasons. Meet RED, which keeps buffer
occupation small, leaving it available for dealing with
transient congestion so already-transmitted unacknowledged
data from far away don't get lost when someone clicks on a
web page or starts an FTP or something.

at least one satellite ip provider is evaluating transparent caching as a
way to obviate these effects by aggregating dumb-client connections into
something smarter for the sky link.

At least six such companies have wandered into the offices
of either me or my managing director in the past three months.
At least four other companies have telephoned. Just FYI. --:slight_smile:

> So, if you have a system with per-connection bufffering, and
> that hits a "speed limit" of 512 kbits/sec over your satellite T1,
> you can raise the speed up to the full T1 by opening 3 streams. Thus
> more connections == faster downloads.

it's still the same amount of buffering per end host though.

The amount of per-interface buffering on routers is a
function of the propagation delay and bandwidth of the
network, optimally, whether there is one flow or ten flows
or ten thousand flows passing through. This buffering is
more of a "speed limit" based on limited experiments done
with various satellites used for real circuit
transatlantic circuit restoration than host buffering is.
The problem is that a short transient spike comes along
and eats a couple of segments in a very wide window,
causing slow-start. This often happens well away from
the pair of routers on either end of the satellite
connection, from anecdotal evidence.

> ... The same group ... did some satellite testing, trying
> a parallel FTP client they had. They benchmarked 1, 2, 4, 8, and 16
> streams between two SunOS 4 boxes over a T1 satellite (500ms RTT).
> Maximum performance (shortest transfer time) was achieved with 4
> TCP streams.

that's got more to say about the sunos tcp implementation than about tcp
as a protocol.

No, it has more to say about the test methodology.
It is difficult to interfere with your own packets, if you
have only one interface to transmit on...

A better small-scale test would involve several sinks and
sources clustered around a router that does not impose a
round-robin or similar serialization scheme on inbound
packets, combined with a "burst generator" which sends
traffic across that single router, interfering with stuff
coming down from the satellite.

Cisco to their credit has in the past spent lots of time
studying this based on some studies led by Curtis and
others of the performance of the Bellcore ATM NAPs, and
some other experience garnered here and there. Other
vendors are probably not ignorant of these testing
methodoligies...

Anyone at ANS who wants to talk about the IBM traffic
recorder is welcome to do so. :slight_smile:

  Sean.