Latency, TCP ACKs and upload needs

Mailman List Mirror (Read Only)
1

As part of the ongoing CRTC hearings, the incumbents' claim that
continued implementation of the current 5/1 standard would make Canada a
world leader for broadband in the future.

A satellite company who currently can't even deliver its advertised 5/1
now brags its next satellite will deliver 25/1.

So I have a few questions:

Considering a single download TCP connection. I am aware that modern TCP
stacks will rationalize ACKs and send 1 ACK for every x packets
received, thus reducing upload bandwidth requirements. Is this basically
widespread and assumed that everyone has that ?

Also, as you split available bandwidth between multiple streams, won't
ack upload requirements increase because ACK rationalisation happens far
less often sicne each TCP connection has its own context for ACKs?

When one considers the added latency of satellite links, does 25/1 make
sense ? (I need a sanity check to distinguish between marketing spin
presented to the regulator and real life)

I noticed that in the USA, EXEDE Satellite advertises 12/3 plans and
they are also on a VIA Sat satellite, presumably the same vehicle that
Xplornet tries to deliver its measly 5/1 on. Would all beams be
identical on a satellite or can they be configured differently with a
ISP adjustable rate of upload/download inside the same spectrum ?

Also, when you establish a TCP connection, do most stacks have a default
window size that gives the sender enough "patience" to wait long enough
for the ACK ?

If sender sends packet 457, doesn't get ACK in time and resends 457,
doesn't that also result in reduction in window size (the very opposite
of what would be needed in high latency links) ?

And when the first ACK finally arrives, won't the sender assume this ACK
was for the resent 457 ? Or is satellite latency low enough that
stacks all have enough default "patience" to wait for ACKs and this is a
non issue ?

(Note Xplornet refused to answer questions on whether they operate
special proxies at their gound stations to manage TCP connections to
appear "close").

What i am trying to get at here is whether 25/1 on satellite, in real
life with a few apps exchanging data, would actually be able to make use
of the 25 download speed or whether the limited 1mbps upload would choke
the downloads ?

2

There’s a lot of dynamics here, including how long lived the TCP session
is, the windowing behavior of the host/servers and any acceleration done
in the path. There is also TCP Selective ACK which means you don’t need
to explicitly ACK each frame.

I’m not specifically knowledgeable about how this performs with earth to
orbit and back dynamics, and hope to never personally need to debug it :slight_smile:
I have used satellite based internet over the atlantic ocean before and
it didn’t seem that bad aside from the pricing involved.

I would expect that if all endpoints are doing the right TCP options you
will see reasonable performance. Make sure nobody is masking the window
scaling options and such. (I have heard some carriers do this to prevent
the window from getting too large).

- Jared

3

One of the things to consider is that geostationary satellite operators
operate based entirely on the economics of oversubscription.

If you were to purchase a full duplex 1 Mbps x 1 Mbps connection via VSAT
terminal in North America (whether C, Ku or Ka-band) you'd be looking at
$2000/month or more. You'd have a fixed FDD pipe at 495ms latency end to
end.

32:1 oversubscription or more is normal. It costs close to $150 million to
build and launch a 5000 kilogram satellite into geostationary orbit before
you build any teleport infrastructure. The entire satellite has far less
aggregate data throughput capacity than two strands of singlemode.

Modern Ka-band satellites as used by consumer grade VSAT services in the
United States use dozens of individual spot beams. The FDD capacity in each
spot beam may be exhausted or significantly oversubscribed in one
geographical region, yet relatively unused in others. Compare, for example,
real world user reported speeds at 10pm on Exede service in western WA
state vs. somewhere in a very rural part of Wyoming.

Spot beam TDMA contention ratios are carefully managed by satellite
operators - they're very much aware of the issue you describe, and do their
best to mitigate it. Extensive massaging of TDMA parameters in spot beams
is the only way that it's economical to offer service for between $75 to
$150/month even with a 2 or 3 year contract.
There are a number of physics, OSI layer 1 and 2 issues to consider with
satellite before discussing anything TCP related.

4

As part of the ongoing CRTC hearings, the incumbents' claim that
continued implementation of the current 5/1 standard would make Canada a
world leader for broadband in the future.

A satellite company who currently can't even deliver its advertised 5/1
now brags its next satellite will deliver 25/1.

So I have a few questions:

Considering a single download TCP connection. I am aware that modern TCP
stacks will rationalize ACKs and send 1 ACK for every x packets
received, thus reducing upload bandwidth requirements. Is this basically
widespread and assumed that everyone has that ?

with an mss of 1460 the inbound packets for reasonably well packed flow
will be 36.5x larger than the acks which are 40 bytes.

sack rfc 2018 means you don't have to acknowledge all of the inbound
packets.

Also, as you split available bandwidth between multiple streams, won't
ack upload requirements increase because ACK rationalisation happens far
less often sicne each TCP connection has its own context for ACKs?

When one considers the added latency of satellite links, does 25/1 make
sense ? (I need a sanity check to distinguish between marketing spin
presented to the regulator and real life)

satellite vendors use various forms of tcp acceleration which may
involve among other things having middle boxes that pre-emptively send
the ack, play with the window size etc.

I noticed that in the USA, EXEDE Satellite advertises 12/3 plans and
they are also on a VIA Sat satellite, presumably the same vehicle that
Xplornet tries to deliver its measly 5/1 on. Would all beams be
identical on a satellite or can they be configured differently with a
ISP adjustable rate of upload/download inside the same spectrum ?

Also, when you establish a TCP connection, do most stacks have a default
window size that gives the sender enough "patience" to wait long enough
for the ACK ?

retransmission timeouts are typically measure in seconds...

and geostationary orbits typically have RTTs under 800ms.

If sender sends packet 457, doesn't get ACK in time and resends 457,
doesn't that also result in reduction in window size (the very opposite
of what would be needed in high latency links) ?

And when the first ACK finally arrives, won't the sender assume this ACK
was for the resent 457 ? Or is satellite latency low enough that
stacks all have enough default "patience" to wait for ACKs and this is a
non issue ?

(Note Xplornet refused to answer questions on whether they operate
special proxies at their gound stations to manage TCP connections to
appear "close").

seems likely

5

a geostationary orbit based connection will have a minimum latency of
492-495ms in a dedicated-carrier configuration between two earth stations,
varying very slightly with the modem overhead time for FEC.

In a TDMA network all bets are off, if you're in wyoming on exede and
everyone is asleep, you could see very close to 500ms, you could also see
1200ms during peak hours. Or worse. Consumer grade satellite operators have
only the blunt force tool of GB quota/month to prevent the TDMA capacity in
a particular spotbeam from becoming a complete wasteland of modems stepping
on each other (effectively reducing everyone to 32kbps or worse, and
2000ms).

Go over 8 or 10GB/month and you either need to pay a lot more money, or go
in the TDMA penalty box for $TIME.

As usage patterns tend to rise and fall in a sine wave pattern not very
different than a 10GbE circuit feeding a huge number of downstream DSL
customers, many operators offer 'free' late night/evening hours that don't
count towards the quota. Example:
http://www.exede.com/the-free-zone-internet-details/

6

Others have already answered with the technical details. Let me take a
stab at some more, uh, variable items.

In a message written on Tue, Apr 19, 2016 at 09:29:12PM -0400, Jean-Francois Mezei wrote:

Also, when you establish a TCP connection, do most stacks have a default
window size that gives the sender enough "patience" to wait long enough
for the ACK ?

Your question is phrased backwards. All will wait for the ACK, the
timeouts are long (30-120 seconds). The issue is that you only get
one window of data per RTT, so if the window is too small, it will
choke the connection.

90%+ of the stacks deployed will be too small. Modern Unix generally
has "autotuning" TCP stacks, but I don't think Windows or OS X has
those features yet (but I'd be very happy to be wrong on that point).
Regardless of satellite uplink/downlink speeds, boxes generally need
to be tuned to get maximum performance on satellite.

What i am trying to get at here is whether 25/1 on satellite, in real
life with a few apps exchanging data, would actually be able to make use
of the 25 download speed or whether the limited 1mbps upload would choke
the downloads ?

With a properly tuned stack what you're describing is not a problem.

1460 byte payloads down, maybe 64 byte acks on the return, and with SACK
which is widely deployed an ACK every 2-4 packets. You would see about
2,140 packets/sec downstream (25Mbps/1460), and perhaps send 1070 ACKs
back upstream, at 64 bytes each, or about 68Kbps. Well under the 1Mbps
upstream bandwidth.

7

As part of the ongoing CRTC hearings, the incumbents' claim that
continued implementation of the current 5/1 standard would make Canada a
world leader for broadband in the future.

A satellite company who currently can't even deliver its advertised 5/1
now brags its next satellite will deliver 25/1.

Take a look at https://www.rfc-editor.org/rfc/rfc3449.txt
  TCP Performance Implications of Network Path Asymmetry

So I have a few questions:

Considering a single download TCP connection. I am aware that modern TCP
stacks will rationalize ACKs and send 1 ACK for every x packets
received, thus reducing upload bandwidth requirements. Is this basically
widespread and assumed that everyone has that ?

The usual defaults are to ack every other packet & wait 200ms before
acking a single packet.

Also, as you split available bandwidth between multiple streams, won't
ack upload requirements increase because ACK rationalisation happens far
less often sicne each TCP connection has its own context for ACKs?

Yes. And multiple streams will interfere with each other.
It used to be popular to split a download into multiple streams but I
thought that went out of style now that tcp window scaling is
generally enabled by default.

When one considers the added latency of satellite links, does 25/1 make
sense ? (I need a sanity check to distinguish between marketing spin
presented to the regulator and real life)

I noticed that in the USA, EXEDE Satellite advertises 12/3 plans and
they are also on a VIA Sat satellite, presumably the same vehicle that
Xplornet tries to deliver its measly 5/1 on. Would all beams be
identical on a satellite or can they be configured differently with a
ISP adjustable rate of upload/download inside the same spectrum ?

Also, when you establish a TCP connection, do most stacks have a default
window size that gives the sender enough "patience" to wait long enough
for the ACK ?

There's an initial timeout that's on the order of 3-5 seconds. Once
the xfer starts ... I've forgotten the details, but the retransmission
timer is based on the "smoothed round trip time" (srtt).

If sender sends packet 457, doesn't get ACK in time and resends 457,
doesn't that also result in reduction in window size (the very opposite
of what would be needed in high latency links) ?

The window size is what the receiver advertises, the congestion window
(cwnd) is what the sender computes based on the advertised window size
& packet loss. cwnd is what gets reduced when there's packet loss.
& yes, reducing cwnd is bad for thruput as is not having selective
acks (sack) enabled on the receiver.

And when the first ACK finally arrives, won't the sender assume this ACK
was for the resent 457 ?

The sender keeps track of the 'smoothed round trip time' (srtt).
Since it can't tell if the ack is for the original or retransmitted
pkt, it's not supposed to use that ack for updating the rtt

  Or is satellite latency low enough that
stacks all have enough default "patience" to wait for ACKs and this is a
non issue ?

should be a non-issue

(Note Xplornet refused to answer questions on whether they operate
special proxies at their gound stations to manage TCP connections to
appear "close").

What i am trying to get at here is whether 25/1 on satellite, in real
life with a few apps exchanging data, would actually be able to make use
of the 25 download speed or whether the limited 1mbps upload would choke
the downloads ?

dunno. Assuming the bandwidth is available, I suspect you could get
25Mb/s doing something like downloading a movie from archive.org but
for anything interactive like web surfing / gaming I'd bet no - but
because of latency, not the 1Mb/s uplink speed.

Regards,
Lee

8

Thanks to all for the sanity check.

Always depressing when you think you may have a good argument but after
much reading, you find out you don't :frowning:

BTW, in case someone knows. With the recent "beam" satellites having a
lot of different focused antennas, how does the uplink work ?

Does all traffic pass through a central "switch" which then directs
packets to the approperiate antenna ?

Would a each beam directed at a served area be paired with its own
dedicated beam directed at ground station ? Or does the uplink from
ground station carry traffic for multiple beams and thus becomes the
bottleneck ?

(Xplornet bragged about its next satellite having 20gbps capacity, but
IF the uplink from ground station is also at 20gps and serves 5 beams
aimed at Canada, then on average each beam only gets 4gbps ?)

With regards to the "dream" of having 350 low orbit satellites covering
the globe for Internet, does anyone know how the uplink will be done?
Won't there be a bottleneck if in serving Canada's north, satellites
currently speeding over the region have to use satellite-to-satellite
links to carry information until it reaches a satellite that is over a
ground station in the south ?

or is it expected that ground stations will be built "near" each area to
be served ?

(Am trying to justify that satellite should be reserved for people truly
isolated and that Nunavut communities should get undersea fibre and work
need to start now because of long construction times during which
satellites will fall further back in terms of capacity needs).

9

Others have already answered with the technical details. Let me take a
stab at some more, uh, variable items.

  [.. snip lots ..]

90%+ of the stacks deployed will be too small. Modern Unix generally
has "autotuning" TCP stacks, but I don't think Windows or OS X has
those features yet (but I'd be very happy to be wrong on that point).

Windows has had an autotuning stack since at least Vista.

Regards,
Lee

10

Typically you'll see one ACK per 2 packets, so you need approximately 1:50 bytes up/down ratio for the ACKs.

It's possible to have middle boxes suppress some ACKs, please see thread here:

https://www.ietf.org/mail-archive/web/aqm/current/msg01482.html

11

Note that with delayed ACKs (RFC 1122) there is an ACK for every other
packet; SACK should do better than that.

Tony.

12

Wed, Apr 20, 2016 at 07:27:53AM -0700, Leo Bicknell wrote:

90%+ of the stacks deployed will be too small. Modern Unix generally
has "autotuning" TCP stacks, but I don't think Windows or OS X has
those features yet

OS X since ~10.5 has autotuning, here are some hints from ESnet
  Mac OSX Tuning
Personally never tried this on the sat-type RTT. As explained,
scaling factor of 3 is a limiting one for high-performance transfers.
For sat links may limit the things too, since 64K * 2^3 / 0.9 ms RTT * 8 ~
4.5 Mbit/sec, but one's mileage may vary. And, of course, packet loss
can turn down this BDP-derived speed drastically.

13

Windows also has TCP buffers auto-tuning since Windows 7/Vista up to 16MB, however only receiver-side tuning on their client versions,
for sender-side tuning you will need a server version.
That means uploading stuff from a regular windows "client" machine to a remote host with a large RTT will be very slow.