A friend of mine just recently got Xplornet satellite service at his
rural home. I'm well aware of the latency issues with satellite
although frankly his latency is much better than I had feared it would
be and is around 600-700ms.
But what seems to be worse than the latency is the "burstiness" of the
traffic and I am just wondering if that is normal/expected for
satellite service in general, and/or expected from Xplornet's service,
or if what I am seeing is not expected at all (i.e. not an artifact of
the satellite signal but rather a network management issue).
Here's iperf3 for 30 seconds sending data (i.e. upload speed):
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.21 sec 12.9 KBytes 87.4 Kbits/sec
[ 5] 1.21-2.00 sec 6.47 KBytes 67.2 Kbits/sec
[ 5] 2.00-3.00 sec 22.0 KBytes 180 Kbits/sec
[ 5] 3.00-4.00 sec 41.4 KBytes 339 Kbits/sec
[ 5] 4.00-5.00 sec 41.4 KBytes 339 Kbits/sec
[ 5] 5.00-6.00 sec 55.6 KBytes 456 Kbits/sec
[ 5] 6.00-7.00 sec 69.9 KBytes 572 Kbits/sec
[ 5] 7.00-8.00 sec 89.3 KBytes 731 Kbits/sec
[ 5] 8.00-9.00 sec 120 KBytes 986 Kbits/sec
[ 5] 9.00-10.00 sec 86.7 KBytes 710 Kbits/sec
[ 5] 10.00-11.00 sec 133 KBytes 1.09 Mbits/sec
[ 5] 11.00-12.00 sec 184 KBytes 1.51 Mbits/sec
[ 5] 12.00-13.00 sec 186 KBytes 1.53 Mbits/sec
[ 5] 13.00-14.00 sec 159 KBytes 1.30 Mbits/sec
[ 5] 14.00-15.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 15.00-16.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 16.00-17.00 sec 93.2 KBytes 763 Kbits/sec
[ 5] 17.00-18.00 sec 264 KBytes 2.16 Mbits/sec
[ 5] 18.00-19.00 sec 124 KBytes 1.02 Mbits/sec
[ 5] 19.00-20.00 sec 157 KBytes 1.28 Mbits/sec
[ 5] 20.00-21.00 sec 120 KBytes 986 Kbits/sec
[ 5] 21.00-22.00 sec 86.7 KBytes 710 Kbits/sec
[ 5] 22.00-23.00 sec 369 KBytes 3.02 Mbits/sec
[ 5] 23.00-24.00 sec 197 KBytes 1.61 Mbits/sec
[ 5] 24.00-25.00 sec 90.6 KBytes 741 Kbits/sec
[ 5] 25.00-26.00 sec 193 KBytes 1.58 Mbits/sec
[ 5] 26.00-27.00 sec 192 KBytes 1.57 Mbits/sec
[ 5] 27.00-28.00 sec 189 KBytes 1.55 Mbits/sec
[ 5] 28.00-29.00 sec 193 KBytes 1.58 Mbits/sec
[ 5] 29.00-30.00 sec 179 KBytes 1.46 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-32.20 sec 4.41 MBytes 1.15 Mbits/sec 388 sender
[ 5] 0.00-30.00 sec 3.57 MBytes 998 Kbits/sec receiver
which averaged the overall prescribed "upload" speed, but notice that
it's not 1Mb/s in any kind of a steady stream but rather bursts of
higher than 1Mb/s speed followed by low/no speed. At one point it was
2 seconds with no transfer at all even.
and here's receiving (i.e. "download"):
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.35 sec 46.6 KBytes 283 Kbits/sec 0 12.9 KBytes
[ 5] 1.35-2.00 sec 0.00 Bytes 0.00 bits/sec 0 12.9 KBytes
[ 5] 2.00-3.00 sec 67.3 KBytes 551 Kbits/sec 0 37.5 KBytes
[ 5] 3.00-4.00 sec 46.6 KBytes 382 Kbits/sec 0 40.1 KBytes
[ 5] 4.00-5.00 sec 105 KBytes 858 Kbits/sec 0 44.0 KBytes
[ 5] 5.00-6.00 sec 88.0 KBytes 721 Kbits/sec 0 54.3 KBytes
[ 5] 6.00-7.00 sec 141 KBytes 1.16 Mbits/sec 0 69.9 KBytes
[ 5] 7.00-8.00 sec 124 KBytes 1.02 Mbits/sec 0 101 KBytes
[ 5] 8.00-9.00 sec 186 KBytes 1.53 Mbits/sec 0 146 KBytes
[ 5] 9.00-10.00 sec 248 KBytes 2.04 Mbits/sec 0 206 KBytes
[ 5] 10.00-11.00 sec 311 KBytes 2.54 Mbits/sec 0 257 KBytes
[ 5] 11.00-12.00 sec 0.00 Bytes 0.00 bits/sec 43 194 KBytes
[ 5] 12.00-13.00 sec 0.00 Bytes 0.00 bits/sec 75 199 KBytes
[ 5] 13.00-14.00 sec 435 KBytes 3.56 Mbits/sec 0 199 KBytes
[ 5] 14.00-15.00 sec 0.00 Bytes 0.00 bits/sec 34 114 KBytes
[ 5] 15.00-16.00 sec 0.00 Bytes 0.00 bits/sec 34 140 KBytes
[ 5] 16.00-17.00 sec 373 KBytes 3.05 Mbits/sec 0 149 KBytes
[ 5] 17.00-18.00 sec 0.00 Bytes 0.00 bits/sec 0 162 KBytes
[ 5] 18.00-19.00 sec 373 KBytes 3.05 Mbits/sec 0 168 KBytes
[ 5] 19.00-20.00 sec 0.00 Bytes 0.00 bits/sec 0 171 KBytes
[ 5] 20.00-21.00 sec 373 KBytes 3.05 Mbits/sec 0 172 KBytes
[ 5] 21.00-22.00 sec 0.00 Bytes 0.00 bits/sec 14 141 KBytes
[ 5] 22.00-23.00 sec 0.00 Bytes 0.00 bits/sec 0 120 KBytes
[ 5] 23.00-24.00 sec 373 KBytes 3.05 Mbits/sec 0 131 KBytes
[ 5] 24.00-25.00 sec 0.00 Bytes 0.00 bits/sec 1 146 KBytes
[ 5] 25.00-26.00 sec 0.00 Bytes 0.00 bits/sec 14 104 KBytes
[ 5] 26.00-27.00 sec 0.00 Bytes 0.00 bits/sec 0 104 KBytes
[ 5] 27.00-28.00 sec 373 KBytes 3.05 Mbits/sec 0 107 KBytes
[ 5] 28.00-29.00 sec 0.00 Bytes 0.00 bits/sec 0 119 KBytes
[ 5] 29.00-30.00 sec 373 KBytes 3.05 Mbits/sec 0 123 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-30.00 sec 3.94 MBytes 1.10 Mbits/sec 215 sender
[ 5] 0.00-30.80 sec 3.13 MBytes 853 Kbits/sec receiver
Again, very bursty with periods of 1-2 seconds with no transfer.
As you can imagine, the bursiness of this makes for horrible video
conferencing since that cannot "buffer" the way single-direction
streams like streaming video can and the codec ends up using the "worst
case" dips as the speed of the connection and encodes for that.
Cheers,
b.
Brian,
Satellite services are shared bandwidth broadcast systems. The behavior you’re seeing is pretty common at times when you’re competing for access with other users. Just like the regular Internet, there are times of day when people tend to move more data, and because of latency and other limitations on bidirectional traffic, packets get delivered in batches. It’s not possible to interleave bytes, or even packets themselves.
So when you see low or no throughput, it’s because the transponder is addressing packets to other users.
-mel beckman
Although I don’t know of a way to solve this for videoconferencing, historically one way to mitigate the radio/vsat “batchiness” issue and its effect on end-to-end latency was to use a caching proxy server connected to a, er, “real” network somewhere, preferably as near as possible to the headend/uplink station. The modern web’s move to TLS also means this technique is becoming pointless even for HTTP/S (although MITMing remains a way around that - many a HOWTO abounds, describing how to do this with Squid).
FWIW, some very old radio systems behaved similarly… albeit not with 600+msec latency :-/. Some of the really old asymmetric TV systems (dial-up for uplink, CATV for downlink) exhibited similar characteristics and were similarly difficult to mitigate.
Good luck!
Adam Thompson
Consultant, Infrastructure Services
![[MERLIN LOGO]](https://mail.merlin.mb.ca/owa/%0Ab0lEQVR42u1bbVBTVxp+6LIrXMFGuXZBSUgy5SOKQwIWdwlCEJnZVpmgTovMjjWoO926dAzVwZ2u%0AFtTOrDJ1Sbfqtq5KqLOidRRYxaWCEmCjK1tImBGT0A4fAcWRWMNXQEdlfwQuCd58YQXp3ucPyc05%0A59573ue87/O+5+A1MjIyAhdQNeqhOF2JsloNACAqlA1hGAeyVWJIoiPAYGbCy8vLy8sVAZTlamTu%0AO+7w90RROBTZGRCGcZgZ/akRQNWoR9LWfLcGK5BnQL4+hZnVGUaAV5w1UJyudHuwbEUxZE48BYOX%0AlATOPIDXrzbZfWf5E1DIM7BxlRgAYO63oKxWA8WZSmhbjJQ+UB3ZCZY/wczuTA4BdO6/+kiOQ9Gn%0ALFcjW1EMc78F0gQRSvM/YGZ4pocAW0gTRE4Vv2yVGNWHc8ANIlFWq2HCwQyB2wRIS4x22UYYxoHm%0AqzwIwzgoKldDWa5mZvinQgBuUIBb7Vj+BEoOZIHlT0BecArt3SZmlmciAbhB5KQH5QaRKDmQhd6B%0AIcj2MqFgxhLgeZS8JDoC29JTUKMxoLSmkZnpmRgCEkXhNlmBwePB87ZIwQ0iIS8oZmZ6JhLAVvVP%0AJpaz/AnkbpGi4+59RhDORAKkJYqoz5N147JVYnCDSChOX2Zm+yWEtysxJ4mOgKpRj96BIaga9ZPa%0A/cvdIkXmvuPQthghDOOg/V4fSuvbUHqjFdq2HvRaHlFto7gk5KujIFshYKwzBfBoN3DjKjGUuzd7%0AfBNzvwVzU7KwOT0N3//wBDXNtyGJXAjJ4oXjbQYfQlmth3nwIQBg2+ooKDYtZyz0Io3vznYwAPDW%0A5FAaoK0k3+MUsfRGK+Qn6vDkyVPsSIvGxqQIsGbPepYogw+R9HEJtG3We2kOroeQRzKWeoEEcKsQ%0AlLtFOq7sj5VRn7V3Bpz207aZINldgsxDV5AimIeum3WQxiykNT4AsGbPQu47sePEqW9lrDSdGmAM%0AQlEUFgnCcEvXgqJyNZqeBkI7ZF8jiAryg4TPgiwmCMIFflBcbEL2iTrIV0eh9I9vQWtow7HT1nRS%0Atsrxqk5bxmes8jIQwDz0GMqGbigb7qKpewBRkcsBXQsA4KHhBi5+9D78CF9rivhgGKrWByhqvAtl%0A/W1EPv0Bzcb7KNn51jMGdZVOjmkABtOYBuZVtYF74DqyL36PDvMw5PFslG6KwbZ064kfXWsXzpZc%0AQiKfhUQ+CxtjAlH4tgD/TA/FSFsLTMNPkZOZQruazf0Wpw+kunl7vA5hIxIZTIEH0N4ZgOysDk3d%0A1tiet5KHbeJgsHytzRTZGajRGKBtMaKoXA1hKIc6BlZ6oxWyz6vA/+UcCJZG4KPLbXgzgoRwgZ+9%0ALhg9OOJQMI7GfdbsWZBEMgSYMgJo7wxAclSD3uHHYPl6o2TDEkj4rGc6VB/OAW9tDsz9FmQriq37%0ABb7zkHnoCoQ8EtV710DV3o/iph6U3uqhCKBt6bTWFhY4jv/t9/pQVK2n0kAGUxQCtHcGIPrrfynj%0AV/9ORGt8wFrerT6cQ20UZe47DuW//oO0ZXxU710D1uxZlMdg+Yw7mLFKoLMUMu9MPbX65QwBpsYD%0AmIceQ3JUQ10oWB36jNt+JisI46D6cA6S/pAPc78FNXV1iAplQ2vgQxIdgdLmHquiXzwfACAvKEbH%0A3fvWvqFsh7HfdvXbpooWiwUdxk63X4okScwnnZ9f8GRMgiAQwrE+t05v8LiPLTqMnbBYLG4/py16%0ATPdhMo2LaEFEOO31EA4bBEG4NYa3Qt2J3uHH1tU51weymEC3HoYXHIh316XisqoO+vY7aPquE0lb%0A85H6myTUgot1AhaU576BqkGPGo2B8h50J4vMgw+Rtr/cShAeibx0ay2gVn0NJ0+dgWVoyGNmE76+%0ASIiPw4aMdLvr58ouoOJylUdjCsLDsGvnDgDAJ/kHPXqOtdJUrJOmUt9PFp+BztBC+5sr1Kqv4XzZ%0ABer7P04cpb3OYQfjz3s+dmuMV5QNd+1yeXfx24LL+EptxJLoX0M2ekr4VdarMBB8+P/CC5WnirDn%0AWBllfADI3SylNb5kdwm1H1CYtRIA8G2jFl8eV07K+ABgGRpCReUVnLN52YrKKpwvuzDpMSeD82UX%0A3PYaPxaMnV344niheyGg48HwuAiz+ewMf6u4CbW+GwDw6aYEcEh/RL3xBnKvm+EFL4xovkFfb59d%0AH/n6FNp/HEnbfwlN7aZR4ydTpd9a9TW7dkujhUgQx4EgCHQYO3Gy+Az124aMdIRw2LBYLLilb0FF%0AZRX1m05vAEZX2beNWodjuvImdEgQxyEhPo7Wzdeqr1EhRmdoodz1VKFOfR0A8PvNmc4JwJ3rQxm+%0AqXsApc0mpC12LNSMpn7sL2kAAMQLgvD0Zz+H/MJ3+Ex9HxI+C3vj5uLqvBioGv1G9QKb9kSx1e1f%0AQk2zNe+XrRDY7QCOxUkAmE8GIDtr63jnCdsXIexgCMLDAAAxIiEGLYPUBDiK8yEctv2YkwBJBlD3%0AnRgyyIAAFBw6Mq0Czx0SeOeu5CHzrI66IDurg7w7GHIxm1Lztth/vhF9lkeIiuTh8by54B24Dpav%0ANxSpodgmDgYALI/kItdFtU+yu4Ra+bIVAhRmJTtsH0Ezyc4wnyTtQgGt8QICnlvQOQ9BlmkzPEEQ%0A1AJyRQJvWUwgmroHoPi3daX0Dj/Gnqp27KlqRyKfBWGQH0WE9gfDaBr0gXnh66h5AHDx6JlikSuM%0AFYzGYr4r40806I+FBo0WDRqtxyLQFjq9Aedo2uv0BkroTQd25WzHJ/kH3SKBtzX1ex3SRSSKRmv/%0AY6hpNaOm1WzXITzAB4kLfDDc24cv3lni9natefAh8s7U47OLTePexg3jv8zQGVpcGpogCFqd8CIR%0AwmE7JMHExUQtWwmfBQmfhcK3BZTRVaN/x4pCfyqshvr6bYw5TpUo0C0CKK/qID9RZ3fypzAreVpP%0A/TgScJ6IQJeeIyIc72akgwwIcFiLmGoSTHwXWr+dOGrwxAnVQHXzbfvK3ekbkCxeSEuC9nt9UFbr%0AobyqQ0dPP3Wd+9ocFGYlT3udv8dkohVwnmCdNBVrRzMMY2cX9h34lJrsBHEc3tssc9r/1gtOD0M4%0AbHyY9b5d7WKiJvJ+nhv0Wh5BtP00pLE8CHnzKcNr20yUwLNL+ZbxUZiV7PBAyFS774rKKsSIRB5V%0A4xyBww7G7p07KBKMpbETSWCbcho7u/CXz49gabQQ8wMCnBrSVarqzAu9t1mGL48r6UWgJ4OxZs+i%0A3a8vq29DWX2bw37c1+agIDN+2g97CCLC7WL2yeKvcbL460mLwMmQYFFEmJ34dEeM7srZ/lx1hASx%0ANdTRkeAVTwaSxvLgKWHy0mOhOZj+Upz0eTMl+YUXZMZIYFt6tZ34BHHclBeFnIUkjzyAYtNyNLWb%0AqEObzlZ8XnospLG8Sbv7hPg4LBqdqImxmiRJuxo6OUHZLgoPgxdNjZ0gCOzK2W7dkBkacrsOQNqE%0ACNv7OjIkhx2MDz/YaleEMnZ2gcMOntQzjL2fo/dydJ2OBIQvAWOn9blOFf4dDP7P8T+TSF9YFJq8%0AQgAAAABJRU5ErkJggg==)
100 - 135 Innovation Drive
Winnipeg, MB, R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
athompson@merlin.mb.ca
www.merlin.mb.ca
Hi,
I used to work for a satellite ISP, and if I'm not mistaken Xplornet is buying bandwidth of my previous employer. That does not mean that your friend is on the same service.
Where I worked, phy transmissions are scheduled based on tokens. A UT must have a token to transmit data. If there is no congestion, a token will be available and the UT or ground station may transmit. Congestion does not need to exist in the ground network or even the transponder. It can even be in the spectrum of that geographical area.
To overcome the latency, a few things are done:
- prefetching on the UT
- WAN acceleration on the UT and in the network (basically syn/ack spoofing)
and a few other proprietary things.
Satellite is obviously not the optimal medium for video conferencing, but I would recommend that your friend tries to ratelimit their transmissions.
The reason why your latency is higher than you expect, is because you expect latency to be equivalent to the round-trip time from earth to GEO and back. However, most of the time your UT will not be with the shortest line-of-sight, meaning the distance is longer. Also, the satellite merely acts as a smart mirror, and the traffic must still be backhauled from a groundstation to a processing location. They can also be several tens of milliseconds away. Then they must follow their normal path along the internet.
Ping me off-list if you have specific questions.
Thanks,
Sabri
Hi,
Hi,
Where I worked, phy transmissions are scheduled based on tokens. A UT
must have a token to transmit data. If there is no congestion, a
token will be available and the UT or ground station may transmit.
Congestion does not need to exist in the ground network or even the
transponder. It can even be in the spectrum of that geographical
area.
Interesting. So basically as Mel said, over-sold network. 
To overcome the latency,
Latency (AFAIU) is not really his primary issue. it's the lack of
consistency in bandwidth. Periods of a second or two even where there
is no transmission of anything at all followed by a second or two of
transmission bursting even beyond his subscribed "rate". This effects
his subscribed rate but in a really bad way for real-time traffic such
as live/two-way video. He'd much, much more rather get a consistent
pipe at his prescribed rate rather than an average of it over longer
periods of time because then the codec would not have be encoding for
those super bad periods of time where there are 1-2 seconds of no
bandwidth at all.
Satellite is obviously not the optimal medium for video conferencing,
Indeed.
but I would recommend that your friend tries to ratelimit their
transmissions.
He doesn't need to. The over-congested network is doing that for him.
In any case, I don't know that he has any way to put a rate limit
on the tools he is using.
The reason why your latency is higher than you expect,
It actually isn't. It's nowhere near as high as I had come to
(anecdotally -- I'd never had reason to do the math on the latency
before now) believe it would be.
Fortunately he might be a candidate for Xplornet (or others') WISP
services. Hopefully they are a bit more stable.
Cheers,
b.
It’s not really oversold bandwidth. It’s just that the turnaround time for a bolus of data is too long for two-way video conferencing to be smooth or reliable. It’s like video conferencing using post cards 
-mel
This is pretty typical of consumer VSAT and such. You can of course get better performance...if you're willing to pay for it. If you find the right carrier/re-seller, you can perhaps find one whose "system" (to include ground station, terminals, and smarts on the bird) can respect DSCP and flag at least your voice traffic appropriately (probably EF) to perhaps lower the jitter and make conferencing more palatable. Finding competent folks at a typical consumer provider's helpdesk to talk to about such things is probably the limiting factor. The higher up the foodchain you go towards the folks who "own" the spectrum rather than re-sell will perhaps get you more luck on something like this though probably also at higher MRC.
Unfortunately, it's just what happens when you spread an already limited resource (transponder bandwidth) out over essentially an entire continent or at least substantial portions of it. Imagine if you had a cable provider with a single node for an entire, say, US state.
Are consumer satellite provider networks TDD or FDD? I guess I had assumed the latter, but maybe not?
Assuming FDD, I don't see why the downstream needs to necessarily have problems with extremely long user-data-striping periods unless there's some factor I am just totally not considering.
The upstream is of course more of a problem. I assume it's TDMA, and the terminals have imperfect clocks.
Could be interesting to do one-way-delay measurements in parallel to see how the packets are travelling
and where and how big the queues are (Bufferbloat).
Likeways if you have a unix system you could look at tcp statistics
to see if you have packet retransmits (netstat -s).
As previously noted TDMA between uploading or even ACK'ing users could be a bottleneck for the download speed.
Olav
Except that videoconferencing is just the victim of the problem, and
the problem is bursty bandwidth not latency. In fact, the back-and-
forth of conversation is actually surprisingly decent for satellite.
Not as much "talking over" as I would have suspected.
But put the victim application aside, the real data is in the iperf3
results I posted, demonstrating how bursty the throughput is. The
problem with that of course is that the "lowest" bandwidth "valleys"
becomes the "constant bandwidth" that the codec uses rather than the
average -- which of course cannot be used for real-time VC.
Cheers,
b.