Date: Thu, 12 Jun 2008 19:26:56 -0400
From: Robert Boyle <robert@tellurian.com>
>I'm looking for input on the best practices for sending large files
>over a long fat pipe between facilities (gigabit private circuit, ~20ms RTT).
>I'd like to avoid modifying TCP windows and options on end hosts
>where possible (I have a lot of them). I've seen products that work
>as "transfer stations" using "reliable UDP" to get around the
>windowing problem.
>
>I'm thinking of setting up servers with optimized TCP settings to
>push big files around data centers but I'm curious to know how
>others deal with LFN+large transfers.
In our experience, you can't get to line speed with over 20-30ms of
latency using TCP regardless of how much you tweak it. We transfer
files across the US with 60-70ms at line speeds with UDP based file
transfer programs. There are a number of open source projects out
there designed for this purpose.
Clearly you have failed to try very hard or to check into what others
have done. We routinely move data at MUCH higher rates over TCP at
latencies over 50 ms. one way (>100 ms. RTT). We find it fairly easy to
move data at over 4 Gbps continuously.
If you can't fill a GE to 80% (800 Mbps) at 30 ms, you really are not
tying very hard. Note: I am talking about a single TCP stream running
for over 5 minutes at a time on tuned systems. Tuning for most modern
network stacks is pretty trivial. Some older stacks (e.g. FreeBSD V6)
are hopeless. I can't speak to how Windows does as I make no use of it
for high-speed bulk transfers.
It's also fairly easy to run multiple parallel TCP streams to completely
fill a 10 Gbps pipe at any distance. This is the preferred method for
moving large volumes of data around the world in the R&E community as
it requires little or no system tuning and is available in several
open-source tools.
Clearly you have failed to try very hard or to check into what others
have done. We routinely move data at MUCH higher rates over TCP at
latencies over 50 ms. one way (>100 ms. RTT). We find it fairly easy to
move data at over 4 Gbps continuously.
That's impressive.
If you can't fill a GE to 80% (800 Mbps) at 30 ms, you really are not
tying very hard. Note: I am talking about a single TCP stream running
for over 5 minutes at a time on tuned systems. Tuning for most modern
network stacks is pretty trivial. Some older stacks (e.g. FreeBSD V6)
are hopeless. I can't speak to how Windows does as I make no use of it
for high-speed bulk transfers.
Let me refine my post then...
In our experience, you can't get to line speed with over 20-30ms of latency using TCP on _Windows_ regardless of how much you tweak it. >99% of the servers in our facilities are Windows based. I should have been more specific.
-Robert
Tellurian Networks - Global Hosting Solutions Since 1995 http://www.tellurian.com | 888-TELLURIAN | 973-300-9211
"Well done is better than well said." - Benjamin Franklin
Clearly you have failed to try very hard or to check into what others
have done. We routinely move data at MUCH higher rates over TCP at
latencies over 50 ms. one way (>100 ms. RTT). We find it fairly easy to
move data at over 4 Gbps continuously.
That's impressive.
If you can't fill a GE to 80% (800 Mbps) at 30 ms, you really are not
tying very hard. Note: I am talking about a single TCP stream running
for over 5 minutes at a time on tuned systems. Tuning for most modern
network stacks is pretty trivial. Some older stacks (e.g. FreeBSD V6)
are hopeless. I can't speak to how Windows does as I make no use of it
for high-speed bulk transfers.
Let me refine my post then...
In our experience, you can't get to line speed with over 20-30ms of latency using TCP on _Windows_ regardless of how much you tweak it. >99% of the servers in our facilities are Windows based. I should have been more specific.
I'll stipulate that I haven't looked too deeply into this problem for Windows systems.
But I can't imagine it would be too hard to put a firewall/proxy (think Socks) and set the FW/proxy to adjust (or use an always on, tuned, tunnel) the TCP settings between the two FW/proxies on either side of the link.
It has reasonably little invasion or reconfiguration and is probably reasonably solid.
"Enable TCP window scaling and time stamps by using the Registry Editor
to browse to location
[HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters]
and add the key
Tcp1323Opts
with value
3"
are "hard". If you think otherwise, pick up the phone, pretend to
work for an ISP Help Desk, and walk someone who doesn't work in IT
through the changes.
Microsoft scatter the tuning information for Windows Xp all across
their website. Some of it is undocumented by Microsoft (and thus may
change without notice). The only saving grace is DrTCP, a third party
application which hides all of the registry detail (and potential for
disaster) under a nice GUI.
Then there's the deliberate nobbling of the TCP implementation,
such as the restriction to ten of connections to Windows Xp SP3.
Apparently you're meant to buy Windows Server if you are running
P2P applications
Windows Vista is a vast improvement over Windows Xp (and I bet that
isn't said of many components of Vista). It has a autotuning TCP with
a 16MB buffer, which makes the defaults fine for ADSL and cable, but
still requires machines at universities to be altered. Vista offers
an alternative TCP congestion control algorithm -- Compound TCP. Not
much is known of the performance attributes of this algorithm, mainly
because I.P claims prevented its incorporation into Linux, the corral
where most TCP algorithm shoot-outs take place.
For what it's worth, I did first-tier for about 4 years, and yes, I had
to walk some people through that... and as long as they weren't the
type to *get frustrated* whild doing things they didn't understand, it
generally went swimmingly.
While I was driving down the interstate.
In my 21 year old *stickshift* BMW.
With a Big Mac in the other hand.
So as long as *you* know how to drive the tools, and you're not in a
hurry, that's not "hard". Just "complex".
