My company is looking for ways to improve throughput for data transfers
between individual servers. We’re exploring the creation of Etherchannels
using multiple server NICs, but Etherchannel seems to have the limitation of
not supporting per-packet load-balancing, therefore limiting traffic between
two individual hosts to 1 Gig.
In most of my research, I’ve seen 10-GigE used for traffic aggregation and
for the “unified fabric” solution that Cisco and others are pushing. I’m
interested in knowing if any of you have attempted to implement 10-GigE at
the host level to improve network throughput between individual servers and
what your experience was in doing so.
Thanks in advance,
Jason
As long as it's not a single connection that you're looking to get over 1Gb, etherchannel should actually work. It uses a hash based on (I believe) source and destination IP and port, so it should roughly balance connections between the servers. The other option, if you're using Linux, is to use balance-rr mode on the bonding driver. This should deliver per-packet balancing and the switch doesn't have to know anything about the bonding. Documentation for the bonding driver is here:
http://www.cyberciti.biz/howto/question/static/linux-ethernet-bonding-driver-howto.php
Once upon a time, Alex Thurlow <alex@blastro.com> said:
As long as it's not a single connection that you're looking to get over
1Gb, etherchannel should actually work. It uses a hash based on (I
believe) source and destination IP and port, so it should roughly
balance connections between the servers.
That depends on the devices on each end. For example, some switches can
only hash on MAC addresses, some can look at IPs, and some can look at
ports.
It's also host dependent. The switch will hash based on the algorithms it supports, however it can be asymmetrical if the host doesn't support the same ones. Most host OS's don't hash outbound to the switch based on layer 4 or above. Most only hash at layer 2.
There are 10GE cards for most hardware and OS platforms. Getting them to run at a fraction of that speed depends on application and IP stack tuning. Even then, there are significant bottlenecks. That's one reason Infiniband for HPC has taken off.
My company is looking for ways to improve throughput for data transfers
between individual servers. We�re exploring the creation of Etherchannels
using multiple server NICs, but Etherchannel seems to have the limitation of
not supporting per-packet load-balancing, therefore limiting traffic between
two individual hosts to 1 Gig.
Have you thought about Infiniband? Dual 10 gig cards cost about $50 and 24 port switches about $1200 on ebay. Infiniband has just a fraction of the latency of ethernet (even 10 get eth). You get the lowest latency if your application supports Infiniband, but if not you can run IP over IB.
<>
Nathan Stratton CTO, BlinkMind, Inc.
nathan at robotics.net nathan at blinkmind.com
http://www.robotics.net http://www.blinkmind.com
Athens GA, tried to call in a ticket (Metro Ethernet) and was told a
master ticket was already in place for my circuits. Other than the
ticket # they wouldn't give me any details. Anybody know anything?
--p
The 10G cards from Neterion (http://www.neterion.com/products/xframeE.html) perform extremely well.
Much of your potential limiting rate will be in the CPU and networking resources.
I agree with many of the other folks that etherchannel should work exceedingly well, but if you want to consolidate, there are many choices for single-system 10G ethernet cards.
-alan
Jason Shoemaker wrote:
My company is looking for ways to improve throughput for data transfers
between individual servers. We’re exploring the creation of Etherchannels
using multiple server NICs, but Etherchannel seems to have the limitation
of
not supporting per-packet load-balancing, therefore limiting traffic
between
two individual hosts to 1 Gig.
Have you thought about Infiniband? Dual 10 gig cards cost about $50 and 24
port switches about $1200 on ebay. Infiniband has just a fraction of the
latency of ethernet (even 10 get eth). You get the lowest latency if your
application supports Infiniband, but if not you can run IP over IB.
Infiniband is dying technology..