High throughput switches...

I'm looking for people's experiences with Voltaire switches in general
and the Vantage 6048 in particular.

We'd like to use a central switch and use 16 10g ports trunked via LACP
to two other switches and a SAN (to clarify, the central switch would
have three data channels, each one consisting of 16 trunked 10g ports,
the two downstream switches would have 10gb clients hanging off of them
and I'm not sure how the SAN is supposed to use 16 10g ports; they
haven't felt the need to give me specs on it, but for now I'm assuming
it either has a built in switch fabric that supports LACP or will have
another switch dedicated to it can handle different L1 media). Assuming
we can bond that many channels (which I'm not sure about, but none of
the docs I've read on the IEEE protocol involved indicate a limit on the
number ports that can be aggregated, and the manufacturer docs don't
mention it either), how realistic is the expectation of getting a full
160gb throughput? Are the switches in question up to the task?

This is a research project we're building for a customer, so I'm trying
to manage expectations, but this isn't the sort of thing I've personally
ever built before and I'm hoping someone here has done something close
enough and is willing to share experiences.



Bonding usually involves some sort of flow-based hashing mechanism for
balancing across the links. The logic behind it being that reordering TCP
packets (among other things) negates the benefit of the aggregation in CPU
costs and retransmissions. So it's important to keep one flow (really one
direction of one flow) on one link to prevent out of order delivery.

So if you want to utilize that bandwidth, you need a lot of flows. I
remember numbers being thrown around that you need at least 1000 flows to
balance two paths. I don't know how many you'd need for that to scale to 16
links. Obviously if this is SAN traffic, the usage patterns are going to be
completely different from desktop, so ymmv.

Also, this information may be out of date and/or irrelevant now, but it's
something I've run into before with link aggregation.