Colleagues,
I'm looking for a recommendation on a smallish 10G Ethernet switch for a
small virtualization/SAN implementation (4-5 hosts, 2 SAN boxes) over
iSCSI with some legacy boxes on GigE.
Preferably
- 8-16 10G ports
- several GigE ports for legacy GigE hosts or cross connect to a legacy
GigE switch
- preferably not a large chassis based solution with blades
The hosts aren't going to be driving full line rate, nor the SAN boxes
providing full line rate, but their offered loads will definitely exceed
1Gbps. Assessing whether it is better to go 10G now vs. multi-pathing
with quad GigE cards. Trying to find the best solution for > 1G on a
trunk and < $50K per box.
Any recommendations appreciated.
Thanks
EKG
This topic has been covered completely and often on the Beowulf list.
http://www.beowulf.org/pipermail/beowulf/ or do a site search via your
friendly search engine.
If you're looking at sub 50k, the Nexus 5k isn't a terrible option.
It gives you 32 10Gig SFP slots for ~$25,000 or less if you don't mind
used from ebay.
Also note that new devices hit the market everyday. For example
Supermicro has an offering at
http://www.supermicro.com/products/accessories/Networking/SSE-X24S.cfm
which at the very least will be supported forever.
You may want to take a look at the Brocade VDX 6720, it provides 16 10gb ports,
with 8 ports on demand with addl license.
They are very reasonable, esp. if you only need 16 ports. Maintenance costs are less
than cisco.
Certainly the days of doing NxGE to servers should be behind us.
There are many good 10GE switch offerings. The Juniper QFX and Arista
(insert one of three good product lines here) have been excellent for
us. We like them for different reasons -- Arista is quite good if you
want to integrate with a provisioning system; QFX is our choice when
most provisioning is done manually. Both are way under $50k per box.
The biggest difference between the TOR-style switches and chassis
offerings, aside from the obvious, is buffers. All the TOR-type 10G
switches have really small buffers and that can be a performance issue
for iSCSI when utilization is high. Most of the chassis-type switches
have very generous buffers so you do not run into problems with
micro-bursting, etc. The vendors will all tell you about lossless
ethernet, flow control, etc. and that crap sounds great on paper. Try
making it actually work. You'll want those days of your life back. 
$0.02.
The biggest difference between the TOR-style switches and chassis
offerings, aside from the obvious, is buffers. All the TOR-type 10G
switches have really small buffers and that can be a performance issue
for iSCSI when utilization is high
not particularly when utilisation is high, but in situations where
congestion occurs, e.g. when you either have a high write load from
multiple clients to a single server, or if you're down-shifting from a 10G
server to 1G clients or something.
The vendors will all tell you about lossless ethernet, flow control,
etc. and that crap sounds great on paper. Try making it actually work.
flow control on a switch port can lead to hol blocking, which is bad bad
news - guaranteed to trash multi-access network performance. Some vendors
actually push this as a feature. I don't completely understand why, but
maybe it has something to do with customers mistakenly believing that it
will make their lives better. People believe in all sorts of odd
superstitions though: black cats, spilling salt, having full features on
ethernet LAGs, vendor marketing blurb, fear of the number 13, etc. All
very odd stuff.
Nick
I'm curious if anyone here has experience with the Extreme Summit X650
or X670 switches? I'm also currently looking for 10GBASE-T switch for
my first SAN, but one with 48 ports. The price for X670V-48T seems
reasonable, but I have no experience with 10 GbE over copper, so it's
difficult to figure out what criteria to use (other than price) when
comparing Extreme, Arista, Dell, and others. Any tips?
- Max
Kevin, no thank you, I did not start this thread. If I ever need
products I reach out to my contacts at each manufacturer or
distributor. It would be much less embarrassing for you if the
website in your signature actually finished loading the images
containing the text for your company.
I have used the summit x650s for cloud and they work fine for public and
SAN traffic on the same switch. I have seen then as low as 7k if you can
work with extreme directly. The only downside being the configuration for
large VLAN amounts.
Also netgear makes the xsm7224s which does everything the summit does aside
from only supporting 1024 vlans. I have a pair of these and they work very
well.
You can look ar Brocade TurboIron 24. It has 24 ports of 1/10G
depending on the SFP you put in.
The Juniper ex4500/4550 could work, small chassis, can be made part of a
virtual chassis. Works well in an enterprise setup but can cause
configuration headaches if within a service provider environment where
vlans need to be translated.
Darius
Dell Force10 S4810 is a decent ToR switch: 48 dual-speed 1/10GbE (SFP+) ports and four 40GbE (QSFP+) uplinks
Peter
The page does not render at all for me -- it is completely blank once the gratuitous code stripper is finished with it. I presume security has to be disabled to view it -- and I'm not going to do that. As a matter of policy I never deal with anyone who is so incompetent as to create web sites like this. Generally I have found that such companies product is of the same quality as the web page.