Hello NANOG,
We are thinking of utilizing some InfiniBand stuff for some specific
application in our data centres. We are new to InfiniBand however so we
want to get some equipment and see if it does what we need. Does anyone
know of a good vendor in the US? East or West coast, doesn't matter. If
anyone has any good advice or information about InfiniBand that would be
nice to hear too as we are totally new to it at present.
Hi Landon:
We deliver Infiniband based servers and switches. We have been working with Infiniband for many years already. What are you looking for?
Alex
www.pssclabs.com
VAR or Manufacturer? Mellanox are essentially the defacto standard for
IB switches and HCAs.
Oh by vendor I mean VAR I guess. Mostly I'm also wondering how an IB
network handles IPoIB and how one uses IB with a gateway to layer 3
Ethernet switches or edge routers. If anyone has any resources that
provide details on how this works and how ethernet VLANs are handled I'd
appreciate it.
My limited IB experience has been that the IB switch acts much like a dumb ethernet switch, caring only about which IB hardware addresses are reachable via which port. Routing between IPoIB and IP over ethernet can be done by any host with interfaces on both networks and IP forwarding enabled. In our setups, we've used IPoIB, but with 1918 addresses and not routed beyond the IB network.
IPoIB looks more like an application than a network protocol to Infiniband.
The IB fabric doesn't have a concept of broadcast, so ARP works much
differently than it does in IPv4/ethernet world - basically an all-nodes
multicast group handles the distribution of ARP messages. That said, the ib
drivers that come with redhat/centos are pretty good, and you can always
download the official OFED drivers from the OFA at
https://www.openfabrics.org/linux-sources.html if the stuff in your linux
distribution is missing something.
I've set up IPoIB routers running 10G NICs on the ethernet side and QDR
HCAs on the IB side, using quagga to plug in to the rest of my OSPF
network, and it works fine. Basically you just need to set up quagga like
you would if you were going to turn a linux box into an ethernet router and
don't worry about the fact that it's actually IB on one side of the router
- your network statements, etc., in OSPF in quagga won't change at all.
You'll find that some things in IB have no equivalent to ethernet. For
example, if you want to have gateway redundancy for traffic exiting the IB
fabric, your first instinct will be to look for VRRP for IB, but you won't
find it, because of the ARP differences I talked about above. To get around
this you can set up linux-ha or some other type of heartbeat arrangement
and bring up a virtual IP on the active gateway, which can be shifted over
to the standby gateway when the ha scripts detect a problem. Some vendors
also have proprietary solutions to this problem but they tend to be
expensive.
So, I'd say, read up on quagga and give that a try, and I think you'll find
that as long as the IB drivers are up to snuff (the sminfo command returns
valid results, etc.) it'll pretty much just work for you. I'm also happy to
discuss more offline if you prefer.
Tom
Tom
I wanted to bring attention to the following draft proposal from
Mellanox to export traffic information from InfiniBand switches:
http://sflow.org/draft_sflow_infiniband.txt
If you are an InfiniBand user, this is a great opportunity to think
about the types of metrics that you woud want from your switches in
order to better understand performance. The operational sensibility
that the NANOG audience brings is particularly valuable.
Comments on the proposal are welcome on the sFlow discussion group: