IPv6 Server Load Balancing - DSR

Mailman List Mirror (Read Only)
1

Dear Colleagues,

I've been scratching my head over this for the past couple of months and have come up with blanks, and several weeks of scouring various resources on the net have not yielded anything more fruitful.

I'm looking at server load balancing for IPv6 and specifically need DSR (direct server return). Additionally, I need to support both TCP and UDP.

I have evaluated a number of different load balancing solutions purporting to support IPv6 with varying results (and costs)...

a few examples:

F5 : according to marketing blurb supposedly supports IPv6 in NAT and DSR mode, both UDP and TCP. Their documentation, however, has no mention of IPv6 capability. Other disadvantage = cost...

Brocade/Foundry: Similar situation to F5

Zeus: IPv6 in NAT only, and even more expensive than F5.

Exceliance Aloha: IPv6 in NAT only, and ONLY in TCP (no UDP)

A few others also tested... including LVM/HAProxy (same situation as Exceliance Aloha), and others...

Finally in the end, only OpenSolaris ILB seems to put all the checks in the right boxes for my requirements. But there is still a problem.

1. IPv4 TCP and UDP work fine in NAT, Half-NAT, and DSR
2. IPv6 I've managed to get working, complete with healthchecks, in TCP and UDP in NAT only although the documentation stipulates that DSR is also possible (but not HalfNAT for the moment).

The problem with #2:

Using the same server farm behind, but in dual-stack, and configuring ILB for TCP and UDP services using NAT, everything is fine. If I configure it for DSR, immediately it fails (both with and without healthchecks). Although from the ILB host itself, I can certainly do a manual heathcheck.. (e.g. telnet <server_real_ipv6_addr> 80 and do GET / or HEAD / with no problems. Using ARP poisoning from the shell I can also perform the healthcheck on the real server via telnet using the virtual ip.

The servers are configured normally for DSR.. with the virtual IP attached to a local dummy or loopback interface, and with IPv4 DSR works fine.

Nevertheless, I've been unable to get DSR working with ILB -- and have found absolutely nothing around the net with working examples of IPv6 SLB with DSR. NAT mode works fine, but the real server loses visibility of the end user's IP as the requests come from the internal IP of the ILB host, and with a system that uses client IP address as part of the various criteria for session tracking, it creates a few problems...

I am suspecting that the issue may be related to ND, as the behaviour is similar to the old story with doing DSR on real-servers using older linux distributions that do not by default disable proxy-ARP replies by the server for IP addresses on dummy or loopback interfaces, and of course the proxy ARP causes confusion to the load balancer and breaks the whole thing. But the real servers are recent Debian distributions, and both ipv4 ARP and ipv6 ND is disabled on the dummy interfaces, as is proxy ARP.

Would anyone happen to have any useful pointers, tips, or other on how to resolve the issue?

Many thanks in advance.

Leland

2

This is easily done with OpenBSD. See here for starters:

http://www.undeadly.org/cgi?action=article&sid=20080617010016

Simon

3

Hi Leland,

Seems that hardware vendors doesn't like IPv6... for load balancing.

I had a look to relayd from OpenBSD, and it seems this can be used a LoadBalancing with DSR... Even if they don't recommand this ...

Maybe the is is the time to move from hardware / closed solutions to open ones.. ?

Xavier

4

OpenSolaris ILB is open solution :wink:

but yea, that's what we've started looking at -- hence LVM / HAProxy as well.. (though LVM is IPv4 only, and HAProxy is NAT only for IPv6)

does relayd support UDP as well as TCP or is it layer7 only like HAProxy ?

In the case of ILB, I'm not convinced that it's a problem with the LB itself, but rather the idiosyncrasies of ND in IPv6 that is causing the problem.. but I may be wrong... at any rate, something's amiss ...

cheers,

Leland

5

Hi Leland,

OpenSolaris ILB is open solution :wink:

but yea, that's what we've started looking at -- hence LVM / HAProxy as well.. (though LVM is IPv4 only, and HAProxy is NAT only for IPv6)

does relayd support UDP as well as TCP or is it layer7 only like HAProxy ?

It does everything... :slight_smile: L2 -> L7...

In the case of ILB, I'm not convinced that it's a problem with the LB itself, but rather the idiosyncrasies of ND in IPv6 that is causing the problem.. but I may be wrong... at any rate, something's amiss ...

Maybe on some setup you should desactivate ND...

Xavier

6

Yea.. well. .that's the point... can't deactivate ND on the real interface of the server as that's required for the server itself.. but it, according to the kernel, deactivated on the dummy interface carrying the virtual IP of the server farm... exactly as is done for IPv4 and ARP manipulation.

Hmmmmm...

L.

7

I'm looking at server load balancing for IPv6 and specifically need
DSR (direct server return). Additionally, I need to support both TCP
and UDP.

This is easily done with OpenBSD. See here for starters:

Direct Server Return support in OpenBSD

And FreeBSD:

8

IPVS has had IPv6 support for a while:

http://www.mindbasket.com/ipvs/

We're using it on our mirror site, http://ftp.heanet.ie, with DSR for
http, ftp and rsync load balancing.

rg

9

If you're putting the DSR address on an interface other than loopback, you probably need to turn of DAD on the interface with the DSR address otherwise DAD
will shut down that address on the interface when it sees other servers with the same address. Sometimes it will shut down all but one, sometimes it will
shut down all.

Owen

10

Hi Owen,

The DSR address is indeed on a loopback in our case.

lo Link encap:Local Loopback
          inet6 addr: ::1/128 Scope:Host
          inet6 addr: xxxx:xxxx:x:xxxx::xx/128 Scope:Global

The mystery continues...

Leland

11

Brocade basically sucks when it comes to loadbalancing IPv6, the old serveriron platform is EOL and a complete mess which offers some IPv6 support, but not much. The new ADX platform seems to be in a pre-alfa stage at the moment. So normally I would say stand clear, however we do run a (larger) usenet platform on v6 which uses DSR and that part works on the serveriron, running a pre-relase of the 11.0.0f software.

Must admit we don't do anything fancy, it's all unprotected and statically routed, ACLs are all done on the reals and on the Juniper in front of the serveriron etc. But it seems to hold, haven't heard any complains yet. But be warned this is a really specifc subset of features. For regular operations like web we still have loads and loads of issues.

Basically the other choice is F5. We are busy setting up a PoC with A10, who claim IPv6 support. Hopefully in a few weeks time they can be added to the list of potential suppliers. Other then these two I haven't come across any dedicated stuff and what's left is Linux/BSD based solutions.

MarcoH

12

Well, Frankly our "culture" is very much open source, so if we can find something along those lines, then it would be preferred. (Hence looking at OpenSolaris and ILB). -- having said that, we do have both F5 and Foundry kit here, but it's all pre-IPv6 so doesn't have the support built in. Not really looking to replace what is in existence already for IPv4 with something new to do both, so really that reinforces the open-source avenue really.

I think the biggest problem is really the DSR aspect for IPv6, since the OS/ILB solution works perfectly in NAT mode, and DSR works perfectly with IPv4 on this solution. So either I'm missing something critical on the "real" server configuration, or ILB's implementation of DSR for IPv6 doesn't really work. The "virtual" IP is bound to loopback on the real servers, exactly the same was as for IPv4. So other than something quirky going on with ND, or simply ILB not correctly rewriting the L2 frame, or there's something else more sinister afoot that I'm unable to put my finger on.

Back to the drawing board... :slight_smile:

Thanks,

Leland