Experience with Open Source load balancers?

Greetings all.

I've been tasked with comparing the use of open source load balancing software against commercially available off the shelf hardware such as F5, which is what we currently use. We use the load balancers for traditional load balancing, full proxy for http/ssl traffic, ssl termination and certificate management, ssl and http header manipulation, nat, high availability of the physical hardware and stateful failover of the tcp sessions. These units will be placed at the customer prem supporting our applications and services and we'll need to support them accordingly.

Now my "knee jerk" reaction to this is that it's a really bad idea. It is the heart and soul of our data center network after all. However, once I started to think about it I realized that I hadn't had any real experience with this solution beyond tinkering with it at home and reading about it in years past.

Can anyone offer any operational insight and real world experiences with these solutions?

TIA, replies off list are welcomed.

Regards,

Bryan

S/W vs H/W is really a question rooted in performance and feature
needs vs cost... weigh your options carefully.

We used Pound (http://www.apsis.ch/pound) on a couple of FreeBSD servers
some years ago.

Configuration is simple and the software has lots of good and interesting
features.

The only problem was that always our traffic had a spike, serving pages
through it became a nightmare.

Eventually we ended up buying a couple of Foundry/Brocade load balancers
(Server Iron).

I don't know what is software's current development state but if they
managed to solve those performance issues it would be an interesting choice,
if you really want to go that way.

HTH

Load balancers are _both_ hardware and software.
There's really no such thing as a load balancer solution
that is just hardware or one that's just software (aka Firmware);
it does not exist -- real load balancers are both hardware and software.

And all software has bugs; some of which might (some day) disrupt the intended
operation.

There are packaged solutions that are integrated hardware and software
sold as a product.

There are do-it-yourself solutions that are off-the-shelf hardware,
and software of your choice, including open source options on commodity
hardware, that require manual selection of the software programs used
to facilitate load balancing functions, and maintain them in case of
exceptions,
hardware failures, or other unexpected exceptions.

Both packaged and custom built have their own advantages and disadvantages,
there are tradeoffs, and no universal best; they both meet different
sets of requirements.

Devices packaged by a load balancer manufacturer; usually have
things like user interfaces, instruction manuals, configuration
guidance, and warranty/support for the entire device, both hardware
and software.

In addition, the whole thing is tested by the manufacturers as one product,
and there may be specially integrated hardware functions supported
by the load balancer, that are not found in commodity hardware
(e.g. specialized crypto processors for RSA/AES/SSL acceleration,
ASICs, etc).

The manufacturer's work to deliver their product comes at a high price, so
taking a do-it-yourself with open source software will have much lower prices
paid to the vendor: instead of paying a manufacturer for a product.

Except, if you actually went through the same effort as the manufacturer
of the product, it might be more expensive than buying the product,
so chances are you will do less, and have a less refined result.

Building a load balancer with cheap hardware and open source software,
puts you in the manufacturer's seat.

This provides a huge amount of flexibility, but also adds complexity,
and gives you a lot of choices about what software to put in the box,
and how to set up each package.

Your engineering team may need a great deal of familiarity with both
the software and hardware, and a lot of patience to achieve
performance equivalent to an integrated unit.

Your exact choices of software and package versions for load balancing
or high availability, might (or might not) have been tested by someone
with a similar scenario, on similar hardware.

In case something goes wrong, you won't be getting a replacement unit
overnighted in by the manufacturer, ready to plug in and load a 5kb config.
You have only the troubleshooting and configuration guidance you yourself
developed, before/during use of that, so if an unexpected condition arises,
chances are you won't have as much guidance for troubleshooting or likely
causes, as a vendor would.

Requiring different procedures for dealing with a failure or
malfunctioning of the
load balancing.

In case you do have hardware/software support, for a commodity
hardware solution,
you may find your vendors pointing fingers at one another.

Ø I have had great success with ha proxy http://haproxy.1wt.eu/ -Randy

Can you elaborate on your config?

Honestly I think to get *all* those features you're much better off
with commercial solutions like the ones you're already using from F5,
or something from Cisco, Coyote Point, Brocade, or others. You can
absolutely put together a solution based on any number of open source
products, but you won't get the single integrated front end for
management and configuration that any of the commercial options will
provide, you may be missing features, and ultimately, you're on the
hook for making it work. In particular the stateful failover has been
problematic in open source solutions in my experience. They've come a
VERY long way, but it is a hard problem to tackle.

I've worked with open source and commercial solutions, and while the
open source systems were almost always far more flexible, and cheaper
up front, they certainly required more work to get going.. Once setup
and running though both types of solutions had pretty equal amounts of
maintenance, with the commercial solutions requiring somewhat less
time/babysitting for upgrades and to enable or use new features or
functionality.

We've use Linux LVS for many many years with success.
http://www.linuxvirtualserver.org/

> Greetings all.
>
> I've been tasked with comparing the use of open source load balancing software against commercially available off the shelf hardware such as F5, which is what we currently use. We use the load balancers for traditional load balancing, full proxy for http/ssl traffic, ssl termination and certificate management, ssl and http header manipulation, nat, high availability of the physical hardware and stateful failover of the tcp sessions. These units will be placed at the customer prem supporting our applications and services and we'll need to support them accordingly.
>
> Now my "knee jerk" reaction to this is that it's a really bad idea. It is the heart and soul of our data center network after all. However, once I started to think about it I realized that I hadn't had any real experience with this solution beyond tinkering with it at home and reading about it in years past.
>
> Can anyone offer any operational insight and real world experiences with these solutions?

Honestly I think to get *all* those features you're much better off
with commercial solutions like the ones you're already using from F5,
or something from Cisco, Coyote Point, Brocade, or others. You can
absolutely put together a solution based on any number of open source
products, but you won't get the single integrated front end for
management and configuration that any of the commercial options will
provide, you may be missing features, and ultimately, you're on the
hook for making it work. In particular the stateful failover has been
problematic in open source solutions in my experience. They've come a
VERY long way, but it is a hard problem to tackle.

+1. I think the list of features covers more than just one FOSS project.

Whilst I've had no end of good experiences using LVS (as some others
have mentioned), I wouldn't expect it to do all that is requested in the
original post. At least, not by itself.

I've worked with open source and commercial solutions, and while the
open source systems were almost always far more flexible, and cheaper
up front, they certainly required more work to get going.. Once setup
and running though both types of solutions had pretty equal amounts of
maintenance, with the commercial solutions requiring somewhat less
time/babysitting for upgrades and to enable or use new features or
functionality.

I worry far more about upgrades to proprietary appliances (where it's
often the whole system image), than I do about a few package updates on
a Linux machine (followed by a service restart, or two).

But still, pretty well worded. :slight_smile:

Tom

Can't speak for other brands these days but F5s have two hard disks in them. You can upgrade the software on the hot-spare, boot off that and confirm everything is working. If it isn't you can just switch back.

Paul

Greetings all.

I've been tasked with comparing the use of open source load balancing software against commercially available off the shelf hardware such as F5, which is what we currently use. We use the load balancers for traditional load balancing, full proxy for http/ssl traffic, ssl termination and certificate management, ssl and http header manipulation, nat, high availability of the physical hardware and stateful failover of the tcp sessions. These units will be placed at the customer prem supporting our applications and services and we'll need to support them accordingly.

Now my "knee jerk" reaction to this is that it's a really bad idea. It is the heart and soul of our data center network after all. However, once I started to think about it I realized that I hadn't had any real experience with this solution beyond tinkering with it at home and reading about it in years past.

Can anyone offer any operational insight and real world experiences with these solutions?

I've used LVS and other Open Source solutions in the past. As others
have alluded to, these require knowledge and experience with the
underlying OS and network stack that's often lacking in many
organizations. A good hybrid solution which implements all (I think) of
your requirements is Zeus (http://www.zeus.com/) It's a software
solution which you can deploy on your own hardware. It's been very
solid in my experience. You can deploy the software in a clustered
configuration if needed, though I've only used it in an HA pair.

LaDerrick

Just make sure the DNS components return valid responses to AAAA
queries as well as valid responses to A queries. Many load balancers
get this wrong. They return NXDOMAIN instead of NOERROR, they drop
AAAA queries, they don't return CNAMEs when the A response returns
a CNAME, they return the wrong SOA record (doesn't match the zone
delegated to the box).

Better still would be for them to return AAAA records but until one
is ready to do that the negative responses need to be correct.

If they are returning AAAA queries check NS, SOA, TXT and MX responses
for similar errors. AAAA is just more visible as browsers make AAAA
queries and the others are done in the background.

Mark

[snip]

Better still would be for them to return AAAA records but until one
is ready to do that the negative responses need to be correct.

Hm... better would be for load balancers operate transparently at Layer 3 and
not tamper with the contents of answers from proper DNS servers.

Eating traffic based on application content, or turning NOERROR,
0 matches into NXDOMAIN is seriously f***'ed up.

I look forward to more domains having DS records published by TLDs w/
signed zones...
and possibly browsers displaying warnings trying to visit HTTPS
domains without a signed zone.

perhaps load balancers/middle box manufacturers will start to become a
little bit more honest
in what they do with DNS traffic :slight_smile:

+1 for Zeus. Use it in our production network with great success.
Magnitudes cheaper than a solution from F5, and doesn't hide the inner
workings of the product if you want to do some things outside the
scope of support.
Zeus also does licensing just based on throughput, not arbitrary
transactions per second like F5 does/did. If you're hardware can push
the traffic, theres no limitations on the number of transactions or
sessions.

I'll pile on here too - there's very little of Mozilla's web infrastructure that isn't behind Zeus.

I've worked with everything over the years. BigIP, CSS, CSM, ACE (blows),
NetScaler, say when. I've been thru a few RFPs and bake offs and also
evaluated open source options.

1. If you are looking for simple round robin load balancing with decent load
capabilities then there are several open source options in this thread that
may work. As long as you understand that you are going to be expected to
support them.....

2. If you are pushing features. SSL termination. Header rewrites. Payload
inspection (NetScaler does application firewalling on the same appliance).
Or other complexities and you are having to deal with enterprise traffic
volume you might be better off with one of the big vendors. Applications
these days are more and more complicated and a high end load balancer with a
stable feature set can often rescue your AppDev team and make you a hero.

Recommend: F5 and Citrix Netscaler. If you are looking to combine your L7 FW
into your LB then you might lean towards NetScaler. If you are looking at
seperating those duties you can look at F5. IRules (F5) are the bomb.

-Hammer-

"I was a normal American nerd."
-Jack Herer

Recommend: F5 and Citrix Netscaler. If you are looking to combine your L7 FW into your LB then you might lean towards NetScaler. If you are looking at seperating those duties you can look at F5. IRules (F5) are the bomb.

Except that under (Mozilla) load, Netscaler fell apart. F5, at the time, could not handle the logging rate I required. Mozilla load is typically defined as high connection rate, low traffic per connection and mostly all SSL.

During the Firefox 4 release, we peaked globally at 12Gbps, a significant portion of which was pushed out of three Zeus clusters with L7 rules and some non-trivial traffic script rules and a heck of a lot of content caching. Of all the systems seeing increased usage during the Fx4 release, this wasn't where my worries were :slight_smile:

A slightly older post,

http://blog.mozilla.com/mrz/2008/12/04/load-balancer-performance-issues-fxfeedsmozillaorg-versioncheck/

We're using both an F5 BigIP as well as Nginx (open source software) in a
production environment.

They both have their merits, but when we recently came under some advanced
DDoSes (slowloris, slow POST, and more), we couldn't process certain types
of layer 7 insepction/modification because it was too heavy for the F5 to
handle. Nginx was more cost effective because we could scale laterally with
cheap commodity hardware.

This isn't a knock on the BigIP though; it's a much better piece of
equipment, has commercial support, and a fantastic web interface. With Nginx
you might find yourself compiling modules in by hand and writing config
files.

Ultimately, the open source solution is going to stand the test of time
better. It all depends on who's paying the bills, and what your time is
worth. Nginx was specifically worth the effort for us because we had unique
traffic demands that change too quickly for a commercial solution.

Thanks,
Andreas

Mattew,
      We run high volume SSL but not nearly the 12Gbps you are talking about
so that hasn't been an issue for us. Thanks for the information. Looks like
the Citrix ANG rep owes me another lunch to explain himself. :slight_smile:

I'm gonna do some research on NGINX...

-Hammer-

"I was a normal American nerd."
-Jack Herer