I second the vote for Postfix. I haven't used it with a ton of users, but I
built a large mail cluster for my old employer which was used to send stock
pricing alerts to users that have set up thresholds for certain stocks.

Initially, the cluster was qmail (that was back in 2000). I used qmail
because it's performance was better than anything else I tested, including
postfix. Qmail is a pain to administer, especially when you have a bunch of
them. After getting fed up with qmail, I tested Postfix again, and the
newer versions of it outperformed qmail by a longshot, and I could spawn
many more connections with it without running out of file descriptors.
Adding features to qmail sucks because it's always a patch, and it's usually
something that most other mailservers already have built in. When you
install it and have 10 patches that you need to apply, you need to apply
them in the right order so they don't step on each other. There are some
limits in qmail that you need to patch for in order to make it work in a
very large environment, like the big-concurrency patch and the patch to
increase the size of the queue.

Administration of postfix can all be done with the postconf command, you
don't have to edit config files at all. This makes it great for clusters
because I had a script I wrote that would execute a command across all of
the machines in the cluster. So a config change for every machine involved
grun 'postconf -e whatever = something'
grun 'postfix reload'

I've heard some good things about exim, but in my testing, I was unable to
get the performance I could get with postfix. Maybe it was my fault, I
didn't spend that much time with Exim because the ease of administration
with Postfix was vastly better than any other mailserver I've used.

Postfix rocks. Apple just ripped sendmail out of OS X and replaced it with
Postfix if that says anything.