SendGate: Sendmail Multiple Vulnerabilities (Race Condition DoS, Memory Jumps, Integer Overflow)

Tech details:
Sendmail vulnerabilities were released yesterday. No real public
announcements to speak of to the security community.

SecuriTeam released some data:
"Improper timeout calculation, usage of memory jumps and integer
overflows allow attackers to perfom a race condition DoS on sendmail, and
may also execute arbitrary code."
More here: http://www.securiteam.com/unixfocus/5RP0L0UI0S.html

ISS only reported the Race Condition (DoS?). The Sendmail Advisory
reported the Race Condition DoS, the Memory Jumps and a
"theoretical" Integer Overflow.

To begin with, anyone noticed the memory leak they (Sendmail) silently
patched?
I wonder how many other unreported silently-patched
vulnerabilities are out there?

Second, the Integer Overflow is practical, not theoretical.

ISS reported the Race Condition last mounth. There is NO data available on
when the other vulnerabilities were discovered. Any guesses?

They also patched many non-security related bugs, added checks and more
informative error messages, etc.

Sendmail is, as we know, the most used daemon for SMTP in the world. This
is an International Infrastructure vulnerability and should have been
treated that way. It wasn't. It was handled not only poorly, but
irresponsibly.

Here's what ISS releasing the Race Condition vulnerability has to say:
http://xforce.iss.net/xforce/alerts/id/216
They say it's a remote code execution. They say it's a race condition. No
real data available to speak of. I can't see how it's remotely
exploitable, but well, no details, remember? From what we can see it seems
like a DoS.

Bottom line

(I feel obligated to mention that there's 16 mentions of my name in the Sendmail
release notes, and zero of Gadi's. This of course influences my opinions and
commentary, and possibly Gadi's as well...)

ISS only reported the Race Condition (DoS?). The Sendmail Advisory
reported the Race Condition DoS, the Memory Jumps and a
"theoretical" Integer Overflow.

Well, it *is* mostly a theoretical overflow - for it to work, a site would have to:

a) Override the default value of "O MaxHeadersLength=32768" to something in the
billions *and*

b) Be running on a box with enough RAM/swap to allow a call to malloc(2**31) or
so to succeed.

Then the attacker actually has to (c) *send* some 2 gig of headers.

Well, OK. If you can find somebody still running something pre-8.10.0 (2000/03/01),
you can bypass (a) - still need (b) and (c) though...

To begin with, anyone noticed the memory leak they (Sendmail) silently
patched?

I suspect it's not as silently as you think. If you actually bothered to ask
Claus, he'll probably be able to tell you which RELEASE_NOTES entry the leak
was attached to, or why it wasn't listed (see below).

And it's hard for them to be *too* silent, given that anybody and their pet
llama Ralphie can get the 8.13.5 and 8.13.6 sources and diff them.

There's also the *practical* aspect. Well over a quarter of the actual code
changes were due to the API change of *one* function. You start change-logging
all this in the release notes, important things become easier to overlook...

I wonder how many other unreported silently-patched
vulnerabilities are out there?

It depends how you define "silently patched vulnerabilities". I've found my
share of Sendmail bugs - and 100% of the ones I've reported that ever shipped
in a non-beta version are in the RELEASE_NOTES file. (Those bugs others and I
swatted in new, as-yet-unshipped code in alpha and beta versions are *not* in
there - 8.12's multiple queue support didn't just spring forth essentially
bug-free, for instance. Also, bugs found and fixed by Sendmail Inc employees
aren't listed - although if reported from outside, they will get a "problem
noted by" entry).

A bigger problem is that quite often, what gets fixed is a simple *bug* (for
instance, two I found in 8.12.2). Are they vulnerabilities? Good question.
Some of the "bugs" I've found could possibly be extended into a DoS (for instance,
what I *reported* in 8.12.2 was spontaneous queue-runner hangs due to losing a
timer event - but that *could* be crafted into a DoS if an attacker found a way
to control when things happened to force a timer event loss).

So was that a "silently patched vulnerability"? Or just a plain bug?

Second, the Integer Overflow is practical, not theoretical.

You have an actual exploit that works on a config that isn't especially set
up to allow it to work (i.e. out of the box defaults, and a non-absurd amount
of network traffic)? Or can you at least point out enough details of what
the exploit would have to do to convince me it's not just hand-waving?

In particular, how do you get past the MaxHeadersLength check?

ISS reported the Race Condition last mounth. There is NO data available on
when the other vulnerabilities were discovered. Any guesses?

The code drop for the fix of your "practical integer overflow" happened between
the Alpha0 on Dec 23 and Alpha1 on Feb 12. You'll have to ask either Phil
Brass or Claus when that one got reported.

Sendmail.com's patch is so big they may as well have re-released the whole
program.

I don't know what shipped to sendmail.com's paying customers, but in fact,
sendmail.org *did* "re-release the whole program", as a quick check of
ftp://ftp.sendmail.org/pub/sendmail/ would have revealed....

And in fact, if you actually *diff* the two trees to create a patch, it's not
that large at all:

(numbers fudged very slightly to remove the diff of 2 versions of a Postscript
documentation file (doc/op/op.ps), which by itself was well over half the total
changes):

% diff -ur sendmail-8.13.[56] | wc -l
5253
% diff -ur sendmail-8.13.? | diffstat | grep change
58 files changed, 1411 insertions(+), 886 deletions(-)
% wc sendmail-8.13.6/*/*.[ch] | grep total
122092 394315 2780756 total

Yeah. Whatever. Hell of a lot of code churn there. Almost 2%. A very large
chunk of it looking like:

@@ -4546,10 +4569,12 @@
        {
                if (bitset(MCIF_INHEADER, mci->mci_flags))
                {
- putline("", mci);
+ if (!putline("", mci))
+ goto writeerr;
                        mci->mci_flags &= ~MCIF_INHEADER;
                }
- putline("<<< No Message Collected >>>", mci);
+ if (!putline("<<< No Message Collected >>>", mci))
+ goto writeerr;
                goto endofmessage;
        }

(and another 104 similar fixes of putline alone - that *by itself* is about 30%
of the code changes).

But of course, I'm writing this having actually *looked* at the diff, and having
bothered to ask Claus and Greg what the policy was for listing fixes in the release
notes...

Also, it would help if instead of FUD-mongering, you actually went to Claus
(or asked somebody else to) with *specific suggestions* of how to improve the
process. He may be stubborn about the way he does things, but if you include
specific changes, and show how those changes make *visible* improvements to
the product, he'll listen.

(visible improvements - saying "It Would Be Nice If" doesn't cut it. On the other
hand, replacing the IWBNI with "If X were done, then the security community would
be able to do Y, and the network operations community could do Z, with benefits
A, B, and C" - now *that* might get some traction...)

Here's what ISS releasing the Race Condition vulnerability has to say:
http://xforce.iss.net/xforce/alerts/id/216
They say it's a remote code execution. They say it's a race condition. No
real data available to speak of. I can't see how it's remotely
exploitable, but well, no details, remember? From what we can see it seems
like a DoS.

ISS's Mark Dowd is very clever guy. And if duke says it's exploitable
I would believe him :-). It's an interesting new vector anyway.

But like all timing related attacks, the question is reliability.
Though gossip has it, this one is repeatable with sub-100 attempts
and you get infinite shots at it because even if the process
does die it's a child of the parent listener. (So it is not really
a DoS per se in any case.)

Bottom line
-----------
What they did behind the smoke-screen is replace a lot of setjmp() and
longjmp() functions (not very secure ones at that) with goto's
(interesting choice).

Smoke screen seems like unfarily loaded terminology to use.

OpenBSD fixed (removed) many setjmp/longjmp functions in their
tree a long time ago as a class of bugs. (Though this sendmail
exploitable collecttimeout() longjmp one is new and they patched
it yesterday with everyone else, because as you noted, replacing
it was kinda hairy...)

I don't think its fair to bitch about people fixing bugs and then not
having the time to send out advisories for every little tweak.
The important thing is to fix the bug. And often times the
developer won't understand the real impact of fixing a bug
until someone clever like Mark comes up with some innovative
way to exploit an "unexploitable" bug like this one.

What will be interesting to see when the PoC exploits are
finally released, is if any of the memory/stack protection schemes
mitigate it.

<humor>
Besides, there is only one true mailer to mail them all,
and its name is Postfix.
</humor>

Now if we could only convince Mr. Venema to switch
to a BSD license _everyone_ would switch to Postfix
and everything would be much better. If it weren't for
that "poison pill" clause in its license, I'm sure most
OSes and commercial systems would have swapped
out Sendmail for Postfix long ago.

cheers,
--dr

No offense Valdis, you know I both like you and consider you a friend,
but if you (sendmail) can't take the heat and/or stand up to the task of
being International Infrastructure, step down.

This isn't about processes, it's about something that has been around for
a while, many reply on and keeps ******* up. Where it simply can't.

  Gadi.

Given the scope of the changes you describe -- you wrote "Sendmail.com's
patch is so big they may as well have re-released the whole program."
-- I can't get upset at taking a month to fix it. You're dealing with
asynchronous events, which are really hard to start with. I suspect
that they spent some time deciding how to fix it -- you don't appear
thrilled with their choice, but I don't know what other options they
considered -- and then actually tested the new code. Given how many of
our security problems are due to buggy and inadequately-tested code, I
suspect that taking a month was actually being quite responsible.

    --Steven M. Bellovin, http://www.cs.columbia.edu/~smb

Heat we can take. Whining we can't.

Did you have a *specific* suggestion as to how it could be done better, or
did you just feel the need to flame about how it was handled?

Specific things that you did *not* consider, as far as I can tell:

1) You don't, as far as I know, contribute anything to paying the salaries
of the people actually doing the coding, and who know the way the code works.
This means that a choice has to be made:

a) Handle it in a way that doesn't upset the people paying the bills, even if
the people who aren't paying the bills don't like it.

b) Finding somebody else to pay the bills.

c) Dumping it and finding a project that *does* pay the bills, and fix it
as a hobby rather than a full-time job.

d) Dump it, move on, and let somebody else who doesn't know the code as well
do it, possibly worse.

So Gadi, what do you suggest we do, keeping in mind that the guy who's actually
doing the work needs to pay his rent and buy groceries?

2) You're complaing about the huge size of the patch, *and* the fact it took a
month to get it done. If you want it faster, you can either have less bugs
fixed, or less testing. Choose one. (Also, 2% code churn between releases is
*nothing*. Take a look at the Linux kernel sometime - it has a *far* higher
churn rate between releases that Linus is trying to keep on a 3-month
schedule....)

Steve Bellovin is right - these asynchronous events are a *pain* to debug,
because the human brain doesn't deal with race conditions very well. I
mentioned a timer event issue in a previous note - that one took Claus and I a
good *six months* to debug and understand, and resulted in all of 3 or 4 lines
of code. It was one that was not reproducible on command, only tripped a few
times a month, and by its nature never left any really useful state wreckage
behind. Trying to attach a debugger or adding debugging instrumentation to the
code would change the timing, and as a result scare the bug back into hiding...

Yes, it could potentially have been

Hey I'm the first guy to line up to knock Sendmail a.k.a. the pit of
infinite flaws... but lets not beat people up who don't deserve it.

My understanding of it was they found out from ISS and fixed it
ASAP. (I'm sure they read this list, maybe they would care to comment)
They took two weeks to update their customers and released it to CERT
who was supposed to take two weeks - starting three days ago... but something
happened (as it always does :slight_smile: and it was sent out prematurely - widely.

Hence again some people got caught off guard.
I'm increasingly forming the opinion that waiting for patches on disclosure
actually does harm. (Though I conceed that this is a religious issue
likely never to be resolved.)

Responsible disclosure may in fact be immediate disclosure so
that people can measure impact without waiting for the inevitable
vague wordings. That way we can take any countermeasures possible
immediately - rather than being vulnerable in the silence while
"special" people know. Resulting in not doing anything or being
increasingly watchful as would be warranted. If you know your
mailserver is vulnerable but you can't fix it you can always
start to watch it like a hawk - if you know.

I guess it depends on your security posture. If you are agressive
on security you want information dispersed as widely as possible.
If you are putting in minimal effort, then the less people know
the better it is for you.

Though only time will tell, I would also bet that this is not the last
sendmail signal handler issue we will see... at least until the more
Postfix-like commercial-only next version comes out... and then we
will have a brand new code base full of untested code to deal with.
And since its taken a few decades to stabilize Sendmail up to
_this_ point.... I'll just hug my Postfix code and resolve to buy
Mr. Venema beer at the first available opportunity :).

cheers,
--dr

P.s. I still maintain that the right solution is to convince IBM to tweak
the Postfix license so those who absolutely need fully open source
can use it instead. I've personally audited a lot of that code and know
how incredibly robust, nee paranoid it is. :-). Because when Sendmail
X comes out the open source folks will have a big issue to deal with.

I wonder how many other unreported silently-patched
vulnerabilities are out there?

You seem to be inferring that it is a bad thing to silently
patch bugs which may have security implications. The OpenBSD
team makes a habit of auditing software for flaws and fixing
them without waiting to find out whether they create actual
security vulnerabilities. They consider this to be a GOOD thing.

I think that people who use software also consider it to
be good for software flaws to be fixed as quickly as possible.
Inevitably, this means that if the DEVELOPERS discover a
flaw, they will fix it before they tell anyone about it. The
reason that security researchers publish bulletins about
security flaws is because they are unable to fix them
either due to lack of skill, or more commonly, they just
don't have permission to commit changes to the source code.

Network operators are users of software and not developers,
therefore most network operators are happy when flaws are
fixed early and often.

--Michael Dillon

Steven M. Bellovin wrote:

It took Sendmail a mounth to fix this. A mounth.

A mounth!

With such Vendor Responsibility, perhaps it is indeed a Good Thing to go
Full Disclosure. It seems like history is repeating itself and Full
Disclosure is once again not only a choice, but necessary to make vendors
become responsible.

Given the scope of the changes you describe -- you wrote "Sendmail.com's
patch is so big they may as well have re-released the whole program."
-- I can't get upset at taking a month to fix it. You're dealing with
asynchronous events, which are really hard to start with. I suspect
that they spent some time deciding how to fix it -- you don't appear
thrilled with their choice, but I don't know what other options they
considered -- and then actually tested the new code. Given how many of
our security problems are due to buggy and inadequately-tested code, I
suspect that taking a month was actually being quite responsible.

I'd usually agree, compared to a year and a half with Microsoft or 3 years with Oracle.

The point here, though, if that the patch was released almost with no notification _to_the_security_community_ (bugtraq, fd, etc.). It was obfuscated (open source, funny notion) and released. Exploits are already out there.

When you are critical infrastructure, you have higher responsibility. You either practice non-disclosure and patch your users over-time, then disclose, or simply disclose. It depends on needs and/or how responsive the vendor is.

One can't have it both ways, unfortunately.

  Gadi.

Exploit is out there. How long did that take?

  Gadi.

I wonder how many other unreported silently-patched
vulnerabilities are out there?

You seem to be inferring that it is a bad thing to silently
patch bugs which may have security implications. The OpenBSD

Full disclosure, we believe in it.

team makes a habit of auditing software for flaws and fixing
them without waiting to find out whether they create actual
security vulnerabilities. They consider this to be a GOOD thing.

It is a good thing.

I think that people who use software also consider it to
be good for software flaws to be fixed as quickly as possible.
Inevitably, this means that if the DEVELOPERS discover a flaw, they will fix it before they tell anyone about it. The
reason that security researchers publish bulletins about
security flaws is because they are unable to fix them either due to lack of skill, or more commonly, they just don't have permission to commit changes to the source code.

Network operators are users of software and not developers,
therefore most network operators are happy when flaws are
fixed early and often.

I wonder if the same network operators will be happy about potentially millions of compromised sendmail servers globally.

Hi,

    My only big issue with all of this is that a lot of us are depending on sendmail and its developer to keep it bug free...

    But who are we to ask them to be accountable to us?

    What are we paying them for it?

    What make us so special that to command them, or whinne about it when its not as perfect as ones mind think things should be?

    We should praise them and accept whatever scrap they send us.

    Because at the end of the day, sendmail, its staff, and other contributor saves our butts.

    (time to go put on my fireproof jammies)

Gadi Evron wrote:

Gadi Evron wrote:
(time to go put on my fireproof jammies)

or your .procmailrc

randy

Randy Bush wrote:

Gadi Evron wrote:
   

(time to go put on my fireproof jammies)
   
or your .procmailrc

randy

/usr/local/etc/procmailrc in my case. ( but not in production (; )

Is the exploit actually effective in the wild? The conditions Valdis
spoke of are improbable -- are there actually vulnerable sites? Or is
the attack much easier than he had indicated?

    --Steven M. Bellovin, http://www.cs.columbia.edu/~smb

Date: Thu, 23 Mar 2006 19:28:16 -0600 (CST)
From: Gadi Evron <ge@linuxbox.org>
Subject: Re: SendGate: Sendmail Multiple Vulnerabilities (Race Condition DoS,
    Memory Jumps, Integer Overflow)

[ ... ]
No offense Valdis, you know I both like you and consider you a
friend, but if you (sendmail) can't take the heat and/or stand up to
the task of being International Infrastructure, step down.

This isn't about processes, it's about something that has been around for
a while, many reply on and keeps ******* up. Where it simply can't.

What world do you live in were everything is done perfect ? If you
don't like sendmail because of its history or that it can contains
flaws, vote with your feet and choose something that you do think can
be trusted to do a better job, is more secure, is more actively
developed and is developed more securely then sendmail. [*]

Heck, if I were to have kids one day and would like them to get to
school safely by car, I'd like to have something short of a tank to be
absolutely certain. Instead I'll probably make them aware of the
risks, give them good protection and bicyle helmets... Now if I were a
head of state or something, I'd probably have people to get me that
tank... Note the "have people"...

  Gadi.

Regards,
JP Velders

[*] it happened with ssh.com versus OpenSSH... (though the
    license played a role, people did vote with their feet)

Steven M. Bellovin wrote:

[..]

> This isn't about processes, it's about something that has been around for
> a while, many reply on and keeps ******* up. Where it simply can't.

What world do you live in were everything is done perfect ? If you
don't like sendmail because of its history or that it can contains
flaws, vote with your feet and choose something that you do think can
be trusted to do a better job, is more secure, is more actively
developed and is developed more securely then sendmail. [*]

Indeed, and it is is not like there are no alternatives and of course
one can always roll it's own :wink:

And one even didn't have to pay for it, but complaining, and not helping
out by providing patches or research is always the easy way out.

/me chose postfix btw, but mostly also because the config is much
simpler :wink: Rolling my own would also be an option, the ones out there
work fine already and so what that they have bugs, no way that one can
code bugfree, just make sure that you can upgrade in time.

Heck, if I were to have kids one day and would like them to get to
school safely by car, I'd like to have something short of a tank to be
absolutely certain. Instead I'll probably make them aware of the
risks, give them good protection and bicyle helmets... Now if I were a
head of state or something, I'd probably have people to get me that
tank... Note the "have people"...

I guess you mean something like a 400.000 EUR tractor (vendor-C term):

The thing is, that might help for the collision case or a small bomb,
but one can still walk up to the guy when he gets out and shoot him
directly in the head or try to cut it off as has been demonstrated twice
before in that country. Bit futile thus to protect yourself with such
spendings when it doesn't cover the obvious cases.

Analogous, starting over using a new product might introduce other
security risks and of course never forget the migration path which in
larger installs includes training and upgrades, problem shooting and
then finding out that new bugs exist in the new code. Even the folks who
moved over from SSH.com to OpenSSH have found out that they had to
upgrade a large number of times, some times even with very troublesome
vulnerabilities, in the end causing most people to rate-limit port 22 or
to move it to another port altogether because of the automated scanning
happening.

Greets,
Jeroen

(Fortunately it was not my tax money that bought that tractor :slight_smile:

>
> > Well, it *is* mostly a theoretical overflow - for it to work, a site woul

d have to:

>
> Exploit is out there. How long did that take?
>
Is the exploit actually effective in the wild? The conditions Valdis
spoke of are improbable -- are there actually vulnerable sites? Or is
the attack much easier than he had indicated?

The race condition is easily winnable in the wild.

The integer overflow is essentially unexploitable in the wild, as it involves
*two* buffers, one of which is a compile-time constant bigger than the other.

The compile time constant is 1024 by default.

To trigger the overflow, the first buffer has to be *under* 2G (2**31) in size,
and the second is (by default) 1024 bigger and *over* 2**31 in size. At this point,
the attacker has sent 2 gigabytes of data over the wire, and the victim has
grown a buffer by 1024 bytes, copied, grown, copied, grown, copied, a total of
2,097,152 or so times. Oh, and you need to fit those almost 2G buffers,
*plus* 500K or so of Sendmail binary, in 1 4 gigabyte address space. That's
if you're on a 32-bit machine.

Oh dear, you seem to be about 497K short. At least.

I suppose some idiot site *could* have recompiled their sendmail to allocate
in 8 megabyte chunks rather than 1K. But performance would suck eggs.

Oh, and on a 64-bit machine, it's not any better. You *still* have to fit
2 buffers plus the 500K in under the 2**64 line. And you need to send
that much data too.

There's exploits for the race condition.

I was *specifically* talking about the integer overflow, which looks pretty
damned hard to exploit unless the victim site deliberately recompiled their
sendmail binary with a very sub-optimum configuration.

But then, you'd know that if you either actually *looked* at what I wrote,
or looked at the diff of the 8.13.[56] trees.