Pattern matching odd HTTP request

Mailman List Mirror (Read Only)
1

### On Wed, 19 Sep 2001 00:20:19 +0200, "Karsten W. Rohrbach"
### <karsten@rohrbach.de> casually decided to expound upon
### mike@biggorilla.com the following thoughts about "Re: Pattern matching
### odd HTTP request":

mike@biggorilla.com(mike@biggorilla.com)@2001.09.18 17:03:44 +0000:
[...]
> Doesn't seem new...
>=20
> 195.188.192.18 - - [13/Sep/1999:02:23:43 -0500] "-" 408 - "-" "-"
> 195.188.192.18 - - [14/Sep/1999:02:18:54 -0500] "-" 408 - "-" "-"
>=20
> But just a little more increased.

--- rfc2616 - http 1.1:
10.4.9 408 Request Timeout

   The client did not produce a request within the time that the server
   was prepared to wait. The client MAY repeat the request without
   modifications at any later time.
---

take care,

Yes... but when you're seeing this:

...
208.178.31.134 - - [18/Sep/2001:15:22:21 -0700] "-" 408 -
208.178.176.105 - - [18/Sep/2001:15:22:23 -0700] "-" 408 -
208.178.47.36 - - [18/Sep/2001:15:23:19 -0700] "-" 408 -
208.178.144.36 - - [18/Sep/2001:15:23:30 -0700] "-" 408 -
208.178.120.13 - - [18/Sep/2001:15:23:37 -0700] "-" 408 -
208.178.31.138 - - [18/Sep/2001:15:23:42 -0700] "-" 408 -
208.35.212.156 - - [18/Sep/2001:15:23:49 -0700] "-" 408 -
208.178.176.105 - - [18/Sep/2001:15:23:49 -0700] "-" 408 -
208.178.176.105 - - [18/Sep/2001:15:23:49 -0700] "-" 408 -
208.178.31.134 - - [18/Sep/2001:15:23:51 -0700] "-" 408 -
208.178.176.105 - - [18/Sep/2001:15:23:52 -0700] "-" 408 -
208.178.47.36 - - [18/Sep/2001:15:24:49 -0700] "-" 408 -
208.178.144.36 - - [18/Sep/2001:15:25:00 -0700] "-" 408 -
208.178.120.13 - - [18/Sep/2001:15:25:07 -0700] "-" 408 -
208.178.31.138 - - [18/Sep/2001:15:25:12 -0700] "-" 408 -
208.178.176.105 - - [18/Sep/2001:15:25:18 -0700] "-" 408 -
208.178.176.105 - - [18/Sep/2001:15:25:19 -0700] "-" 408 -
208.35.212.156 - - [18/Sep/2001:15:25:20 -0700] "-" 408 -
208.178.31.134 - - [18/Sep/2001:15:25:22 -0700] "-" 408 -
208.178.176.105 - - [18/Sep/2001:15:25:23 -0700] "-" 408 -
208.178.47.36 - - [18/Sep/2001:15:26:19 -0700] "-" 408 -
208.178.120.13 - - [18/Sep/2001:15:26:37 -0700] "-" 408 -
...

You start to suspect a DDOS port-flood attack. It's certainly causing me to
spawn a lot of httpds and occupying a lot of ports.

2

This isn't good. I wrote a bit of test code to see what would happen if I had alot of timeouts:

3

Bill McGonigle(mcgonigle@medicalmedia.com)@2001.09.18 18:58:42 +0000:

>
> You start to suspect a DDOS port-flood attack. It's certainly causing
> me to
> spawn a lot of httpds and occupying a lot of ports.

[...]

On Apache 1.3, this brings the number of httpd processes up to
MaxClients, then each one waits 300 seconds (the default timeout) for
the connections to time out, at which point the other connections are
made, and the cycle continues. A DDOS of this nature would be
particularly nasty. One client (happened to be on localhost) tied up
the server for 6 minutes this way with the default Apache config.

indeed, that's nasty.
the quick fix action would be setting
    Timeout 5
in the httpd.conf, but this won't really fix the problem and make the
objects inaccessible for users with high latency links.

source ip based connection rate limiting would perhaps solve the
problem. are there any modules available out there to accomplish this
task?

Here's what the logfile for these attempts looks like:

127.0.0.1 - - [18/Sep/2001:18:43:06 -0400] "-" 408 -

Doh!

yup, i see them from time to time in some of my servers' logs, but not
at that rate jake reported. i cc'ed brian from the apache project,
perhaps they got some solution for this...

/k

4

Date: Tue, 18 Sep 2001 18:58:42 -0400
From: Bill McGonigle <mcgonigle@medicalmedia.com>

[ snip ]

On Apache 1.3, this brings the number of httpd processes up to

[ snip ]

Jake, what server OS? Iff FreeBSD >= 4.0, check out the man page
for accf_http(9). Consider hacking Ap to use it... accept(2)
won't even see the connection until the socket is valid. I don't
know what the timeout is, but that'll help prevent Ap from
forking just to generate 408s.

Disclaimer:

I've not used accf_http on Ap. It seems to work well on some R&D
work that I've done, although I've not tested the stock accf_http
extensively.

Eddy

5

Brian Behlendorf(brian@collab.net)@2001.09.18 18:50:56 +0000:

> source ip based connection rate limiting would perhaps solve the
> problem. are there any modules available out there to accomplish this
> task?

http://modules.apache.org/search?id=241

is the only one I know of. I've not used it myself, and it's not a part
of the "core" distribution. If people use it and it works, I'd appreciate
knowing, as this comes up frequently enough that I'd agitate for getting
it included.

update: mod_throttle/3.1.2 seems to do the thing (as far as i can see
from the source) but i cannot get the thing running on my freebsd 4.3
box here, i keeps dumping core all the time. it is designed to already
check the source ip address in the fixup handler.
_all_ other modules i downloaded do not do this. i tried the following
ones:
- mod_throttle/3.1.2 dumps core at runtime, dang thing
- mod_throttle_access/0.2 starts processing after GET request
- mod_bandwidth/2.0.3 starts processing after GET request
- mod_bwshare/0.1.2 shmget: invalid argument but SHM is
        there, i can see it with ipcs(1)!
- mod_conn/1999 starts processing after GET request
- mod_limitipconn/0.0.3 starts processing after GET request

[http://modules.apache.org/search?id=123] is the url to mod_throttle
which looks very mature (but runs buggy on my box, whyever). i am
pretty tired now from my long uptime hunting that stupid nimda worm and
getting rid of it on some of my customers' servers. i hate iis, yuck.

mod_throttle looks like it could do it, because it contains the relevant
code in the fixup handler. However the policies are pretty weird -- at
least for me in my current state. man, it's been a long time since i
hacked my last apache module. must be 5 years or so... if somebody gets
mod_throttle running on freebsd4, drop me a line how you did it. it's
nearly midnight and i'm going to take a looooong nap now.

btw, if your log files keep growing and growing, use multilog from
daemontools to autorotate them and keep them at a specified maximum
size:
    ErrorLog "|exec setuidgid log multilog s200000 n5 /path/to/errlogdir"
daemontools are at [http://cr.yp.to/daemontools.html\]

you could even do
    ErrorLog "|exec grep -v '/scripts/' | setuidgid log multilog s200000 n5 /path/to/errlogdir"
to get rid of the darn worm-generated errors.

lowering the timeout does not make much sense, the best solution would
be having the max connections from one ip in the scoreboard structures
and connection handler because it _must_ be processed at connection
time, not when the request is already in -- there never will be a GET or
any other request... i see that this can be doen in the fixup_handler in
an module, but i am quite rusty concerning the apache api :wink:
brian, i'd appreciate if you would forward this to the core people.
thanks.

i got an important appointment with my bed now...
take care,
/k (tired like hell)

6

Thanks for all your work on this one, Karsten, and I hope you had a good nap. :slight_smile:

mod_throttle looks like it will stop a DOS from one client effectively, though the configuration is a bit complex for just that use of it. I plan to implement it for that. It doesn't appear to be useful though for the type of DDOS that seems to be brewing (which I hope fizzles and dies).

The traffic pattern I was seeing (one request every 1.5 minutes) means it would take 45 attackers to tie up a stock Apache indefinitely. If this was implemented as a nimda-like worm, using random IP scanning, and it attacked as found servers, I think there would be a pretty good chance of defending against it (firewall the ip if there are n number of timeouts in a time period). If it did discovery first, though, and kept a cache (I'm not going to throw a flag on someone looking for my /index.html) then attacked at a predetermined time I can't think of a way to defend against it with a per-IP configuration. I'd probably never set my per-IP limit below 5, and this would use 3.33 connections per IP.

If, however, Apache had a limit on 'barely-open connections' with some sort of timeout function, I think that would help. For instance, it might look like:

BarelyOpenConnectionTimeout 10
BarelyOpenConnectionLimit 50

Such that if there were 50 connections open that hadn't sent a request for 10 seconds, it would stop dropping them in a FIFO manner.

I mostly hack on higher-level modules in mod_perl, so I don't know enough about apache internals to speak to the feasibility of such a function.

-Bill

7

Noah(sitz@onastick.net)@2001.09.20 10:36:17 +0000:

>
> Thanks for all your work on this one, Karsten, and I hope you had a good
> nap. :slight_smile:
>
> mod_throttle looks like it will stop a DOS from one client effectively,
> though the configuration is a bit complex for just that use of it. I
> plan to implement it for that. It doesn't appear to be useful though
> for the type of DDOS that seems to be brewing (which I hope fizzles and
> dies).

I haven't tried it myself (yet... it's on the plate), but mod_tsunami
(http://bert.tuxfamily.org/mod_tsunami/) may work better. It limits the
number of connections on a per-directory basis, using the apache
scoreboard. The primary issue I had with mod_throttle when I was looking
at it (a few months back) was that it was fairly CPU intensive. This may
or may not be an issue depending on how much traffic you push on a
"normal" day (whatever that is).

ALL modules that use directory/location base config for resource
limiting won't work, because they start processing the reuqets after the
request is in. to be more verbose:

1 client connects to server
2 when socket is connected, client send http headers (accept/host/...)
3 client issues a request (see rfc2616 for that) like GET / HTTP/1.1
  followed by two carriage returns
4 server starts processing the request (filtering headers and uri
  information through the stack of configured modules)

richt after step 2, there is no per-directory/per-location handler
trigger, since the request has not yest triggered the corresponding
handler because it was not issued by the client.

the malicious client would just open a lot of sockets and wait.
mod_throttle is the only implementation that could process those
connections because it has an additional 'fixup' handler that gets
called _before_ request processing has begun.

the ideal location for some logic limiting incoming dead connections is
in the core itself, because the api is somewhat limited in what you can
do before the request processing an a module.

i hope this shed a little light on the issue.

take care,
/k

8

[snip]

1 client connects to server
2 when socket is connected, client send http headers (accept/host/...)
3 client issues a request (see rfc2616 for that) like GET / HTTP/1.1
  followed by two carriage returns

I hate to be pendantic, but the following is the order of HTTP headers:

nedominic@is:~ > netcat -l -p 5000
GET / HTTP/1.0
Connection: Keep-Alive
User-Agent: Mozilla/4.74 [en] (X11; U; Linux 2.2.16 i686)
Host: is:5000
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png,
*/*
Accept-Encoding: gzip
Accept-Language: en
Accept-Charset: iso-8859-1,*,utf-8

dominic@is:~ >

In other words, your step 2 and 3 should be in the opposite order.

4 server starts processing the request (filtering headers and uri
  information through the stack of configured modules)

i hope this shed a little light on the issue.

Ditto.