TCP session disconnection caused by Code Red?

Mailman List Mirror (Read Only)
1

I'm seeking the collective wisdom of this list to try to prove that I'm
not halucinating.

For the last few days, our network seems to be basically unreachable from the
outside. Most incoming TCP sessions (web requests, incoming mail, telnet
sessions, etc.) often fail with a simple "Connection refused" like nobody is
listening on that port (but of course there is a server running on the port).
It does not matter which service I'm trying to connect to, it also does not
matter which server I'm connecting to. It happens randomly, i.e. three
successive connections fail, then 5 connections succeed, then 2 connections
fail, etc. Outgoing connections (i.e. our customers surfing the web) work just
fine - except active FTP, which of course contains an implicit "incoming" TCP
session.

I've been fighting with this since friday and have done some packet traces. By
comparing successful and unsuccesful packet traces I came to the conclusion
that my problems are being caused by incoming TCP packets with the RST bit
set. So I applied the following access list on the link to our upstream:

access-list 170 deny tcp any any rst
access-list 170 permit ip any any

After doing this, our incoming TCP sessions magically started working. Looking
at the packet counters, I see about 20% of our incoming packets are TCP RST
packets. Putting the same filter on an internal link, I see about 1% of TCP
RST packets.

Turning on access list logging, I see that most of the packets are destined
for port 80 on unused IP addresses (which are nullrouted) - which I guess is
Code Red searching for victims.

So now tell me, am I dreaming or has Code Red found a bug in Cisco IOS? :slight_smile:
Or is this some kind of new (or old?) denial of service attack disguising as
Code Red? I've reported this to psirt@cisco.com and I'm waiting to see what
they come up with.

Anyone seen anything like this? I see the same thing with 12.0(14)S2,
12.0(17)S, 12.0(18)S and 12.2(1). Machine is a 7206 VXR, NPE-400, PA-2E3, 2FE
I/O controller.

Blaz Zupan, Medinet d.o.o, Trzaska 85, SI-2000 Maribor, Slovenia
E-mail: blaz@amis.net, Tel: +386-2-320-6320, Fax: +386-2-320-6325

2

For the last few days, our network seems to be basically unreachable from the
outside. Most incoming TCP sessions (web requests, incoming mail, telnet
sessions, etc.) often fail with a simple "Connection refused" like nobody is

Your routers are brain dead from the load.. routers that are used to
handling a few thousand connections are being asked to handle 10's of
thousands. 1 good 1000+ address scan from an ISDN user kills my
Lucent/Ascend TNT unless we filter for it.

3

Your routers are brain dead from the load.. routers that are used to
handling a few thousand connections are being asked to handle 10's of
thousands. 1 good 1000+ address scan from an ISDN user kills my
Lucent/Ascend TNT unless we filter for it.

Hmmm, a 7206 should surely be able to handle more than 600 packets per second
or am I wrong here? Our upstream E3 is currently used a maximum of 15Mbps and
at peak time we see about 3000 pps on that link. If 20% of that is TCP RST
packets, that would be 600 packets per second. And I'm sure somebody else on
this list would be noticing this as well, especially with higher speed links.

Blaz Zupan, Medinet d.o.o, Trzaska 85, SI-2000 Maribor, Slovenia
E-mail: blaz@amis.net, Tel: +386-2-320-6320, Fax: +386-2-320-6325

4

Hmmm, a 7206 should surely be able to handle more than 600 packets per second
or am I wrong here? Our upstream E3 is currently used a maximum of 15Mbps and

It's not the packets per second that seems to kill them, its
the amount of arp cache and sessions (figure 600 packets per second,
each packet to a different host...Thats a lot of sessions in 5 minutes)

5

Curious, in that case consider null routing unused blocks, perhaps take
the opportunity to improve on subnet and vlan distribution to help the
null routing.

Steve

6

> It's not the packets per second that seems to kill them, its
> the amount of arp cache and sessions (figure 600 packets per second,
> each packet to a different host...Thats a lot of sessions in 5 minutes)

Curious, in that case consider null routing unused blocks, perhaps take
the opportunity to improve on subnet and vlan distribution to help the
null routing.

That's exactly the case. All the unused IP addresses are nullrouted and most
of the traffic was destined for the nullrouted addresses. I don't think a lot
of arp activity was going on.

Blaz Zupan, Medinet d.o.o, Trzaska 85, SI-2000 Maribor, Slovenia
E-mail: blaz@amis.net, Tel: +386-2-320-6320, Fax: +386-2-320-6325