Cisco IOS Failure due to Virus

Hey Everyone -

We have two 7507 routers configured with dual RSP4s w/256MB RAM,
VIP2-50s with 128/8MB RAM, Gig, POSIP OC3 and Fast Ethernet
interfaces.

These routers have run flawlessly for over two years now. But about
two weeks ago, all of a sudden we started having serious crashing
problems with these two routers. The routers will lose bgp
connectivity (one at a time) to our upstreams (configured on each
router). First, we would see a keepalive not sent message, then a bgp
hold timer expire, then the bgp peering session would go down. OSPF
would start crashing, then we would see the memory error messages,
then all interfaces would blink off-line. (Note - we are running the
max memory we can on both the RSPs and the VIPs).

Within 1 minute, the exact same thing would happen to the other
router. Often times we would have to reboot the router to get it to
come back online. We would see the following errors and have to reboot
multiple times to get the router back:

%SYS-2-MALLOCFAIL: Memory allocation of 704 bytes failed from
0x60329F00, alignment 0
Pool: Processor Free: 92744 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "Pool Manager", ipl= 0, pid= 6
-Traceback= 6038049C 60382200 60329F08 6038DEDC

%TCP-6-NOBUFF: TTY0, no buffer available
-Process= "BGP Router", ipl= 0, pid= 132

%% Low on memory; try again later

GigabitEthernet1/1/0: keepalive not sent

We are running the latest S train IOS patched for the IPV4 issue -
however downgrading to the code we had run for the previous year did
not solve the problem, nor did replacing the RSPs, VIPs and interfaces
with new cards. In addition, we have complied with the Cisco
recommendations for mitigating the effects of the Nachi Worm.

http://www.cisco.com/en/US/products/sw/voicesw/ps556/products_tech_note09186a00801b143a.shtml

We also shut down one of the routers totally and the other router
still experienced the same issue.

None of these updates or fixes have solved the problem.

I am thinking it may have something to do with all the virus stuff
running around (same thing was crashing my Lucent TNT's), but I cannot
seem to get an answer from Cisco, nor can
I find anyone seeing the same issues.

Hopefully someone here can shed some light on this problem.

Thanks in Advance

Richard

I fly because it releases my mind
from the tyranny of petty things . .

Did you enable CEF?
Are you dropping 92 byte ICMP packets where needed?

Hi Robert,

Thanks for the info. We are running dCEF...routers show about 4% CPU
load and the following memory:

BR02#sh mem
               Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 613AE340 247798976 106515996 141282980 140653360 134546752
     Fast 6138E340 131080 37240 93840 93840 93788

Also, we are not blocking 92 byte ICMP due to the traceroute problems on
customers networks...

Thanks

I am not sure if this post belongs here, so I apologize if it does not.
I have been experiencing some weirdness while traveling and wondered if
the group has any insight into what seems to be a pretty ugly situation.

I am traveling and have my lap top with me. I am staying in a hotel that
offers broadband support. There are 2 of us (with 2 lap tops) sharing a
room. I acquire an internet connection and sign up for the service, so
get an IP address. In my case that IP address is 12.44.189.24.

I disconnect my cable and pass it to my roommate. He plugs in and
acquires IP address 12.44.189.47. He does the email thing for a while
and then passes the cable back to me. Imagine my surprise when the
network routes packets destined for his IP address (from his email
server no less) to my computer. My firewall (Zone alarm) detects these
incoming packets and blocks them since they are unsolicited.

In further analysis of the logs, I see that there are a large number of
IP addresses that are packet destinations and routed to my computer Zone
Alarm detects them and blocks them. According to Zone Alarm I am getting
packets for destination IP addresses as follows:12.44.189.244.
12.44.189.178 12.44.189.181 12.189.44.244 and some others too. They are
all port 80 requests, identified by Zone Alarm as TCP (flags:S).

This seems strange to me since they are arriving at an IP address that
is different from mine.

How can this happen? Is there the potential for a problem (I am thinking
particularly about future guests who may not have the degree of
protection (limited though it is) that Zone Alarm is affording me.)?

This then got me thinking about corporate security. If I have taken my
laptop and put it on an external network (e.g. the hotel network) what
protections can I realistically expect, and what should my corporate IT
department do to make sure my compute hasn't contracted something nasty
while it was away from home. I could see that the kind of network
behavior that I observed could infect a less well protected computer and
thus cause me to bring an infection back to my office where it can
attack from behind the corporate shields and firewalls.

Any comments would be very welcome.

Regards

Chris Bird

Hi,
we've seen this.. yuo need to make sure you filter the nachi worm 92 byte icmp
echo's on your interfaces and it will be fine. The problem seems to be input
buffers which use all the memory up for some reason.

Steve

Whoa stop press! You connected a computer to a public IP and zone alarm starts
buzzing away.. FBI!

Depends on how the hotel system works, it may be broadcasting or doing some
other IP weirdness, either way its not surprising.

But there is no security threat from some left over packets from old TCP/IP
sessions... as for the question on corporate security, I would hope that any
connection to the Internet be it a corporate LAN or a travelling user on a
remote network is done from a computer which has been adequately setup to be
protected from the latest vulnerabilities and is locked down as much as
possible, goes without saying!

This is one such way as you mention of how office networks with their fancy
one stop, protects all ills firewalls are still succumb to viruses and other
nasties. I'd assume your IT department enforces policies on regularly installing
OS patches and updating local virus scanners as part of its security policy...
right?

Steve

Christopher Bird wrote:

This seems strange to me since they are arriving at an IP address that
is different from mine.

That's the function of a hub, and the reason why you don't ever want to send out sensitive information in plaintext. Your neighbor in the next room over could run a packet sniffer and capture your logins.

Mike Lewinski wrote:

Christopher Bird wrote:

> This seems strange to me since they are arriving at an IP address that
> is different from mine.

That's the function of a hub, and the reason why you don't ever want to
send out sensitive information in plaintext. Your neighbor in the next
room over could run a packet sniffer and capture your logins.

Yes and no. On a broadcast network, like Ethernet, you expect to see traffic
for other hosts. The issue here is that it looks like the traffic for these
other hosts was getting delivered to Christopher's link-layer address.

If he kicks his NIC into promiscuous mode and sees all of this traffic
with a sniffer, then this is expected. That's the scenario you highlight.

If, OTOH, he's sitting there and traffic for other IP addresses is getting
delivered to his NIC with the NIC's MAC address in there, something strange
is going on. Unless he has his NIC switched to promiscuous mode and the
OS's IP stack is misbehaving badly, Zone Alarm should not see the traffic
on the LAN that does not have his MAC address on it.

How would a switch/router be deciding that these other IP addresses
should go to his PC's NIC (MAC address)?

Stephen J. Wilcox wrote:

Hi,
we've seen this.. yuo need to make sure you filter the nachi worm 92 byte icmp
echo's on your interfaces and it will be fine. The problem seems to be input
buffers which use all the memory up for some reason.

This sounds vaguely similar to the recent IOS buffers stuck issue.

Pete

No, its quite different

1:
On the vuln. the buffer filled up and could not be emptied without a reboot

On nachi the buffer doesnt seem to fill and an acl or shutting the interface
will solve the problem whilst the router stays up

2:
On the vuln. the outcome was that the particular interface stopped forwarding
traffic

On nachi the router runs out of main memory and starts dropping processes
because of malloc failure

FYI I have only encountered the nachi problem on a few PE routers which were old
and had little memory anyway eg Cisco 2500.. presumably the buffer filling isnt
a memory leak and providnig there is enough spare memory the router wont be
affected in this way.

Steve