How do you guys troubleshoot high CPU utilization on the ASR-9K platform?
Detailed guides are available for IOS platforms, but I can't seem to find
anything useful for the ASR.
The average line-card (0/0/CPU0: A9K-24x10GE-TR) CPU utilization of my
routers is about 10%, however recently I have noticed that 3-5 times a day
it increases to 40% and stays there for about an hour (20% spp + 10% netio
+ the rest).
I know this is well withing the acceptable range, but I am the kind of
person who likes to understand every change in his network and during the
investigation I had to realize that I simply don't have the tools to
troubleshoot the ASR CPU.
It coincides with nothing else? More traffic? CPU increasing at regular intervals every day without any obvious reasons is probably something worth looking into!
I've experienced similar behavior on other platforms as well. Sometimes
the output of the box is not correct. We were able to prove this to the
vendor by conducting experiments and graphing the CPU. One of the
protocols they said "couldn't possibly be causing this" turned out to be
the root of the problem.
I live by one rule when troubleshooting:
The box is a lie.
You should check a log files during the time of high cpu load. ASR9K do most of the packet processing on NP.
High CPU load may happen during some control plane processing, like bgp neighbor flapping.
This document (BRKARC-2017) turned out to be very useful to determine the
possible root-cause. The utilization of the spp and netio processes
increase if the router/line-card is software switching traffic, in our case
ICMP. We will test the policing feature and implement it.