Odd router brokenness

Since this list likes to speculate with little facts on a regular basis (and I'll admit to being as guilty as anyone) I throw this one out for opinions :

We were seeing very odd behavior on a Cogent circuit following a software upgrade to tol01.atlas. Two traceroutes:

mark@angola-gw> traceroute
traceroute to (, 30 hops max, 40 byte packets
  1 * * gi1-1.ccr01.tol01.atlas.cogentco.com ( 110.315 ms
  2 te4-2.ccr01.sbn01.atlas.cogentco.com ( 139.520 ms 196.910 ms 5.728 ms
  3 * * *
  4 * te0-5-0-5.ccr21.ord03.atlas.cogentco.com ( 8.310 ms te0-0-0-7.ccr21.ord03.atlas.cogentco.com ( 8.752 ms
  5 te0-0-0-0.ccr22.ord03.atlas.cogentco.com ( 8.983 ms te0-1-0-0.ccr22.ord03.atlas.cogentco.com ( 7.948 ms *
  6 * * te-9-1.car4.Chicago1.Level3.net ( 26.127 ms
  7 GOOGLE-INC.car4.Chicago1.Level3.net ( 38.132 ms 25.120 ms *
  8 * * ( 24.539 ms
  9 * ( 26.134 ms ( 25.021 ms
      MPLS Label=666803 CoS=4 TTL=1 S=1
10 ( 31.816 ms 35.702 ms 32.249 ms
11 ( 32.897 ms * *
12 * yyz06s05-in-f6.1e100.net ( 33.319 ms *

and a ping over the same path:

--- www.l.google.com ping statistics ---
675 packets transmitted, 323 packets received, 52.1% packet loss
round-trip min/avg/max/stddev = 12.834/28.831/129.743/28.987 ms

and at the same time:

mark@angola-gw> traceroute
traceroute to (, 30 hops max, 40 byte packets
  1 gi1-1.ccr01.tol01.atlas.cogentco.com ( 4.445 ms 1.841 ms 1.713 ms
  2 te7-7.ccr02.cle04.atlas.cogentco.com ( 5.318 ms te3-2.ccr02.cle04.atlas.cogentco.com ( 4.755 ms te7-7.ccr02.cle04.atlas.cogentco.com ( 4.982 ms
  3 te4-2.ccr01.pit02.atlas.cogentco.com ( 7.997 ms te3-2.ccr01.pit02.atlas.cogentco.com ( 7.736 ms te4-2.ccr01.pit02.atlas.cogentco.com ( 8.177 ms
  4 te0-0-0-5.mpd21.dca01.atlas.cogentco.com ( 17.197 ms te0-0-0-5.ccr22.dca01.atlas.cogentco.com ( 16.907 ms te0-0-0-5.mpd21.dca01.atlas.cogentco.com ( 17.008 ms
  5 te0-1-0-0.mpd22.dca01.atlas.cogentco.com ( 17.358 ms te0-0-0-0.mpd22.dca01.atlas.cogentco.com ( 17.196 ms te0-1-0-0.mpd22.dca01.atlas.cogentco.com ( 18.690 ms
  6 te4-2.mpd01.iad03.atlas.cogentco.com ( 17.885 ms * 18.537 ms
  7 cogentco.com ( 17.836 ms !<10> 17.918 ms !<10> 17.833 ms !<10>

--- ping statistics ---
236 packets transmitted, 236 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 22.717/27.942/128.011/12.236 ms

Works perfectly. There is no asymmetric routing in this scenario (only 1 BGP peer running during this test), and it is not due to traffic congestion. Initial speculation over the dropped packets in the trace to was ICMP depriortization. The results are too consistent for that to make sense (I have dozens of traceroutes to the same destination - they all appear similar).

I realize there is a long history of Cogent/L3 ugliness but I'm pretty sure that this issue has nothing to do with that subject.

Traceroutes and pings from the control plane of tol01.atlas sourced from do not show any odd behavior. Inbound traffic (to us) is not affected by this. Our workaround while resolving this issue was to change local-pref on the affected prefixes to send traffic out our other providers.

The issue started after a software upgrade to tol01.atlas and resolved after a (reported) reboot of tol01.atlas.

The question is: How does a router break in this manner? It appears to unintentionally be doing something different with traffic based on the source address, not the destination address. I realize this can be done intentionally - but that is not the case here (unless somebody isn't telling me something).

I don't think we can determine that it has anything to do with source
address based on data shown. could very well be 6500 and somehow broken adjacency to, perhaps hardware adjacency having MTU of 0B, causing punt
which is rate-limited by different policer than TTL exceeded policer.

an ether-channel go south without the port showing down. Stuff hashed over
the good like was fine, stuff hashed over the bad like wasn't. Led to some
painful support calls from customers. I agree this list is a haven of
speculation and OT comments. In order to avoid making a bad problem worse
you should probably contact cogent.

I was told the router was reloaded to resolve a CEF issue. Not sure what was wrong with 'clear cef linecard'.

Now *that* brings back memories!

It's fixed at this point. You are correct in that it was quite painful getting this escalated far enough to get it fixed. The tools that are available (at least that I know of) to try to prove the issue to level 1 and 2 support just doesn't get the job done.

It's the eternal problem of convincing L1/2 support that you really have a problem not of your own making.


Or just fixing the broken prefixes/adjacencies and opening CTAC case about
what was wrong with them.