Packet Loss on Level3

Is anyone else seeing packet loss on Level3.

  6. 2.9% 35 372.1 148.6 19.4 704.8 127.4
  7. 8.6% 35 268.1 161.5 21.3 691.8 156.0
  8. 0.0% 35 173.6 142.8 34.5 532.4 117.1
  9. 5.7% 35 78.1 210.3 35.7 631.2 157.9

If the loss doesn't continue for every hop along the path, it isn't real
loss. For example, if there was actually a problem at hop 6, it would
also affect the packets that are traveling to hop 8, so hop 8 wouldn't
come up loss-free. Of course it's always possible that there is loss on
a particular circuit that is part of a link-aggregate bundle, which
causes only certain packets hashed certain ways to go down the "bad"
channel, but typically you wouldn't see any loss at all since it's so
hard to target that one bad channel channel.

The far more likely scenario is that you're seeing routers with
overloaded control-plane or exception processor rate-limits, due to
excessive icmp generation demands. Running mtr is actually a leading
cause of this type of overload, so you might actually be helping
contribute to the problem. Unfortunately many routers have hard-coded
rate-limits for icmp generation which can't be bumped, despite the fact
that we routinely hit them due to "ordinary" things like excessive mtr
use. I don't have a single router which hasn't bumped the rate limits by
pretty significant amounts, and most are dropping through the day under
normal traffic loads:

ras@arandomrouter> show pfe statistics ip icmp | match throttled
  5195868898 throttled icmps

Take a look at a presentation I did at NANOG 45 for more details.