I am wondering if the packets might be MD5 packets addressed to your router.
They might be small with no valid content but your router gets busy validating
the MD5 sum for each packet before the router then drops the packet. That
keeps the router busy doing nothing useful. Did you check your CPU utilization
during the high latency periods then check to see which process is consuming
the CPU cycles if it does look high?