I'm attempting to get ahold of someone at CenturyLink/Quantum/Lumen about an unusual hardware failure that's causing issues for a lot of customers in southwest Denver. It appears that a buffer in one link in a LAG or ECMP group is flipping a bit around 10MB in to a zero. They have a bunch of tech visits scheduled to various people, and I assume after enough of those fail to fix the issue it'll eventually filter up to the NOC or neteng in a couple of weeks, but I'm hoping to short-circuit that a bit.
Symptoms/troubleshooting:
No change on home router replacement
No difference across devices
ICMP ping works fine, even padded to 1460 with 00 or FF
SSH/SCP work (generally limited to 2MB or 4MB buffers/window depending on the version)
HTTPS fails frequently
Downloading a 10MB 0-filled file via HTTP succeeds consistently.
Downloading a 10MB 1-filled file fails frequently
Specifically, any 1-filled file with 10482846 or more bytes sees frequent failures
For a consistent 5-tuple and large 1-filled file, failures are consistent.