Anyone seeing ping corruption?

Deepak_Jain · December 9, 2021, 4:39pm

Haven’t seen this before. This is a Nexus 9K as a testing platform. Getting sporadic complaints about data transfers aborting, but data moves well through the platform.

Hop 13 doesn’t respond to our 1400 byte ping, hop 12 does a normal response, Google’s 14 corrupts the packet or maybe deliberately manipulates it? 1.1.1.1 doesn’t do that.

Any insights welcome.

Thanks,

trace 8.8.8.8

traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 40 byte packets

[deleted 3 hops]

4 154.24.71.85 (154.24.71.85) (AS 7922) 1.199 ms 154.24.32.5 (154.24.32.5) (AS 7922) 1.32 ms 154.24.32.1 (154.24.32.1) (AS 7922) 3.162 ms

5 154.54.81.173 (154.54.81.173) (AS 7922) 1.564 ms 154.54.81.141 (154.54.81.141) (AS 7922) 1.691 ms 154.54.81.221 (154.54.81.221) (AS 7922) 1.616 ms

6 154.54.40.109 (154.54.40.109) (AS 7922) 7.041 ms 154.54.40.105 (154.54.40.105) (AS 7922) 8.035 ms 154.54.40.109 (154.54.40.109) (AS 7922) 7.002 ms

7 154.54.47.218 (154.54.47.218) (AS 7922) 7.183 ms 8.595 ms 7.186 ms

8 154.54.12.18 (154.54.12.18) (AS 7922) 7.55 ms 16.791 ms 16.32 ms

9 * 66.110.96.62 (66.110.96.62) (AS 7922) 16.9 ms 8.202 ms

[Label=24004 E=0 TTL=1 S=1]

10 66.110.96.58 (66.110.96.58) (AS 7922) 18.65 ms 15.823 ms 11.381 ms

11 72.14.195.232 (72.14.195.232) (AS 7922) 8.473 ms 8.128 ms 8.241 ms

12 108.170.248.33 (108.170.248.33) (AS 7922) 8.771 ms 108.170.248.1 (108.170.248.1) (AS 7922) 9.004 ms 9.223 ms

13 142.251.60.237 (142.251.60.237) (AS 7922) 8.138 ms 142.251.65.95 (142.251.65.95) (AS 7922) 7.768 ms 142.251.64.7 (142.251.64.7) (AS 7922) 8.171 ms

14 8.8.8.8 (8.8.8.8) (AS 7922) 8.271 ms 14.179 ms 8.185 ms

ping

Vrf context to use [default] :

Target IP address or Hostname: 8.8.8.8

Repeat count [5] :

Packet-size [56] : 1400

Timeout in seconds [2] :

Sending interval in seconds [0] :

Extended commands [no] : yes

Interface :

Source address or interface :

Data pattern [0xabcd] :

Type of service [0] :

Set DF bit in IP header [no] :

Time to live [255] :

Loose/Strict, Record, Timestamp, Verbose [None] :

Sweep range of sizes [no] :

Sending 5, 1400-bytes ICMP Echos to 8.8.8.8

Timeout is 2 seconds, data pattern is 0xABCD

76 bytes from 8.8.8.8: icmp_seq=0 ttl=111 time=8.732 ms

wrong data byte #68 should be 0xcd but was 0x0

37 2d b2 61 af 2f 4 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0

cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0

cd ab 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

76 bytes from 8.8.8.8: icmp_seq=1 ttl=111 time=8.395 ms

wrong data byte #68 should be 0xcd but was 0x0

37 2d b2 61 e7 68 4 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0

cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0

cd ab 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

76 bytes from 8.8.8.8: icmp_seq=2 ttl=111 time=8.036 ms

wrong data byte #68 should be 0xcd but was 0x0

37 2d b2 61 45 b2 4 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0

cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0

cd ab 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

76 bytes from 8.8.8.8: icmp_seq=3 ttl=111 time=9.54 ms

wrong data byte #68 should be 0xcd but was 0x0

37 2d b2 61 42 e1 4 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0

cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0

cd ab 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

76 bytes from 8.8.8.8: icmp_seq=4 ttl=111 time=8.758 ms

wrong data byte #68 should be 0xcd but was 0x0

37 2d b2 61 be 15 5 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0

cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0

cd ab 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

— 8.8.8.8 ping statistics —

5 packets transmitted, 5 packets received, 0.00% packet loss

round-trip min/avg/max = 8.036/8.692/9.54 ms

Lukas_Tribus1 · December 9, 2021, 4:50pm

8.8.8.8 truncates ICMP's responses, this is well known. Different
platforms will react differently to it.

lukas@dev:~$ ping 8.8.8.8 -c1 -s1000
PING 8.8.8.8 (8.8.8.8) 1000(1028) bytes of data.
76 bytes from 8.8.8.8: icmp_seq=1 ttl=116 (truncated)

--- 8.8.8.8 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 5.657/5.657/5.657/0.000 ms
lukas@dev:~$ ping 1.1.1.1 -c1 -s1000
PING 1.1.1.1 (1.1.1.1) 1000(1028) bytes of data.
1008 bytes from 1.1.1.1: icmp_seq=1 ttl=58 time=15.8 ms

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 15.893/15.893/15.893/0.000 ms
lukas@dev:~$

J_Doe · December 20, 2021, 2:41am

Hi,

Out of curiosity - does anyone know why Google is truncating ICMP responses ?

Thanks,

- J

Sean_Donelan · December 20, 2021, 9:50pm

Assuming (not confirmed) mitigating old-style DOS attacks. See "ping of death." ICMP packets should be small enough to never require fragmentation. But large enough to contain at least the header of the original packet.

When I used to write firewall stuff, there was a great debate about how much data to return in an ICMP error/reply packet. I don't think the debate was ever resolved.

Jon_Lewis1 · December 20, 2021, 10:13pm

My guess would be to reduce "garbage traffic bandwidth". They're running an open resolver...not a lighthouse.

Steven_Champeon · December 20, 2021, 10:45pm

Are there even enough dialup connections and ancient modems left that POD
is a thing anyone needs to worry about? I mean, yeah, I enjoyed knocking
spammer idiots offline with that but it's been twenty years or more since
I had that uniquely weird pleasure.

n.b.: alias pod='ping -p 2b2b2b415448300d -c 5'

Sean_Donelan · December 20, 2021, 11:17pm

Likely not. But ICMP is spoofable, and anyone operating extremely large attack bounce points, should try to mitigate the attack potential. Limiting ICMP responses, DNS answers, reducing SYN retries, gratuitous firewall RSTs, etc, etc.

But I'm just guessing why Google does it. I don't know, and unless a googler is willing to share; we won't ever know. Sometimes I disagree with their reasons, but google usually has a reason.

Oh, and also every network operator should implement source address validation (beating the dead horse, for the nanog mail topic moderators).

Valdis_Kletnieks · December 21, 2021, 12:26am

It wasn't just dialup and modems. The Ping of Death had to do with sending
a packet that was already 64K in size, that would require an ICMP response that
would try to include the entire packet - corrupting the memory following the
buffer the response was built in. Lots of ethernet-connected Windows systems
got BSOD'd that way.

Having said that, I'm sure there's still unpatched systems out there. Probably
a few that are *still* unpatched against the 1990s IPV4 version, but a lot more
likely for the 2013 and 2020 IPv6 versions against Microsoft systems.

Hank_Nussbacher3 · December 21, 2021, 7:11am

As Google has stated in many forums and I quote:
"Google Public DNS is a Domain Name System service, not an ICMP network testing service."

-Hank

Mark_Tinka4 · December 21, 2021, 8:00am

:-)... can't argue with that.

In the 90's and early 2000's, it was:

 ping \-t yahoo\.com

Mark.

Lukas_Tribus1 · December 21, 2021, 8:34am

The core issue is that many watchdogs implemented in all kinds of
devices use ICMP for health checking (and just one host), as opposed
to DNS or HTTP services of multiple hosts. Those users have to point
their watchdogs somewhere, and "ping.crappy-iot-vendor.com" just
sounds less reliable (and probably is), then one of those quad8. quad1
or quad9 services.

It's obvious that it's not a DNS Servers job to respond to ICMP
requests, and under DoS they'd probably rate-limit it.

I think Google choice makes sense, at least it does not allow 1:1
reflection at big sizes.

Nobody is interested in running reliable anycast ICMP responders on
the internet, after all, no useful data can be collected with it.
Device vendors will keep shipping code with watchdogs that can only
ping a single host, so users will keep putting DNS servers in there.

Lukas

Sean_Donelan · December 21, 2021, 9:09am

And SRI-NIC.ARPA

see

https://archive.nanog.org/mailinglist/mailarchives/old_archive/2003-10/msg00069.html

Masataka_Ohta · December 21, 2021, 2:33pm

Valdis Klētnieks wrote:

Are there even enough dialup connections and ancient modems left that POD
is a thing anyone needs to worry about?

It wasn't just dialup and modems. The Ping of Death had to do with sending
a packet that was already 64K in size, that would require an ICMP response that
would try to include the entire packet - corrupting the memory following the
buffer the response was built in. Lots of ethernet-connected Windows systems
got BSOD'd that way.

No, an ICMP echo reply does not include the entire request packets but
can be constructed by rewriting several fields of the request without
needing extra memory.

But, if a request packet size is just 64K or a little larger than that,
which is a invalid packet after reassembly, its fragments may be sent
as, with improper check by receivers, valid fragments, reassembly of
which will cause buffer overflow.

Another possibility of DOS by sending large ICMP echo requests should
be that they require tens of fragmentation reassembly, which consumes
a lot of resource of receivers especially when some fragments are
missed (maybe intentionally by senders) because fragmentation reassembly
buffers won't be released unless all the fragments are received and
reassembled.

Masataka Ohta

Bjorn_Mork · December 21, 2021, 4:38pm

Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp> writes:

No, an ICMP echo reply does not include the entire request packets

RFC792:
The data received in the echo message must be returned in the echo
reply message.

Masataka_Ohta · December 22, 2021, 12:24am

Bjorn Mork wrote:

No, an ICMP echo reply does not include the entire request packets

RFC792:
The data received in the echo message must be returned in the echo
reply message.

RFC792:

Echo or Echo Reply Message

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1