Re: Level3 4.2.2.1 and 4.2.2.2 not responding to pings, DNS queries work

Mailman List Mirror (Read Only)
1

I’m seeing some of the resolver IPs being filtering from various locations while responding from others, no doubt due to their use of anycast. I rarely ping test 4.2.2.2 anymore, having switched to 8.8.8.8 some time ago which I have a window up running 24x7 as a quick way to detect if my laptop is having connectivity issues from wherever it currently resides. It’s usually reliable, but I understand such traffic is the first to get filtered/dropped when needed so it’s just an initial indicator for me from which further testing can be performed when needed.

I will also mention it’s possible to detect which NS server/pool/location you are reaching on Level3’s network via the below dig query. This has been an invaluable tool over the years as IME they tend to break DNS sub-delegation at least once or twice a year and the more data you can provide to them about where the breakage is the faster you can get them to engage the DNS team to actually fix it vs. arguing with you that it’s not broken because it "works for them” (facepalm). And at least once I found they broke resolution on their authoritative name servers (ns1.l3.net or ns2.l3.net) and again they first told me it wasn’t a problem because DNS again the broken one(s) would timeout and then get the answer from another, working server (double-facepalm).

dig +short @4.2.2.2 hostname.bind CH TXT

I’ve also been provided this query to derive the same for Cloudflare’s NS servers:

dig CHAOS TXT id.server @1.1.1.1 +nsid

And the following for Cleanbrowsing NS servers (these power content filtering on Unifi networks):

nslookup -type=txt iptest.whois.dnscontest.cleanbrowsing.org

Below are the results I got for Level3 from various locations (pardon the wall of text). It looks like they’re making some changes to serve DNS queries off their NTP servers

From Level3 WDC1/McLean:

PING 4.2.2.1 (4.2.2.1) 56(84) bytes of data.

— 4.2.2.1 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.987/0.987/0.987/0.000 ms
“pubntp1.wdc12”
PING 4.2.2.2 (4.2.2.2) 56(84) bytes of data.

— 4.2.2.2 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.926/0.926/0.926/0.000 ms
“cns4.sjo1”
PING 4.2.2.3 (4.2.2.3) 56(84) bytes of data.

— 4.2.2.3 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.799/0.799/0.799/0.000 ms
“pubntp1.wdc12”
PING 4.2.2.4 (4.2.2.4) 56(84) bytes of data.

— 4.2.2.4 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.855/0.855/0.855/0.000 ms
“pubntp2.wdc12”
PING 4.2.2.5 (4.2.2.5) 56(84) bytes of data.

— 4.2.2.5 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.897/0.897/0.897/0.000 ms
“cns4.sjo1”
PING 4.2.2.6 (4.2.2.6) 56(84) bytes of data.

— 4.2.2.6 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.973/0.973/0.973/0.000 ms
"pubntp1.wdc12”

From Level3 SFO1/Sunnyvale:

PING 4.2.2.1 (4.2.2.1) 56(84) bytes of data.

— 4.2.2.1 ping statistics —
2 packets transmitted, 0 received, 100% packet loss, time 999ms

“cns3.sjo1”
PING 4.2.2.2 (4.2.2.2) 56(84) bytes of data.

— 4.2.2.2 ping statistics —
1 packets transmitted, 0 received, 100% packet loss, time 0ms

“cns2.sjo1”
PING 4.2.2.3 (4.2.2.3) 56(84) bytes of data.

— 4.2.2.3 ping statistics —
1 packets transmitted, 0 received, 100% packet loss, time 0ms

“cns1.sjo1”
PING 4.2.2.4 (4.2.2.4) 56(84) bytes of data.

— 4.2.2.4 ping statistics —
1 packets transmitted, 0 received, 100% packet loss, time 0ms

“cns2.sjo1”
PING 4.2.2.5 (4.2.2.5) 56(84) bytes of data.

— 4.2.2.5 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.275/0.275/0.275/0.000 ms
“cns3.sjo1”
PING 4.2.2.6 (4.2.2.6) 56(84) bytes of data.

— 4.2.2.6 ping statistics —
1 packets transmitted, 0 received, 100% packet loss, time 0ms

"cns3.sjo1”

From Comcast Atlanta:

PING 4.2.2.1 (4.2.2.1) 56(84) bytes of data.

— 4.2.2.1 ping statistics —
1 packets transmitted, 0 received, 100% packet loss, time 0ms

“cns2.atl2”
PING 4.2.2.2 (4.2.2.2) 56(84) bytes of data.

— 4.2.2.2 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 13.442/13.442/13.442/0.000 ms
“cns4.atl2”
PING 4.2.2.3 (4.2.2.3) 56(84) bytes of data.

— 4.2.2.3 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 13.431/13.431/13.431/0.000 ms
“cns2.atl2”
PING 4.2.2.4 (4.2.2.4) 56(84) bytes of data.

— 4.2.2.4 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 13.632/13.632/13.632/0.000 ms
“cns2.atl2”
PING 4.2.2.5 (4.2.2.5) 56(84) bytes of data.

— 4.2.2.5 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 12.656/12.656/12.656/0.000 ms
“cns3.atl2”
PING 4.2.2.6 (4.2.2.6) 56(84) bytes of data.

— 4.2.2.6 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 12.041/12.041/12.041/0.000 ms
"cns4.atl2”

From AT&T Atlanta:

PING 4.2.2.1 (4.2.2.1) 56(84) bytes of data.

— 4.2.2.1 ping statistics —
1 packets transmitted, 0 received, 100% packet loss, time 0ms

“cns2.atl2”
PING 4.2.2.2 (4.2.2.2) 56(84) bytes of data.

— 4.2.2.2 ping statistics —
1 packets transmitted, 0 received, 100% packet loss, time 0ms

“pubntp1.atl2”
PING 4.2.2.3 (4.2.2.3) 56(84) bytes of data.

— 4.2.2.3 ping statistics —
1 packets transmitted, 0 received, 100% packet loss, time 0ms

“cns3.atl2”
PING 4.2.2.4 (4.2.2.4) 56(84) bytes of data.

— 4.2.2.4 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 15.430/15.430/15.430/0.000 ms
“pubntp1.atl2”
PING 4.2.2.5 (4.2.2.5) 56(84) bytes of data.

— 4.2.2.5 ping statistics —
1 packets transmitted, 0 received, 100% packet loss, time 0ms

“cns3.atl2”
PING 4.2.2.6 (4.2.2.6) 56(84) bytes of data.

— 4.2.2.6 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 16.103/16.103/16.103/0.000 ms
"cns4.atl2”

From AT&T SF Bay Area Peninsula:

PING 4.2.2.1 (4.2.2.1) 56(84) bytes of data.

— 4.2.2.1 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 7.880/7.880/7.880/0.000 ms
“cns3.sjo1”
PING 4.2.2.2 (4.2.2.2) 56(84) bytes of data.

— 4.2.2.2 ping statistics —
1 packets transmitted, 0 received, 100% packet loss, time 0ms

“cns2.sjo1”
PING 4.2.2.3 (4.2.2.3) 56(84) bytes of data.

— 4.2.2.3 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 6.888/6.888/6.888/0.000 ms
“cns3.sjo1”
PING 4.2.2.4 (4.2.2.4) 56(84) bytes of data.

— 4.2.2.4 ping statistics —
2 packets transmitted, 0 received, 100% packet loss, time 1000ms

“cns4.sjo1”
PING 4.2.2.5 (4.2.2.5) 56(84) bytes of data.

— 4.2.2.5 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 7.173/7.173/7.173/0.000 ms
“cns1.sjo1”
PING 4.2.2.6 (4.2.2.6) 56(84) bytes of data.

— 4.2.2.6 ping statistics —
1 packets transmitted, 0 received, 100% packet loss, time 0ms

"cns2.sjo1”

From DRFortress Honolulu:

PING 4.2.2.1 (4.2.2.1) 56(84) bytes of data.

— 4.2.2.1 ping statistics —
1 packets transmitted, 0 received, 100% packet loss, time 0ms

“cns3.sjo1”
PING 4.2.2.2 (4.2.2.2) 56(84) bytes of data.

— 4.2.2.2 ping statistics —
1 packets transmitted, 0 received, 100% packet loss, time 0ms

“cns3.sjo1”
PING 4.2.2.3 (4.2.2.3) 56(84) bytes of data.

— 4.2.2.3 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 54.233/54.233/54.233/0.000 ms
“cns3.sjo1”
PING 4.2.2.4 (4.2.2.4) 56(84) bytes of data.

— 4.2.2.4 ping statistics —
1 packets transmitted, 0 received, 100% packet loss, time 0ms

“cns3.sjo1”
PING 4.2.2.5 (4.2.2.5) 56(84) bytes of data.

— 4.2.2.5 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 54.195/54.195/54.195/0.000 ms
“cns3.sjo1”
PING 4.2.2.6 (4.2.2.6) 56(84) bytes of data.

— 4.2.2.6 ping statistics —
1 packets transmitted, 0 received, 100% packet loss, time 0ms

"cns4.sjo1”

From Hawaiian Telecom Honolulu (oddly from here I get no response to the hostname.bind dig queries):

PING 4.2.2.1 (4.2.2.1) 56(84) bytes of data.

— 4.2.2.1 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 54.727/54.727/54.727/0.000 ms
PING 4.2.2.2 (4.2.2.2) 56(84) bytes of data.

— 4.2.2.2 ping statistics —
1 packets transmitted, 0 received, 100% packet loss, time 0ms

PING 4.2.2.3 (4.2.2.3) 56(84) bytes of data.

— 4.2.2.3 ping statistics —
1 packets transmitted, 0 received, 100% packet loss, time 0ms

PING 4.2.2.4 (4.2.2.4) 56(84) bytes of data.

— 4.2.2.4 ping statistics —
1 packets transmitted, 0 received, 100% packet loss, time 0ms

PING 4.2.2.5 (4.2.2.5) 56(84) bytes of data.

— 4.2.2.5 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 56.765/56.765/56.765/0.000 ms
PING 4.2.2.6 (4.2.2.6) 56(84) bytes of data.

— 4.2.2.6 ping statistics —
1 packets transmitted, 0 received, 100% packet loss, time 0ms

From Unitas Seacacus:

PING 4.2.2.1 (4.2.2.1) 56(84) bytes of data.

— 4.2.2.1 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.062/2.062/2.062/0.000 ms
“cns2.nyc6”
PING 4.2.2.2 (4.2.2.2) 56(84) bytes of data.

— 4.2.2.2 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.012/2.012/2.012/0.000 ms
“cns2.nyc6”
PING 4.2.2.3 (4.2.2.3) 56(84) bytes of data.

— 4.2.2.3 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.651/1.651/1.651/0.000 ms
“cns3.nyc6”
PING 4.2.2.4 (4.2.2.4) 56(84) bytes of data.

— 4.2.2.4 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.055/2.055/2.055/0.000 ms
“cns2.nyc6”
PING 4.2.2.5 (4.2.2.5) 56(84) bytes of data.

— 4.2.2.5 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.102/2.102/2.102/0.000 ms
“cns3.nyc6”
PING 4.2.2.6 (4.2.2.6) 56(84) bytes of data.

— 4.2.2.6 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.626/1.626/1.626/0.000 ms
"cns3.nyc6”

From Verizon FIOS New York (oddly from here I get no response to the hostname.bind dig queries):

PING 4.2.2.1 (4.2.2.1) 56(84) bytes of data.

— 4.2.2.1 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 4.355/4.355/4.355/0.000 ms
PING 4.2.2.2 (4.2.2.2) 56(84) bytes of data.

— 4.2.2.2 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 3.764/3.764/3.764/0.000 ms
PING 4.2.2.3 (4.2.2.3) 56(84) bytes of data.

— 4.2.2.3 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 4.053/4.053/4.053/0.000 ms
PING 4.2.2.4 (4.2.2.4) 56(84) bytes of data.

— 4.2.2.4 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 3.394/3.394/3.394/0.000 ms
PING 4.2.2.5 (4.2.2.5) 56(84) bytes of data.

— 4.2.2.5 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 3.442/3.442/3.442/0.000 ms
PING 4.2.2.6 (4.2.2.6) 56(84) bytes of data.

— 4.2.2.6 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 4.944/4.944/4.944/0.000 ms

From Allied Telecom in Washington DC:

PING 4.2.2.1 (4.2.2.1) 56(84) bytes of data.

— 4.2.2.1 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.683/1.683/1.683/0.000 ms
“pubntp1.wdc12”
PING 4.2.2.2 (4.2.2.2) 56(84) bytes of data.

— 4.2.2.2 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.592/1.592/1.592/0.000 ms
“pubntp2.wdc12”
PING 4.2.2.3 (4.2.2.3) 56(84) bytes of data.

— 4.2.2.3 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.776/1.776/1.776/0.000 ms
“cns1.wdc12”
PING 4.2.2.4 (4.2.2.4) 56(84) bytes of data.

— 4.2.2.4 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.810/1.810/1.810/0.000 ms
“pubntp2.wdc12”
PING 4.2.2.5 (4.2.2.5) 56(84) bytes of data.

— 4.2.2.5 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.729/1.729/1.729/0.000 ms
“cns1.wdc12”
PING 4.2.2.6 (4.2.2.6) 56(84) bytes of data.

— 4.2.2.6 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.432/1.432/1.432/0.000 ms
"cns4.sjo1”

-T

2

Dan,

That dig tip for identifying the NS phy loc is very nice! That’s something I can put in our support procedures for DNS troubleshooting.

-mel

3

Dan’s not here, man! But I’m glad to spread the word so it can help more people in their troubleshoot efforts.

-T

4

A tangent came to mind while reading over this thread: Google has a similar feature to aid in troubleshooting.
https://redirector.googlevideo.com/report_mapping

It will indicate which YouTube node/cluster your current IP prefix is currently served by; I have found this to be helpful in the past.

5

Yup. Note that many authoritative DNS servers also support RFC5001 - “DNS Name Server Identifier (NSID) Option” (NSID). This can be really valuable for figuring out which instance of an Anycast server you are hitting.

E.g:

$ dig +nsid +edns NS . @b.root-servers.net

[[SNIP]]
;; OPT PSEUDOSECTION:

; EDNS: version: 0, flags:; udp: 1232

; NSID: 62 31 2d 69 61 64 (“b1-iad”)

;; QUESTION SECTION:

;. IN NS

;; ANSWER SECTION:

. 518400 IN NS a.root-servers.net.

. 518400 IN NS b.root-servers.net.

. 518400 IN NS c.root-servers.net.

[[SNIP]]

W