Unable to Reach m.root-servers.net from Comcast

Mailman List Mirror (Read Only)
1

Debated whether to bring this up on the OARC DNS Ops list or NANOG. Seem more of a network-y thing.

I happened to notice that I cannot query or ping m.root-servers.net’s IPv4 address from my home Comcast connection (SF Bay Area). IPv6 works. All of the other root servers are reachable via both IPv4 and IPv6.

Just me or all Comcast users?

Obviously, nothing critical. My DNS servers will work just fine with one root unavailable on one protocol. But curious what’s broken. Been this way for at least a couple of weeks.

Compare IPv4 and IPv6 traceroutes,

Mac-Air:~ fred$ traceroute -I m.root-servers.net

traceroute to m.root-servers.net (202.12.27.33), 64 hops max, 72 byte packets

1 moat (192.168.64.90) 3.004 ms 1.409 ms 1.155 ms

2 10.60.234.131 (10.60.234.131) 14.354 ms 17.153 ms 14.451 ms

3 po-313-1222-rur202.pleasanton.ca.sfba.comcast.net (96.110.179.157) 14.296 ms 11.150 ms 11.279 ms

4 po-200-xar02.pleasanton.ca.sfba.comcast.net (68.85.155.117) 11.752 ms 10.195 ms 11.783 ms

5 be-248-rar01.pleasanton.ca.sfba.comcast.net (96.108.99.129) 14.261 ms 11.228 ms 13.187 ms

6 be-398-ar01.hayward.ca.sfba.comcast.net (162.151.87.225) 13.577 ms 13.552 ms 13.725 ms

7 * * *

8 * * *

9 * * *

10 * * *

11 * * *

12 *^C

Mac-Air:~ fred$ traceroute6 -I m.root-servers.net

traceroute6 to m.root-servers.net (2001:dc3::35) from 2601:644:8f01:3b0:c842:51d6:e589:5203, 64 hops max, 16 byte packets

1 2601:644:8f01:3b0:4262:31ff:fe01:b8 1.753 ms 1.765 ms 1.343 ms

2 2001:558:101e:54::2 14.526 ms 26.113 ms 32.365 ms

3 po-313-1221-rur201.pleasanton.ca.sfba.comcast.net 28.170 ms 14.109 ms 11.374 ms

4 po-200-xar01.pleasanton.ca.sfba.comcast.net 11.746 ms 20.031 ms 42.526 ms

5 * * *

6 * * *

7 * * *

8 lo-0-v6.ear2.sanjose1.level3.net 15.106 ms 14.454 ms 13.240 ms

9 pt.telekomu.edge1.losangeles9.level3.net 15.921 ms 17.188 ms 16.294 ms

10 m.root-servers.net 17.032 ms 15.169 ms 14.892 ms

2

I would be opening a ticket with comcast. This appears to be off charter as it is a individual home user issue.

3

I’m getting similar results on residential Comcast up here in Michigan, US (Comcast’s Heartland region). Checking from a Google Cloud vm in Iowa, US, I was able to reach m.root-servers.net via both v4 and v6, but it looks like it’s routing to the Sao Paulo cluster in both cases (looking at https://m.root-servers.org/, I would have assumed it would have routed to SF’s cluster)

test-instance-3:~$ ping m.root-servers.net
PING m.root-servers.net(M.ROOT-SERVERS.NET (2001:dc3::35)) 56 data bytes
64 bytes from M.ROOT-SERVERS.NET (2001:dc3::35): icmp_seq=1 ttl=63 time=141 ms
64 bytes from M.ROOT-SERVERS.NET (2001:dc3::35): icmp_seq=2 ttl=63 time=141 ms
64 bytes from M.ROOT-SERVERS.NET (2001:dc3::35): icmp_seq=3 ttl=63 time=141 ms
^C
— m.root-servers.net ping statistics —
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 140.752/140.903/141.091/0.140 ms

test-instance-3:~$ dig +short m.root-servers.net
202.12.27.33

test-instance-3:~$ ping 202.12.27.33
PING 202.12.27.33 (202.12.27.33) 56(84) bytes of data.
64 bytes from 202.12.27.33: icmp_seq=1 ttl=254 time=143 ms
64 bytes from 202.12.27.33: icmp_seq=2 ttl=254 time=140 ms
64 bytes from 202.12.27.33: icmp_seq=3 ttl=254 time=141 ms
^C
— 202.12.27.33 ping statistics —
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 140.387/141.368/143.146/1.259 ms

test-instance-3:~$ traceroute -I m.root-servers.net
traceroute to m.root-servers.net (202.12.27.33), 64 hops max
1 * * *
2 142.251.64.24 140.646ms 140.640ms 140.749ms
3 187.16.215.148 140.371ms 140.292ms 140.314ms
4 202.12.27.33 139.841ms 140.324ms 139.980ms

test-instance-3:~$ dig -x 187.16.215.148
…
148.215.16.187.in-addr.arpa. 6423 IN PTR as7500.saopaulo.sp.ix.br.

…

test-instance-3:~$ traceroute6 m.root-servers.net
traceroute to m.root-servers.net (2001:dc3::35) from 2600:1900:4000:d977:0:1::, 30 hops max, 24 byte packets
1 * * *
2 2001:4860:0:1::7cd8 (2001:4860:0:1::7cd8) 140.1891 ms 140.0114 ms 140.0064 ms
3 as7500.saopaulo.sp.ix.br (2001:12f8::215:148) 140.7670 ms 140.7445 ms 140.7408 ms
4 M.ROOT-SERVERS.NET (2001:dc3::35) 140.5523 ms 140.4191 ms 140.4163 ms

4

Same here in East Central Illinois. v4 fails; v6 works.

Trace stops in Chicago.

5

Yes, definitely Comcast-wide. And I should have realized that was almost certainly true earlier. I did look at the Comcast looking glass listed at PeeringDB and even saw the problem, but didn’t put 2-and-2 together at the time. All I saw was that the path for the IPv4 prefix for m-root, 202.12.27.0/24, matched the IPv6 prefix, 2001:dc3::/32, and the origin was WIDE Project, AS7500

RP/0/RSP0/CPU0:route-server.newyork.ny.ibone#show bgp 202.12.27.0/24

Mon Aug 26 05:12:03.793 utc

BGP routing table entry for 202.12.27.0/24

Versions:

Process bRIB/RIB SendTblVer

Speaker 1100876456 1100876456

Last Modified: May 6 23:17:18.578 for 3y16w

Paths: (1 available, best #1)

Not advertised to any peer

Path #1: Received by speaker 0

Not advertised to any peer

3257 7500, (aggregated by 7500 202.12.27.167), (received & used)

66.208.229.9 from 66.208.229.9 (96.109.22.227)

Origin IGP, metric 0, localpref 250, valid, internal, atomic-aggregate, best, group-best

Received Path ID 0, Local Path ID 0, version 1100876456

Community: 7922:403 7922:3000

Originator: 96.109.22.227, Cluster list: 96.109.22.250, 96.109.22.50

RP/0/RSP0/CPU0:route-server.newyork.ny.ibone#show bgp ipv6 unicast 2001:dc3::/32

Mon Aug 26 05:14:43.682 utc

BGP routing table entry for 2001:dc3::/32

Versions:

Process bRIB/RIB SendTblVer

Speaker 873076831 873076831

Last Modified: May 6 23:15:42.581 for 3y16w

Paths: (1 available, best #1)

Not advertised to any peer

Path #1: Received by speaker 0

Not advertised to any peer

3257 7500, (aggregated by 7500 202.12.27.167)

2001:559::9 from 2001:559::9 (96.109.22.227)

Origin IGP, metric 0, localpref 250, valid, internal, atomic-aggregate, best, group-best

Received Path ID 0, Local Path ID 0, version 873076831

Community: 7922:403 7922:3000

Originator: 96.109.22.227, Cluster list: 96.109.22.250, 96.109.22.50

But there’s a /32 route for m.root-servers.net itself,

RP/0/RSP0/CPU0:route-server.newyork.ny.ibone#show bgp 202.12.27.33

Mon Aug 26 05:10:50.040 utc

BGP routing table entry for 202.12.27.33/32

Versions:

Process bRIB/RIB SendTblVer

Speaker 918848884 918848884

Last Modified: Jun 18 02:40:28.576 for 3y10w

Paths: (1 available, best #1, not advertised to EBGP peer)

Not advertised to any peer

Path #1: Received by speaker 0

Not advertised to any peer

64650, (received & used)

66.208.229.9 from 66.208.229.9 (96.109.22.250)

Origin IGP, metric 0, localpref 300, valid, internal, best, group-best

Received Path ID 0, Local Path ID 0, version 918848884

Community: 7922:403 7922:2900 no-export

Which is a private AS number. Looks like Comcast is blackholing m-root?

I honestly can’t imagine trying to explain this to frontline Xfinify support. Any Comcast lurkers around who might be able see what’s going on here?

6

Seems they fixed it.
I did check their route server to see what was going on, here’s the before and after:

Broken:
RP/0/RSP0/CPU0:route-server.newyork.ny.ibone#show bgp 202.12.27.33
Mon Aug 26 19:51:23.615 utc
BGP routing table entry for 202.12.27.33/32
Versions:
Process bRIB/RIB SendTblVer
Speaker 918848884 918848884
Last Modified: Jun 18 02:40:29.577 for 3y10w
Paths: (1 available, best #1, not advertised to EBGP peer)
Not advertised to any peer
Path #1: Received by speaker 0
Not advertised to any peer
64650, (received & used)
66.208.229.9 from 66.208.229.9 (96.109.22.250)
Origin IGP, metric 0, localpref 300, valid, internal, best, group-best
Received Path ID 0, Local Path ID 0, version 918848884
Community: 7922:403 7922:2900 no-export

Fixed:
RP/0/RSP0/CPU0:route-server.newyork.ny.ibone#show bgp 202.12.27.33
Mon Aug 26 22:20:06.647 utc
BGP routing table entry for 202.12.27.0/24
Versions:
Process bRIB/RIB SendTblVer
Speaker 1100876456 1100876456
Last Modified: May 6 23:17:19.747 for 3y16w
Paths: (1 available, best #1)
Not advertised to any peer
Path #1: Received by speaker 0
Not advertised to any peer
3257 7500, (aggregated by 7500 202.12.27.167), (received & used)
66.208.229.9 from 66.208.229.9 (96.109.22.227)
Origin IGP, metric 0, localpref 250, valid, internal, atomic-aggregate, best, group-best
Received Path ID 0, Local Path ID 0, version 1100876456
Community: 7922:403 7922:3000
Originator: 96.109.22.227, Cluster list: 96.109.22.250, 96.109.22.50

For whatever reason they had 202.12.27.33/32 entered into their routing table. I Wonder why they would have done that…