I'm posting this here in hopes of getting the attention of someone that
can get this issue resolved, or at least an internal ticket filed. I've
tried the customer-facing tech support and not been able to get such a
thing done.
In short, from within Spectrum's US IPv6 network (verified in both North
Carolina and Ohio), dfw.source.kernel.org (2604:1380:4641:c500::1) is
unreachable and connections time out. This site is otherwise fine and
globally accessible via IPv6, tested on both Qwest and T-Mobile hosted
systems. This is a regression from some time in early April this year.
We’ve been having IPv6 issues for weeks in Texas on Spectrum, we have an Enterprise support ticket open on our IP transit circuits but I’ve seen no real movement.
Our issue is being tracked internally as INC000026440632, but I believe it’s specific to the DFW area.
If it is helpful, I am on Spectrum in Ohio and I can reach that address from my IPv6 test system. This is on a commercial account with a provider assigned network. My system is at ip6echo.net. You are welcome to ping and traceroute to it if that is helpful to you.
A friend CANNOT get to that address from a Spectrum residential account also in my area.
For what it’s worth, on Spectrum Residential here in LAX and I can reach ip6echo.net and Google’s public ipv6.google.com sites via residential service… I have my own modems and routers, which may make the difference in my case… Spectrum does indicate that IPv6 my need to be enabled on their routers…
Actual data from a Spectrum residential customer in DFW.
First, IPv4:
<ns:~> trace dfw.source.kernel.org
traceroute to dfw.source.kernel.org (139.178.84.217), 64 hops max, 40
byte packets
1 my.router 0.389 ms 0.350 ms 0.292 ms
2 142-254-130-077.inf.spectrum.com (142.254.130.77) 8.423 ms 8.408
ms 8.080 ms
3 lag-63.artrtx2801h.netops.charter.com (24.28.88.17) 27.167 ms
25.065 ms 21.977 ms
4 lag-22.artntxaf01r.netops.charter.com (24.175.49.233) 10.718 ms
10.083 ms 15.886 ms
5 lag-23.mcr11crtntxjt.netops.charter.com (24.175.36.224) 13.386 ms
11.560 ms 11.297 ms
6 lag-21.rcr01dllatx37.netops.charter.com (24.175.49.0) 11.339 ms
lag-28.rcr01dllatx37.netops.charter.com (24.175.33.246) 11.904 ms
128.186 ms
7 lag-414.dllstx976iw-bcr00.netops.charter.com (66.109.6.52) 12.603 ms
lag-14.dllstx976iw-bcr00.netops.charter.com (66.109.6.88) 12.172 ms
lag-414.dllstx976iw-bcr00.netops.charter.com (66.109.6.52) 12.299 ms
8 lag-302.pr3.dfw10.netops.charter.com (209.18.43.77) 21.570 ms
lag-0.pr3.dfw10.netops.charter.com (66.109.5.121) 11.763 ms
lag-302.pr3.dfw10.netops.charter.com (209.18.43.77) 12.182 ms
9 dls-b23-link.ip.twelve99.net (62.115.156.208) 11.515 ms * 11.706 ms
10 packethost-ic-369414.ip.twelve99-cust.net (213.248.72.3) 11.870
ms 30.246 ms 18.199 ms
11 * * *
12 * * *
13 dfw.source.kernel.org (139.178.84.217) 12.021 ms 12.076 ms 11.922 ms
ping dfw.source.kernel.org
PING dfw.source.kernel.org (139.178.84.217): 56 data bytes
64 bytes from 139.178.84.217: icmp_seq=0 ttl=50 time=11.590 ms
64 bytes from 139.178.84.217: icmp_seq=1 ttl=50 time=11.785 ms
IPv6:
<ns:~> trace6 dfw.source.kernel.org
traceroute6 to dfw.source.kernel.org (2604:1380:4641:c500::1) from
2603:8080:REDACTED, 64 hops max, 20 byte packets
1 2603-8080-REDACTED.res6.spectrum.com 0.404 ms 0.340 ms 0.322 ms
2 2603-90c5-0003-000e-0000-0000-0000-0001.inf6.spectrum.com 10.308
ms 7.901 ms 9.902 ms
3 lag-63.artrtx2801h.netops.charter.com 17.008 ms 10.523 ms 11.077 ms
4 lag-22.artntxaf01r.netops.charter.com 14.638 ms * *
5 lag-23.mcr11crtntxjt.netops.charter.com 11.090 ms 11.612 ms 12.234 ms
6 * * *
7 lag-414.dllstx976iw-bcr00.netops.charter.com 12.572 ms *
lag-24.dllstx976iw-bcr00.netops.charter.com 12.160 ms
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 *^C
This has been “resolved", I finally got through to some awesome engineer at Spectrum who has rerouted traffic while they work with their hardware vendor (thanks Jake):
1 2605:6000:0:8::f:7acc (2605:6000:0:8::f:7acc) 0.372 ms 0.323 ms 0.282 ms
2 ae15.ARTNTXAF02H.chtrse.com (2605:6000:0:4::f:87e9) 4.002 ms 3.977 ms 10.884 ms
3 * * *
4 lag-23.mcr11dllbtxlb.netops.charter.com (2605:6000:0:4::2:1e) 1.464 ms 1.574 ms 1.560 ms
5 * * *
6 * *
lag-26.hstqtx0209w-bcr00.netops.charter.com (2001:1998:0:4::528) 6.351 ms
7 * * *
8 lag-0.pr2.atl20.netops.charter.com (2001:1998:0:4::51f) 18.577 ms 18.572 ms 18.428 ms
9 * * *
10 e0-36.core2.bna1.he.net (2001:470:0:429::2) 32.553 ms 32.720 ms 32.563 ms
11 * * *
12 * * *
13 equinix-ix.dfw2.packet.net (2001:504:0:5:0:5:4825:1) 24.591 ms 24.553 ms 24.604 ms
14 * * *
15 * * *
16 dfw.source.kernel.org (2604:1380:4641:c500::1) 24.423 ms 26.020 ms 24.415 ms
One of the tools that I’ve used in the past is the RIPE Atlas service to measure these things. It’s helped me isolate IP space reachability issues for new announcements, because you can get enough of a random sample of hosts to isolate things, and enough data about that endpoint to launch follow-up measurements.
My issue was just trying to convince Spectrum to look into the problem in the first place, I brought the Atlas probe receipts because it’s such a helpful tool, but wasn’t able to get through to anyone helpful (acct mgr, noc email, even the escalation list) until I started lighting fires filing FCC complaints and using social media (which thankfully worked).
Not sure how accurate it is (I hope it isn’t), but some of the techs I spoke to said a lot of the internal tooling for troubleshooting is incapable of dealing with IPv6, so they weren’t able to do things like run traceroutes to confirm what I was seeing. My guess is that this issue was caught in a catch-22 where they needed impossible to obtain proof on their end to escalate to a team who can actually deal with the issue.
Sucks for us folk who went all in on v6 only to find out not even the ISP can help us.
My issue was just trying to convince Spectrum to look into the problem in the first place, I brought the Atlas probe receipts because it’s such a helpful tool, but wasn’t able to get through to anyone helpful (acct mgr, noc email, even the escalation list) until I started lighting fires filing FCC complaints and using social media (which thankfully worked).
Sucks for us folk who went all in on v6 only to find out not even the ISP can help us.
While this situation presented itself on IPv6, it’s certainly not a problem uniquely experienced by or related to IPv6. There are all manners of issues, regardless of address family, that can take too much time to resolve because of issues with knowledge / tooling / contacts / etc .
There is a reason “Does anyone know someone over at FOO?” has been a common phrase for decades.