Spectrum networks IPv6 access issue

Mailman List Mirror (Read Only)
1

Hey all,

I'm posting this here in hopes of getting the attention of someone that
can get this issue resolved, or at least an internal ticket filed. I've
tried the customer-facing tech support and not been able to get such a
thing done.

In short, from within Spectrum's US IPv6 network (verified in both North
Carolina and Ohio), dfw.source.kernel.org (2604:1380:4641:c500::1) is
unreachable and connections time out. This site is otherwise fine and
globally accessible via IPv6, tested on both Qwest and T-Mobile hosted
systems. This is a regression from some time in early April this year.

2

We’ve been having IPv6 issues for weeks in Texas on Spectrum, we have an Enterprise support ticket open on our IP transit circuits but I’ve seen no real movement.

Our issue is being tracked internally as INC000026440632, but I believe it’s specific to the DFW area.

Daniel Marks
d@nielmarks.com

3

If it is helpful, I am on Spectrum in Ohio and I can reach that address from my IPv6 test system. This is on a commercial account with a provider assigned network. My system is at ip6echo.net. You are welcome to ping and traceroute to it if that is helpful to you.

A friend CANNOT get to that address from a Spectrum residential account also in my area.

Brandon

4

For what it’s worth, on Spectrum Residential here in LAX and I can reach ip6echo.net and Google’s public ipv6.google.com sites via residential service… I have my own modems and routers, which may make the difference in my case… Spectrum does indicate that IPv6 my need to be enabled on their routers…

https://www.spectrum.net/support/internet/ipv6-faq

jeffp@jeffp-Linux:~$ ping -6 ip6echo.net
PING ip6echo.net(ip6echo.net (2600:5c01:f876:10::53)) 56 data bytes
64 bytes from ip6echo.net (2600:5c01:f876:10::53): icmp_seq=1 ttl=51 time=76.1 ms
64 bytes from ip6echo.net (2600:5c01:f876:10::53): icmp_seq=2 ttl=51 time=78.4 ms
64 bytes from ip6echo.net (2600:5c01:f876:10::53): icmp_seq=3 ttl=51 time=76.5 ms

jeffp@jeffp-Linux:~$ ping -6 ipv6.google.com
PING ipv6.google.com(lax31s06-in-x0e.1e100.net (2607:f8b0:4007:811::200e)) 56 data bytes
64 bytes from lax31s06-in-x0e.1e100.net (2607:f8b0:4007:811::200e): icmp_seq=1 ttl=116 time=12.6 ms
64 bytes from lax31s06-in-x0e.1e100.net (2607:f8b0:4007:811::200e): icmp_seq=2 ttl=116 time=14.4 ms

I normally use the Google IPv6 site for such tests to avoid irritating private hosts, so a big thanks to Brandon for offering that alternative!

Jeffp

5

I’m in the northeast and I can confirm on a Spectrum Enterprise connection I have issues tracerouting to 2604:1380:4641:c500::1.

Gets to e0-33.core1.nyc7.he.net and then packets just disappear:

Host Loss% Snt Last Avg Best Wrst StDev

  1. _gateway 0.0% 108 1.0 1.7 0.8 26.3 3.3
  2. 2602:XXX:XXXX:X::X 0.0% 108 4.4 6.6 4.2 27.9 4.0
  3. 2602:XXX:XXXX:X::XXX 0.0% 108 3.1 3.0 3.0 3.5 0.0
  4. 2604:XXXX:X:X::x:XXXX 0.0% 108 5.2 4.8 3.3 37.1 5.0
  5. lag-15.srspny0101h.netops.charter.com 0.0% 108 3.9 20.8 3.6 371.6 45.4
  6. lag-32.albynyyf02r.netops.charter.com 44.4% 108 4.9 4.9 4.8 5.5 0.1
  7. lag-26.rcr01albynyyf.netops.charter.com 80.4% 108 5.1 5.3 5.0 7.7 0.6
  8. lag-416.nycmny837aw-bcr00.netops.charter.com 57.9% 108 37.4 14.5 12.7 54.4 7.1
  9. lag-1.pr2.nyc20.netops.charter.com 0.0% 108 49.8 17.1 12.3 65.7 11.0
  10. e0-33.core1.nyc7.he.net 0.0% 108 12.9 13.2 12.8 19.8 0.7
  11. (waiting for reply)

Trying from another network and I was able to get through:

Host Loss% Snt Last Avg Best Wrst StDev

  1. ca.tor.ix.caramelfox.net 0.0% 25 45.7 45.9 43.9 50.9 2.0
  2. (waiting for reply)
  3. (waiting for reply)
  4. ve955.core2.stl1.he.net 56.0% 25 62.0 63.9 62.0 67.6 2.0
  5. (waiting for reply)
  6. (waiting for reply)
  7. (waiting for reply)
  8. equinix-ix.dfw2.packet.net 0.0% 25 81.2 86.0 77.8 103.3 8.4
  9. (waiting for reply)
  10. (waiting for reply)
  11. dfw.source.kernel.org 0.0% 24 77.6 78.1 77.0 82.5 1.1

Maybe something in Hurricane Electric’s network potentially not getting these packets where they need to go?

6

Just adding another data point of a failure from Spectrum IPv6 to that host in question.

From here in Milledgeville Georgia that host seems to be unreachable, can I get to hurricane electrics network and seems to die with them.

Apologies for this output but I’m just using an app that I already had on my phone. Didn’t have a PC available…

#1

  • RTT [ms]: 7.8
  • Probe Send Time: 21:26:22
  • IP Address: 2600:6c5a:47f:xxxx:9e53:22ff:fe95:d858
  • TTL: 64
  • AS Number: AS20115
  • AS Name: CHARTER-20115
  • Country Name: United States
  • Country Code: US
  • Time Zone: America/New_York
  • Region: GA
  • City: Milledgeville
  • Latitude: 33.08
  • Longitude: -83.24

#2

  • Probe Send Time: 21:26:23

#3

  • RTT [ms]: 16.5
  • Probe Send Time: 21:26:26
  • IP Address: 2001:506:100:548f::4
  • Hostname: lag-22-10.dtr01mdvlga.netops.charter.com
  • TTL: 62
  • Country Name: United States
  • Country Code: US
  • Time Zone: America/New_York
  • Region: GA
  • City: Columbus
  • Latitude: 32.47
  • Longitude: -84.98

#4

  • Probe Send Time: 21:26:26

#5

  • RTT [ms]: 21.1
  • Probe Send Time: 21:26:29
  • IP Address: 2001:506:100:5507::4
  • Hostname: lag-20.rcr01sghlgaao.netops.charter.com
  • TTL: 59
  • Country Name: United States
  • Country Code: US
  • Time Zone: America/New_York
  • Region: GA
  • City: Sugar Hill
  • Latitude: 34.11
  • Longitude: -84.00

#6

  • Probe Send Time: 21:26:29

#7

  • Probe Send Time: 21:26:32

#8

  • Probe Send Time: 21:26:35

#9

  • RTT [ms]: 35.3
  • Probe Send Time: 21:26:38
  • IP Address: 2001:470:0:42a::2
  • Hostname: e0-35.core2.bna1.he.net
  • TTL: 56
  • AS Number: AS6939
  • AS Name: HURRICANE
  • Country Name: United States
  • Country Code: US
  • Time Zone: America/Chicago

#10

  • Probe Send Time: 21:26:38

#11

  • Probe Send Time: 21:26:41

#12

  • Probe Send Time: 21:26:44

#13

  • Probe Send Time: 21:26:47

#14

  • Probe Send Time: 21:26:50

#15

  • Probe Send Time: 21:26:53

#16

  • Probe Send Time: 21:26:56

#17

  • Probe Send Time: 21:27:00

#18

  • Probe Send Time: 21:27:03
7

you have permission to traverse this host but not talk to it.

8

Actual data from a Spectrum residential customer in DFW.

First, IPv4:

<ns:~> trace dfw.source.kernel.org
traceroute to dfw.source.kernel.org (139.178.84.217), 64 hops max, 40
byte packets
1 my.router 0.389 ms 0.350 ms 0.292 ms
2 142-254-130-077.inf.spectrum.com (142.254.130.77) 8.423 ms 8.408
ms 8.080 ms
3 lag-63.artrtx2801h.netops.charter.com (24.28.88.17) 27.167 ms
25.065 ms 21.977 ms
4 lag-22.artntxaf01r.netops.charter.com (24.175.49.233) 10.718 ms
10.083 ms 15.886 ms
5 lag-23.mcr11crtntxjt.netops.charter.com (24.175.36.224) 13.386 ms
11.560 ms 11.297 ms
6 lag-21.rcr01dllatx37.netops.charter.com (24.175.49.0) 11.339 ms
    lag-28.rcr01dllatx37.netops.charter.com (24.175.33.246) 11.904 ms
128.186 ms
7 lag-414.dllstx976iw-bcr00.netops.charter.com (66.109.6.52) 12.603 ms
    lag-14.dllstx976iw-bcr00.netops.charter.com (66.109.6.88) 12.172 ms
    lag-414.dllstx976iw-bcr00.netops.charter.com (66.109.6.52) 12.299 ms
8 lag-302.pr3.dfw10.netops.charter.com (209.18.43.77) 21.570 ms
    lag-0.pr3.dfw10.netops.charter.com (66.109.5.121) 11.763 ms
    lag-302.pr3.dfw10.netops.charter.com (209.18.43.77) 12.182 ms
9 dls-b23-link.ip.twelve99.net (62.115.156.208) 11.515 ms * 11.706 ms
10 packethost-ic-369414.ip.twelve99-cust.net (213.248.72.3) 11.870
ms 30.246 ms 18.199 ms
11 * * *
12 * * *
13 dfw.source.kernel.org (139.178.84.217) 12.021 ms 12.076 ms 11.922 ms

ping dfw.source.kernel.org
PING dfw.source.kernel.org (139.178.84.217): 56 data bytes
64 bytes from 139.178.84.217: icmp_seq=0 ttl=50 time=11.590 ms
64 bytes from 139.178.84.217: icmp_seq=1 ttl=50 time=11.785 ms

IPv6:

<ns:~> trace6 dfw.source.kernel.org
traceroute6 to dfw.source.kernel.org (2604:1380:4641:c500::1) from
2603:8080:REDACTED, 64 hops max, 20 byte packets
1 2603-8080-REDACTED.res6.spectrum.com 0.404 ms 0.340 ms 0.322 ms
2 2603-90c5-0003-000e-0000-0000-0000-0001.inf6.spectrum.com 10.308
ms 7.901 ms 9.902 ms
3 lag-63.artrtx2801h.netops.charter.com 17.008 ms 10.523 ms 11.077 ms
4 lag-22.artntxaf01r.netops.charter.com 14.638 ms * *
5 lag-23.mcr11crtntxjt.netops.charter.com 11.090 ms 11.612 ms 12.234 ms
6 * * *
7 lag-414.dllstx976iw-bcr00.netops.charter.com 12.572 ms *
    lag-24.dllstx976iw-bcr00.netops.charter.com 12.160 ms
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 *^C

ping6 dfw.source.kernel.org
PING6(56=40+8+8 bytes) 2603:8080:REDACTED --> 2604:1380:4641:c500::1
^C
--- dfw.source.kernel.org ping6 statistics ---
5 packets transmitted, 0 packets received, 100.0% packet loss

I have a Linode VM in Dallas that I also can't get to via IPv6.
Traffic appears to take the same path for IPv4.

9

This has been “resolved", I finally got through to some awesome engineer at Spectrum who has rerouted traffic while they work with their hardware vendor (thanks Jake):

 1  2605:6000:0:8::f:7acc (2605:6000:0:8::f:7acc)  0.372 ms  0.323 ms  0.282 ms
 2  ae15.ARTNTXAF02H.chtrse.com (2605:6000:0:4::f:87e9)  4.002 ms  3.977 ms  10.884 ms
 3  * * *
 4  lag-23.mcr11dllbtxlb.netops.charter.com (2605:6000:0:4::2:1e)  1.464 ms  1.574 ms  1.560 ms
 5  * * *
 6  * *
    lag-26.hstqtx0209w-bcr00.netops.charter.com (2001:1998:0:4::528)  6.351 ms
 7  * * *
 8  lag-0.pr2.atl20.netops.charter.com (2001:1998:0:4::51f)  18.577 ms  18.572 ms  18.428 ms
 9  * * *
10  e0-36.core2.bna1.he.net (2001:470:0:429::2)  32.553 ms  32.720 ms  32.563 ms
11  * * *
12  * * *
13  equinix-ix.dfw2.packet.net (2001:504:0:5:0:5:4825:1)  24.591 ms  24.553 ms  24.604 ms
14  * * *
15  * * *
16  dfw.source.kernel.org (2604:1380:4641:c500::1)  24.423 ms  26.020 ms  24.415 ms
10

One of the tools that I’ve used in the past is the RIPE Atlas service to measure these things. It’s helped me isolate IP space reachability issues for new announcements, because you can get enough of a random sample of hosts to isolate things, and enough data about that endpoint to launch follow-up measurements.

- Jared

11

I can confirm things have been resolved here, many thanks all!

12

My issue was just trying to convince Spectrum to look into the problem in the first place, I brought the Atlas probe receipts because it’s such a helpful tool, but wasn’t able to get through to anyone helpful (acct mgr, noc email, even the escalation list) until I started lighting fires filing FCC complaints and using social media (which thankfully worked).

Not sure how accurate it is (I hope it isn’t), but some of the techs I spoke to said a lot of the internal tooling for troubleshooting is incapable of dealing with IPv6, so they weren’t able to do things like run traceroutes to confirm what I was seeing. My guess is that this issue was caught in a catch-22 where they needed impossible to obtain proof on their end to escalate to a team who can actually deal with the issue.

Sucks for us folk who went all in on v6 only to find out not even the ISP can help us.

-Daniel Marks

13

We know the feeling well. Try porting from them…..

14

My issue was just trying to convince Spectrum to look into the problem in the first place, I brought the Atlas probe receipts because it’s such a helpful tool, but wasn’t able to get through to anyone helpful (acct mgr, noc email, even the escalation list) until I started lighting fires filing FCC complaints and using social media (which thankfully worked).

Sucks for us folk who went all in on v6 only to find out not even the ISP can help us.

While this situation presented itself on IPv6, it’s certainly not a problem uniquely experienced by or related to IPv6. There are all manners of issues, regardless of address family, that can take too much time to resolve because of issues with knowledge / tooling / contacts / etc .

There is a reason “Does anyone know someone over at FOO?” has been a common phrase for decades.