IPv6 version of www.qwest.com/www.centurylink.com has been down for 10 days

The IPv6 version of www.qwest.com has been down for 10 days. Wget shows a
301 to www.centurylink.com, but that also fails. Emails to the nocs at both
companies have gone unanswered. Unless HE is deployed in a web browser,
this behavior leads to a bad end-user experience.

If anyone can prod either of these two companies that would be much
appreciated.

Frank

nagios:/home/fbulk# wget -6 www.qwest.com
--2011-08-18 00:32:40-- http://www.qwest.com/
Resolving www.qwest.com... 2001:428:b21:1::20
Connecting to www.qwest.com|2001:428:b21:1::20|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://www.centurylink.com/ [following]
--2011-08-18 00:32:40-- http://www.centurylink.com/
Resolving www.centurylink.com... 2001:428:b21:1::22
Connecting to www.centurylink.com|2001:428:b21:1::22|:80... failed:
Connection timed out.
Retrying.

--2011-08-18 00:33:02-- (try: 2) http://www.centurylink.com/
Connecting to www.centurylink.com|2001:428:b21:1::22|:80... failed:
Connection timed out.
Retrying.

--2011-08-18 00:33:25-- (try: 3) http://www.centurylink.com/
Connecting to www.centurylink.com|2001:428:b21:1::22|:80... failed:
Connection timed out.
Retrying.

--2011-08-18 00:33:49-- (try: 4) http://www.centurylink.com/
Connecting to www.centurylink.com|2001:428:b21:1::22|:80... failed:
Connection timed out.
Retrying.

Etc...

It seems that any IPv6 efforts by organisations are best effort at most with of course some notable exceptions who seem to offer a really very good service (HE for example). It's starting to get to a point now, I think, that some end users have IPv6 (Andrews and Arnold have offered IPv6 for years) and issues such as these are just going to start to give IPv6 a bad name in the eyes of consumers.

It'd really suck for end users to start actively avoiding IPv6 connectivity because it keeps breaking and for organisations that have active AAAA records to break peoples connectivity to their resources.

Thus spake Leigh Porter (leigh.porter@ukbroadband.com) on Thu, Aug 18, 2011 at 11:47:19AM +0000:

It seems that any IPv6 efforts by organisations are best effort at most with of course some notable exceptions who seem to offer a really very good service (HE for example). It's starting to get to a point now, I think, that some end users have IPv6 (Andrews and Arnold have offered IPv6 for years) and issues such as these are just going to start to give IPv6 a bad name in the eyes of consumers.

It'd really suck for end users to start actively avoiding IPv6 connectivity because it keeps breaking and for organisations that have active AAAA records to break peoples connectivity to their resources.

This, as Frank points out is why getting Happy Eyeballs support into
applications like web browsers is so important. I think modern versions
of Chrome & Firefox do this. Safari does something similar, but
arguably more naive. I don't know about IE.

Dale

+1 -- I'm all for publishing AAAA records as everyone knows, but, if you publish AAAA records for a consumer facing service, please support and monitor that service with a similar level to what you do for your IPv4 versions of the service.

The coming years are going to be difficult enough for end-users without adding unnecessary anti-IPv6 sentiments to the mix.

Owen

It'd really suck for end users to start actively avoiding IPv6 connectivity because it keeps breaking and for organisations that have active AAAA records to break peoples connectivity to their resources.

+1 -- I'm all for publishing AAAA records as everyone knows, but, if you publish AAAA records for a consumer facing service, please support and monitor that service with a similar level to what you do for your IPv4 versions of the service.

The coming years are going to be difficult enough for end-users without adding unnecessary anti-IPv6 sentiments to the mix.

Owen

+1 to Owen's comment.

I'd also add some more comments:

A lot of eyeballs that have v6 right now are the people with a lot of clue. Do you want these people, who'll often be buying or recommending your services to rate your ability to deliver as a fail? Our experience with IPv6 consumer broadband has been that the early adopters are the people who, well, goto IETF meetings, follow standards and ask the bloody hard questions.

Even given the Happy Eyeballs (Did Hurricane PAY for it to be abbrievated as HE?? :slight_smile: ) most end users prefer IPv6 over IPv4. Deeply this means there is a tendency for v6 traffic to grow and be more important to connectivity than you may imagine. The tipping point for IPv6 traffic being dominant I suspect is going to be a lower threshold of take up than people might expect. Consider this when thinking about the level of thought you give to IPv6 infrastructure and PPS rates.

MMC

FYI, the issue is not resolved and I've not heard from either of the
companies suggesting that they're working on it.

Note their commitment to IPv6 in these releases:
http://www.prnewswire.com/news-releases/centurylink-joins-internet-community
-in-world-ipv6-day-123089908.html
http://news.centurylink.com/index.php?s=43&item=2129

Frank

I just noticed that the quad-A records for both those two hosts are now
gone. DNS being what it is, I'm not sure when that happened, but our
monitoring system couldn't get the AAAA for www.qwest.com about half an hour
ago.

Hopefully CenturyLink is actively working towards IPv6-enabling their sites
again.

Frank

Charter.com has also remove the quad-A's for www.charter.com. My monitoring
system alerted me this afternoon that it couldn't get to the v6 version of
their website. Because of DNS caching, I don't know how many hours or days
ago it was removed.

Frank

The Qwest one died roughly around the time of their merger/migration to
Centurylink web sites. I did bring up the issue with them as a customer,
and it seems the response was to disable publicly-facing IPV6 services (and
associated AAAA records) for the time being, as you observed.

Not that I agree with the "fix", but it is what it is.

A Chrome plugin alerted me to the fact that savvis.com has an AAAA for
www.savvis.com. Unfortunately access to that host over IPv6 is down, too.

Frank

A Chrome plugin alerted me to the fact that savvis.com has an AAAA for
www.savvis.com. Unfortunately access to that host over IPv6 is down, too.

Frank

The fault must be local to you. Works fine from here.

Mark

Strange, not for me.

nagios:/etc/nagios3# ping6 www.savvis.com
PING www.savvis.com(2001:460:100:1000::37) 56 data bytes
64 bytes from 2001:460:100:1000::37: icmp_seq=1 ttl=239 time=55.5 ms
64 bytes from 2001:460:100:1000::37: icmp_seq=2 ttl=239 time=55.4 ms
64 bytes from 2001:460:100:1000::37: icmp_seq=3 ttl=239 time=55.6 ms
64 bytes from 2001:460:100:1000::37: icmp_seq=4 ttl=239 time=55.4 ms
^C
--- www.savvis.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2999ms
rtt min/avg/max/mdev = 55.465/55.517/55.608/0.176 ms
nagios:/etc/nagios3# wget -6 www.savvis.com
--2011-09-05 20:57:08-- http://www.savvis.com/
Resolving www.savvis.com... 2001:460:100:1000::37
Connecting to www.savvis.com|2001:460:100:1000::37|:80... failed: Connection
refused.
nagios:/etc/nagios3#

Frank

I'm with Frank on this one: ICMP yes, HTTP/HTTPS no, via native IPv6 (multiple locations). No, wait -- it shows as open from a couple tunnels (both HE & SixXS). So it's not consistent. Lovely.

  Closed from:
2607:ff50::/32 (native)
2607:fcd0::/32 (native)

  Open from:
2001:1938::/32 (SixXS tunnel)
2001:4978::/32 (SixXS tunnel)
2001:470::/32 (HE tunnel)

  That gives me a really bad feeling of what might be wrong, but I'll leave it to the professionals.

      Jima

$ telnet -6 www.savvis.com 80
Trying 2001:460:100:1000::37...
telnet: Unable to connect to remote host: Connection refused

I checked, it's a TCP RST packet, not ICMP unreachable. This is from native IPv6.

via gogo6 tunnel box (http://gogo6.com/) from my UK location
( not tested other tunnels nor native)

$ telnet -6 www.savvis.com 80
Trying 2001:460:100:1000::37...
Connected to www.savvis.net.

$ ping6 www.savvis.com
PING6(56=40+8+8 bytes) 2001:5c0:1110:8000:217:f2ff:fee6:ab79 --> 2001:460:100:1000::37
16 bytes from 2001:460::::xx, icmp_seq=0 hlim=243 time=149.971 ms

Christian

...and the AAAA's are back! And port 80 responds.

Frank

Good news: access to the v6 version of www.qwest.com came up at 12:30 pm
today -- it redirects to www.centurylink.com, but at least it's working.

Only www.savvis.com remains in my list of service provider websites that
have non-working IPv6.

Frank

Well, sometime yesterday www.centurylink.com removed it AAAA record(s).

www.qwest.com still has them.

Frank

AAAA and IPv6 access to www.centurylink.com were restored around 11:30 am
U.S. Central.

Frank

HTTP both www.qwest.com and www.centurylink.com have been in and out since
December 27. Sometimes it responds in less than 10 seconds, other times it
connects and there's no TCP response for minutes. This was tested from two
different networks.

If anyone from CenturyLink is lurking, could you please notify your NOC or
IT department?

Frank