pay.gov and IPv6

Mailman List Mirror (Read Only)
1

Following up on a two year old thread, one of my clients just hit this
problem. The failure is not that www.pay.gov is not reachable over ipv6
(2605:3100:fffd:100::15). They accept (TCP handshake) the port 443
connection, but the connection then hangs waiting for the TLS handshake.

openssl s_client -connect www.pay.gov:443

openssl s_client -servername www.pay.gov -connect 199.169.192.21:443

Browsers (at least firefox) see that as a very slow site, and it does
not trigger their happy eyeballs fast failover to ipv4.

2

:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Following up on a two year old thread, one of my clients just hit this
problem. The failure is not that www.pay.gov is not reachable over ipv6
(2605:3100:fffd:100::15). They accept (TCP handshake) the port 443
connection, but the connection then hangs waiting for the TLS handshake.

openssl s_client -connect www.pay.gov:443

openssl s_client -servername www.pay.gov -connect 199.169.192.21:443

Browsers (at least firefox) see that as a very slow site, and it does
not trigger their happy eyeballs fast failover to ipv4.

Happy eyeballs is about making the connection not whether TCP
connections work after the initial packet exchange.

I would send a physical letter to the relevent Inspector General
requesting that they ensure all web sites under their juristiction
that are supposed to be reachable from the public net get audited
regularly to ensure that IPv6 connections work from public IP space.

While you are sending the letter can you also ask why pay.gov's DNS
servers are broken.

Checking: 'pay.gov' as at 2016-11-16T20:21:28Z

pay.gov @199.169.194.28 (ns1.twai.gov.): edns=ok edns1=timeout edns@512=noopt ednsopt=ok edns1opt=timeout do=ok ednsflags=ok docookie=ok edns@512tcp=ok optlist=ok
pay.gov @2605:3100:fffc:100::7 (ns1.twai.gov.): edns=ok edns1=timeout edns@512=noopt ednsopt=ok edns1opt=timeout do=ok ednsflags=ok docookie=ok edns@512tcp=ok optlist=ok
pay.gov @199.169.192.28 (ns2.twai.gov.): edns=ok edns1=timeout edns@512=noopt ednsopt=ok edns1opt=timeout do=ok ednsflags=ok docookie=ok edns@512tcp=ok optlist=ok
pay.gov @2605:3100:fffd:100::7 (ns2.twai.gov.): edns=ok edns1=timeout edns@512=noopt ednsopt=ok edns1opt=timeout do=ok ednsflags=ok docookie=ok edns@512tcp=ok optlist=ok

Mark

3

I fixed it (and Netflix) by turning off IPv6 for all my users... but any
chance this is a path MTU issue causing the apparent hang?

Matthew Kaufman

4

I fixed it by using the rpz feature of bind to disable the AAAA record
for www.pay.gov. I lookup the real A record, and then put

www.pay.gov IN A %s

into the local rpz zone. That suppresses the AAAA record, so local
clients are forced into IPv4 for that site. That allows them to use IPv6
for other sites.

path MTU - hm, I need to check that.

5

openssl s_client -connect www.pay.gov:443

I’m not seeing the issue here, but they do have some possible issues the way they’re setting cookies (See details below).

What path are you seeing to them? I’m also not having the issue from the IETF97 network here in Seoul which has IPv6 as well.

puck:~$ traceroute6 www.pay.gov.
traceroute to www.pay.gov. (2605:3100:fffd:100::15), 30 hops max, 80 byte packets
1 ge-0-7-0-22.r05.chcgil09.us.bb.gin.ntt.net (2001:418:3f4::1) 0.751 ms 0.871 ms 0.994 ms
2 verio-gw.cgcil.ipv6.att.net (2001:1890:1fff:307:192:205:32:193) 2.008 ms 1.991 ms 2.837 ms
3 cgcil22crs.ipv6.att.net (2001:1890:ff:ffff:12:122:132:198) 27.333 ms 27.167 ms 27.070 ms
4 sl9mo22crs.ipv6.att.net (2001:1890:ff:ffff:12:122:2:178) 27.602 ms 27.646 ms 27.628 ms
5 sl9mo21crs.ipv6.att.net (2001:1890:ff:ffff:12:122:2:217) 30.055 ms 29.894 ms 29.855 ms
6 dlstx22crs.ipv6.att.net (2001:1890:ff:ffff:12:122:2:1) 28.888 ms 27.016 ms 26.933 ms
7 dlstx84crs.ipv6.att.net (2001:1890:ff:ffff:12:123:18:249) 28.126 ms 26.757 ms 26.645 ms
8 2001:1890:ff:ffff:12:122:124:141 (2001:1890:ff:ffff:12:122:124:141) 26.142 ms 26.269 ms 26.179 ms
9 2001:1890:c00:610b::1138:7d27 (2001:1890:c00:610b::1138:7d27) 27.273 ms 27.255 ms 27.544 ms
10 2001:1890:1c08:cf01::2 (2001:1890:1c08:cf01::2) 27.673 ms !X 27.559 ms !X 27.465 ms !X

curl -v Pay.gov - Home
* Trying 2605:3100:fffd:100::15...
* TCP_NODELAY set
* Connected to www.pay.gov (2605:3100:fffd:100::15) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* ALPN/NPN, server did not agree to a protocol
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* start date: May 28 14:58:43 2015 GMT
* expire date: May 29 06:16:02 2018 GMT
* common name: www.pay.gov
* issuer: CN=Entrust Certification Authority - L1K,OU="(c) 2012 Entrust, Inc. - for authorized use only",OU=See www.entrust.net/legal-terms,O="Entrust, Inc.",C=US

GET /public/home HTTP/1.1
Host: www.pay.gov
User-Agent: curl/7.51.0
Accept: */*

< HTTP/1.1 200 OK
< Date: Wed, 16 Nov 2016 21:52:08 GMT
< Content-type: text/html; charset=ISO-8859-1
< Strict-transport-security: max-age=31536001; includeSubDomains
< Cache-Control: no-cache
< Cache-Control: no-store
< Pragma: no-cache
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< X-XSS-Protection: 1; mode=block
< Strict-Transport-Security: max-age=31536000
< Set-Cookie: JSESSIONID=949QYsVLKQqBB42HTy91pJnGfnfJthLfQTv02CvDnt7rNQnpSvb1!1259175335!-1040755441!1479333128223; path=/public; secure; HttpOnly
< Set-Cookie: JSESSIONID=949QYsVLKQqBB42HTy91pJnGfnfJthLfQTv02CvDnt7rNQnpSvb1!1259175335!-1040755441; path=/public; HttpOnly
< Set-Cookie: ClientId=14793331282345260; path=/public; HttpOnly; secure
< Set-Cookie: ClientId=1479333128244363; path=/public; HttpOnly; secure
< X-FRAME-OPTIONS: DENY
< Content-Language: en-US
< X-Powered-By: Servlet/2.5 JSP/2.1
< Transfer-encoding: chunked

6

That will absolutely work.

NIST is still monitoring ipv6 .gov sites
  https://usgv6-deploymon.antd.nist.gov/cgi-bin/generate-gov
so the IG isn't going to do anything there & pay.gov has a contact us page
  https://pay.gov/public/home/contact
that I'd bet works much better than a letter to the IG

Regards,
Lee

7

It happens too often, unfortunately.

People deploying IPv6 at web sites and other services, don’t check if PMTUD is broken by filtering, ECMP, load balancers, etc.

This is the case here:

tbit from 2001:df0:4:4000::1:115 to 2605:3100:fffd:100::15
server-mss 1440, result: pmtud-fail
app: http, url: https://www.pay.gov/
[ 0.009] TX SYN 64 seq = 0:0
[ 0.165] RX SYN/ACK 64 seq = 0:1
[ 0.166] TX 60 seq = 1:1
[ 0.166] TX 371 seq = 1:1(311)
[ 0.325] RX 1500 seq = 1:312(1440)
[ 0.325] RX 1500 seq = 1441:312(1440)
[ 0.325] TX PTB 1280 mtu = 1280
[ 0.325] RX 1362 seq = 2881:312(1302)
[ 3.325] RX 1500 seq = 1:312(1440)
[ 3.325] TX PTB 1280 mtu = 1280
[ 9.326] RX 1500 seq = 1:312(1440)
[ 9.326] TX PTB 1280 mtu = 1280
[ 21.325] RX 1500 seq = 1:312(1440)
[ 21.325] TX PTB 1280 mtu = 1280
[ 45.325] RX 1500 seq = 1:312(1440)

Regards,
Jordi

8

>
> :
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA512
>>
>> Following up on a two year old thread, one of my clients just hit this
>> problem. The failure is not that www.pay.gov is not reachable over ipv6
>> (2605:3100:fffd:100::15). They accept (TCP handshake) the port 443
>> connection, but the connection then hangs waiting for the TLS handshake.
>>
>> openssl s_client -connect www.pay.gov:443
>>
>> openssl s_client -servername www.pay.gov -connect 199.169.192.21:443
>>
>> Browsers (at least firefox) see that as a very slow site, and it does
>> not trigger their happy eyeballs fast failover to ipv4.
>
> Happy eyeballs is about making the connection not whether TCP
> connections work after the initial packet exchange.
>
> I would send a physical letter to the relevent Inspector General
> requesting that they ensure all web sites under their juristiction
> that are supposed to be reachable from the public net get audited
> regularly to ensure that IPv6 connections work from public IP space.

That will absolutely work.

NIST is still monitoring ipv6 .gov sites
  https://usgv6-deploymon.antd.nist.gov/cgi-bin/generate-gov

Which show green which means that the tests they are doing are not
sufficient. They need to test from behind a 1280 mtu link.

The DNSSEC testing is also insufficient. 9-11commission.gov shows
green for example but if you use DNS COOKIES (which BIND 9.10.4 and
BIND 9.11.0 do) then servers barf and return BADVERS and validation
fails. QWEST you have been informed of this already.

Why the hell should validating resolver have to work around the
crap you guys are using? DO YOUR JOBS which is to use RFC COMPLIANT
servers. You get PAID to do DNS because people think you are
compentent to do the job. Evidence shows otherwise.

.GOV domains (GSA + Umbrella) EDNS Compliance Report show the broken
servers for .gov. It isn't hard to check.

so the IG isn't going to do anything there & pay.gov has a contact us page
  Pay.gov
that I'd bet works much better than a letter to the IG

You have to be able to get to Pay.gov to use
it. Most people don't have the skill set to force the use of IPv4.

If it is production it should work. It is the I-G's role to ensure this
happens. Butts need to kicked.

Mark

9

I think it is not just a matter of testing behind a 1280 MTU, but about making sure that PMTUD is not broken, so it just works in any circumstances.

Regards,
Jordi

10

I think it is not just a matter of testing behind a 1280 MTU, but about makin
g sure that PMTUD is not broken, so it just works in any circumstances.

Regards,
Jordi

If you don't do MSS fix up a 1280 link in the middle will find PMTUD issues
provided the testing host has a MTU > 1280.

Mark

11

The good news is that I reported this particular site as a problem two and
three years ago, both, and it isn't any worse.

Matthew Kaufman

12

The good news is that I reported this particular site as a problem two and
three years ago, both, and it isn't any worse.

did you contact Pay.gov Customer Service at:
800-624-1373 (Toll free, Option #2)
or send an email to
pay.gov.clev@clev.frb.org

I just called, but I can't duplicate the problem and they need to work
with someone that is having a problem reaching the site.

Regards,
Lee

13

I sent email there and to another contact I had at the time.

And I'm not going to break my users by turning IPv6 back on, so someone
else will need to work with them.

Matthew Kaufman

14

I sent email there and to another contact I had at the time.

and the response was?

And I'm not going to break my users by turning IPv6 back on, so someone
else will need to work with them.

That's fine, but until someone is willing to work with them don't
expect it to get fixed.

Regards,
Lee

15

I am working with pay.gov.clev@clev.frb.org, trying to explain the
problem. They seem to think I should provide "application name and ID"
before they can research this. I responded as below. I will let you know
if there is any progress on this. It is 100% reproducible here - and
should be reproducible from anywhere with a slightly smaller than normal
MTU.

We try to get to https://www.pay.gov/public/home - which fails. I
presume that is before there is any application name or ID.

Try to reach that page from a browser on a machine with ipv6
connectivity that goes thru a tunnel with a 1300 byte MTU. It will fail.

16

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

That's fine, but until someone is willing to work with them don't
expect it to get fixed.

I am working with pay.gov.clev@clev.frb.org, trying to explain the
problem. They seem to think I should provide "application name and ID"
before they can research this.

They probably want that so they know where to route the ticket. I
pointed them at
  RFC 4890 - Recommendations for Filtering ICMPv6 Messages in Firewalls
and suggested they have someone on the network team use ipv6 to
connect to pay.gov over a 1280 byte MTU link. Hopefully that's enough
to get the ticket routed to the correct group.

I responded as below. I will let you know
if there is any progress on this.

I'd appreciate that.
And thanks for being willing to work with them to fix the problem.

Lee

17

Engineer complains the universe doesn't match his assumptions.

18

* Mark Andrews:

The DNSSEC testing is also insufficient. 9-11commission.gov shows
green for example but if you use DNS COOKIES (which BIND 9.10.4 and
BIND 9.11.0 do) then servers barf and return BADVERS and validation
fails. QWEST you have been informed of this already.

Why the hell should validating resolver have to work around the
crap you guys are using?

The protocol doesn't have proper version negotation, and again and
again, implementers have tried to force backwards-incompatible
implementations on the Internet at large.

19

> I am working with pay.gov.clev@clev.frb.org, trying to explain the
problem.

The intersection of government bureaucracy and technical issues is
frustrating to say the least. I just sent the message below, but have no
expectation that it will change anything.

20

I tested from my home and happy eyeballs is not falling back to IPv4.

So, I tend to suspect that is not ICMPv6 filtering, but something else, such as wrong load balancer or ECMP configuration.

Regards,
Jordi