AAAA's for www.netflix.com

Mailman List Mirror (Read Only)
1

I started monitoring IPv6 access to www.netflix.com after seeing this
posting
(http://www.personal.psu.edu/dvm105/blogs/ipv6/2012/06/netflix-is-back.html)
and what I found, over the week, was that access was coming and going
(www.premieronline.net/~fbulk/netflix.png). But not because of IPv6
connectivity, but because the AAAA's were coming and going. Netflix's DNS
TTL is pretty short.

I assume Netflix has some global DNS load balancing so my perspective may
not be complete. Has anyone else been seeing this?

I contacted a Netflix employee (he's well known on this list) and he
responded once but I haven't heard back since Saturday.

Here's some sample queries from Saturday and today.

2

UltraDNS is doing something strange with its CNAME responses. www.netflix.com is a CNAME to a name with both A and AAAA, but the authoritative server for netflix.com only returns that CNAME for A queries, not AAAA. So, if you do an A query first, your resolver will cache the CNAME and use it for the subsequent AAAA query (returning an AAAA), but if you do an AAAA query first, it will cache the no-records response and return no AAAA record.

$ dig ns netflix.com
;; QUESTION SECTION:
;netflix.com. IN NS
;; ANSWER SECTION:
netflix.com. 162 IN NS pdns5.ultradns.info.
netflix.com. 162 IN NS pdns6.ultradns.co.uk.
netflix.com. 162 IN NS pdns4.ultradns.org.
netflix.com. 162 IN NS pdns2.ultradns.net.
netflix.com. 162 IN NS pdns1.ultradns.net.
netflix.com. 162 IN NS pdns3.ultradns.org.

$ dig @pdns1.ultradns.net. www.netflix.com
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61357
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;www.netflix.com. IN A
;; ANSWER SECTION:
www.netflix.com. 300 IN CNAME dualstack.wwwservice--frontend-313423742.us-east-1.elb.amazonaws.com.

$ dig @pdns1.ultradns.net. aaaa www.netflix.com
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34855
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
;; QUESTION SECTION:
;www.netflix.com. IN AAAA
;; AUTHORITY SECTION:
netflix.com. 1800 IN SOA dns.netflix.com. nicadmin.netflix.com. 2012060120 900 600 1209600 1800

-Ben

3

In message <5F907BC1-9344-4187-BA12-CEAF7E1C3E73@bjencks.net>, Ben Jencks write
s:

> I started monitoring IPv6 access to www.netflix.com after seeing this
> posting
> =
(http://www.personal.psu.edu/dvm105/blogs/ipv6/2012/06/netflix-is-back.htm=
l)
> and what I found, over the week, was that access was coming and going
> (www.premieronline.net/~fbulk/netflix.png). But not because of IPv6
> connectivity, but because the AAAA's were coming and going. Netflix's =
DNS
> TTL is pretty short. =20
>=20
> I assume Netflix has some global DNS load balancing so my perspective =
may
> not be complete. Has anyone else been seeing this?
>=20
> I contacted a Netflix employee (he's well known on this list) and he
> responded once but I haven't heard back since Saturday. =20

UltraDNS is doing something strange with its CNAME responses. =
www.netflix.com is a CNAME to a name with both A and AAAA, but the =
authoritative server for netflix.com only returns that CNAME for A =
queries, not AAAA.

It's not strange. IT IS BROKEN. There is zero, nada, none, no
excuse for not returning a CNAME to the AAAA in this situation.

4

Just to close the loop on this - UltraDNS has an issue with CNAMEs and their Directional DNS service. We (Netflix) have applied a workaround and it appears stable.

-Dave

5

Hm, looking at http://v6launch.ripe.net/, whatever you changed didn't
improve visibility of the AAAA, but decreased it.

Best regards,
Daniel

6

> Just to close the loop on this - UltraDNS has an issue with CNAMEs and
> their Directional DNS service. We (Netflix) have applied a workaround and
> it appears stable.

Hm, looking at http://v6launch.ripe.net/, whatever you changed didn't
improve visibility of the AAAA, but decreased it.

TTL's of zero don't help. The A query has a TTL of 3600 in the
response the AAAA query has a zero TTL. This is rocket science.
This isn't hard to do correctly. How to handle CNAMEs has been
specified 1/4 of a century.

7

well, something appears to be working..

http://www.betterbroadbandblog.com/2012/06/world-ipv6-daywe-have-liftoff/

Netflix moved up to second in the IPv6 list – as noted above, Netflix has
been rolling out IPv6 coverage over the last few weeks. Interestingly, it
appears as if Netflix may have created its own IPv6-specific domain which
is responsible for almost a third of all IPv6 traffic. If this is the case
it might not be in full compliance with the spirit of World IPv6 Day, as
the aim should have been for Netflix to operate one single domain with both
AAAA records for IPv6 and A records for IPv4.

8

Joly,

What do you mean? www.netflix.com is dual stacked, which represents
availability of our website (and PC/Mac streaming clients) to100% of our
users who have IPv6.

-Dave

9

The zero TTL on the CNAME an AAAA RRs makes www.netflix.com
zero-stacked at least for some resolvers:

$ dig @pdns3.ultradns.org www.netflix.com. A +norec +short
wwwservice--frontend-313423742.us-east-1.elb.amazonaws.com.
$ dig @pdns3.ultradns.org www.netflix.com. AAAA +norec +short
dualstack.wwwservice--frontend-313423742.us-east-1.elb.amazonaws.com.
$ dig @pdns3.ultradns.org www.netflix.com. ANY +short +norec
$

Resolving www.netflix.com using ANY RRtype fails with an empty answer
section in the DNS response.

This DNS trickery seems to be from the "taking a shower, trying not
to get wet" department. And has adverse effects in corner cases. While
playing around, I had periods of time where I couldn't resolve the FQDN
at all, possibly due some caching of the empty response.

Best regards,
Daniel

10

Correction... I don't really know wether the zero TTL on the CNAME
provokes problems, but not returning any RR on ANY RRtype queries seems
to do.

Best regards,
Daniel

11

> What do you mean? www.netflix.com is dual stacked, which represents
> availability of our website (and PC/Mac streaming clients) to100% of our
> users who have IPv6.

The zero TTL on the CNAME an AAAA RRs makes www.netflix.com
zero-stacked at least for some resolvers:

$ dig @pdns3.ultradns.org www.netflix.com. A +norec +short
wwwservice--frontend-313423742.us-east-1.elb.amazonaws.com.
$ dig @pdns3.ultradns.org www.netflix.com. AAAA +norec +short
dualstack.wwwservice--frontend-313423742.us-east-1.elb.amazonaws.com.
$ dig @pdns3.ultradns.org www.netflix.com. ANY +short +norec
$

Resolving www.netflix.com using ANY RRtype fails with an empty answer
section in the DNS response.

Which is just plain BROKEN.

This DNS trickery seems to be from the "taking a shower, trying not
to get wet" department. And has adverse effects in corner cases. While
playing around, I had periods of time where I couldn't resolve the FQDN
at all, possibly due some caching of the empty response.

It's not DNS trickery. It's not reading the RFC or doing any testing
for common cases. Not every query has type A. Way too many load
balancers get the answers to anything other than A queries wrong.
If your load balancer get answers to queries other than A wrong
demand your money back.

This sort of !@$E in authoritative servers makes it hard for recursive
servers to do the right thing.

pandora.tv servers are the poster child for getting it wrong currently.

* They don't set "aa" as per RFC 103[45].
  (If you set it in the query they clear it.)
* They don't clear "ad" as per RFC 103[45].
* They have extra data after the DNS response.
* They send back malformed responses to EDNS queries.

bsdi# dig9 ns pandora.tv @N1.pandora.tv +noedns +norec

; <<>> DiG 9.9.1-P1 <<>> ns pandora.tv @N1.pandora.tv +noedns +norec
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36153
;; flags: qr ra ad; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: Messages has 27 extra bytes at end

;; QUESTION SECTION:
;pandora.tv. IN NS

;; ANSWER SECTION:
pandora.tv. 300 IN NS N1.pandora.tv.
pandora.tv. 300 IN NS N2.pandora.tv.
pandora.tv. 300 IN NS N15.pandora.tv.
pandora.tv. 300 IN NS N16.pandora.tv.
pandora.tv. 300 IN NS N17.pandora.tv.

;; Query time: 385 msec
;; SERVER: 61.111.8.236#53(61.111.8.236)
;; WHEN: Fri Jun 8 11:56:22 2012
;; MSG SIZE rcvd: 143

bsdi# dig9 ns pandora.tv @N1.pandora.tv +edns=0 +norec
;; Got bad packet: FORMERR
143 bytes
cd c9 80 a0 00 01 00 05 00 00 00 01 07 70 61 6e .............pan
64 6f 72 61 02 74 76 00 00 02 00 01 c0 0c 00 02 dora.tv.........
00 01 00 00 01 2c 00 05 02 4e 31 c0 0c c0 0c 00 .....,...N1.....
02 00 01 00 00 01 2c 00 05 02 4e 32 c0 0c c0 0c ......,...N2....
00 02 00 01 00 00 01 2c 00 06 03 4e 31 35 c0 0c .......,...N15..
c0 0c 00 02 00 01 00 00 01 2c 00 06 03 4e 31 36 .........,...N16
c0 0c c0 0c 00 02 00 01 00 00 01 2c 00 06 03 4e ...........,...N
31 37 c0 0c 00 00 00 00 00 00 00 00 00 00 00 00 17..............
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ...............
bsdi#

Mark

12

> $ dig @pdns3.ultradns.org www.netflix.com. A +norec +short
> wwwservice--frontend-313423742.us-east-1.elb.amazonaws.com.
> $ dig @pdns3.ultradns.org www.netflix.com. AAAA +norec +short
> dualstack.wwwservice--frontend-313423742.us-east-1.elb.amazonaws.com.
> $ dig @pdns3.ultradns.org www.netflix.com. ANY +short +norec
> $
>
> Resolving www.netflix.com using ANY RRtype fails with an empty answer
> section in the DNS response.

Which is just plain BROKEN.

Yup.

> This DNS trickery seems to be from the "taking a shower, trying not
> to get wet" department. And has adverse effects in corner cases. While
> playing around, I had periods of time where I couldn't resolve the FQDN
> at all, possibly due some caching of the empty response.

It's not DNS trickery.

The "trickery" is returning different CNAMEs for QTYPE=A and QTYPE=AAAA.
I'm not sure what's the goal of that is, but it's 4am here so I have an
excuse of not seeing the light. :slight_smile:

Best regards,
Daniel

13

We've confirmed that UltraDNS had "additional" issues caused by the push that they fixed for the previously reported problem. We are actively engaged with them to come to a resolution.

-Dave

14

Netflix may have created its own IPv6-specific domain which is responsible
for almost a third of all IPv6 traffic. If this is the case it might not be
in full compliance with the spirit of World IPv6 Day, as the aim should
have been for Netflix to operate one single domain with both AAAA records
for IPv6 and A records for IPv4.

What about that?

j

15

    Netflix may have created its own IPv6-specific domain which is
    responsible for almost a third of all IPv6 traffic. If this is the
    case it might not be in full compliance with the spirit of World
    IPv6 Day, as the aim should have been for Netflix to operate one
    single domain with both AAAA records for IPv6 and A records for IPv4.

What about that?

I've asked Sandvine to explain exactly what this means. We haven't created anything IPv6 specific, aside from starting to serve 100% of our IPv6 traffic from the Netflix CDN instead of our CDN partners, which is irrelevant to this actual exercise.