I work for a regional ISP and very recently there has been an influx of
calls reporting "slowness" when accessing certain websites (i.e google.com/voice/b) via HTTP. After performing a tcpdump and analyzing the
session, I have been able to pinpoint the latency at the application
layer. After the tcp session has been established, it takes up to 15-20
seconds before the application begins sending data. The root of the
problem was that the PTR record for our customer(s) address does not
exist. As soon as the record is created, latency from the application is
eliminated. This is analogous to latency when accessing a server over SSH
when no PTR is available.
A seperate packet capture from another customer exhibiting similar
performance behavior showed many TCP retransmissions. At first glance, I
assumed this was network related however this correlates with the
application not responding and inducing retransmissions at the TCP layer
(different symptoms, same problem).
Historically, there was no compelling reason to create PTR records for our
CPE however more and more applications seem to be dependent on it.
Although we will be assigning a record for each address, my question is why
is the application (specifically HTTP) dependent on a reverse record ?
What is the purpose?
HTTP has no requirement that the connecting client have reverse DNS setup. Some servers have reverse lookups enabled, and some of those undoubtedly block until the record has been retrieved or all avenues of discovery have been exhausted... and this is likely where the issue exists. As to why the server or the script/application its running needs the record, you'd have to ask the developer.
You're returning NXDOMAIN, right? If they're doing a reverse lookup and you return NXDOMAIN it should fail quickly, or else the application is even more horribly broken than just doing reverse lookups would imply. On the other hand, if you're not responding to the PTR request then that could be causing the timeout.
What happens if the ISP never defines a name server with their RIR for
their provider-independent address space? Does ARIN point to somewhere
which supplies NXDOMAIN? Just a thought -- I don't have a clue.
It is entirely possible they have it pointed to their non-existent or
broken DNS. Given current best practices, I see no reason not to assign a
generic x.x.x.x-dynamic.customer.isp.com DNS across their netblock.
What happens if the ISP never defines a name server with their RIR for
their provider-independent address space? Does ARIN point to somewhere
which supplies NXDOMAIN? Just a thought -- I don't have a clue.
It is entirely possible they have it pointed to their non-existent or
broken DNS. Given current best practices, I see no reason not to assign a
generic x.x.x.x-dynamic.customer.isp.com DNS across their netblock.
I think that returns SERVFAIL somewhere down the line?
Does it make sense to reinforce the behaviour (good and bad terms left for
another time), while looking forward to v6?
We really have no objections to creating records for our IPs however there was no compelling reason previously. With the manifestation of performance issues, we are currently creating a generic record for our addresses.
I assumed that the applications would take absent records into consideration instead of waiting and timing out before responding with data. Trying to troubleshoot this issue from the limited visibility is difficult ; the latency the application is introducing is abstracted (unless I am unaware of that troubleshooting technique).
When you mis-place your keys do you only look in one place and then give
up? The calling server does not know there is "no" record until it exhausts
its list of DNS servers, which is probably what is introducing the delay you
are seeing (each server trying to find a PTR with each of its upsteams) until
they all time out...
If the reverse zone is properly configured, but just the PTR record is missing,
you get NXDOMAIN, which is not "you mis-place your keys"; it's
"someone told you authoritatively that your keys don't exist", never existed
or no longer existed.
If you ask where your key ring went, and Frodo Baggins informs you that
it doesn't exist, because it was tossed down into a pool of magma on mount doom,
and you trust his reply, you stop looking for it.
The only way you don't trust a valid DNS reply is if you are
implementing DNSSEC,
and the "authoritative proof of non-existence" doesn't validate
We recently encountered a similar issue with a customer.
The sites that had slowness issues were configured to
use the traditional Google Analytics javascript instead
of the newer asynchronous code.
The problem was not the lack of a PTR record, but rather
the in-addr delegation was pointing to lame servers that were
no longer responding (hence the long timeouts as the
Google servers attempted to perform reverse lookups
on the client IP's). As others have pointed out, as long
as there's a valid nameserver responding, a lack of PTR
record would not cause issues as an NXDOMAIN record would
be sent back immediately and the Google Analytics server
will move on.
paul4004> It is entirely possible they have it pointed to their
paul4004> non-existent or broken DNS. Given current best practices, I
paul4004> see no reason not to assign a generic
paul4004> x.x.x.x-dynamic.customer.isp.com DNS across their netblock.
It's already been pointed out that lame delegations are more likely
problems for many. But the "we'll just pre-fill in-addr to avoid
problems" isn't going to work for ip6.arpa. If anyone has enough
hardware to serve the zone for a /48 (64k * 4bil * 4bil *
bytes-in-record), I'd love to see it.
We need to get web and app folks to stop counting on
ip6.arpa/in-addr.arpa as a validation of trustworthiness. PTR make some
sense for validating servers, MTAs, etc. and it's handy for traceroute
but it was never a great tool and it's getting less useful with time.
It's already been pointed out that lame delegations are more likely
problems for many. But the "we'll just pre-fill in-addr to avoid
problems" isn't going to work for ip6.arpa. If anyone has enough
hardware to serve the zone for a /48 (64k * 4bil * 4bil *
bytes-in-record), I'd love to see it.
If PTR exists in zone file, serve it.
Else, synthesize generic reverse.
Jobsagoodun.
We need to get web and app folks to stop counting on
ip6.arpa/in-addr.arpa as a validation of trustworthiness. PTR make some
sense for validating servers, MTAs, etc. and it's handy for traceroute
but it was never a great tool and it's getting less useful with time.
I've always seen it as a reasonable indication of a) minimum level of clue and b) giving a damn. If you can't be bothered or don't know how to provide even basic generic rDNS for your network, there's a reasonable chance you're lacking in other areas of network / user management. (Not "you" personally, of course).
[snip]
I can serve the zone for a /48 by creating a "sparse" zone.
That is... when you ask my DNS server what such and such PTR reverses too,
what I don't have a DNS entry for, it can tell you something generic.
There is no need for me to physically create 64k*4bil*4bil on a disk
or memory area
somewhere. I can make a plugin for my DNS server to hand you the
generic result
when you ask my DNS server what something reverses to...
That is, I can serve you an ephemeral record without requiring an
extra byte of storage beyond
the life of your query
This is precisely the approach taken by the DNS server package written by
the tech folks at the late, lamented MindSpring -- the name of which eludes
me for the moment.
> If PTR exists in zone file, serve it. Else, synthesize generic
> reverse. Jobsagoodun.
If all we're doing is lying with some generic answer that we hack our
server to produce, why are we bothering?
Because some applications rely on it working (for some definition of "working").
My contention is that (at least for end hosts), PTR records are mostly
pointless and just overhead for DNS servers.
If you haven't set up PTR records for your end hosts, realistically you're going to be serving NXDOMAINs for them anyway, so there's not really any overhead introduced by supplying something generic instead...
In message <FA83E0C6-DAFC-4C8C-AF46-826EDF4727F8@oneshoeco.com>, Tom Lanyon wri
tes:
> > If PTR exists in zone file, serve it. Else, synthesize generic
> > reverse. Jobsagoodun.
>=20
> If all we're doing is lying with some generic answer that we hack our
> server to produce, why are we bothering?
Because some applications rely on it working (for some definition of =
"working").
> My contention is that (at least for end hosts), PTR records are mostly
> pointless and just overhead for DNS servers.
If you haven't set up PTR records for your end hosts, realistically =
you're going to be serving NXDOMAINs for them anyway, so there's not =
really any overhead introduced by supplying something generic instead...
Tom
Except you also have to supply the A/AAAA records as well.
MacOS and Windows can both populate the reverse zone for you as can
dhcp servers.
The practice of filling out the reverse zone with fake PTR record
started before there was wide spread support for UPDATE/DNS. There
isn't any need for this to be done anymore. Machines are capable
of adding records for themselves.
MacOS and Windows can both populate the reverse zone for you as can
dhcp servers.
The practice of filling out the reverse zone with fake PTR record [...]
OK.. let's say you're a DSL provider. Are you going to have your
DHCP server populating the forward and reverse DNS? With what, the
account holder's name? somename.example.com ?
Wouldn't you say blahblah192-168-0-2.city.state.dsl.example.com
provides more useful information?
First of all, you know that the IP address is an end user, an access
network's end user's one IP address,
an endpoint, rather than a subnet assigned to an actual multinode network.
Second of all, you know it's an ISP, and you have city and state
information of the network service.
This is more useful than arbitrary user made up hostname.
The hostname is more meaningful on "real networks" such as SMB LANs,
Enterprise intranets, web farms, server networks, and other places
where generic records should not be assigned, but the PTR should be
the actual hostname.
If the IP address is dynamic or autoconfigured for _those_ types of
networks, then yes, automatic RDNS registration makes sense. If it's
static, not so much.
Dynamic DNS registration is also complicated to make secure.... as
in preventing hosts from updating other hosts' records or mucking
around the zone in other unwanted ways requires complex key
management and ACL configuration
From nanog-bounces+bonomi=mail.r-bonomi.com@nanog.org Sun Nov 6 19:58:58 2011
Date: Sun, 6 Nov 2011 19:57:51 -0600
Subject: Re: Performance Issues - PTR Records
From: Jimmy Hess <mysidia@gmail.com>
To: Mark Andrews <marka@isc.org>
Cc: nanog@nanog.org
> MacOS and Windows can both populate the reverse zone for you as can
> dhcp servers.
> The practice of filling out the reverse zone with fake PTR record [...]
OK.. let's say you're a DSL provider. Are you going to have your
DHCP server populating the forward and reverse DNS? With what, the
account holder's name? somename.example.com ?
I'll suggest that (a) IF the addresses do migrate among different customers
of the ISP, (b) the addresses handed out are publicly routable, AND (c) the
CPE has to 'authenticate' itself to the head-end, then it is _very_ useful
*to*the*ISP* to have dynamically-assigned DNS records of the form:
cust.{accountid}.{locationid}.ISP.{com/net/TLD}
or something of the sort.
Something of that sort can save a -lot- of time/effort in identifying the
customer involved in a complaint.
Once upon a time, Robert Bonomi <bonomi@mail.r-bonomi.com> said:
I'll suggest that (a) IF the addresses do migrate among different customers
of the ISP, (b) the addresses handed out are publicly routable, AND (c) the
CPE has to 'authenticate' itself to the head-end, then it is _very_ useful
*to*the*ISP* to have dynamically-assigned DNS records of the form:
cust.{accountid}.{locationid}.ISP.{com/net/TLD}
or something of the sort.
Putting a customer ID in reverse DNS would probably be a violation of
privacy policies.
Surely that's only useful if they're still allocated the address at the time of investigating said complaint; a dynamically updating DNS record like this is really no substitution for accurate accounting records in your RADIUS system.
