Interesting thread on /. --
http://ask.slashdot.org/article.pl?sid=05/04/18/198259&tid=95&tid=128&tid=4
- ferg
The original poster doesn't identify what his starting TTL was before he
made his change. I didn't read all the responses, but I can see how an
original 14 day TTL could take 4 weeks to propagate from master to end-
user, given the amount of testing he/she was doing *prior* to making the
change.
-Jim P.
Fergie (Paul Ferguson) wrote:
Interesting thread on /. --
FWIW, I did some 'dig'ing on my Comcast home service. The DHCP is handing
out 204.127.198.4 and 63.240.76.4 for DNS at the moment.
I ran a query for a name in a zone I control that has a five minute TTL
on 204.127.198.4. The first query came up with 5 minutes. I quickly made
a change to the zone. hirty seconds after the initial query, I try again...
err... and come up with the change. Hmm... Not caching at all? Another
30 seconds and I get the change, with 5m TTL. Thirty seconds later, I
get the original response with appropriately decremented TTL. Another
thirty seconds, I get the change, with 4m TTL.
My findings: Comcast is now using some kind of load balancing that messes
with this kind of testing. 204.127.198.4 is not a single resolver. However,
as far as I could tell, they were obeying the TTL. After 5 minutes, all
of the responses were coming back with the change. The TTL values in the
responses were decrementing as expected.
I'd rather expect this sort of behavior with anycasted servers...
With a cache, the behavior is confusing, but also harms DNS TCP support,
just like that described for authoritative servers.
Further there isn't a good reason to have anycasted caches. Indeed, with
DHCP-learned nameservers, there is negative reasons to have anycast.
Anycast balancing will never be as good as random selection from the
appropriate set given by DHCP.
Frequently, [judging by the questions asked on DNSOP about setting up
anycast caches, anyway], the goal of such gyration is high availability.
However, its [been] fairly easilly shown that this goal isn't achieved.
--Dean
I'd rather expect this sort of behavior with anycasted servers...
Where do you see any connection between anycast and ignoring DNS TTL?
Or is this just part of your usual rant against anycast DNS service?
We use anycast for our caching (recursive) DNS servers. It works well
for us, and we certainly intend to continue to use it. The actual DNS
software used is Nominum CNS and BIND 9.3.1, both of which honor the
DNS TTL.
Steinar Haug, Nethelp consulting, sthaug@nethelp.no
Dean Anderson wrote:
I'd rather expect this sort of behavior with anycasted servers...
I would not expect this kind of behavior from an anycasted address.
You'd need a LOT of routing churn to see different caches every few
seconds. It's much more likely some kind of load balancer in front
of a DNS server farm.
With a cache, the behavior is confusing, but also harms DNS TCP support, just like that described for authoritative servers.
I verified it wasn't anycast by trying to exploit this very issue. I
did a query that fell back to TCP while doing multiple small queries.
I ran a network capture to pick out the short quries that occurred while
the TCP query was going on. Short quries during the TCP connection
came back with verying TTLs indicating that I was talking to different
caches, i.e. different servers. Yet the TCP query continued without
any hiccups. This indicates there is some type of per-session load
balancing going on, not anycast routing.
Further there isn't a good reason to have anycasted caches. Indeed, with
DHCP-learned nameservers, there is negative reasons to have anycast. Anycast balancing will never be as good as random selection from the appropriate set given by DHCP.Frequently, [judging by the questions asked on DNSOP about setting up
anycast caches, anyway], the goal of such gyration is high availability. However, its [been] fairly easilly shown that this goal isn't achieved.
Since this isn't anycast routing, can we save the religious diatribes
for another thread?
> I'd rather expect this sort of behavior with anycasted servers...
Where do you see any connection between anycast and ignoring DNS TTL?
Or is this just part of your usual rant against anycast DNS service?
The data he showed isn't necessarilly "ignoring ttl". If there are
multiple anycasted caching servers behind a specific IP address, then
those several cache's will each have a different state. Since, [as I
explained, and was supposed by the poster], there is "some kind of load
balancing going on", and also since implementors of anycast caches have
posted questions and explained their purposes [which could be seen as
"load balancing"], this is a likely explanation. It may not be the only
explanation: e.g. they could be restarting their nameservers every thirty
seconds. But "anycast loadbalancing" of a caching server is probably the
most likely.
But since you post on DNSOP, I assume that you read DNSOP [indeed, I may
assume too much here], and so you have read the recent questions posed
there on just how to implement just this sort of configuration. So, in
light of that, I take your message to be your "usual [and fact-free] rant
against anyone who explains the harms of anycast"
We use anycast for our caching (recursive) DNS servers. It works well
for us, and we certainly intend to continue to use it. The actual DNS
software used is Nominum CNS and BIND 9.3.1, both of which honor the
DNS TTL.
"worked once for me" doesn't cut it, now. Does it? Probably you didn't
notice that the cache states of different caching servers must be
different. "load balancing" [of nearly any sort] and anycast does not
work so well.
Dean Anderson wrote:
> I'd rather expect this sort of behavior with anycasted servers...I would not expect this kind of behavior from an anycasted address.
You'd need a LOT of routing churn to see different caches every few
seconds. It's much more likely some kind of load balancer in front
of a DNS server farm.
No, you are thinking of the (wrong) claims originally made by ISC about
how anycast would affect TCP to an anycast authoritative server. ISC
wrongly asserted that since BGP routes don't churn very fast compared with
DNS TCP connection lifetimes, that there should be no problem with anycast
and TCP. This view has been shown to be wrong in the face of Per Packet
Load Balancing (PPLB) which has been demonstrated to work on BGP links by
haesu@towardex.com. Further, I showed that if you have PPLB on interior
(eg OSPF) links leading to different BGP peers, the problem also happens.
Packets are sent on a per packet basis to different places.
But caching servers are usually setup to load balance. Usually, the
servers with the same IP address share an ethernet along with multiple
routers. So the packets are switched on essentially a per-packet basis.
Or possibly a per-arp basis that alters the MAC-based-forwarding behavior
of a switch. This is fairly fine grained load balancing.
> With a cache, the behavior is confusing, but also harms DNS TCP support,
> just like that described for authoritative servers.I verified it wasn't anycast by trying to exploit this very issue. I
did a query that fell back to TCP while doing multiple small queries.
I ran a network capture to pick out the short quries that occurred while
the TCP query was going on. Short quries during the TCP connection
came back with verying TTLs indicating that I was talking to different
caches, i.e. different servers. Yet the TCP query continued without
any hiccups. This indicates there is some type of per-session load
balancing going on, not anycast routing.
This additional information would seem to indicate they are behind a more
traditional stateful load balancer, rather than anycast. Without your TCP
connection, I don't think you could distinguish a traditional load
balancer from an anycast cache setup.
--Dean
No, you are thinking of the (wrong) claims originally made by ISC about
how anycast would affect TCP to an anycast authoritative server. ISC
wrongly asserted that since BGP routes don't churn very fast compared with
DNS TCP connection lifetimes, that there should be no problem with anycast
and TCP. This view has been shown to be wrong in the face of Per Packet
Load Balancing (PPLB) which has been demonstrated to work on BGP links by
haesu@towardex.com. Further, I showed that if you have PPLB on interior
(eg OSPF) links leading to different BGP peers, the problem also happens.
Packets are sent on a per packet basis to different places.
And I can show that if you give a pig wings....
Look, it breaks in certain situations. But anycast implementations of TCP apps have worked "well" for a decade now. Deal with the fact that not only do people use it, but users don't notice it.
Or don't. No one here cares if you do. Reality trumps lab tests.
But caching servers are usually setup to load balance. Usually, the
servers with the same IP address share an ethernet along with multiple
routers. So the packets are switched on essentially a per-packet basis.
Or possibly a per-arp basis that alters the MAC-based-forwarding behavior
of a switch. This is fairly fine grained load balancing.
This is complete news to me. Of course, I do not run most of the caching name servers on the Internet, so what do I know. Do you?
Would anyone who runs an anycast recursive name server care to supply data points to support or refute Mr. Anderson's assertion?
Mr. Anderson, do you have any data points to support your assertion?
Once upon a time, Dean Anderson <dean@av8.com> said:
If there are
multiple anycasted caching servers behind a specific IP address, then
those several cache's will each have a different state. Since, [as I
explained, and was supposed by the poster], there is "some kind of load
balancing going on", and also since implementors of anycast caches have
posted questions and explained their purposes [which could be seen as
"load balancing"], this is a likely explanation.
Not really. Normally, anycasting should be stable; if you send a
request to the same IP address you will get a response from the same
server. There may be additional load balancing being done where at a
particular location that IP maps to multiple servers (we've done that
with TruCluster for example), but that has nothing to do with
anycasting.
> But caching servers are usually setup to load balance. Usually, the
> servers with the same IP address share an ethernet along with multiple
> routers. So the packets are switched on essentially a per-packet
> basis.
> Or possibly a per-arp basis that alters the MAC-based-forwarding
> behavior
> of a switch. This is fairly fine grained load balancing.This is complete news to me. Of course, I do not run most of the
caching name servers on the Internet, so what do I know. Do you?Would anyone who runs an anycast recursive name server care to supply
data points to support or refute Mr. Anderson's assertion?
Our recursive name service, using anycast servers, is setup with 3
name servers at 3 different physical locations, with each server
connected to a router at the same physical location. Each server
handles two different anycast addresses. There is no per-packet load
balancing involved.
I can't speak for the rest of the net, of course - but our recursive
anycast service has worked well for several years.
Steinar Haug, Nethelp consulting, sthaug@nethelp.no
While that setup may have worked well, it's not an anycast implementation I would suggest that others follow. Having the same set of servers announcing multiple IP addresses (assuming those addresses are both in the same set of addresses given out to those doing dns lookups) leaves you open to a failure mode where a single server stops responding to queries but doesn't withdraw its routing announcement. If a user sees that server as the closest instance of both DNS server IP addresses, they will see that as a failure of "both" of their DNS servers.
A more reliable way of doing this is to have multiple anycast clouds with their own servers, each serving a single service address. That way, an incomplete failure (on query response but no route withdrawl) of a local server for one service address won't have any effect on access to the second service address.
I should note that what I describe as an "incomplete failure" here is the standard failure mode for non-anycasted servers, so this isn't a new problem created by anycast.
In the case of the roots, there are 13 of those "clouds," although some
of those clouds still consist of just a single server. For less critical infrastructure, like an ISP's local recursive name service, a considerably smaller number of clouds should be just fine.
-Steve
And I can show that if you give a pig wings....
I suppose IF a pig had wings, indeed, it *would* fly. But pigs aren't
growing winglets.
However, there are two relevant facts here:
1) People are starting to deploy PPLB.
2) People are starting to use TCP DNS
Look, it breaks in certain situations. But anycast implementations
of TCP apps have worked "well" for a decade now. Deal with the fact
that not only do people use it, but users don't notice it.Or don't. No one here cares if you do. Reality trumps lab tests.
"Reality" for the last ten years has been that no one did either PPLB or
TCP DNS. That reality is changing. It'll probably start to change faster,
sooner. Then, users will start to notice the problems.
> But caching servers are usually setup to load balance. Usually, the
> servers with the same IP address share an ethernet along with multiple
> routers. So the packets are switched on essentially a per-packet
> basis. Or possibly a per-arp basis that alters the
> MAC-based-forwarding behavior of a switch. This is fairly fine
> grained load balancing.This is complete news to me. Of course, I do not run most of the
caching name servers on the Internet, so what do I know. Do you?Would anyone who runs an anycast recursive name server care to supply
data points to support or refute Mr. Anderson's assertion?Mr. Anderson, do you have any data points to support your assertion?
Discussion of this very question on DNSOP. I refer you to the DNSOP
archives. (I keep my own archives, but you can find them through the IETF
pages at www.ietf.org). If you really can't find the relevant discussion,
I'll be happy to forward a slew of selected messages to you.
All fine and good. But nowhere in your long paragraph do you actually
address Steinar's question, which is "What does this rant have to do with
*ignoring* *DNS* *TTL*?"
People have been using TCP applications on anycast for at least a decade, as I mentioned before. Since DNS responses tend to be very short lived TCP session, it seems to me that if it works for other applications (e.g. HTTP), it should work for DNS.
Either way, reality still trumps lab tests, or mailing lists posts. Since it has worked, and continues to work, in _THE REAL WORLD_ for TCP applications much longer lived than DNS, I suggest that your assertion that "users will start to notice the problems" is incorrect. Of course, time will tell which of us is correct.
Maybe I'm insane. Or maybe you are. Although I think time has already told which of us is....
While that setup may have worked well, it's not an anycast implementation
I would suggest that others follow. Having the same set of servers
announcing multiple IP addresses (assuming those addresses are both in the
same set of addresses given out to those doing dns lookups) leaves you
open to a failure mode where a single server stops responding to queries
but doesn't withdraw its routing announcement. If a user sees that server
as the closest instance of both DNS server IP addresses, they will see
that as a failure of "both" of their DNS servers.
Agreed, that's a possibility. In practice it hasn't really turned out to
be a problem for us.
A more reliable way of doing this is to have multiple anycast clouds with
their own servers, each serving a single service address. That way, an
incomplete failure (on query response but no route withdrawl) of a local
server for one service address won't have any effect on access to the
second service address.
Yes, we have been doing some of this also. Part of the problem is the
desire to spread the load among the servers: Some of this comes naturally
from the different server locations in our network - but with 2 addresses
per server we can also balance the load somewhat by announcing either one
or two addresses per server.
Steinar Haug, Nethelp consulting, sthaug@nethelp.no
I fail to see the correlation still.. anycasted caches should all be operating
independently getting their DNS data from authoritative sources.
If at any point one of them uses a TTL that it has not received from the
authoritative source it is ignoring the ttl, where does anycast get involved
with this particular problem?
thanks
Steve
>> Or don't. No one here cares if you do. Reality trumps lab tests.
>
> "Reality" for the last ten years has been that no one did either
> PPLB or
> TCP DNS. That reality is changing. It'll probably start to change
> faster,
> sooner. Then, users will start to notice the problems.People have been using TCP applications on anycast for at least a
decade, as I mentioned before. Since DNS responses tend to be very
short lived TCP session, it seems to me that if it works for other
applications (e.g. HTTP), it should work for DNS.
Its funny how I give you TWO conditions, and you ignore one of them. I'll
try to use little tiny baby words:
TCP Anycast does NOT work with PPLB (Per - packet - load - balancing)
Say it slowly several times.
Either way, reality still trumps lab tests, or mailing lists posts.
Since it has worked, and continues to work, in _THE REAL WORLD_ for
TCP applications much longer lived than DNS, I suggest that your
assertion that "users will start to notice the problems" is
incorrect. Of course, time will tell which of us is correct.Maybe I'm insane. Or maybe you are. Although I think time has
already told which of us is....
Yes, indeed, it has. I've been vindicated on a number of issues on a
number of subjects. You?
Try actually reading the long paragraph. It starts with the interesting
sentence 'The data he showed isn't necessarilly "ignoring ttl".', and I
then explain why that is.
The queries produce different data, but none of the data is inconsistent
if there are different caches responding on the same address. Here is the
original description: (slightly reformated with roman numerals)
(I) I ran a query for a name in a zone I control that has a five minute
TTL on 204.127.198.4. The first query came up with 5 minutes.
(II) I quickly made a change to the zone.
(III) Thirty seconds after the initial query, I try
again...err... and come up with the change. Hmm... Not caching at all?
(IV) Another 30 seconds and I get the change, with 5m TTL.
(V) Thirty seconds later, I get the original response with appropriately
decremented TTL.
(VI) Another thirty seconds, I get the change, with 4m TTL.
Here is the detailed anycast explanation:
(I) Cache 1 gets answer to query X? = Y
(II) Authority changes X? to Z
(III) Cache 2 gets answer to query X? = Z
(IV) Cache 3 gets answer to query X? = Z
(V) Cache 1 responds
(VI) Cache 3 responds
No TTLs were ignored.
--Dean