This was Vixie's last post on the subject of Anycast on DNSOP.
NB: Patrick Gilmore and Chris Morrow, note that Vixie agrees that HTTP
anycast is a bad idea.
Note the nonsense about anycast being "completely coherent".
Note also that Vixie continues to ignore per-packet load balancing issues,
and focuses on route-change times, instead.
Note the nonsense about anycast being "completely coherent".
If you check, I think you'll see that he actually said "ultradns's
anycast for .ORG is completely coherent".
Reasonable people can disagree. I think Paul is reasonable, I hope he thinks I am reasonable. We disagree. Doesn't mean I won't buy him a drink at the next NANOG. 
Tell you what, if you come to Seattle, I will buy you a drink too, just to smooth things over. Maybe we can agree to disagree, reasonably.
Also, whether Paul thinks it is a bad idea, or even if it really is a bad idea, that does not change the fact it was in wide use in the past, and probably still is used by many people without any serious problems.
NB [translation, "operational content"]: Akamai does not use any anycast for HTTP. I am not at all certain why Paul is telling us this is a bad idea, since we don't do it. Then again, we might in the future, I am not privy to every decision in the company. (No, that is not a "hint", I really do not think we will do anycast HTTP for content delivery, but I also really do not know everything we will do in the future.)
Either way, Paul's advice is always welcome, even if we don't listen sometimes. =)
There seems to be no possibility for anycast to be "completely coherent",
so ultradns' anycast couldn't be "completely coherent" either. But Vixie
mentions it to respond to comments by others about Ultradns' particularly
pervasive use of anycast.
--Dean
it may not be possible to make every service *consistent*, but it is
perfectly possible to make it coherent (i'm talking about coherency of
copies of a shared resource). i'm curious to see how you can substantiate
this claim, since any backend which supports distributed transaction
semantics will give you this. i can't comment on the veracity of paul's
statement comme applique ultradns, since i'm not familiar with how they do
things, but that doesn't change the fact that you've just made a statement
which appears blatantly false to anyone with any distributed systems
experience.
-p
Date: Tue, 3 May 2005 18:03:12 -0400
From: Patrick W. Gilmore
NB [translation, "operational content"]: Akamai does not use any
anycast for HTTP. I am not at all certain why Paul is telling us
this is a bad idea, since we don't do it. Then again, we might in
the future, I am not privy to every decision in the company. (No,
that is not a "hint", I really do not think we will do anycast HTTP
for content delivery, but I also really do not know everything we
will do in the future.)
One also should distinguish between TCP _to_ an anycasted address and
TCP _from_ an anycasted address. The latter is trickier, as asymmetric
routing increases the chances that the session will need to be
transferred to another pod:
ac LAX --> uc PHX
uc PHX --> ac DFW
This presses the question of synchronization (how do LAX and DFW know
about each other?) and source-based routing. IMNSHO there's more
overlap between these two topics than one might expect, too.
When anycast gets _really_ interesting is when an anycasted client makes
a request [from an anycasted address] to an anycasted server. The
results resemble persistent BGP oscillation due to poor MED choices:
ac LAX --> ac PHX
ac PHX --> ac DFW
ac DFW --> ac ORD
ac ORD --> ac SJC
ac SJC --> ac LAX
I could see that happening with anycasted SMTP, but it would be less
likely (although certainly not impossible) with HTTP. i.e., an HTTP
client isn't as likely to be anycasted as an SMTP client.
Far too much to cover in a noisy NANOG thread. 
Hence the need to
agree on just what "anycast" is before debating the merits.
Eddy
From: "Dean Anderson" <dean@av8.com>
To: "Mark Boolootian" <booloo@ucsc.edu>
Cc: <Nanog@merit.edu>
Sent: Tuesday, May 03, 2005 6:33 PM
Subject: Re: [dnsop] DNS Anycast revisited (fwd)>
>
> >
> > > Note the nonsense about anycast being "completely coherent".
> >
> > If you check, I think you'll see that he actually said "ultradns's
> > anycast for .ORG is completely coherent".
>
> There seems to be no possibility for anycast to be "completely coherent",
> so ultradns' anycast couldn't be "completely coherent" either. But Vixie
> mentions it to respond to comments by others about Ultradns' particularly
> pervasive use of anycast.it may not be possible to make every service *consistent*, but it is
perfectly possible to make it coherent (i'm talking about coherency of
copies of a shared resource). i'm curious to see how you can substantiate
this claim, since any backend which supports distributed transaction
semantics will give you this. i can't comment on the veracity of paul's
statement comme applique ultradns, since i'm not familiar with how they do
things, but that doesn't change the fact that you've just made a statement
which appears blatantly false to anyone with any distributed systems
experience.
Coherency of DNS service is dependent upon the operational clue of the
hostmaster in charge of such DNS systems and ensuring that each specific
system is maintained to provide consistent service. (i.e. ensuring zones
are properly updated on all servers, ensuring each system can answer
the queries by enduser with proper answer)
Anycast obviously opens a small set of can of caveats and notes while
providing benefits. A properly configured and maintained anycast DNS
setup can provide just as much coherency than using different unicast
addresses, and ofcourse YMMV. And this has been done and is already being
done by more ISPs to simplify their resolver DNS service. Honestly I've
heard more success stories with anycast deployments than failure stories
from the operational community thus far. Failure stories I've heard so
far with respect to operational caveats of anycast were primarily brought
up by those researching different and varying scenarios, many of which
are very uncommon on the normal internet, but not primarily from those
who actually operate a network and striving to provide service.
Just as with anycast, people implementing per-packet load balancing should
be aware of its caveats and implications before turning it on without
reading the fine manual. Also note that most of popular setup of
per-packet load balancing is done on a setup where multiple circuits are
bound to the _same_ router, to provide added throughput. For most people,
the extensive jitter provided by enabling PPLB accross different uplink
POPs/ISPs is enough to fail the Return On Investment (ROI) of doing so.
Before we even get to jitter, I would hate to see a sporadic sick-looking
traceroute for all hops beyond the point where PPLB is enabled, due to
PPLB being enabled between different IGP paths or uplink carriers.
I myself would agree with the most of the anycast-supportive operators
in that we should rather worry about tangible reality operational problems,
not theories or lab-environment-generated proof of concepts.
To really understand just what anycast is, I find this pdf introduction
pretty useful for many folks who are new to the anycast topic and are
looking to find some value in this long thread 
--
<http://www.nanog.org/mtg-0310/miller.html>
-J
(another person who deploys anycast for a few networks and have been
very happy with the results it provided)
This seems to be a trivial interpretation of "coherent". It is assumed
that the copies of DNS _zones_ are kept in sync regardless of whether the
servers are to traditional replicas or to anycasted replicas. No one ever
claimed that zone transfers between the copies would be affected by
anycast. The "in-sync"-ness of the zone data is competely orthogonal to
anycast. Roots are updated via back channels on non-anycast addresses, and
not with AXFR.
Though, the whole "complete coherency" statement seems to be nonsense, so
I can't really say that it wasn't what he meant. If it was, it had no
bearing on anything, certainly not on any concerns having to do with
Ultradns' operations. The concern about Ultradns was the pervasive use of
anycast. As just pointed out, it seems they've altered this, perhaps as a
result of these concerns being aired. (You're welcome, but save the thanks
until we have a solid solution)
The following is interesting: *some* Vixie associates thought that asking
Nanog how many roots were anycast was considered off topic: (I think this
query also resulted from DNSOP discussions in Oct/Nov '04.)
(now, wasn't randy bush one of the people who was doing ad hominems on me
just now? So he knows I have a valid point, but took a shot anyway. Nice.
Get 'em while you can, I guess)
> > There seems to be no possibility for anycast to be "completely
coherent",
> > so ultradns' anycast couldn't be "completely coherent" either. But
Vixie
> > mentions it to respond to comments by others about Ultradns'
particularly
> > pervasive use of anycast.
>
> it may not be possible to make every service *consistent*, but it is
> perfectly possible to make it coherent (i'm talking about coherency of
> copies of a shared resource).This seems to be a trivial interpretation of "coherent". It is assumed
that the copies of DNS _zones_ are kept in sync regardless of whether the
servers are to traditional replicas or to anycasted replicas. No one ever
claimed that zone transfers between the copies would be affected by
anycast. The "in-sync"-ness of the zone data is competely orthogonal to
anycast. Roots are updated via back channels on non-anycast addresses, and
not with AXFR.
i'm terribly sorry, but i'm unable to extract any meaning at all from these
statements. when i parse them, they make no sense at all (not in terms of
being wrong, just not understandable). could you rephrase them?
coherency and consistency are well-defined terms in systems engineering. we
are talking about dns queries and hence coherency of zone data (the shared
resource). i fail to see how this is open to any interpretation at all.
i snipped the rest for obvious reasons.
-p
Sorry, The original statement Vixie made is nonsense. Here is the original
statement again:
Vixe writes:
lest anyone be confused, ultradns's anycast for .ORG is completely
coherent and doesn't admit the possibility of giving out different
responses from different anycast nodes for policy reasons or any other
reason, and so it's an example of "good" anycast the way i count such
things.
Vixie seems to be responding to concern raised for Ultradns' pervasive use
of anycasting. This was the only issue raised involving Ultradns.
During the anycast discussion on DNSOP, the subject of zone coherency (as
normally used) was not an issue. So there is no question of zone
coherency for Ultradns' servers. We assumed (and did not dispute) that
zone updates were unaffected by anycast. Zone updates happen over private
secure channels on non-anycasted IP addreses. They ought to be as coherent
as DNS gets. They ought not be affected by anycast.
Vixie ends by saying essentially, that because of Ultradns' coherency, it
is an example of "good anycast". But the two issues (coherency and
anycast) have no relationship. There is no reason to conclude that
coherency means anycast is either good or bad. Hence, his statement is
nonsense.
Alternately, one might read "coherency" as refering to the shared TCP
state of the group of servers. The "shared TCP state" of a group of
anycast servers is conceptual. There is no actual coherency for this
shared TCP state. So that is nonsense, too.
I suspect the statement is just nonsense, that is hoped to sound good.
Many people will not really read or parse his whole statement, but will
remember 'Ultradns is an example of good anycast, according to Vixie'
But, given that the original statement is nonsense, I can't really say
that any of these are correct.
--Dean
Just to make life fun, there is the whole "anycast a bunch of name servers, each with different zone files pointing at local HTTP servers". Since the "anycast" portion is over UDP, it avoids a lot of the problems (real or otherwise) mentioned here, and the HTTP is still unicast but distributed and can be made resilient to failure.
Of course, the DNS backend is then .. uh .. "de-coherent"? But it works, and works well, in many currently operational configurations.
Does PPLB (or anything else) break this? I'm certain I could find things that would break this if I looked hard enough. But as I've said many times, reality trumps NANOG posts. Since this is a _working_ configuration today, I would say that disproves any claims that it cannot or will not work.
And last time I checked -- on this list, mind you -- it certainly was not.
Cf. people trying to run and hide, or lash out at me for complaining, when I
pointed out how two anycast routes pointing to the same dead node made the
.ORG anycast implementation unusable.
I reserve judgment on whether their implementation has been fixed in the
meantime; I have no evidence either way at the moment.
Date: Tue, 3 May 2005 21:58:37 -0400
From: Patrick W. Gilmore
Just to make life fun, there is the whole "anycast a bunch of name
servers, each with different zone files pointing at local HTTP
servers". Since the "anycast" portion is over UDP, it avoids a lot
of the problems (real or otherwise) mentioned here, and the HTTP is
still unicast but distributed and can be made resilient to failure.
Actually, it's not because the anycast portion is over UDP _per se_.
Someone hits an anycasted [Akamai] DNS server, which then transfers the
session to a unicast IP address c/o DNS.
Once the data is flowing via unicast, there's no further chance of
transferring the session to another anycast box. DNS --> <whatever> is
a quick-and-dirty way of accomplishing the session transfer once and for
all without requiring additional protocol intelligence. It's also
visible to the world, as opposed to behind-the-scenes session transfer.
Of course, the DNS backend is then .. uh .. "de-coherent"?
Comparing anycasted DNS service with CDNs is about as valid as comparing
TCP with BGP. They serve different purposes at different levels, and
one relies on the other.
it works, and works well, in many currently operational configurations.
Indeed. One could move beyond CDNs and cite many corporate WANs as
examples of working configurations that lack coherent DNS. Unicast
nameservers can lack coherency. As several others have said, anycast
and coherency are orthogonal.
Does PPLB (or anything else) break this? I'm certain I could find
things that would break this if I looked hard enough. But as I've
The trick is finding something that breaks your config without affecting
"typical" ones.
said many times, reality trumps NANOG posts. Since this is a
_working_ configuration today, I would say that disproves any claims
that it cannot or will not work.
I'd not go so far as to say "disproves", but it shifts the burden of
proof.
Eddy
Dean Anderson wrote:
i'm terribly sorry, but i'm unable to extract any meaning at all from these
statements. when i parse them, they make no sense at all (not in terms of
being wrong, just not understandable). could you rephrase them?coherency and consistency are well-defined terms in systems engineering. we
are talking about dns queries and hence coherency of zone data (the shared
resource). i fail to see how this is open to any interpretation at all.Sorry, The original statement Vixie made is nonsense. Here is the original
statement again:Vixe writes:
lest anyone be confused, ultradns's anycast for .ORG is completely
coherent and doesn't admit the possibility of giving out different
responses from different anycast nodes for policy reasons or any other
reason, and so it's an example of "good" anycast the way i count such
things.Vixie seems to be responding to concern raised for Ultradns' pervasive use
of anycasting. This was the only issue raised involving Ultradns.During the anycast discussion on DNSOP, the subject of zone coherency (as
normally used) was not an issue. So there is no question of zone
coherency for Ultradns' servers. We assumed (and did not dispute) that
zone updates were unaffected by anycast. Zone updates happen over private
secure channels on non-anycasted IP addreses. They ought to be as coherent
as DNS gets. They ought not be affected by anycast.Vixie ends by saying essentially, that because of Ultradns' coherency, it
is an example of "good anycast". But the two issues (coherency and
anycast) have no relationship. There is no reason to conclude that
coherency means anycast is either good or bad. Hence, his statement is
nonsense.
Context helps.
In the previous paragraph Vixie said:
while i'm on the subject, i also remain convinced that using anycast to do
distributed load balancing for applications like WWW, on the assumption
that the path you heard a dns query on is instructive as to what content
would be best to answer with, is silly, and will more often do harm or do
nothing than do good. (and i've told akamai and speedera this many times.)
("but it makes for great marketing slideware.")
In other words this is a bad idea:
[FT@fenrir FT]$ dig a248.e.akamai.net @69.45.79.10
;; ANSWER SECTION:
a248.e.akamai.net. 20 IN A 80.67.72.214
a248.e.akamai.net. 20 IN A 80.67.72.201
FT@inuyasha:~$ dig a248.e.akamai.net @69.45.79.10
;; ANSWER SECTION:
a248.e.akamai.net. 20 IN A 69.45.79.15
a248.e.akamai.net. 20 IN A 69.45.79.16
While I'm not a mind reader, It seems he's saying that, since Ultradns doesn't use anycast to do this, it is an example of 'good anycast.'
Date: Tue, 3 May 2005 22:21:45 -0400 (Eastern Daylight Time)
From: Todd Vierling
[ trimming CC list before it grows too long ]
And last time I checked -- on this list, mind you -- it certainly
was not. Cf. people trying to run and hide, or lash out at me for
complaining, when I pointed out how two anycast routes pointing to
the same dead node made the .ORG anycast implementation unusable.
Akamai's service uses non-coherent DNS by design. Your post referenced
a failure case in which DNS service was not coherent by virtue of
certain pods not responding; UDNS attempts to provide coherent DNS
service.
I reserve judgment on whether their implementation has been fixed in the
"me too"
meantime; I have no evidence either way at the moment.
One of the challenges of anycast is failure detection and mitigation.
<mumbles>
flooding clusters via source-based routing
tunneling anycast-destined OAM packets via unicast
ns-to-machine affinity within pods
tight coupling of DNS service to anycast route injection
</mumbles>
Anycast implementation _does_ present new operational challenges, but
they're hardly insurmountable.
Eddy
If you want to be taken seriously, perhaps you might want to research the
use of the term coherency in terms of DNS.
Or not.
Hint: It doesn't mean that one node responds and another doesn't.
Rodney Joffe
CenterGate Research Group, LLC
http://www.centergate.com
"Technology so advanced, even WE don't understand it"(R)
Comments on (and contributions to) draft-ietf-grow-anycast-00 would be gratefully received (by private mail, probably, rather than on this list).
Joe
<feedback loop="trollbait">
It certainly breaks coherency in my mind when one query host receives
SERVFAIL for a while, then nothing at all, but another query host is
receiving records that the first query host *should* have seen.
</feedback>
After all these years, I should know better than to feed back the bait.
Maybe it's just fun to do.
That said, I'll credit you for catching my brain fart. I read right past
'coherent' in vix's text and saw the phrase '"good" anycast' in relation to
the .ORG zone. It was not technically a *coherency* flaw to which I
referred, but rather a fundamental *architecture* flaw. (It's too bad that
the architectural flaw was never really discussed, or for that matter
subjected to a mea culpa and/or public postmortem. But I suppose that is
the pride/ego component of running such a large zone -- even one intended
for the "public interest".)
I wrote and maintain a query host deterministic, but inter-site coherent,
anycasted DNS server for my $orkplace, so don't presume to tell me what you
think I know. It only makes you look like a holier-than-thou ass [again].
Feel free to feed the trolls elsewhere, though, if you wish; it can be fun
to watch.
Sorry if it was not clear, but my post had _nothing_ to do with "CDNs".
I was just talking about people setting up anycast name servers, each of which pointed at a different HTTP server (or other service), to spread load. In many cases, the two servers are the same.
Date: Tue, 3 May 2005 23:56:48 -0400
From: Patrick W. Gilmore
I was just talking about people setting up anycast name servers, each
of which pointed at a different HTTP server (or other service), to
spread load. In many cases, the two servers are the same.
Ah, okay... which again helps demonstrate the lack of coupling between
*cast and coherency. A single unicast DNS server can serve split views
just as easily.
No, it disproves. You say "it will not / cannot work". Showing you
a working instance in production absolutely disproves your statement.You can say "it might break", but that's a different statement.
Yes. I misread/misthought "disproves will not / cannot work" as "proves
can / will work".
Speaking of things that, and people who, are incoherent... 
Eddy