Hello everyone.
I am working on creating a small anycasting based setup with 3-4 servers in
US. Plan is to use this for DNS and later for CDN setups. I have few
confusions in mind and was wondering if you guys here can put some light on
them:
1. For anycasting does announcing a /24 from different ASNs (of
different datacenters) makes sense or it will be an issue to have a block
being announced from different ASNs and I should avoid and prefer having
own router below datacenters network and eventually use one single ASN to
announce the anycasting block?
2. We plan to use this anycasting based setup for DNS during initial few
months. Assuming low traffic for DNS say ~10Mbps on average (on 100Mbps
port) and transit from just single network (datacenter itself) - is this
setup OK for simple software based BGP like Quagga or Bird? Certainly
colocating routers will be slow & expensive. Does it offer any direct
advantage in such simple setups?
3. IPv6! - I am looking at possibility of having support of IPv6 in
anycast right from start. Can't really find a good prefix size for
anycasting announcement. I can see Hurricane Electric as well as Google
using whole /32 block for IPv6. So is /32 is standard? We have only one /32
allocation from ARIN and thus if using /32 seems like hard deal - we have
to likely get another /32 just for anycasting? or we can use /48 without
issues? Also, is /48 a good number for breaking /32 so that we can do /48
announcements from different datacenters in simple uni casting setup?
I apologize for any wrong questions/logic - really new to this. Please
correct me if I am wrong on any concept.
Appreciate your help.
Thanks.
Hello, Anurag.
1. For anycasting does announcing a /24 from different ASNs (of
different datacenters) makes sense or it will be an issue to have a block
being announced from different ASNs?
Keeping a consistent announcing ASN for your prefix is thought to be best-practice, and if you don't do so, eventually there will be people who will undoubtedly complain, but there is no technical difficulty with announcing your same prefix from multiple origin ASNs. Any difficulties you encounter will be because of people aggressively filtering what they choose to listen to.
2. We plan to use this anycasting based setup for DNS during initial few
months. Assuming low traffic for DNS say ~10Mbps on average (on 100Mbps
port) and transit from just single network (datacenter itself) - is this
setup OK for simple software based BGP like Quagga or Bird?
Yes, and in fact, that's how nearly all large production anycast networks are built… Each anycast instance contains its own BGP speaker, which announces its service prefix to adjacent BGP-speaking routers, whether those be your own, or your transit-provider's. Doing exactly as you describe is, in fact, best-practice.
3. IPv6! - Is /32 is standard? We have only one /32
allocation from ARIN and thus if using /32 seems like hard deal - we have
to likely get another /32 just for anycasting? or we can use /48 without
issues? Also, is /48 a good number for breaking /32 so that we can do /48
announcements from different datacenters in simple uni casting setup?
A /48 is quite reasonable. Announcing a whole /32 just for your anycast service would be wasteful.
Good luck!
-Bill
Bill,
woody@pch.net (Bill Woodcock) wrote:
> 2. We plan to use this anycasting based setup for DNS during initial few
> months. Assuming low traffic for DNS say ~10Mbps on average (on 100Mbps
> port) and transit from just single network (datacenter itself) - is this
> setup OK for simple software based BGP like Quagga or Bird?
Yes, and in fact, that's how nearly all large production anycast networks are built??? Each anycast instance contains its own BGP speaker, which announces its service prefix to adjacent BGP-speaking routers, whether those be your own, or your transit-provider's. Doing exactly as you describe is, in fact, best-practice.
Well, let's say, using Quagga/BIRD might not really be best practice for
everybody... (e.g., *we* are using Cisco equipment for this)
Using anycasting for DNS is, to my knowledge, best practice nowadays.
> 3. IPv6! - Is /32 is standard? We have only one /32
> allocation from ARIN and thus if using /32 seems like hard deal - we have
> to likely get another /32 just for anycasting? or we can use /48 without
> issues? Also, is /48 a good number for breaking /32 so that we can do /48
> announcements from different datacenters in simple uni casting setup?
A /48 is quite reasonable. Announcing a whole /32 just for your anycast service would be wasteful.
Why? It's simply another prefix, no matter how big. It might look
wasteful, but if *that* is the allocation you *have*, it's the
one you ought to use.
One should be careful - people do filter on allocation lengths, so
breaking out a /48 out of a /32 allocation and advertising it on its
own can lead to it being filtered.
Elmar.
if you know anyone who is filtering /48 , you can start telling them to STOP doing so as a good citizen of internet6.
I agree with Woody anything more than /48 for anycast is waste.
mehmet
Bill,
2. We plan to use this anycasting based setup for DNS during initial few
months. Assuming low traffic for DNS say ~10Mbps on average (on 100Mbps
port) and transit from just single network (datacenter itself) - is this
setup OK for simple software based BGP like Quagga or Bird?
Yes, and in fact, that's how nearly all large production anycast networks are built??? Each anycast instance contains its own BGP speaker, which announces its service prefix to adjacent BGP-speaking routers, whether those be your own, or your transit-provider's. Doing exactly as you describe is, in fact, best-practice.
Well, let's say, using Quagga/BIRD might not really be best practice for
everybody... (e.g., *we* are using Cisco equipment for this)
Actually there is a *very* good reason why many (most?) anycast
instances use quagga/BIRD/gated/etc
to speak bgp (or even ospf for internal anycast) which using a Cisco (or
any separate router) usually won't accomplish.
-- Pete
How does your Cisco know whether an adjacent nameserver is heavily loaded, and adjust its BGP announcements accordingly?
-Bill
Re Bill,
pete@altadena.net (Pete Carah) wrote:
> Well, let's say, using Quagga/BIRD might not really be best practice for
> everybody... (e.g., *we* are using Cisco equipment for this)
Actually there is a *very* good reason why many (most?) anycast
instances use quagga/BIRD/gated/etc
to speak bgp (or even ospf for internal anycast) which using a Cisco (or
any separate router) usually won't accomplish.
Please enlighten me...
Elmar.
Re Bill,
woody@pch.net (Bill Woodcock) wrote:
> Well, let's say, using Quagga/BIRD might not really be best practice for
> everybody... (e.g., *we* are using Cisco equipment for this)
How does your Cisco know whether an adjacent nameserver is heavily loaded, and adjust its BGP announcements accordingly?
It doesn't have to.
I don't know how you guys do it, but we take great care to
keep min. 70% overhead capacity during standard operation.
Elmar.
RFC 2870 section 2.3 suggests 33%. How us guys do it is 2%-3%, since "standard operation" is only the case when nobody's getting DDoSed. And then we have a backup plan, which is to be able to redirect queries away from nodes that are overloaded. And we have backup plans for the backup plans. But then, we've been doing anycast DNS for twenty years now, so we've had some time to develop those plans.
I think what you're hearing from other people, though, is that having a backup plan is, indeed, best practice.
-Bill
My point had to do with resilience in the face of hardware/OS/software
failures in the box providing the
service. Bill's has more to do with resilience in the face of other
network events (e.g. the upstream for one
of the boxes has a DDOS; you cannot reasonably provide enough excess
capacity to handle that...) Neither of these is addressed by using a
separate router to announce the server's anycast route. (unless somehow
the Cisco is providing the anycasted service, which would address my
concern but still not Bill's.)
Also, Bill is probably talking root (or bigger public) servers whose
load comes from "off-site"; the average load characteristics for those
are well known but there can be extremes that would be hard to plan for
(hint - operating at 30% isn't really good enough, probably not 10%
either. Bill (and the other Bill) have pretty good stats for this that
I've only glanced at...) And it is easy to see where one of the
extremes might hit only one or two of the anycast instances. He implies
having the instances talking to each other in background to adjust bgp
announcements to maybe help level things. Fortunately at least for the
root servers, the redundancy is at two levels and anycast is only one of
them.
-- Pete
Thanks for guidance everyone!
Appreciate it.
And yes, I can see another thread running on discussion about /48 - I am
listening silently about it.
Multiple AS doing anycasting was little concern for me, but now seems good
since I can see everyone's suggestion to use single own ASN for anycasting.
I've done this two ways.
I've used Quagga to announce routes directly from the anycast servers. This guarantees you that the route will go away if the server completely goes away, and that traffic will be directed elsewhere. It also allows you to run scripts on the servers that can withdraw the routes in other circumstances, such as if a script running on the server detects that the server is non-responsive (or overloaded).
I've used load balancers in front of the name servers. Like Quagga running directly on the server, a load balancer can withdraw routes when all servers behind it stop responding. It has some advantages, in that it can withdraw routes to non-responsive servers even in cases where the server may be too confused to detect its own problems and send the appropriate messages to Quagga. It can spread load among a larger collection of servers than a router would be able to on its own, sit in front of the servers and do rate limiting, and things like that. It could help with the overload issue Bill mentions by selectively sending some queries to other sites without the all or nothing effect you get from a BGP route withdrawal. On the other hand, load balancers aren't cheap, and and once installed in the middle of a network they become one more device to fail.
I have no idea what Cisco equipment Elmar is using, but I wouldn't jump to the conclusion that it can't withdraw routes when needed.
-Steve
Steve Gibbard (scg) writes:
I have no idea what Cisco equipment Elmar is using, but I wouldn't jump to the conclusion that it can't withdraw routes when needed.
Wouldn't the dns bit of ip sla do most of what's needed on IOS ?
Cisco IOS Software Releases 12.4 Mainline - Retirement Notification - Cisco
some interesting examples at www.cisco.com/web/CA/events/pdfs/CNSF2011-Automations_for_Monitoring_and_Troubleshooting_your_Cisco_IOS_Network-Dan_Jerome.pdf
(slide 29 and onwards)
Note: this is more of a question than an assertion,
I've used quagga/ospfd for DNS anycasting within ISPs, and a script
to monitor the nameserver response, but I'd love to hear what people are
doing that's not host based.
Morn' Steve,
scg@gibbard.org (Steve Gibbard) wrote:
I have no idea what Cisco equipment Elmar is using, but I wouldn't jump to the conclusion that it can't withdraw routes when needed.
We use scripts external to both the routing platform and the
service delivery platform to check the service and reconfigure
L3 equipment (which is all kinds of L3 capable hardware).
Elmar.
Hello everyone
Thought to re-open to this thread and discuss couple of doubts I have in
mind regarding the same.
I tried doing anycasting with 3 nodes, and seems like it didn't worked well
at all. It seems like ISPs prefer their own or their customer route (which
is our transit provider) and there is almost no "short/local route" effect.
I am presently using Charter + Cogent/Gblx + Tinet for this setup.
I was looking for some advise over this issue:
1. How to deal with ISPs not prefering local node on their peers network
and going to far off node on their own/or customer network? Is it just
normal and there's no fix by any BGP announcement method/conf file
parameter? Should one prefer taking transit for all locations from same ISP?
2. I am using Quagga on all instances. Any advise on configuration file
parameters specifically for anycasting instance? (I would be happy to share
existing conf. is required). I did tried AS path prepending, but seems like
it didn't helped much (or may be I fail to get it working). What are
possible parameters one can have in conf for anycasting case (BGP MED's)?
3. Is putting singlehommed nodes with no peering and transit from single
ISP was poor idea? I wonder how other small players do it? Do you take
multiple upstreams for all anycasting DNS nodes?
Though route pull off is pretty much instance, and I see no problems with
it. Also, I have not got the node in EU up yet (which will be under colo's
network which is under Telia). I guess since distance in EU with US is way
higher then between our domestic US nodes, so probably I will see
local/near routing effect with EU server. But still clueless on how to get
it working in current setup.
Appreciate everyone's comments and help.
Thanks.
Correct. That's why you need to use the same transit providers at each location.
http://www.pch.net/resources/papers//dns-service-architecture/dns-service-architecture-v11.pdf
Slides 20-29.
-Bill
It could be a nightmare to try to balance the traffic when you are using different providers.
You can go ahead and try using path prepending but you will always find some strangeness going on regardless.
As Bill mentioned using the same transit will help, especially if you use a transit provider that has some communities pre-defined which will allow you to automatically advertise or not (even geographically) , and path prepend by simply sending communities out , you will save lots of time.
Mehmet