Well, the recent jumbo AS path issue had an interesting effect of resource starvation on a few routers. Still, I think the softest targets are the root name servers. I was glad to hear at the Toronto NANOG meeting that this was being looked into from a routing perspective. Not sure what is being done from a DoS perspective.
---Mike
mike@sentex.net (Mike Tancsa) writes:
... Still, I think the softest targets are the root name servers. I was
glad to hear at the Toronto NANOG meeting that this was being looked into
from a routing perspective. Not sure what is being done from a DoS
perspective.
Now that we've seen enough years of experience from Genuity.orig, UltraDNS,
Nominum, AS112, and {F,K}.root-servers.net, we're seriously talking about using
anycast for the root server system. This is because a DDoS isn't just against
the servers, but against the networks leading to them. Even if we provision
for a trillion packets per second per root server, there is no way to get
the whole Internet, which is full of Other People's Networks, provisioned at
that level. Wide area anycast, dangerous though it can be, works around that.
See www.as112.net for an example of how this might work. "More later."
mike@sentex.net (Mike Tancsa) writes:
> ... Still, I think the softest targets are the root name servers. I was
> glad to hear at the Toronto NANOG meeting that this was being looked into
> from a routing perspective. Not sure what is being done from a DoS
> perspective.Now that we've seen enough years of experience from Genuity.orig, UltraDNS,
Nominum, AS112, and {F,K}.root-servers.net, we're seriously talking about
using
anycast for the root server system. This is because a DDoS isn't just
against
the servers, but against the networks leading to them. Even if we provision
for a trillion packets per second per root server, there is no way to get
the whole Internet, which is full of Other People's Networks, provisioned at
that level. Wide area anycast, dangerous though it can be, works around
that.
Is this the anycast based on MSDP ?
Regards
Marshall Eubanks
Date: Thu, 04 Jul 2002 19:46:42 -0400
From: Marshall Eubanks
Is this the anycast based on MSDP ?
No.
explains it well. Think of it as multihoming via several
separate systems.
Eddy
Anycast, not multicast.
-Bill
But the only IPv4 anycast
that I know of does use MSDP :
http://www.ietf.org/internet-drafts/draft-ietf-mboned-anycast-rp-08.txt
Is there a different proposal ? What's the RFC / I-D name ?
Regards
Marshall
But the only IPv4 anycast
> that I know of does use MSDP :
> http://www.ietf.org/internet-drafts/draft-ietf-mboned-anycast-rp-08.txt
> Is there a different proposal ? What's the RFC / I-D name ?
You seem to be confusing anycast with something complicated. It's not a
protocol, it's a method of assigning and routing addresses.
-Bill
FYI - for those scratching their heads on "anycast" .....
I just pushed out a paper on anycast by Chris Metz. Good foundation
material.
http://www.cisco.com/public/cons/isp/essentials/ip-anycast-cmetz-03.pdf
Doesnt announcing the same routing prefix into BGP from multiple locations do
the same thing without needing a new range or enhancement in IGMP etc ?
We do this in IGP currently..
Steve
FYI - for those scratching their heads on "anycast" .....
I just pushed out a paper on anycast by Chris Metz. Good foundation
material.http://www.cisco.com/public/cons/isp/essentials/ip-anycast-cmetz-03.pdf
Thanks - and the AS112 project seems to use static BGP spoofing, where
different locations announce the address via different paths :
"As a way to distribute the load for RFC1918-related queries, we use IPv4
anycast addressing. The address block is 192.175.48.0/24 and its origin AS is
112. This address block is advertised from multiple points around the Internet,
and these distributed servers coordinate their responses and back end
statistical analyses."
Marshall
Doesnt announcing the same routing prefix into BGP from multiple locations do
the same thing without needing a new range or enhancement in IGMP etc ?We do this in IGP currently..
Steve
As I see it, the problems with doing this in BGP are
- it's static - no failover. If AS 701 and AS 1239 are both
announcing a route to foo, and your preferred route is "through" AS701,
and the AS701 foo goes down, then you do not
automatically switch over to the AS1239 foo, even if you could reach it.
- there is no way to have multiple anycast addresses within an AS
- load balancing is tough
These may all be solved, though... it's hard to tell without a protocol
description.
Regards
Marshall Eubanks
Hi Marshall
If a route isnt withdrawn when the end network/device fails then no system will
fix that.
Presumably anycast wouldnt enable load balancing anyway as BGP only installs a
single route?
Or are you thinking both of these would be solved with a BGP enhancement?
Dont understand the multiple anycast comment, do you mean as it stands now? If
so it works fine if you inject the same route into an IGP providing you ensure
theres no IGP load balancing if you intend on doing TCP (altho most applications
for this appear to be UDP single request-responses)
Steve
Uhm it seems to me people are trying to make this whole AS112-thing sound more
complex than it really is...
We use the BGP anycast-method in our backbone, and have been doing so for a
long time. Basically, we have multiple caching DNS-servers scattered around
our network, but all of them use the same IP-adress (well, actually two -
since customers expect to configure a primary and a secondary DNS on their
computers). The DNS resolvers all run zebra and identify themselves as a
private AS, announcing two single host routes (the two DNS resolver-IP's) to
the border-router they are connected to.
Our customers' DNS queries will be routed to the nearest available server, by
the same mechanisms as any other hot-potato routing setup (i.e. MEDs). This
works beautifully since we are only dealing with DNS UDP packets. (The
servers do also have a unique IP adress for management traffic etc, and these
are normally routed in the IGP - but they do not respond to DNS traffic on
this IP). That way, we have both "load-balancing" (customer queries are
spread out to the servers who are closest to the customer) and redundancy -
if one resolver fails, BGP will use the next available route to get to this
prefix. The only difference with the AS112 setup is the fact that you are
doing it across several AS'es instead of just inside a single one, but the
principle is the same - and just as simple.
This is an extremely simple anycast setup for DNS servers, and potentially
other simple UDP-based services, we have been using it for a couple of years,
and it works beautifully. No new protocols, no complex setups, just normal
BGP operation. I even think someone wrote a very good paper on setting up DNS
resolvers this way once, though I can't remember where I saw it.
--Lars Erik
Date: Fri, 05 Jul 2002 09:05:44 -0400
From: Marshall Eubanks
- it's static - no failover. If AS 701 and AS 1239 are both
announcing a route to foo, and your preferred route is
"through" AS701, and the AS701 foo goes down, then you do not
automatically switch over to the AS1239 foo, even if you
could reach it.
???
- there is no way to have multiple anycast addresses within
an AS
???
- load balancing is tough
Just as tough as load-balancing over different upstreams in a
multihomed network. That's all anycast really is: multihoming
with the added twist of using multiple, separate systems instead
of one.
Each system has a unique, non-anycast IP address bound as the
primary IP, allowing communication between the disjoint parts.
Secondary IP(s) live(s) in the anycast range, and is/are routed
appropriately.
You can bind the appropriate 192.175.48/24 addresses to your NSen
and run an authoritative copy of the root TLD. IIRC, Paul even
mentioned doing this a few weeks ago... I believe the thread was
on dynamic DNS updates and Win2000's broken implementation.
Think of anycast as DDoS in reverse: Instead of distributed
traffic sources, one has distributed traffic sinks. Hence the
attractiveness in surviving DDos attacks.
Eddy
Marshall,
First, I hope you don't mind that I cut all the additional cc's. I don't
think any of the folks really needed extra copies 
Now...
Marshall Eubanks wrote:
>
> Doesnt announcing the same routing prefix into BGP from multiple locations do
> the same thing without needing a new range or enhancement in IGMP etc ?
>
> We do this in IGP currently..
Well, this doesn't need anything to change with normal BGP. It really
has very little to do with IGMP per se. The anycast routing prefix is
announced into many different networks, and as the end user, you will
see many paths, hopefully. If you only see one because of your IBGP,
then that's the path you'll take. If you see many, you'll take the one
that *your* ospf or isis setup prefers.
As I see it, the problems with doing this in BGP are
- it's static - no failover. If AS 701 and AS 1239 are both
announcing a route to foo, and your preferred route is "through" AS701,
and the AS701 foo goes down, then you do not
automatically switch over to the AS1239 foo, even if you could reach it.
No. Its not static. You may have misunderstood. Anycast is not just
multiple routes. It is also multiple machines in different places. So
there is really no single "foo". There are many "foos". Each one may
have more than one connection to the net. The announcements appear
behind many ASs. When your system sees many paths to "foo", it does not
know that in fact, each path goes to a different machine entirely, on a
different network even, in a different physical location. There's
another part that goes with anycast use, and dns; when any particular
foo goes down, or fails in any way, not just by physically failing, it
stops announcing itself (the router or routing software it uses
withdraws the route) and it is no longer one of the paths your network
will see. So if you were seeing it from 701, and 1239, and if anycast is
truly being used, you'll actually see the route being withdrawn from the
network(s) that has the foo that went bad. Unless, of course, there are
multiple foos in that network. In which case you will see no change and
you will still get to foo via the original route you preferred, just not
the foo you had used previously. And it makes no difference to you,
because in almost all of the cases, the query is answered in a single
packet, so persistence is irrelevant.
- there is no way to have multiple anycast addresses within an AS
Huh? What in the world do you mean here?
- load balancing is tough
Yes, which is why the load balancing services in the world are sold at a
premium. And it is not all that tough. With anycast, it is not
tough, at all, until you have to deal with the subject that brought this
thread up, ddos attacks. In which case it need real engineering.
These may all be solved, though... it's hard to tell without a protocol
description.
If you're talking about anycast and the way we're all using is in the
dns, there is no protocol as such. It uses existing mechanisms. All the
same protocols. You're currently making use of dns that uses anycast,
but you didn't have to modify anything, or download any new software, or
make any changes, did you?
> > >
> > > > But the only IPv4 anycast
> > > > that I know of does use MSDP :
> > > You seem to be confusing anycast with something complicated. It's not a
> > > protocol, it's a method of assigning and routing addresses.
> > >
> > > -Bill
You really do seem to be fixated on multicast still. anycast /=
multicast.
HTH
Dear Rodney;
Thanks for the info.
Rodney Joffe wrote:
Marshall,
First, I hope you don't mind that I cut all the additional cc's. I don't
think any of the folks really needed extra copiesNow...
Marshall Eubanks wrote:
Doesnt announcing the same routing prefix into BGP from multiple locations do
the same thing without needing a new range or enhancement in IGMP etc ?We do this in IGP currently..
Well, this doesn't need anything to change with normal BGP. It really
has very little to do with IGMP per se. The anycast routing prefix is
announced into many different networks, and as the end user, you will
see many paths, hopefully. If you only see one because of your IBGP,
then that's the path you'll take. If you see many, you'll take the one
that *your* ospf or isis setup prefers.As I see it, the problems with doing this in BGP are
- it's static - no failover. If AS 701 and AS 1239 are both
announcing a route to foo, and your preferred route is "through" AS701,
and the AS701 foo goes down, then you do not
automatically switch over to the AS1239 foo, even if you could reach it.No. Its not static. You may have misunderstood. Anycast is not just
multiple routes. It is also multiple machines in different places. So
That's the point 
there is really no single "foo". There are many "foos". Each one may
have more than one connection to the net. The announcements appear
behind many ASs. When your system sees many paths to "foo", it does not
know that in fact, each path goes to a different machine entirely, on a
different network even, in a different physical location. There's
another part that goes with anycast use, and dns; when any particular
foo goes down, or fails in any way, not just by physically failing, it
stops announcing itself (the router or routing software it uses
withdraws the route) and it is no longer one of the paths your network
will see. So if you were seeing it from 701, and 1239, and if anycast is
Let's go through this a little.
Let's say that you and I are running the foo service in anycast. You announce the foo IP address (say in a /24) behind your AS, I announce the same /24 behind my AS. Now, if my foo server goes down, how do my routers know to withdraw the announcements ? If they don't, why wouldn't people "closer" to me still try and get the foo service from me, alas, without success. That's what I meant.
Or, are you saying that an anycast host has to be a router running BGP ? So if it goes down, so would the service and the announcements? This works for DNS, but not for the things I would like to anycast.
truly being used, you'll actually see the route being withdrawn from the
network(s) that has the foo that went bad. Unless, of course, there are
multiple foos in that network. In which case you will see no change and
you will still get to foo via the original route you preferred, just not
the foo you had used previously. And it makes no difference to you,
because in almost all of the cases, the query is answered in a single
packet, so persistence is irrelevant.- there is no way to have multiple anycast addresses within an AS
Huh? What in the world do you mean here?
Sorry, too early in the AM. Withdrawn.
- load balancing is tough
Yes, which is why the load balancing services in the world are sold at a
premium. And it is not all that tough.
tough, at all, until you have to deal with the subject that brought this
thread up, ddos attacks. In which case it need real engineering.These may all be solved, though... it's hard to tell without a protocol
description.If you're talking about anycast and the way we're all using is in the
dns, there is no protocol as such. It uses existing mechanisms. All the
same protocols. You're currently making use of dns that uses anycast,
but you didn't have to modify anything, or download any new software, or
make any changes, did you?
Nope. Thanks for the info.
Marshall
Date: Fri, 05 Jul 2002 12:28:46 -0400
From: Marshall Eubanks
Let's go through this a little.
Let's say that you and I are running the foo service in
anycast. You announce the foo IP address (say in a /24)
behind your AS, I announce the same /24 behind my AS. Now, if
my foo server goes down, how do my routers know to withdraw
The server must have some routing intelligence. The simplest
case is a machine running Zebra speaking BGP or OSPF; if Zebra
is up, so is the route. A process can monitor DNS and kill the
route if needed.
Better yet, hack Zebra. Use Unix domain sockets and hack BIND to
send keepalives to Zebra. Or have Zebra launch BIND (a la DJB's
daemontools) and watch for SIGCHLD or use kqueue() on FreeBSD or
OpenBSD. Remember to apply some dampening before spewing IGP
equivalent into global tables.
the announcements ? If they don't, why wouldn't people
"closer" to me still try and get the foo service from me,
alas, without success. That's what I meant.
Yes, shortest path wins. That's why the routes must be yanked
when DNS dies.
If you have an internal backbone, anycast gets easier. Hint: no
MEDs needed (or even wanted), many BGP speakers, aggregation.
Stable routes to the outside world, and your IGP deals with dead
servers.
Or, are you saying that an anycast host has to be a router
running BGP ? So if it goes down, so would the service and
Perhaps not BGP, but some routing intelligence.
the announcements? This works for DNS, but not for the things
I would like to anycast.
What would you like to anycast?
Eddy
Yes, this document correctly described IPv4 anycast. It somewhat
overstates the severity of the issue with TCP and the dynamicism of the
underlying network topology... Although that's often brought up by people
who've never used anycast before and think they're being clever, in actual
deployed networks it's accounted for less than 0.001% of total traffic
volume, or far less than is generally lost across the network anyway. It
also describes the group-membership-announcement issue, which is basically
a non-issue now that all the host vendors can support OSPF, which neatly
fills the need.
-Bill
Or, are you saying that an anycast host has to be a router running BGP ?
No, typically they run OSPF.
> So if it goes down, so would the service and the announcements?
Correct. If a device wants to witdraw itself from a service pool, it
withdraws the host route associated with that service.
> This works for DNS, but not for the things I would like to anycast.
Mmm, like what? This is all ancient history at this point... It seems
unlikely that anyone would discover something that didn't work at this
late date.
-Bill
Correct. That's _all anycast is_. Nothing tricky here. At all.
-Bill