oh k can you see

Mailman List Mirror (Read Only)
1

so a few of us are still looking at routing through the anycast
sunglasses. a particular probe is seeing instability [0] for
k.root-servers.net [1]. so we hop on to a router nearby, and
have some fun looking at things. we discover an anomaly which
takes a while to sort out

  o some of the anycasted servers mark their announcements with
    the magic NO_EXPORT community in an attempt to localize
    their distribution (it would be good if a server in kenya
    did not take load from nyc)

  o they also have a server or two which does not so mark their
    announcements on the presumption that the rest of the world
    can get to those non-marking server(s)

this last assumption is not safe

  o consider large providers p0 and p1 which are connected to
    k0 which announces with NO_EXPORT

  o there is also server k1 which is out there somewhere and
    announcing without NO_EXPORT

  o test point t0 is in a multi-homed asn connected to p0 and
    p1

  o the routers in p0 and p1 at which t0's router is connected
    have their best path to k being the one to k0

  o this obscures their path to k1

  o and, as they obey k0's NO_EXPORT, they can not export any
    route to a k.root-servers.net server to t0

  o so t0 sees no route to k.root-servers.net

this implies that a non-trivial part of the net can not see
anycast services for which some of the servers are marking
their announcements as NO_EXPORT.

note that we saw this from a multi-homed site in seattle, not
from some more net-isolated probe.

whoops!

randy

2

Randy Bush wrote:

so a few of us are still looking at routing through the anycast
sunglasses. a particular probe is seeing instability [0] for
k.root-servers.net [1]. so we hop on to a router nearby, and

  o this obscures their path to k1

  o and, as they obey k0's NO_EXPORT, they can not export any
    route to a k.root-servers.net server to t0

Isnt this the standard problem? Why does anycast have any special bearing on the problem? (perhaps it doesnt you just want someone to fix it)

My amateurs understanding of this is that:

Sites connected to providers who have chosen a path marked as NO_EXPORT as best over one not so marked will not get any route to that prefix from that provider. They better hope that they are connected to another provider who did not select as best path a NO_EXPORT marked prefix.

A number of conclusions can possibly be made:

NO_EXPORT is not safe to be used while trying to traffic engineer but maintain global connectivity.

NO_EXPORT is only safe for more specific prefixes, as long as there is a less specific prefix that is still usuable.

NO_EXPORT to a large provider with potential large numbers of single homed BGP customers who may not be taking a 0/0 (in an attempt to use SAV?) is probably not a good idea

NO_EXPORT to large providers raises the probablity of there being sites who multihome to only those, therefore NO_EXPORT to multiple large providers is almost certainly dangerous.

traffic engineering done by the core based upon instructions from the edge is dangerous.

3

Isnt this the standard problem?

Correct.

    > Why does anycast have any special bearing on the problem?

It doesn't, except that Vixie decided to do it anyway with F, and it
sounds like now K is doing it as well.

I have no clue why. The problems with doing this had been clearly
understood for a long time, the putative benefit is negliglble, and to the
best of my knowledge, none of those of us who run large anycast networks
commercially have ever found it beneficial to use NO_EXPORT in the general
case.

I guess folks just like to be different.

    > Sites connected to providers who have chosen a path marked as NO_EXPORT as
    > best over one not so marked will not get any route to that prefix from
    > that provider. They better hope that they are connected to another
    > provider who did not select as best path a NO_EXPORT marked prefix.

Correct.

    > NO_EXPORT is not safe to be used while trying to traffic engineer but
    > maintain global connectivity.

Correct.

    > NO_EXPORT is only safe for more specific prefixes, as long as there is a
    > less specific prefix that is still usuable.
    
Correct.

    > NO_EXPORT to a large provider with potential large numbers of single homed
    > BGP customers who may not be taking a 0/0 (in an attempt to use SAV?) is
    > probably not a good idea
    
Correct.

    > NO_EXPORT to large providers raises the probablity of there being sites
    > who multihome to only those, therefore NO_EXPORT to multiple large
    > providers is almost certainly dangerous.
    
Correct.

Which leaves the question of why F, and now K, appear to be trying to do
it.

                                -Bill

4

F's covering prefix, 192.5.5.0/24, is advertised to peers of F-root local nodes with NO_EXPORT. 192.5.5.0/24 is advertised to peers of AS 3557 without NO_EXPORT.

A shorter prefix, 192.5.4.0/23, is also advertised without NO_EXPORT from AS 3557.

In the cases where local nodes of F advertise 192.5.5.0/24 with NO_EXPORT to ASes which also provide transit for 3557, we have taken care to ensure that local policy in those ASes causes the 3557 route (a customer route) always to be selected in preference to that received from the local node (a peer route). In these cases the local node provides local backup service.

A description of F's deployment strategy can be found in <http://www.isc.org/pubs/tn/isc-tn-2003-1.html>.

Joe

5

Randy's description of the issue with NO_EXPORT is correct.
It has never appeared to be particularly widespread.
It is not specific to anycast.

You also describe the rationale correctly by saying "it would be good if
a server in Kenya did not take load from nyc". I'll expand a little
more on that. K does anycast with two objectives: primarily to increase
robustness of the service in the face of serious load increases,
secondarily provide better service to some local areas in the Internet
topology. It is the secondary objective that poses the problem. We
operate "local nodes" which are intended to serve only a local area.

This is clearly a routing problem and routing policy is clearly the
responsibility of ISPs. The local ISPs should make sure that routes of
local nodes do not propagate far enough to cause unwanted load.
Consequently we could just announce one route without doing anything to
it and "wash our hands" as far as routing and network load is concerned.
Server load is not a concern here because in the case of local nodes the
network will saturate far more quickly than the server.
This is a very clean solution. It keeps responsibility where it belongs
and does not introduce extra complexity in BOP routing. I like it. :wink:

However we try to be helpful and provide tools to the ISPs by tagging
the routes from such "local nodes" with NO_EXPORT and we artificially
lengthen the AS-paths to the global nodes in order to make the local
nodes more attractive to ASes that hear both. The latter is problematic
too because it can cause non-optimal node selection but does not lead to
anything worse than that. Remember that these are nothing but hints to
ISPs who determine their own routing policy and are responsible for it.
So if someone barks at K for this it is the wrong tree for the most part.

What should we do? Stop giving the hints? Add complexity by announcing
another less specific covering prefix like F does? Any better ideas?

We are currently in an evaluation phase of our anycast deployment.
We are taking measurements and are analysing them.
Early results:
http://www.ripe.net/ripe/meetings/ripe-51/presentations/pdf/ripe51-anycast_k-root.pdf
We are also seriously considering to reduce the number of local nodes
where this is feasible.

This is a good time to hear good ideas. Let us have them.

Daniel

PS: Bill, I hope this also answers your question on why we do this.
We have been doing it for a long time too.
As I said: suggestions are welcome.

6

mornin' daniel:

You also describe the rationale correctly by saying "it would
be good if a server in Kenya did not take load from nyc".
I'll expand a little more on that. K does anycast with two
objectives: primarily to increase robustness of the service
in the face of serious load increases, secondarily provide
better service to some local areas in the Internet topology.
It is the secondary objective that poses the problem. We
operate "local nodes" which are intended to serve only a
local area.

when it is connected to global providers, this does not work.
and do not count on the hope that small local provider p0 does
not pass the marked prefix to a global provider - that's like
saying 1918 prefixes will never leak.

[ note: i have friends in kenya, and would be happy if this
  stuff would work well. this does not mean that i will
  pretend that it does. ]

This is clearly a routing problem and routing policy is
clearly the responsibility of ISPs.

as you have deployed something that participates in the global
routing mesh, this ploy should be embarrassing. as what you
have deployed attempts to take clever advantage of a kinky, and
not widely used (guess why!), feature of the global routing
system, you would be polite to take responsibility for what
happens.

What should we do?

at the core of the problem is the assumption that anycast will
find the closest server. thus, if the service is deployed in
many places in the topology and geography, each will only take
local load. it is critical to note that this relies on an
assumption of *very* topologically and geographically rich
deployment. it also gets bitten by the abundance of providers
with linear topologies with large geographic reach (but this
will be a short-term problem as tony hain from cisco plans to
abolish us as part of cisco's ipv6 marketing campaign:-).

Add complexity by announcing another less specific covering
prefix like F does?

although this further descends into the dangerous purgatory of
cleverness, you would probably be advised to do something such
as this. otherwise, even if k connected directly to all of
multi-homed t0's upstreams, by default, none of them would give
t0 your prefix because it is poisoned.

my naive view of your current deployment means that k can not
be seen from any multi-homed sites unless one or more of their
upstreams (recurse for tier-n) is even more clever and
implements "t0 is our customer and we ignore NO_EXPORT toward
customers," thus piling on yet another bit of cleverness, the
implications of which we can discover in the next level of
purgatory.

randy

7

ev'nin randy:

Of course the NCC takes resposibility for the K anycast deployment
including the way we announce BGP routes to K. We also clearly describe
and announce what we do. We cannot take responsibility for what others
do with that routing information; we cannot because we have no control
over and little way of knowing what they do. We are doing the best we
can; hence this conversation.

We are considering to add a covering prefix announced from global nodes
relatively quickly. This should solve the particular problem and we
cannot (yet) see any problems it would create. But this is more complex
than the current state and thus brings us further away from salvation ;-).
If there are reasons not to do this, please let us know.

We are also considering seriously to treat "local" nodes and global
nodes the same routing wise wherever possible. This will be done
one-by-one wiht the proper announcements and concurrent measurements.
My poersonal hope is that we can do this for all K nodes and thus remove
all BGP cleverness that originates from us. This does not mean that all
cleverness concerning K's routes would be removed though.

Daniel

8

Here's what we do on the PCH anycast network, to derive our "anycast cleverness" from the network topology rather than from BGP hacks. It seems to work. Other ways of doing this are presumably valid as well:

We've got four global nodes (nodes that have transit, rather than just peering), in Hong Kong, Palo Alto, Ashburn, and London. We get transit from the same transit providers in all four locations. Our transit providers hot potato to us, so as long as their peers hot potato to them, those who can't get to one of our local nodes should get to the topologically closest global node (topology, of course, does not always match geography).

We've then got a much larger number of local nodes, which look just like the global nodes except that they don't have any BGP transit. They're all at exchange points, they all peer openly, and we encourage our peers to peer with us at all common locations and to treat us like a normal peer. That means they don't announce us to their transit providers, but do generally announce us to their customers. Since this is the same thing that pretty much every network that's peering either globally or locally does, this doesn't require anything non-standard or hackish.

Our peers and their customers see us at whatever set of nodes they connect to. Those who don't peer with us, and aren't customers of any networks who do, see us at a more limited set of locations. This does mean we have to turn down offers of donated BGP transit from time to time, and we did have to turn off one peer who decided we were a good cause and was determined to give us transit whether we wanted it or not. There have been a few times when we've found our routes being leaked (fortunately by networks with considerably longer AS paths to most of the world than our transit providers) and have had to turn down sessions until the filters got fixed. These are the same issues we had at a real intra-connected global network I used to work for; it's not anything special about anycast.

The cases of suboptimal routing we see this leading to generally stem from networks that are unwilling to peer with us in their home markets but are eager to peer with us somewhere else. Their generally suggested way around this is that we should buy transit from them, and our response is that we aren't going to pay them to accept free services from us. Again, that's really a standard peering politics issue, and has nothing to do with anycast (and we're still generally closer to them than we would be with an arbitrary unicast location).

The remaining theoretical concern that might be solved by NO_EXPORT would be a situation where a network closer to one of our global nodes was getting transit from a poorly connected network close to one of our local nodes. However, simple economics works against that. Connectivity to or from poorly connected regions tends to be really expensive, so it's unlikely that anybody is going to be hauling traffic over those links when they don't have to.

My impression (and I think this is what Bill was saying yesterday as well) is that most of the weird routing problems that come up with anycast are a result of treating anycast as something different and special, which it doesn't need to be.

-Steve

9

assuming we are talking about the well known community no-export, then i
understand the problem.

a better solution would be to peer only the anycast node, such that transit
providers continue to propagate to the global internet (minus the peers seeing
the shorter path).

for wider distribution within the region, possibly using a transit provider for
the anycast and use communities supported by the upstream to restrict
announcement to its peers or upstreams

or am i naive too?

Steve

10

I think you underestimate the tendencies of ISPs all over the world to leak peering routes towards their transit providers.

Contrary to popular belief, leaks through peers in remote regions do not always result in huge AS_PATHs which are never selected by the rest of the network. For example, some of the most remote and poorly-connected ISPs that F is announced to from local nodes are transit customers of international, default-free carriers.

Joe

11

ok sure, but is this not just normal transit issues, these are not special
because they are a) anycast b) root-servers? if any networks peers leak they
should be reprimanded

Steve

12

Contrary to popular belief, leaks through peers in remote regions do
not always result in huge AS_PATHs which are never selected by the
rest of the network. For example, some of the most remote and poorly-
connected ISPs that F is announced to from local nodes are transit
customers of international, default-free carriers.

bingo!

thanks to, among other techno-colonialist games, the usaid leland
intiative, entire countries are direct non-bgp customers of a local
telco monopoly which is a customer of a european or american telco
monopoly.

randy

13

ok sure, but is this not just normal transit issues, these are
not special because they are a) anycast b) root-servers? if any
networks peers leak they should be reprimanded

rofl!

thanks, i needed a good laugh

randy

14

Here's what we do on the PCH anycast network

steve:

could you tell us more about the pch anycast network so we
can take a look at how its prefixes propagate?

randy

15

You're right -- these are normal issues that any multi-homed AS might see. The effectiveness of knuckle-rapping after the fact is not necessarily great, however, with respect to service uptime.

Anybody who cares about their service availability might think around the subject and take appropriate steps to mitigate or avoid leaks. Alternatively, they might well consider the cure worse than the disease, and live with the occasional leak instead.

I think there is sound, logical reasoning that can result in both conclusions, depending on the peculiarities of the service in general and the habits of local peers in particular (with a dash of personal preference and a sprinkling of past experience). It's the thinking part that is important.

Diversity in approach is good, especially in the delivery of a single, critical service.

Joe

16

ok sure, but is this not just normal transit issues, these are not special

    > because they are a) anycast b) root-servers?

Correct.

    > if any networks peers leak they
    > should be reprimanded

Well, I might phrase that in a different way, but yes, if they do it
enough to matter, the market will teach them the error of their ways.

                                -Bill

17

Hi,

this implies that a non-trivial part of the net can not see
anycast services for which some of the servers are marking
their announcements as NO_EXPORT.

Is it an idea to have anycasted instances using NO_EXPORT announce /25's
instead of /24's? That way you would still have global connectivity for
the /24 and still respect the NO_EXPORT.

Another possibility is for $LARGE_ISP to localpref the NO_EXPORTED down
to $LOW value (some of you will go 'doh' now).

Thanks,

18

Is it an idea to have anycasted instances using NO_EXPORT
announce /25's instead of /24's?

many many folk filter on /24, so the /25 would not be seen.

Another possibility is for $LARGE_ISP to localpref the
NO_EXPORTED down to $LOW value

and then how will the down-preffed prefix be seen within
$large_isp? local-pref is a very heavy hammer.

randy, at the clue edge i guess

19

Maybe I'm missing something, but the core issue is that the NO-EXPORT'ed anycast instance has a higher localpref inside the AS it's being advertised to, and as such supressing the non-NO_EXPORT'ed prefix. The "exportable" prefix gets suppressed at a point on the network such that the peering routers never see it, so it never makes it out of that AS.

Advertising the NO-EXPORT as a /25 (or two /25s) would serve the same purpose as tagging it NO-EXPORT, because as you say most peers filter on the /25. Incidentally it would obviate the need to assign it a higher localpref because it's a shorter prefix. However, unless I'm misunderstanding something, the /25 would not preempt the /24 advertisement, so the /24 would still make it out of the AS.

Just make sure you don't have anything numbered x.x.x.127 on the block...

Is it an idea to have anycasted instances using NO_EXPORT
announce /25's instead of /24's?

many many folk filter on /24, so the /25 would not be seen.

Isn't that the point? The existing /24 is tagged NO-EXPORT after all...

Another possibility is for $LARGE_ISP to localpref the
NO_EXPORTED down to $LOW value

and then how will the down-preffed prefix be seen within
$large_isp? local-pref is a very heavy hammer.

randy, at the clue edge i guess

-C

20

Maybe I'm missing something, but the core issue is that the NO-
EXPORT'ed anycast instance has a higher localpref inside the AS it's
being advertised to, and as such supressing the non-NO_EXPORT'ed
prefix. The "exportable" prefix gets suppressed at a point on the
network such that the peering routers never see it, so it never makes
it out of that AS.

it willl lose the best path war at the point it is down-preffed.
if that is done at the entrance, then the route is useless. if it
is done at some intermediate points, it's hell to create and
maintain a consistent net-wide config.

randy