Anycast but for egress

Mailman List Mirror (Read Only)
1

(Unsure if this is the right forum to ask this question, but here goes:)

From what I understand, IP Anycast can be used to steer traffic into a server that’s close to the client.

I am curious if anyone here has/encountered a setup where they use anycast IP on their gateways… to have a predictable egress IP for their traffic, regardless of where they are located?

For example, a search engine crawler could in principle have the same IP advertised all over the world, but it looks like they don’t… I wonder why?

2

Because there’s no good/reliable way to get the replies back to the correct initiating host.

Owen

3

Unless you’re twisting knobs, egress traffic should already exit your network at the closest possible egress point to its origin. Is your intention to carry the traffic for longer than that?

4

(Unsure if this is the right forum to ask this question

Sure, why not… There isn’t anywhere more appropriate, really.

From what I understand, IP Anycast can be used to steer traffic into a server that's close to the client.

That’s the net effect, as it’s normally used. But anycast is really very simple, and has no concept of client/server… An IP address is assigned to multiple devices or processes, in locations which the routing topology views as diverse.

In practice, that means that services are bound to a common shared address (an “anycast service address”) as those services are deployed on servers in different locations. The service address is advertised into the BGP routing infrastructure. Clients send packets to the service address, and the BGP routing infrastructure routes each packet on the shortest path to its destination, without knowing that there are multiple destinations.

I am curious if anyone here has/encountered a setup where they use anycast IP on their gateways... to have a predictable egress IP for their traffic, regardless of where they are located?

For example, a search engine crawler could in principle have the same IP advertised all over the world, but it looks like they don't... I wonder why?

I think you’re going to need to construct a clearer and more precise explanation of what you’re imagining, because my reading of these two lines is that they’re saying different things; I don’t see the connection between them that you see. That said, a few reactions:

Anycast is often thought to _reduce_ predictability, since it offers multiple exclusive possible termination points for each packet, whereas unicast, multicast or broadcast would each have predictable outcomes by comparison: a specific node would receive the packet, a specific set of nodes would receive the packet, or all (in-scope broadcast domain) nodes would receive the packet.

If you’re asking whether it would make sense for border routers, which have access to full-table transit, to advertise that accessibility as an anycasted service, that’s what the special “default route” 0.0.0.0/0 is. Many people configure full-transit BGP routers to redistribute a 0.0.0.0/0 default route into their IGP, their internal routing protocol (albeit that may well be iBGP, nowadays) in order to accommodate routers which haven’t the resources to hold or use full routes.

A search engine crawler depends upon a unicast return path in order to establish a TCP session with the web sites it’s crawling, and see the return traffic from them. If a search engine crawler shared an anycast service address with other instances of itself in other locations, the outbound queries would head to web sites (which might be unicast or might be anycast, doesn’t matter), which would then try to reply. If the source address of the query is an anycast service address, the reply will go to the nearest instance of that shared address, rather than to the specific instance which originated the query.

It’s for this reason that one normally assigns unique unicast addresses to network-facing interfaces which will originate packets, and anycast addresses to internal loopback interfaces, to which services are bound… The server can receive packets addressed to the anycast shared address, but will originate packets using its unique address.

Here’s a tutorial from twenty years ago (when this was all less than fifteen years old!) that explains in some detail… Things haven’t really changed since then:

https://www.pch.net/resources/Tutorials/anycast/Anycast-v10.pdf

                                -Bill

5

Matt Harris​

Infrastructure Lead

816‑256‑5446

Direct

Looking for help?

Helpdesk

Email Support

We build customized end‑to‑end technology solutions powered by NetFire Cloud.

6

Without any sarcasm: to make it harder to block.
If, say, Google, always crawled your site from 8.8.1.2 (random made-up example) then you would see a not-insignificant number of hosts and networks null-routing that IP. I have no idea why someone would do so, but I’ve seen it done many times. Mostly by people who don’t understand how un-special they are on the internet. Also it would trigger IDS/IPS systems all over the place, having gobs and gobs of connections coming from a single IP.

That’s setting aside the technical issues involved; routing is often asymmetric, i.e. the return packet takes a different path than the inbound packet. So it would, as Owen implied, be nearly impossible to ensure the reply packets got back to the correct TCP stack. As an example, I’m multi-homed and use path-prepending, so if a packet claiming to be from 8.8.8.8 arrived on one of my commercial links, I would send the reply out the cheapest link, which in my case is a flat-rate R&E network (that has a path to Google), thus ensuring the reply does not get to the originating anycast node.

When my clients make connections outbound to anycast addresses, the destination is more-or-less stable, and the replies come back to the client’s unique IP, so anycast works in that direction. The guarantees are not present in the reverse direction.

The logical extremity of this is that it would be nearly impossible for two anycast addresses to establish a TCP connection to each other. (In general. There will be lots of local cases where it does happen to work, by coincidence.)

You’ll find that even anycast nodes do not make connections outbound using their anycast address, pretty much for these reasons.

-Adam

7

We use our IGP (IS-IS) for our Anycast services. We find it to be very basic, and as such, very predictable.

Of course, if you don't run your own AS for the services you provide, or operate your own backbone, this is probably not a viable option.

Mark.

8

If I were in your shoes, I’d pick a VPS provider that assigns external, globally routable IPs to their customers. Linode, Vultr, Digital Ocean, etc.

You may be able to do something on AWS with Elastic IPs but I don’t know enough about Amazon’s infrastructure to give you a qualified answer.

9

Since you mentioned AWS, have you tried AWS Global Accelerator? You get a pair of globally anycasted static IPs.
https://aws.amazon.com/global-accelerator/

Another alternative is to request a contiguous IP range of EIPs (/28 or /24 etc) that you can use for your EC2 instances or VPC resources.

Andras

10

No, there is no such thing as “close”. You could have a direct peering with some ISP and have them still deliver the responses on the other side of earth. You do not control the routing of other networks and can not be sure what they will do.

For larger networks you may also have multiple peering points. Say you have a peering with them in city A and city B. How do you know which of their IP ranges are closer to A or B? You don’t. And the same goes for them, they have no idea if you prefer A or B. Therefore you could select A and they may reply to B. They may even load balance between A and B if you are really unlucky.

Routing is asymmetric. That means you have absolutely no idea where the replies end up going. Often it will not be what you think is “close”.

I do not run anycast, but I understand that the usual way of dealing with these issues is to do as little as possible with anycast before redirecting to a unicast address. For example you could have just your DNS on anycast and each site would reply with unique unicast addresses. Since DNS is just a single pair of UDP request/response, with the first packet originating from a unicast client, this works well.

Regards,

Baldur

11

Here is what I think would happen if you were to try this setup. Let’s assume you deployed in eu-west-2 (London) and eu-central-1 (Frankfurt). You would find that you could successfully connect to a number of networks but also that some of them would work from the “wrong” site. Eg. you would have clients in London that require you to use the Frankfurt instance to connect to. You would also find a lot of networks that you could not connect to from either site. And you would have some instability where some networks at random switch between these states. For example you could have a client that one day works from London, the next day it is Frankfurt and then someday it won’t be working at all.

As you add more sites, you also add more ways for the traffic to end up in the wrong places. You could have something that works with just the two sites above but then when you add eu-west-1 (Ireland) you could suddenly not connect to them from any of the three sites.

There IS a possible solution which is to tunnel the traffic back to the correct site. This still requires you to use unique IP addresses for each site, but all addresses could be in the same subnet. You would have IP a.b.c.1 to be London, a.b.c.2 Frankfurt, a.b.c.3 Ireland. Then if London receives traffic to a.b.c.2 you would have a tunnel to send the traffic to Frankfurt.

Regards,

Baldur

12

In our small experience, not at all.

We are Anycast'ing DNS (authoritative and recursive), NTP and TACACS+. All works well, across 11 or so countries.

Mark.

13

Hi all, great replies. :slight_smile: Let me clarify my initial question, and then respond one by one:

My intention is to run a web-crawling service on a public cloud. This service is geographically distributed, and therefore will run in multiple regions around the world inside AWS… this means there will be multiple AWS VPCs, each with their own NAT gateway, and traffic destined to websites that we crawl will appear to come from this NAT gateway’s IP address.

The reason I want a predictable IP is to communicate this IP to website owners so they can allow access from these IPs into their networks. I chose IP as an example; it can also be a subnet, but what I don’t want to provide is a list of 100 different IP addresses without any predictability.

I understand that this is not perfect, and would frankly not be my preferred approach to solve the problem… but we’ve had requests of this nature from websites to create an allowlist of a limited number of predictable IPs so it doesn’t trip their IDSs/other systems they might have… so we’re trying to see how well it would work in practice. For the moment, let’s set aside the issue as to whether AWS will even let me advertise the same IP on all my VPC NAT gateways, and just look at whether it’s technically feasible. My gut feeling is that this wouldn’t work well in practice, but I wanted to ask the experts here…

Also, pointers on what the best practices for solving this issue are most welcome, so I can reference those who ask for IP addresses to this discussion and follow recommendations here.

Onto the responses:

@owen@delong.com and @woody@pch.net athompson@merlin.mb.ca

Because there’s no good/reliable way to get the replies back to the correct initiating host.

When my clients make connections outbound to anycast addresses, the destination is more-or-less stable, and the replies come back to the client’s unique IP, so anycast works in that direction. The guarantees are not present in the reverse direction.

Yes, this makes sense as the destination can be anywhere around the world, and that routing is asymmetric as others mentioned. However, if the destination service is “close” (in the routing metric sense) to the initiating host, anycast return IP ought to work well, right? I understand this is a very important caveat and impractical to implement correctly in the real world.

We use our IGP (IS-IS) for our Anycast services. We find it to be very
basic, and as such, very predictable.

This is interesting… I wonder whether Anycast will still have some failure modes and break TCP connections if routing (configuration) were to change? I checked the PDF linked by Bill Woodcock… while the methodology is the same from 20y ago, would the data still be the same (order of magnitude)? :slight_smile:

https://www.pch.net/resources/Tutorials/anycast/Anycast-v10.pdf (p38)

“Limited operational data shows underlying instability to be on
the order of one flow per ten thousand per hour of duration.”

@daniel@corbe.net, @matt@netfire.net,

Unless you’re twisting knobs, egress traffic should already exit your network at the closest possible egress point to its origin. Is your intention to carry the traffic for longer than that?

No, but I hope my intention is more clear in this email. It’s to have a predictable egress IP to simplify firewall rules.

thanks all!!

14

On AWS once we purchase EIPs, they are allocated to our account and so we can assign them to VPC NAT gateways. That’s our current plan.

15

AWS Global Accelerator gives anycast IPs that’s good for ingress, but my original question was about having predictable egress IPs.

It looks like having a few EIPs/a contiguous network block is the way to go.

Thanks!

16

we, verio, did anycast tcp streaming (hour long) of the tony awards in
about '96. solid.

randy

17

My intention is to run a web-crawling service on a public cloud. This service
is geographically distributed, and therefore will run in multiple regions
around the world inside AWS... this means there will be multiple AWS VPCs,
each with their own NAT gateway, and traffic destined to websites
that we crawl will appear to come from this NAT gateway's IP address.

Hello,

AWS does not provide the ability to attach anycasted IP addresses to a
NAT gateway, regardless of whether it would work, so that's the end of
your quest.

The reason I want a predictable IP is to communicate this IP to website
owners so they can allow access from these IPs into their networks.
I chose IP as an example; it can also be a subnet, but what I don't want to
provide is a list of 100 different IP addresses without any predictability.

If you bring your own IP addresses, you can attach a separate /24s of
them to your VPCs in each region, providing you with a single
predictable range of source addresses. You will find it difficult and
expensive to acquire that many IP addresses from the regional
registries for the purpose you describe.

Silly question but: for a web crawler, why do you care whether it has
the limited geographically distribution that a cloud service provides?
It's a parallel batch task. It doesn't exactly matter whether you have
minimum latency.

Regards,
Bill Herrin

18

Yes. Predictable and unchanging (but each unique per location) static IP addresses is what you’re looking for.

It would be a huge convenience to others if you could specify a single contiguous CIDR block for others to “permit” in their access control lists, but alas that would be very difficult as well… Since BGP announcements generally need to be aggregated up to at least a /24 or a /48 (though people are less strict on the v6 side), each group of hosts numbered from the same block of that size would need to have internally contiguous convex routing, meaning that it would have to be interconnected by its own network (albeit that could be tunnels) and accept inbound traffic at any point on the surface of that network, backhauling it to the appropriate location. So if you wanted to be able to identify a single CIDR block with eight locations in it, you’d either need to specify a /24 that was 97% wasted, and was fully internally interconnected (i.e. no efficiencies in localizing traffic), or you’d need to advertise eight /24s, which would aggregate up to a single /21, which was 99.6% wasted.

So, you can see why the combination of scarce IPv4 addresses, scarce BGP routing slots, and content routing tricks often don’t play well together.

                                -Bill

19

I was about to say something about us having equal success over 105 or so countries, when I came to the realization that inviting quantitative comparisons of manhood with Mark is the very definition of folly. :slight_smile:

Anyway, yeah, the folks who were scared of anycast in the 1990s were running from shadows, not basing it on experience or data. In the real world, the number of stateful flows affected by route changes is dwarfed by those disrupted by other causes, and is immeasurably small. And when they do crop up on the radar, it’s almost always someone’s equal-cost-multi-path gone wrong, rather than an actual shift. So, not an issue at all in the real world, just in the imaginations of folks who thought TCP was a complex thing reserved for the specific use-cases that they’d already conceived of in the 1980s. Took a while to get beyond their protestations, but here we are in the 21st century. Planck's principle holds. Science progresses one funeral at a time.

                                -Bill

20

I was about to say something about us having equal success over 105 or so countries, when I came to the realization that inviting quantitative comparisons of manhood with Mark is the very definition of folly. :slight_smile:

Well, we are nowhere close to the 105 countries PCH boasts. That's a whole other level of scale :-).

Impressed!

Anyway, yeah, the folks who were scared of anycast in the 1990s were running from shadows, not basing it on experience or data. In the real world, the number of stateful flows affected by route changes is dwarfed by those disrupted by other causes, and is immeasurably small. And when they do crop up on the radar, it’s almost always someone’s equal-cost-multi-path gone wrong, rather than an actual shift. So, not an issue at all in the real world, just in the imaginations of folks who thought TCP was a complex thing reserved for the specific use-cases that they’d already conceived of in the 1980s. Took a while to get beyond their protestations, but here we are in the 21st century. Planck's principle holds. Science progresses one funeral at a time.

100%.

Mark.