I’m looking for solutions do deploy some type of selective high availability and load balance based on the glue between Layer 2 and Layer 3 (ARP or ND).
And I’m coming here to ask help to avoid reinventing the wheel.
I know VRRP / Heartbeat, and their downside is the Active/Passive characteristic.
→ But this project demands something that allows-me to have Active/Active deployments.
I know GLBP, and it almost fits on the needed requirements.
→ Except by his load-balancing methods that do not allow-me define priority and affinity between server nodes and clients.
The basic ideia is something like Cisco GLBP with steroids:
-
Multiple server nodes of same service running on a common bus and answering the “L2 anycast requests” of the clients that are on the same bus and same subnet.
-
Some type of signaling between the multiple nodes making known the status of the other nodes, their load. Maybe complementary information like “which node is serving which client?”
-
Resource Pools and Client Pools, and the crossing between then based on priorities and affinities (Here is the Gotcha!).
-
I want to be able to say “Node X will priorly serve clients A, E, G, and T. Node Y will serve priorly clients B, C D, F. And node Z will server everyone else.”
Answering suggestions in advance:
(I discussed that with some friends and based on those talks I will try to predict some suggestions that we already considered.)
- No, unfortunately tradicional L3 anycast will not fit on the requirements. Servers and clients to be at the same bus, on the same subnet. No L3 hops between then.
- No, the use of some type of connection broker in L2 does not satisfy one of the requirements. Beyond the load balance, that this approach will address, the high availability in case on L2 segregation is also needed.
My v0 draft of idea was using GLBP, and L2 Firewall rules dynamically adjusted, based on the Master-Status, to allow and block L2 communications between each of those nodes and lists of client pools.
(Actually, I’m coming back to this idea again… Since I still don’t have any other better idea until now.)
I friend Suggested that EVPN could help-me, but I must confess that is a hard topic to me.
Unless it can be used depending exclusively on software (no special hardware required), it won’t fit.
Bigip with each host having two nics on public and private via inter switch shared vlan.
Should not cause issue so long as you know service comes via bigip to debug usage of kit via private ip side
Linux has EVPN support. Or you could use network switches with EVPN support.
EVPN will let each server or direct connected switch be a hidden layer 3 gateway. The application will not know about it and it will appear to be layer 2 but routable like layer 3. Which means you can do anycast at layer 3. Physically the packets are moved in tunnels so you can use any layer 2 or layer 3 hardware at your choice.
Regards,
Baldur
Hi Douglas,
Anycast is where you send to one network address and the "nearest"
single server with that address receives the packet.
By definition, every piece of equipment in an L2 broadcast domain is
exactly one hop from every other -- no equipment is "nearer." So
conceptually, there is no anycast.
However, L2 domains aren't built with hubs any more; they're built
with switches. There actually are variable distances between
equipment, they're just not expressed in the protocols. So, in theory
you could build an SDN controller for your switches which sets up
different FIB entries in each switch to select which port receives the
traffic for the designated "anycast" mac address. But you may face
limitations where the hardware can't reasonably be programmed to give
each port its own FIB allowing fine-grained control of which client
reaches which server.
Realistically... that approach would tend to be both expensive to
build and very brittle. There's almost certainly a better way to
accomplish your goal than trying to invent L2 anycast.
If you're load balancing IP traffic, another approach might be a
custom ARP controller which responds to ARP requests with different
MAC addresses depending on the request source. There's no guaranteed
timeout for ARP bindings but if you shared around a pool of MAC
addresses guaranteeing that every MAC address in the pool gets
assigned to a currently-working server it could work. You just have to
keep in mind that gratuitous arp absolutely would not work in this
sort of scenario so you have to have a plan for switching loads
between servers without it.
I don't think anybody has built that sort of arp controller (at least
I haven't heard of one) so you'd have to invent it yourself.
From what I understand of EVPN, it's about creating something
equivalent to VLANs across a distributed virtual server
infrastructure. Basically like what Amazon does under the hood for its
virtual private cloud. Since you're trying to get the machines to
appear on the same subnet, not separate them to different subnets, I
don't think it's what you're looking for.
Regards,
Bill Herrin
tor. 1. jul. 2021 21.06 skrev William Herrin <bill@herrin.us>:
From what I understand of EVPN, it’s about creating something
equivalent to VLANs across a distributed virtual server
infrastructure. Basically like what Amazon does under the hood for its
virtual private cloud. Since you’re trying to get the machines to
appear on the same subnet, not separate them to different subnets, I
don’t think it’s what you’re looking for.
EVPN creates a virtual layer 2 domain, aka a vlan, that can span the internet or be used on a plain old layer 2 switch. It uses vxlan or mpls tunnels and BGP to coordinate. EVPN has support for multiple active/active exits, something almost like lacp. There can be load balancing using layer 3 headers as key, which might be exactly what OP is looking for.
EVPN elimates layer 2 flooding by keeping a database of MAC addresses in BGP. Otherwise it behaves exactly like a vlan with extra features.
Douglas Fischer wrote:
I'm looking for solutions do deploy some type of selective high
availability and load balance based on the glue between Layer 2 and Layer 3
(ARP or ND).
If you are looking for L2 anycast, it was purposelessly
invented as a functionality of ND, though it does not
satisfy your requirements. See rfcs 2461 and 4861.
Glad to know it is totally ignored by most, if not all.
Masataka Ohta
Hello Masataka!
Yes… It probably solves my issues on a v6 only world.
But unfortunately, in this scenario there is IPv4 also, and to make me cry alone in the bathroom there are some IPv4 only…
So, I will need to provide some solution that covers IPv4 also.
Hello William!
An ARP Controller to compose a L2 Cluster solution seems a good Idea to a begging…
(I would include ND)
I will try to think a bit on that…
Any suggestions are welcome.
Maybe a spine and leaf architecture could work for you.
You could install 1 server per leaf or more. I believe this could achieve high-availability and load-balancing at layer 2.
There is a kind of layer 3 overlay, but for the hosts this is transparent and it feels like a real pure layer 2.
I would go that route as it is also a very common setup these days. It scales well horizontally and no active/passive. It’s just active/active/active/active…./active.
Jean
Douglas Fischer wrote:
Yes... It probably solves my issues on a v6 only world.
No, not at all.
I'm afraid you don't understand what your issues are.
At least, neither L2 or L3 anycast has anything to do with
high reliability because reachability to a server and
availability of the server are different things.
Masataka Ohta
As others have pointed out, what you’re describing isn’t anycast, nor anything directly to do with high availability.
There are multiple well-understood frameworks which can be used to do what you’re describing.
This is strictly a layer-7 issue, nothing to do with layer-2 nor anycast. There’s no need to get down into the networking weeds to accomplish what you appear to be trying to accomplish.
Just do it all at layer-7.