Maybe you can explore the in kernel feature call RP filter or reverse path filter. In router gear it's called uRPF.
+100 to rp_filter
There are 2 modes: Loose or strict.
If your server is BGP multi-homed, then you must use loose. Loose is still very powerful and useful.
I think loose with any default will fail to do what you want. If you are running your router without a default, then loose would probably be okay.
Basically, RP is doing what a router does, but the opposite way. When a packet arrives on your server, it checks the routing table for destination next-hop and RP also check whether the frames arrived from the good source interface.
For strict mode, the router allows the incoming packet if the incoming interface would be the outgoing interface when sending a packet to the incoming packet's source IP.
If your routing is asymmetric or spoofed, then RP drops it. It's a nice feature, but it's doing a double route checkup so for sure, it's slightly slower. I'm not sure we can say that it's twice slower though.
I'm confident that it is at least some slower. However ...
I have a lowly AMD E-350 APU (lscpu says it's at 918 MHz) processing multiple hundred Mbps on GPON against a full DFZ feed with no noticeable delay. (I've never felt the need nor desire to instrument it.)
As such, I'm confident that any system that would be used in a greenfield deployment will be able to *easily* handle the traffic that most servers will see.
I assume your network is not asymmetric, so RP would help you for ingress traffic. For egress, then add blackholes routes to /dev/null interface or with the bogon scripts in python. I wouldn't use iptables for that as it's purely routing, but there are many ways to achieve the same goal.
"unreachable" routes (in Linux parlance) or "null" routes (in Cisco parlance) combined with Reverse Path Filtering (RPF) is a HUGE win in my book.
I've expanded this methodology to federate Fail2Ban between multiple systems. EBGP via bird to trade fail2ban specific tables between machines and ip rule to make sure the fail2ban table is processed. Works great in my opinion.
I recommend to explore the rp_filter as it might do what you're looking for.
As a side note, iptables is super slow when under attack and/or under heavy load. There are a lot of limitations, like the kernel can only forward ~1.4 Mpps per cpu/socket with iptables. It's too slow slow in my opinion and this was still true recently, but I can't confirm with the latest 5.x kernel. It could have been fix or improve.
That may be the case. However, that's Apples (iptables) to walnuts (RPF). They are both food (processing packets), but they are significantly different.