I have been asked to investigate moving an entire network to multi-hop on all the eBGP sessions. Basically all upstreams, downstreams and peers will eBGP with a route reflector located in the core. This RR will be some kind of quagga or similar box. The dev guys want to be able to poke at the BGP feeds directly and do *magic* that standard router aren't capable of.
My gut feel is that this is a bad idea. Besides anything else it makes sane link state detection very challenging - especially where we have multiple sessions with a peer.
Is their any BCP or operational experience that agrees or disagrees with my gut.
The dev guys want to be able to poke at the BGP feeds directly and do
*magic* that standard router aren't capable of.
This should scare you in a significant manner.
The last company I worked for moved to eBGP Multi-Hop where there were
two connections to the same provider (same AS). This allowed them to
utilize both links in both directions vs only one link in one direction
and have failover.
As you have mentioned link state detection gets a bit crazy with this.
If you have a MetroE connection (for example) with multiple segments,
this could be problematic. If your side of the link goes down, then you
stop sending traffic to the provider, but the provider still tries to
send traffic to you. If a segment in the middle goes down, then neither
side stop sending traffic.
Due to the fact that the BGP session is still up, and the interface on
your router is still up, BGP sees the link as a valid path.
However there is a fix for this. If your provider supports it that is.
Ethernet OAM (Ethernet Operations, Administration, and Management) will
allow you to monitor the connection on Layer 2 end to end and not switch
to switch. If any part of the link breaks, OAM brings your and the other
side of the link down, telling BGP that the link is no longer usable,
therefore avoiding the issues above.
If you are using a POS, MPLS, or other similar technology, then the
issues talked about above are either less of an issue, or not an issue
at all. The biggest problem with multi segment Ethernet links is that
you need OAM to reliably run eBGP Multihop and OAM isn't supported by a
lot of providers (mainly because it requires a newer software version).
Hope this helps.
I have not tried yet, but you could have a look at . Perhaps it
could be extended to match your needs.
Do you run an IGP?
Of course, this sort of thing is usually great fun and seems like a Very
Good Idea At The Time. You get your cool configuration in place with lots
of local hax and the network hums along. Then the developer who wrote the
hax leaves because of something or another. And the person who configured
the box leaves due to management politics, and then the Windows IT support
person takes over, along with the smart person on the front-line tech
support desk. Then you hit your first major security bug with your local
route reflector and the vendor patch causes your configuration to break
Then hilarity ensues.
I've seen extreme local messing-around-with-systems at some companies.
Hilarity ensued. But there is a silver lining to all this: all these
companies learned from their stupidity and never did things like that
again. At least, the ones which didn't go bust.
As regards collapsing all your bgp requirements into a single BGP box,
well, good luck with that. Can I recommend you call this box
"spof.apolix.co.za"? It seems quite appropriate.
[You could have another single box called "ospf.apolix.co.za", which dealt
with all your ospf requirements... just a thought.]
Incidentally, I presume your devs have found some way of patching quagga in
memory so that every time they write a new local hack or need to fix the
previous one, they don't have to bring down the entire network in order to
bring it into production? That would bring the entire experiment to a new
level of coolness.
Anyway, I wish you well with this experiment in the future of your
Multihop eBGP is a debugging tool that a developer left in the production code.
Even in lab environments I would say that eBGP multihop is a bad idea. Production networks that try to make money should not embark on such a mission. Trust your gut
If some box wants to take a feed of routes, it should just iBGP peer with your RRs.