advice on BGP + CARP setup on FreeBSD

Mailman List Mirror (Read Only)
1

Hi fellow operators,

We are slowly and carefully joining the fray of autonomous systems and
started announcing our own netblock, a first test that started last
week. So far, things are going well, but before going further along this
setup, I would be curious to hear experience from other operators about
the plan we are thinking of deploying.

Our requirements:

* free software, as much as possible
* inexpensive
* using existing operating system expertise (FreeBSD or Debian)

So far, we have:

* our own ASN
* a /21 assigned by ARIN
* two uplinks deployed (Netelligent and Cogent)
* Netelligent announces 3 /24 netblocks for us
* we announce the last /24 through a BGP link with cogent

We have some horrible diagrams describing the setup here:

https://wiki.koumbit.net/RoutingService/RoadMap

As you can see, the uplinks are connected directly into a switch, in two
separate VLANs. The reason for this is we want to be able to hotswap the
routers in case of a hardware failure, but we have understood from
Cogent's documentation that this is not a good practice because the
links appears up even if the router goes down. What is your opinion on
this?

Also, we currently testing OpenBGPd for the announcements, and we are
very pleased with it. The syntax is clear and it just works, with
minimal memory usage:

https://wiki.koumbit.net/OpenBgpdMaintenance#Checking_memory_usage

However, this seems to be a fairly exotic platform, most people running
BGP with Cisco, Juniper or, in some cases Quagga or Bird for Linux
machines. Are there recmomendations on using OpenBGP in production? Good
/ bad experiences? How many people are running Linux routers vs
dedicated Cisco/Juniper/etc routers?

Finally, we are likely to complete this setup with a CARP (the free
equivalent of VRRP) on the inside of the network. FreeBSD can apparently
group interfaces and communicate with OpenBGPd - did anyone deploy such
a thing here? What are your experiences or advice?

Thanks for any advice,

A.

2

Our requirements:

* free software, as much as possible
* inexpensive
* using existing operating system expertise (FreeBSD or Debian)

You need to make a decision on how to spend your money: on commodity router
hardware where you can easily get support if there's a problem, or on more
FOSS operating systems with a routing layer (e.g. openbgpd / bird / quagga
/ etc).

As you can see, the uplinks are connected directly into a switch, in two
separate VLANs. The reason for this is we want to be able to hotswap the
routers in case of a hardware failure, but we have understood from
Cogent's documentation that this is not a good practice because the
links appears up even if the router goes down. What is your opinion on
this?

Cogent is correct and their reasoning is correct.

However, this seems to be a fairly exotic platform, most people running
BGP with Cisco, Juniper or, in some cases Quagga or Bird for Linux
machines. Are there recmomendations on using OpenBGP in production? Good
/ bad experiences? How many people are running Linux routers vs
dedicated Cisco/Juniper/etc routers?

I run lots of different routing systems for a lot of different situations
(am currently using quagga, bird, openbgpd, cisco ios, cisco xr, junos and
brocade ironware for bgp stuff). For small setups, it really doesn't make
a whole lot of difference so long as you run with a configuration which
supports both ibgp and an interior routing protocol like ospf or isis.
It's not going to make a whole lot of difference to you whether you use
quagga, openbgpd or bird because you're not going to stress the RIB engine
with only two providers.

Usually, it's better to run COTS routers (e.g. juniper / cisco / etc). If
you don't want to do this, you will probably end up spending roughly the
same in terms of manpower, so don't be tempted to think that you're going
to save a whole lot with a free unix based system.

If you want a FOSS system and you have no preconceptions about routing, I'd
suggest using linux/freebsd + bird because bird is a truly wonderful RIB
engine. If you are already familiar with cisco syntax, linux/freebsd +
quagga will do the job just fine. If you have decided that you like
openbgpd and want all the features of openbgpd (including md5 passwords),
then you need openbsd + opengpd + openospfd, all of which I have found to
be frankly a pain to operate and maintain, although I think openbsd has
improved since the last time I used it in anger which was 3-4 years ago.

Finally, we are likely to complete this setup with a CARP (the free
equivalent of VRRP) on the inside of the network. FreeBSD can apparently
group interfaces and communicate with OpenBGPd - did anyone deploy such
a thing here? What are your experiences or advice?

linux carp is hopeless and I would strongly advise not to use linux if you
want to implement vrrp / carp. Incidentally if anyone feels this is
unfair, they need to take a long hard look at the linux vmac implementation
and if they don't run screaming, I'll take my hat off.

The FreeBSD CARP implementation (which is borrow directly from openbsd)
usually works fine, but i've seen more than my fair share of kernel panics
on relatively recent freebsd relating to carp.

Srsly, get a cisco / juniper router. Unless you're doing some incredibly
specialised large scale router implementation and you really know what
you're doing and why you're doing it, using a FOSS system will end up being
more expensive in terms of your time.

Nick

3
4

and because VRRP is patent-encumbered

CARP is probably patent encumbered too. I don't understand why people
claim that using CARP as a default gateway isn't patent encumbered because
it uses pretty much exactly the same process that the VRRP patent covers.
To be honest, I wouldn't get too upset about it as the patent expires early
next year, so this is not important.

Also bear in mind that if you decide to migrate to VRRP later on and for
some reason use the same VRRP group as CARP vhid, you will trash your
network connectivity. The OpenBSD people specifically designed CARP so
that it would break VRRP on the same network if they use the same
parameters, but sadly they refuse to document this. It's documented
elsewhere on the web though, so bear it in mind if you ever implement CARP
and then move to VRRP.

I am worried, however, of rumours of kernel panics associated with CARP,
but I am confident that the very responsive FreeBSD community will be
able to help with that.

I hope so. There have been some long-standing problems, but hopefully the
work done on fixing carp in freebsd 10 will finally see the end of these
issues.

Nick