The Making of a Router

Hello Everyone,

We are looking to put together a 2u server with a few PCIe 3 x8
(recommendations appreciated). The router will take a voip transcoding
line card, and will act as an edge router for a telecom company.

For things like BGP (Quagga, Zebra, all that lovely stuff!!!), static
routes, and firewall capabilities we are thinking gentoo linux
stripped for sure however, what about the BSDs? FreeBSD or OpenBSD.
Any comments, feedback, does, and don'ts are much appreciated.

Kind Regards,


I am a believer of not having to re-invent the wheel...

Having said that.. have you looked at 'purpose built appliances' e.g.

If you are looking for a full router....
Consider such as these...

and there are a few others but the concept is the same

Personally, I am not a believer in making a single device be the do all / end all of everything..
While one can do everything on a big server .. however breaking things out e.g. voip trans-coding and routing make maintenance, availability, and ability to create redundancy much more practical.


Faisal Imtiaz
Snappy Internet & Telecom

If you're trying to do this cheaply, I'd recommend an appropriate sized Mikrotik router, and perhaps something running digium's transcoding hardware/Asterisk, or some Adtran hardware.

Don't put all this in one box.


I also wonder about re-inventing the wheel. The router part is easy, you could even do that with a windows box (that's a joke).

Obviously capital cost is part of it, but the man hours involved in doing what you're talking about, especially since you are talking about a telco.... whatever you come up with has to be pretty darn reliable...

Certainly would be interested in a little more information about the use case.


Depends on how skilled you are at maintaining Linux vs BSD, honestly.
Personally, I've accomplished something similar with great performance in
the past on Linux. I ran Debian 7 + latest compiled Quagga + latest
compiled Libreswan + Shorewall. If you're going to have a lot of different
people changing the rules, I would go with Shorewall. The syntax is
brain-dead simple, even though you're stuck with the network stack
limitations of Linux. A lot of my issues with doing this in Linux have to
do with distro's loading a bunch of net filter helpers by default, which
can be a major pain in the ass (I'm looking at you, SIP and SNMP modules).
I had to do a lot of tweaking to the conn track tables to make them large
enough to handle lots of traffic, but obviously YMMV.

Have you tried labbing BSD vs Linux to see which you like better? I'd
probably do that before throwing it in to production.

Look at the ubnt edgemax devices

Jared Mauch

Have to agree on the below. I've seen too many devices be so integrated they do no task well, and can't be rebooted to troubleshoot due to everyone using them.

Jared Mauch

if you want build by yourself I will suggest gentoo and/or freebsd with
bird ( for routing stuff (maybe with 10G nics).

Don't put all this in one box.


Not to mention the fact that this "router" will require support. The build before buy people are silly. Let the smart router guys do their thing and use their box accordingly. When it breaks call to inform them it broke and they will fix it. Diy projects are a nightmare to support.

If he can afford a 10G link... he should be buying real gear... I mean, look, I've got plenty of infrastructure horror stories, but lets not cobble together our own 10gbit solutions, please? At least get one of the new microtik CCR's with a 10gig sfp+? They're only a kilobuck... If you can't afford that I suggest you can't afford to be an ISP.


Totally agree that a routing box should be standalone for tons of reasons. Even separating network routing and call routing.

It used to be that BSD's network stack was much better than Linux's under load. I'm not sure if this is still the case - I've never been put in the situation where the Linux kernel was at its limits. FWIW

I've recently pushed a "large" BSD box to a load of over 300, for more then
an hour, while under test, some things slowed a little, but she kept on


You can build using commodity hardware and get pretty good results.

I've had really good luck with Supermicro whitebox hardware, and
Intel-based network cards. The "Hot Lava Systems" cards have a nice
selection for a decent price if you're looking for SFP and SFP+ cards that
use Intel chipsets.

There might be some benefits in going with something like FreeBSD, but I
find that Linux has a lot more eyeballs on it making it much easier to
develop for, troubleshoot, and support. There are a few options if you
want to go the Linux route.

Option 1: Roll your own OS. This takes quite a bit of effort, but if you
have the tallant to do it you can generally get exactly what you want.

Option 2: Use an established distribution.

Vyatta doesn't seem to be doing much with its FOSS release "Vyatta Core"
anymore, but the community has forked the GPL parts into "VyOS". I've been
watching them pretty closely and helping out where I can; I think the
project is going to win over a lot of people over the next few years.

The biggest point of failure I've experienced with Linux-based routers on
whitebox hardware has been HDD failure. Other than that, the 100+ units
I've had deployed over the past 3+ years have been pretty much flawless.

Thankfully, they currently run an in-memory OS, so a disk failure only
affects logging.

If you want to build your own OS, I'll shamelessly plug a side project of

RAMBOOT makes use of the Ubuntu Core rootfs, and a modified boot process
(added into initramfs tools, so kernel updates generate the right kernel
automatically). Essentially, I use a kernel ramdisk instead of an HDD for
the root filesystem and "/" is mounted on "/dev/ram1".

The bootflash can be removed while the system is running as it's only
mounted to save system configuration or update the OS.

I haven't polished it up much, but there is enough there to get going
pretty quickly.

You'll also want to pay attention to the settings you use for the kernel.
Linux is tuned as a desktop or server, not a router, so there are some
basics you should take care of (like disabling ICMP redirects, increasing
the ARP table size, etc).

I have some examples in:
or (more recent, but includes firewall

Also a note of caution. I would stick with a longterm release of Linux.
I've had good experience with 2.6.32, and 3.10. I'm eager to use some of
the post-3.10 features, though, so I'm anxious for the next longterm branch
to be locked in.

If running a proxy server of any kind, you'll want to adjust
TCP_TIMEWAIT_LEN in the header file and re-compile the kernel, else you'll
run into ephemeral port exhaustion before you touch the limits of the CPU.
I recommend 15 seconds (the default in Linux is 60).

Routing-engine -wise. I currently have a large XORP 1.6 deployment because
I have a need for multicast routing (PIM-SM), but XORP is very touchy and
takes quite a bit of operational experience to avoid problems. Quagga has
much more active development and eyeballs. BIRD is also very interesting.
I like the model of BIRD a lot (more of a traditional daemon than trying
to be a Cisco or Juniper clone). It doesn't seem to be as far along as
Quagga though.

One of the biggest advantages is the low cost of hardware allows you to
maintain spare systems, reducing the time to service restoration in the
event of failure. Dependability-wise, I feel that whitebox Linux systems
are pretty much at Cisco levels these days, especially if running in-memory.

For router with Freebsd+BIRD/Quagga, I suggest BSDRP.

Unless all the money is going into the 10 gig link.


I bought two Maxxwaves with the Core2 Duo processor and it's an Axiomtek NA-822 inside. It's a nifty platform, I'm probably going to put BIRD on it and make some BGP RRs one of these fine days. I'm too OCD to suffer a "server" in the routing/switching bays as the only thing with ports on the back.

I wouldn't imagine trying to route 10 gig with it though, or sharing any functions (like VOIP) on it.


Ahh.. so you are the one who bought those two from Ebay !
I was watching, but got to them rather late.

If you are the one who got them.. you got a great deal.

These have Mikrotik ROS license with them, you can do BGP/ OSPF etc. with them
If you want to reload them with other OS, all you got to do is pull out the FlashCard, and install anything else you want.

The Core2 Duo Model with Mikrotik ROS ver 5 will handle about 1G to 1.5G of traffic with much trouble.
With ROS ver 6 .. they will do about 25-30% more.

The i5 & i7 versions will handle 3-5Gig of traffic easily... and these do support 10G SFP+ Intel nic's with ROS 6.x


Faisal Imtiaz
Snappy Internet & Telecom
7266 SW 48 Street
Miami, FL 33155
Tel: 305 663 5518 x 232

Help-desk: (305)663-5518 Option 2 or Email:

Nope, that wasn't me. I got mine probably a year ago. I pulled the flash and tried Vyatta on it, but Vyatta turned out to be a buggy unpleasant experience. RouterOS had an IPv6 OSPF bug where it ignores some LSAs that made it a showstopper.


Have you tried labbing BSD vs Linux to see which you like better? I'd
probably do that before throwing it in to production.

Great advice Thomas! I will be creating a BSD virtual machine to get a
feel however, with linux I can think broad scale and forcast better.
With BSD, I am concerned with making choices that will shoot me in the

In essence, which BSD operating (Free vs OpenBSD) system is more
widely supported, and proven successful for this task. I am talking
about everything from hardware compatibility to software support for
pfsense and openbgpd from the community. Finally, performance
comparison to a present day Linux Kernel.

PS conntrack optimization for IPTables. Been there done that... I am
strong with Gentoo and am pretty sure can put together a rock solid
machine, just don't want to turn a blind eye though yeah?


Point taken! Transcoding tasks abstracted out :).