Zebra Router???

Mailman List Mirror (Read Only)
1

Hi All,
I'm trying to find out what peoples experiences regarding the throughput
of Zebra on a FreeBSD Box.

My configuration is as follows:

Intel Pentium III 1.40G
1 Gig RAM
2x <Intel Pro 10/100B/100+ Ethernet> (one internal, one external) on
board.

This box is running as a simple static router, i.e. one subnet on the
inside, Internet feed on the other side. No BGP, no RIP, no OSPF. Pretty
simple, eh?

So the goal is to know the bandwidth limitation of this router. Any ideas?
I've heard numbers of 35Meg, 40 Meg, etc, however, I have not recieved a
good reason backing it up. Can anyone offer some input on this?

Regards,
Tom Daly

2

This box is running as a simple static router, i.e. one subnet on the
inside, Internet feed on the other side. No BGP, no RIP, no OSPF. Pretty
simple, eh?

Let me get this straight. No routing protocols? Perhaps Zebra is not what
you need. :slight_smile:

sysctl -w net.inet.ip.forwarding=1 might be more your speed.

So the goal is to know the bandwidth limitation of this router. Any
ideas? I've heard numbers of 35Meg, 40 Meg, etc, however, I have not
recieved a good reason backing it up. Can anyone offer some input on
this?

As much as I hate to say this, stock FreeBSD makes a terrible high
performance router. The route-cache is horribly out of date with modern
techniques, and there just aren't that many wackjobs out there trying to
shove a hungred megs through a unix box to fully debug it (with the
exception of a certain notoriously cheap people who will probably respond
to this email talking about their success with FORE ATM OC3 cards :P).

Then again, as long as it's your network and not mind, who am I to stop
you.

3

Ok, I'll bite. We're routinely shoving 500mbps through our FreeBSD system running zebra, and we've never hit the 50% cpu mark. 3 GigE BGP peers passing me a full table and one GigE LAN interface, a few VLANs, lengthy IPFW rules, and tons of "count" rules so I can MRTG each IP passing through it. In some off network synthetic testing, I easily maxed out our GigE LAN interface before the router dropped a packet. All this on a $1800 Dell server with a $150 Intel PCI-X card slapped in there. This system's been up for 6 months now. Zero crashes, zero hung interfaces, zero problems.

I'm not saying a FreeBSD+Zebra system is going to do everything that your high end router will, but I haven't run into anything that I couldn't find some way of doing with the tools that I had. IPFW, Dummynet, tcpdump and other tools that come "stock" have saved me quite a bit of effort over some other much more costly solutions that couldn't do some things that we depend on now.

I know quite a few would consider this a bit of ghetto networking, but I've even worked out a pretty reliable hot-standby system for all our web servers by running zebra/bgpd on each of them and having them announce /32's for the IP's that the web server is listening on to a route reflector. Have another box setup as a standby system announcing the same /32 to our router with higher metric, and the failover is instant. No extra hardware, no fancy load balancers, and the web servers don't even have to know anything happened.

Zebra/FreeBSD aren't the best things out there, but when you have essentially no budget, there are a lot of us out there who've figured out how to make our networks operate pretty well.

I really don't mean this as a flame, Richard... Just that I think a lot of people out there have discounted this without trying it or researching what others have gotten to work. Not all of us have the luxury of working somewhere where not being "cheap" is an option. :slight_smile:

-- Kevin

4

Empirically, the box will do over 100mbps without breaking
  any semblance of a sweat.

  -a

I'm trying to find out what peoples experiences regarding the throughput
of Zebra on a FreeBSD Box.

My configuration is as follows:

Intel Pentium III 1.40G
1 Gig RAM
2x <Intel Pro 10/100B/100+ Ethernet> (one internal, one external) on
board.

...

So the goal is to know the bandwidth limitation of this router. Any ideas?
I've heard numbers of 35Meg, 40 Meg, etc, however, I have not recieved a
good reason backing it up. Can anyone offer some input on this?

...

5

Hi All,
I'm trying to find out what peoples experiences regarding the throughput
of Zebra on a FreeBSD Box.

     ^^^^^

This box is running as a simple static router, i.e. one subnet on the
inside, Internet feed on the other side. No BGP, no RIP, no OSPF. Pretty
simple, eh?

If you don't use ANY routing protocol, then your Zebra soft is not going
to do anything. (Besides enabling you to telnet to it and do some basic
setup).

You should skip Zebra, and use plain kernel routing.

Kind Regards,
Frank Louwers

6

I had 1 GigE, 2 FastE interfaces, no full feeds, about 16k routes,
pushing 100-120Mbps, and I always hit 100% cpu, causing Zebra's BGP
process to die repeatedly. And, whenever I removed a route from the
Zebra daemon, it didn't get removed from the kernel routing table,
and a simple 'route delete x.x.x.x' did not work, so the route was
effectively STUCK until the next reboot. (No, killing and restarting
the Zebra processes did not work either, nor did a 'route flush').

I've tried different CPUs, motherboards, hard drives, network cards,
kernels, sysctl variables, memory modules, no, some, and all (about 80)
IPFW rules, and still, same results. 100% CPU while the box is sucking
ass at 40-50kpps. Bottom line, from my personal experiences, FreeBSD
sucks as a router, and Zebra sucks as a routing process.

Yeah yeah yeah, Juniper uses FreeBSD for their code, but a) it's heavily
modified, modifications you won't find on the net for yourself, b) they
use ASICs to speed up certain operations, and there is no PCI card with
a PacketGoFaster(tm) chipset you can buy to get better performance out
of your desktop gaming Samba-serving router. I hear Olive has some good
performance, but guess what, it's not public domain and your chances of
finding it are very slim, so you're SOL for now.

But hey, your mileage may vary.

7

From my personal experience, the "Olive" is just plain old JunOS.

When you install it on a PC, it does not include the hardware related
elements, I found this out when the M10 crashed and I installed from scratch
outside the M10 hardware (on a new disk on a PC). For the record, all you
have to do is reinstall the package and the hardware will show up.

I ran several Ethernet ports and it worked fine. I am not sure about PCI GE
ports and if they are supported. Bottom line, if you have JunOS you can make
a stand alone pc work as a router (did this with 5.2R1.4-domestic).

This was a great solution for a lab router doing protocols, not sure about
performance.

All the best,

JK

8

FreeBSD is pretty stable. One such boxen I have has been up for 460 days
doing only routing and some Apache stuff on a GigE link. For the most part
I'd think its
safe to say the comination your working with is accepable for non-critical
use.
Of course if your looking for a failsafe production item your better off
spending the
extra cash on a Cisco/Extreme/Juniper/etc item, but in light of funds...
I'll also point out that CheckPoint used a FreeBSD kernel for some time. I
believe they are now using a Linux variant, but don't quote me on that.
The hardest part is finding drivers and support for the various NIC your
going to
use.

just my 2�

-Joe

9

Hi!

Hi All,
I'm trying to find out what peoples experiences regarding the throughput
of Zebra on a FreeBSD Box.

My configuration is as follows:

Intel Pentium III 1.40G
1 Gig RAM
2x <Intel Pro 10/100B/100+ Ethernet> (one internal, one external) on
board.

Take a closer look to DEVICE_POLLING kernel option, it can highly
increase fxp network card performance and router's max pps parameter at all.

As we tested, typical bottleneck of PC routers is network card irq ratio
that interrupt controller can handle.

10

Well, zebra will have very little to do with the routing performance on the
box. In fact if you are just doing a simple packet forward with no routing
protocols there is no reason to run a routing daemon of any sort.

Your limitations are the box itself, and the FreeBSD kernel. For just
forwarding packets from one side to the other it should perform quite nicely.
Even then there are a number of factors that affect throughput, including packet
size, number of clients, etc....

if you start adding packet filters and what not you may have to be a little
more careful in how you add them so as not to affect performance overly.
I don't see any reason that you shouldn't be able to basically saturate
a 100Mb line with a box of this speed, although I have no publishable information
to back me up.

11

Ok, I'll bite. We're routinely shoving 500mbps through our FreeBSD system
running zebra, and we've never hit the 50% cpu mark. 3 GigE BGP peers
passing me a full table and one GigE LAN interface, a few VLANs, lengthy
IPFW rules, and tons of "count" rules so I can MRTG each IP passing through
it. In some off network synthetic testing, I easily maxed out our GigE LAN
interface before the router dropped a packet. All this on a $1800 Dell
server with a $150 Intel PCI-X card slapped in there. This system's been up
for 6 months now. Zero crashes, zero hung interfaces, zero problems.

I'm not saying a FreeBSD+Zebra system is going to do everything that your
high end router will, but I haven't run into anything that I couldn't find
some way of doing with the tools that I had.

My biggest bitch is the lack of multipath routing (yes I know you can hack it
in), but the ability to load balance of equal cost links would be a really
nice feature.

12

> Hi All,
> I'm trying to find out what peoples experiences regarding the throughput
> of Zebra on a FreeBSD Box.
>
> My configuration is as follows:
>
> Intel Pentium III 1.40G
> 1 Gig RAM
> 2x <Intel Pro 10/100B/100+ Ethernet> (one internal, one external) on
> board.

Take a closer look to DEVICE_POLLING kernel option, it can highly
increase fxp network card performance and router's max pps
parameter at all.

As we tested, typical bottleneck of PC routers is network card irq ratio
that interrupt controller can handle.

> This box is running as a simple static router, i.e. one subnet on the
> inside, Internet feed on the other side. No BGP, no RIP, no OSPF. Pretty
> simple, eh?
>
> So the goal is to know the bandwidth limitation of this router.
Any ideas?
> I've heard numbers of 35Meg, 40 Meg, etc, however, I have not recieved a
> good reason backing it up. Can anyone offer some input on this?

I would be very surprised if an out of the box FreeBSD router couldn't
easily saturate and rate-shape 100Mb/s. Even MBUFS probably wouldn't need to
be reconfigured if its just routing.

I agree with Dmitry that certain cards are notably more "chatty" with their
interrupts than others. It could also be that FreeBSD supports certain
cards' extended operation modes better than others. fxp (Intel EtherExpress)
and certain other boards are what I consider some of the best supported
ones -- Fortunately, they are the most common ones too.

If you go higher than 100Mb/s speeds, look for zero-copy boards, especially
on the Gigabit side that FreeBSD supports. You have drastically low
interrupt overhead and your speeds to move packets around increase. In my
experience, its not the CPU that gets bogged down (first) but the interrupt
load on a properly configured (software) box.

Kevin's 500Mb/s does not suprise me as I have seen those sorts of numbers
myself. Now going above 2Gb/s, that's a little trickier.

Deepak Jain
AiNET

13

... The em driver now supports polling. The author of much of the polling code (luigi@iet.unipi.it) claimed to push 650Kpps on a 2.4Ghz PIV machine. YMMV based on the hardware you use, and the cards you combine.

         ---Mike

14

You can saturate 100mb ether with a low end classic pentium (eg p90).
100mb ethernet is not magic. GbE is a different story though :slight_smile:

-Dan