Software router state of the art

Mailman List Mirror (Read Only)
1

I'm not sure where the claims about "{one, few} flow{s}" are coming from.
Certainly the number of flows on a typical UNIX box acting as a router is
not that relevant unless you specifically configure something like
stateful firewalling, because the typical UNIX box simply doesn't have a
*concept* of "flows." It deals with packets. This has its own problems,
of course, but handling high levels of traffic in many flows is not one of
them.

There are other software routing platforms that DO flows, but the above
mentions "host[s] systems", so I'm reading that as "UNIX router."

On the flip side, packet size is definitely a consideration. This topic
has been beaten to death on the Zebra mailing lists by myself and others
in the past.

With yesterday's technology (P4 3.0G, 512MB RAM, PCI-X, FreeBSD 4) we were
successfully dealing with >300Kpps about 3 years ago, without substantial
work. That was single source/single dest, but with a full routing table.
There's no real optimization for that within the FreeBSD framework, so it
is about the same performance you'd have gotten with multi source/multi
dest.

... JG

2

This is not exactly true. The modern Linux kernel (2.6) uses some amount of flow tracking in order to do route caching. You can check this out on your system by:
"ip route show cache"

It keeps track of Src/Dst/QoS/Ethernet adapters/etc.. Additionally most systems have the iptables modules loaded in kernel and the conntrack module in kernel. This immediately activates connection tracking, therefore considerably slowing down software routing. The most optimal way of speeding this up would be sticking the route cache into somewhat faster memory. Though it would be fairly nice to get rid of the route cache as that can cause problem with eccentric setups. Also, as cache entries take a moment to be deleted, or degrade leading to convergence times being higher.

Joe Greco wrote:

3

Sargun Dhillon wrote:

This is not exactly true. The modern Linux kernel (2.6) uses some amount of flow tracking in order to do route caching. You can check this out on your system by:
"ip route show cache"

Did you mean "route -C" ?

I like the idea and price point of the ImageStream products, but knowing how bad Linux is at being a router and that their products are Linux-based, I'm afraid to give one a try. J products are based on a competing non-Linux platform that has a better reputation for routing.

~Seth

4

but knowing how bad Linux is at being a router and that their
products are Linux-based, I'm afraid to give one a try. J
products are based on a competing non-Linux platform that has
a better reputation for routing.

Enough with the bipartisan politics. There are more choices than
just Linux and FreeBSD for software routing.

Click for instance <http://read.cs.ucla.edu/click/&gt;

--Michael Dillon

5

Anyone have experience with RouterOS (http://www.mikrotik.com/)? Created mostly to run on these guys I think (http://www.routerboard.com/comparison.html) which generally don't get above 200k pps on the higher models.. But will RouterOS run on bigger boxen?

6

Thanks for being oh-so-helpful with a serious question. Got any useful answers for me? Give me a vendor that offers your suggestion. I don't have time for a make-it-myself solution.

~Seth

7

Seth Mattinen wrote:

but knowing how bad Linux is at being a router and that their products are Linux-based, I'm afraid to give one a try. J products are based on a competing non-Linux platform that has a better reputation for routing.

Thanks for being oh-so-helpful with a serious question. Got any useful answers for me? Give me a vendor that offers your suggestion. I don't have time for a make-it-myself solution.

Hmmmm. Well then you probably don't want to use Linux/BSD as a router, as a substantial amount of DIY is required for anything beyond relatively simple routing. MPLS support (on Linux) for example is in early phases and requires integrating separate pieces and is best supported on Fedora9. Needless to say, Fedora isn't designed for reliable/stable operation and long term deployment.

I have yet to look into *BSD based solutions, but hear very good things about firewall performance. I don't know about BGP/OSPF/MPLS etc support on FreeBSD but am going to wager a guess its on par with Linux if not better.

To address another point made in this thread, see http://ols.fedoraproject.org/OLS/Reprints-2007/zhu-Reprint.pdf which addresses hardware multiqueue device support under Linux. Its from 2007. I think there was a question about Linux/multiqueue support in this thread, but I am not 100% sure. :slight_smile:

I think there was mention of Vyatta earlier in the thread and some talk about it switching from Xorp to Quagga, and a supposition that should improve it.

8

Justin Sharp wrote:

9

Andrew D Kirch wrote:

Justin Sharp wrote:

Yes I do, and I'm still in therapy. I was pushing 30mbit, and I can't remember how many PPS through one, and it crashed about once a month requiring onsite intervention (usually at midnight). This was running on a Compaq Deskpro I think. It doesn't have much support for good network cards. I suspect the Realtek's were behind the crashes.

Yeah.... or any number of cheap consumer parts in the Deskpro. I would love to see RouterOS running on an HP or Dell mid range server and how that performs. I'm guessing it would be quite nice.

10

Michael 'Moose' Dinn wrote:

Thanks for being oh-so-helpful with a serious question. Got any useful answers for me? Give me a vendor that offers your suggestion. I don't have time for a make-it-myself solution.

What are your requirements?

The problem I'm facing is that if I want something from Cisco that can do at least line-rate T3, I'm looking at least $20k per router. I don't have a uber-budget, so for me, that's kind of painful when I start to need more than one plus spare parts. But, I have a high level of confidence that I can put cards in, some memory, power it up, configure it and I'm good to go.

Junpier's J-series is a BSD based platform as far as I understand it. ImageStream is *much* more affordable for me, but is Linux-based, and I fear Linux as a router and I don't know what they've done to fix the common gripes with Linux-as-router. I have no idea if either of the two have hardware assist in the cards, but my impression is that they are essentially software platforms with custom interface cards. Interface cards are important to me because I'm operating in an environment where my link to the outside world is probably going to be T1/T3.

I'm aware of Cisco IOS, then BSD-based and Linux-based platforms that are actually sold as routing products. I also know there are a billion "yay, router!" things out there. T1 cards are easy to find. The only other place I know I could buy a T3 card from is Sangoma. Maybe someone has even used it* T3 card before. Rather than reinvent the wheel alone, nanog has to contain the highest concentration of people that have tried various things and already know what will work and what won't work. I'm not looking for OS politics, just operational experience from people who have access to more money and more hardware than I do to have tried more stuff.

If my best option is still from the big players, so be it. If there's something else that's just as stable, I want to hear about it. I'm not adverse to some dirty work, but I just don't have the time right now to jump in over my head into a software router project and then fight my way back to the surface. I'm not trying to create something for educational purposes, I need something suitable for a production environment.

~Seth

* http://www.sangoma.com/products_and_solutions/hardware/data_only/a301.html

11

The problem I'm facing is that if I want something from Cisco that can do at least line-rate T3, I'm looking at least $20k per router. I don't have a uber-budget, so for me, that's kind of painful when I start to need more than one plus spare parts. But, I have a high level of confidence that I can put cards in, some memory, power it up, configure it and I'm good to go.

Junpier's J-series is a BSD based platform as far as I understand it. ImageStream is *much* more affordable for me, but is Linux-based, and I fear Linux as a router and I don't know what they've done to fix the common gripes with Linux-as-router. I have no idea if either of the two have hardware assist in the cards, but my impression is that they are essentially software platforms with custom interface cards. Interface cards are important to me because I'm operating in an environment where my link to the outside world is probably going to be T1/T3.

I'm aware of Cisco IOS, then BSD-based and Linux-based platforms that are actually sold as routing products. I also know there are a billion "yay, router!" things out there. T1 cards are easy to find. The only other place I know I could buy a T3 card from is Sangoma. Maybe someone has even used it* T3 card before. Rather than reinvent the wheel alone, nanog has to contain the highest concentration of people that have tried various things and already know what will work and what won't work. I'm not looking for OS politics, just operational experience from people who have access to more money and more hardware than I do to have tried more stuff.

If my best option is still from the big players, so be it. If there's something else that's just as stable, I want to hear about it. I'm not adverse to some dirty work, but I just don't have the time right now to jump in over my head into a software router project and then fight my way back to the surface. I'm not trying to create something for educational purposes, I need something suitable for a production environment.

[I didn't know what to cut from above, so I left it].

What I've used and seen used before that plays both to the strengths of the PC as a router and addresses some of the T3 related issues -- especially if you control both ends of the T3.

Using an FE to T3 bridge or FE to T1 bridge as the case may be. With a little tuning you can put a rate shaper on the PC (prior art, very stable) to not run into off-PC buffering issues. Your PC has plenty of cheap buffer. The interface to the comms network is done through a dedicated, telco or computer center grade piece of gear.

Everyone here (NANOG) can agree that a 10 or 100Mb/s PC router is a no brainer and as long as you aren't using irresponsible gear, this thing will route packets forever.

Putting telco interfaces into PCs has always been a little more odd, but telco to ethernet bridges are fairly standard and fairly dumb. Depending on how many of these you have etc, you can do creative things with switches, FR, etc. And cost can be all over the map. I know Pairgain used to make good ethernet to T1 bridges, and that's probably the last time I remember playing with this stuff.

YMMV.

Deepak Jain
AiNET

12

Another option (if you want a pure Cisco platform) would be to buy a used Cisco 7500 or 7200 and put a T3 card in there. Those are probably super cheap through reseller channels. (<<$20K for a 1+1).

A quick scan of Ebay shows a PA-MC-T3 for <$3K, a 7505 +RSP4+PS for $300 and a fast ethernet blade for $30.00.

Excluding software licenses (assuming its not running something that its not perpetually licensed to something that will run the T3 card) you are looking at about $3K per T3 in HW.

Deepak

13

Deepak Jain wrote:

The problem I'm facing is that if I want something from Cisco that can do at least line-rate T3, I'm looking at least $20k per router. I don't have a uber-budget, so for me, that's kind of painful when I start to need more than one plus spare parts. But, I have a high level of confidence that I can put cards in, some memory, power it up, configure it and I'm good to go.

Junpier's J-series is a BSD based platform as far as I understand it. ImageStream is *much* more affordable for me, but is Linux-based, and I fear Linux as a router and I don't know what they've done to fix the common gripes with Linux-as-router. I have no idea if either of the two have hardware assist in the cards, but my impression is that they are essentially software platforms with custom interface cards. Interface cards are important to me because I'm operating in an environment where my link to the outside world is probably going to be T1/T3.

I'm aware of Cisco IOS, then BSD-based and Linux-based platforms that are actually sold as routing products. I also know there are a billion "yay, router!" things out there. T1 cards are easy to find. The only other place I know I could buy a T3 card from is Sangoma. Maybe someone has even used it* T3 card before. Rather than reinvent the wheel alone, nanog has to contain the highest concentration of people that have tried various things and already know what will work and what won't work. I'm not looking for OS politics, just operational experience from people who have access to more money and more hardware than I do to have tried more stuff.

If my best option is still from the big players, so be it. If there's something else that's just as stable, I want to hear about it. I'm not adverse to some dirty work, but I just don't have the time right now to jump in over my head into a software router project and then fight my way back to the surface. I'm not trying to create something for educational purposes, I need something suitable for a production environment.

[I didn't know what to cut from above, so I left it].

What I've used and seen used before that plays both to the strengths of the PC as a router and addresses some of the T3 related issues -- especially if you control both ends of the T3.

Using an FE to T3 bridge or FE to T1 bridge as the case may be. With a little tuning you can put a rate shaper on the PC (prior art, very stable) to not run into off-PC buffering issues. Your PC has plenty of cheap buffer. The interface to the comms network is done through a dedicated, telco or computer center grade piece of gear.

Everyone here (NANOG) can agree that a 10 or 100Mb/s PC router is a no brainer and as long as you aren't using irresponsible gear, this thing will route packets forever.

Putting telco interfaces into PCs has always been a little more odd, but telco to ethernet bridges are fairly standard and fairly dumb. Depending on how many of these you have etc, you can do creative things with switches, FR, etc. And cost can be all over the map. I know Pairgain used to make good ethernet to T1 bridges, and that's probably the last time I remember playing with this stuff.

YMMV.

Deepak Jain
AiNET

To echo Deepak's suggestion and draw attention to the original statement "because I'm operating in an environment where my link to the outside world is probably going to be T1/T3." Would lead one to question the PA-MC-T3 even. Could be even cheaper if you don't need the multi-channel component (of course the monthly cost of the DS3 pales here in comparison w/ the h/w setup, but thought Id mention it regardless as it could save you 2 grand.) If all you need is a few t1's just pick up the VIP 2-50 interface card and a 4 x T1 adapter.

This solution can most be definitely be had for under 5 grand. with the RSP4+'s (ECC mem) youd be looking at greater than 99.99 percent uptime if configured with SSO.

-chris

14

Chris Stebner wrote:

This solution can most be definitely be had for under 5 grand. with the RSP4+'s (ECC mem) youd be looking at greater than 99.99 percent uptime if configured with SSO.

But if you end up needing BGP with full routes, throw that out the window. The RSP16's are expensive (even used relative to the RSP4) and usually necessarily for memory due to the current global routing table size. They are still cheap on the used market compared to list of most vendors, though.

Jack

15

* Joe Greco:

I'm not sure where the claims about "{one, few} flow{s}" are coming from.
Certainly the number of flows on a typical UNIX box acting as a router is
not that relevant unless you specifically configure something like
stateful firewalling, because the typical UNIX box simply doesn't have a
*concept* of "flows." It deals with packets.

You are mistaken. Linux routing is flow-based. Ever wondered what
those "dst cache overflow" messages mean you see during a DoS attack?
It's the flow cache complaining that it can't expire records in an
organic manner.

I don't know much about FreeBSD. I think it got a route cache after
FreeBSD 4, too. That's the reason why the FreeBSD 4 IP stack is still
so popular.

16

> Click for instance <http://read.cs.ucla.edu/click/&gt;

Thanks for being oh-so-helpful with a serious question. Got
any useful answers for me? Give me a vendor that offers your
suggestion. I don't have time for a make-it-myself solution.

Sorry, but you're in the wrong place. The IP networking consultants
are over thataway, and if you pay them a nice daily fee they will
sort out your problem for you.

But if you want free suggestions, then you'll have to put up with
half answers, vendor fanboys, and the usual ruckus of NANOG.

--Michael Dillon

P.S. that was a serious suggestion up above. If you have an interest
in software routers, then you should look at it. If you just want to
buy products then all routers are software routers, most especially
the ones that depend on something called IOS or Junos. Focus on the
capabilities that you need and the prices. Don't try to be pretend to
be a router designer.

17

Jack Bates wrote:

Chris Stebner wrote:

This solution can most be definitely be had for under 5 grand. with the RSP4+'s (ECC mem) youd be looking at greater than 99.99 percent uptime if configured with SSO.

But if you end up needing BGP with full routes, throw that out the window. The RSP16's are expensive (even used relative to the RSP4) and usually necessarily for memory due to the current global routing table size. They are still cheap on the used market compared to list of most vendors, though.

Jack

I was "assuming" some level of route filtering/summarization as he did mention a single t1/t3 (at least used the word "link" - singular). Good point though, if you need more than 512mb mem, your gonna have to shell out the extra $10k for the pair of RSP16's

-chris

18

I'm aware of Cisco IOS, then BSD-based and Linux-based platforms that are actually sold as routing products. I also know there are a billion "yay, router!" things out there. Rather than reinvent the wheel alone, nanog has to contain the highest concentration of people that have tried various things and already know what will work and what won't work. I'm not looking for OS politics, just operational experience from people who have access to more money and more hardware than I do to have tried more stuff.

~Seth

19

As much as I hate to contribute to the problem, I'd like to point out
that the barrage of useless, off-topic, empty traffic on this list in
the last week is, in my estimation, quite a bit above the "usual" ruckus
of NANOG.

While I'm not one to thunk down the rulebook, can you all collectively
knock it off?

Cheers,
-jp

20

Rev. Jeffrey Paul wrote: