I find myself in the market for some 10GE routers. As
I don't buy these everyday, I was wondering if any of
you guys had any good resources for evaluating
different vendors and models. I'm mainly thinking
about non-vendor resources as the vendorspeak sites
are not that hard to find.
Also I'd love to hear recommendatios for "budget" 10GE
routers. The "budget" router would be used to hook up
client networks through one 10GE interface and connect
to different transit providers through two 10GE
interfaces.
Also I'd love to hear recommendatios for "budget" 10GE
routers. The "budget" router would be used to hook up
client networks through one 10GE interface and connect
to different transit providers through two 10GE
interfaces.
Today, from Cisco, the smallest router you'll get a 10Gbps
Ethernet port on is the Cisco ASR1000 series. Mind you,
though, FCS for this box isn't until about May. Also, this
box is oversubscribed as the current switch fabric is
10Gbps.
From Juniper, the smallest M-series box you'll get the same
port on is the M120 platform.
You could also look at smaller switches from both vendors,
but if you plan on taking full BGP feeds from your upstream
providers, this might be an issue.
When it comes to budget, force10 are good. I wouldn't be able to confirm if they're worth performance-wise.
I'd strongly suggest Foundry, I'm a big fan of their kits, price-wise and performance-wise, provided you do not need rocket-science features.
MLX/XMR models will surely do the trick perfectly.
When it comes to router purchasing habits, we all tend to get religious...
Bottom line is that most of the 'regular' vendors (namely Cisco, Juniper, Foundry, Force10, Extreme, Riverstone) implement pretty much the same set of features, which are all IETF/IEEE normalized, meaning if you don't need proprietary features (and you'll wish you don't), any router will be fine, the only difference will come from:
- the chassis being non-blocking or not (i.e. backplane design)
- the price per port
- the operating OS
- the feeling you'll get with the salesperson, and the reputation of their Support Teams.
- vendor specific features such as Flow Sampling
To make it simple, most vendors have an IOS like OS, except Juniper which has a really clever and elegant OS, but are very pricey.
Foundry and Force10 have the cheapest price per port
Cisco does only Netflow, Foundry & Force10 only SFlow (which is a true standard) and I think Juniper does JFlow
Cisco's kits are packed with proprietary protocols (HSRP and GLBP instead of VRRP, their own ethernet trunking, EIGRP as their own and yet extremely efficient IGP, TCL scriptable CLI...) , some of them are really good, some are crappy, but I suggest you'd stick with IEEE/IETF protocol to avoid future trouble.
One thing: RSTP/802-1w is very (very, very, very) not often interoperable between vendors who all have their own interpretation of the norm and can quickly turn into a nightmare.
I'd strongly suggest try&buys if (R)STP interoperability is required, but I'm a little paranoid
Greg has laid out a great bit of information and I would like to add just one possibility to the list of budget 10GE routers: Vyatta. According to a recent press release from that company (http://www.vyatta.com/about/pressreleases.php?id=51) they offer a product that is “2 to 3X higher performance at a cost savings of more than 75 percent” when compared to Cisco’s 7200. Unfortunately I have not had the opportunity to test or use the Vyatta routers yet; I have however successfully used other open-source Linux based routers in the past with great success. If you are looking for a truly budget 10GE router, they may be worth adding to the list and looking into.
Greg has laid out a great bit of information and I would like to add just one possibility to the list of budget 10GE routers: Vyatta. According to a recent press release from that company (http://www.vyatta.com/about/pressreleases.php?id=51) they offer a product that is "2 to 3X higher performance at a cost savings of more than 75 percent" when compared to Cisco's 7200. Unfortunately I have not had the opportunity to test or use the Vyatta routers yet; I have however successfully used other open-source Linux based routers in the past with great success. If you are looking for a truly budget 10GE router, they may be worth adding to the list and looking into.
Whether you can actually do 10Gb/s reasonably on a linux or freebsd soft-switched router platform is going to depend a lot on your actual pps rate.
800K pps which is 10Gb/s / 1500 bytes is feasible, but 19M pps which is 10Gb/s / 64 bytes is not.
Susceptibility to dos traffic at relatively low bit, but high pps rates is a general issue with soft-switched platforms. and needs to be accounted for in model deployments.
For the past 8 months, I have problem to use the 10G
in linux system. I have to continuous to upgrade the
hardwares...
my existing system is using the new CPU now, 4G
memory, 1 x 10G card plus several 1G NICs.
Intel 2 Ext CPU X9650 @ 3.00GHz
All CPU is in 100% used when it is in 4G totally
(download + upload).
I'd strongly suggest Foundry, I'm a big fan of their kits, price-wise
and performance-wise, provided you do not need rocket-science features.
MLX/XMR models will surely do the trick perfectly.
I agree too. They still have a bit of development to do on the IPv6 side, but they are getting there. We are using them with Cat 65XXs with SXF Sup720-3BXLs and XMRs. We run ISIS, BGP, and BFD. Everything they say works really does. We have been very pleased. Definitely put them on your short list. The price per port can't be beat and their support is stellar. If you want to reliably route IPv4 and IPv6 at wire speeds regardless of packet size or rate and optionally filter at wire speed too on all ports then they make a great box.
-Robert
Tellurian Networks - Global Hosting Solutions Since 1995 http://www.tellurian.com | 888-TELLURIAN | 973-300-9211
"Well done is better than well said." - Benjamin Franklin
Totally agree.
Foundry support is top notch and the boxes do deliver the promised performance.
The headroom is impressive when the CPU is at 99%. Somehow *cough* we (me) pegged
the CPU on the Server Irons and still had a very very responsive console. Was able to find
the self inflicted error and fix the problem quickly. Out testers on the outside say they did not
notice a performance degradation.
Foundry's performance and support make the price a clear value.
I've only experienced two flavors, Cisco and Foundry.
> Greg has laid out a great bit of information and I would like to add just
> one possibility to the list of budget 10GE routers: Vyatta. According to a
> recent press release from that company
> (http://www.vyatta.com/about/pressreleases.php?id=51) they offer a product
> that is "2 to 3X higher performance at a cost savings of more than 75
> percent" when compared to Cisco's 7200.
"Vyatta operates at Layer 3 wire speed across three Gigabit Ethernet
ports in full mesh when forwarding 512-byte frames or higher."
My experience is similar to those who have posted here, pps is the
limiting factor - usually somewhere between 500-800K. Apparently I
was over eager to believe that more had been achieved.
To Ann's question on resources; I have only used Linux routers with 1G
ports but have surpassed 10G total throughput (up+ down) using various
dual proc set ups, most often Intel Xeon in Dell servers. A gentlemen
by the name of Martin Pels wrote a good paper on the subject early
last year that can be found here: http://docs.rodecker.nl/10-GE_Routing_on_Linux.pdf. He hit a wall at
700K pps and was using two dual core Intel Xeon 64bit 2.33GHz CPUs and
2GB of RAM in a Dell PowerEdge 1950.
Very interesting study I had not seen, and a bummer. That really puts a cramp in my advocation of our CARP+pf load balancers/firewalls/gateways. Than again, what’s a PIX box capable of?
I also had to switch to OpenBSD as there was a fatal crash with the bridge device in FreeBSD when used with my paticular OpenVPN/CARP/pf combination.
AFAIK pf/forwarding only takes place on one core and wouldn’t take advantage of the other 3 cores, correct?
Very interesting study I had not seen, and a bummer. That really puts a
cramp in my advocation of our CARP+pf load balancers/firewalls/gateways.
Than again, what's a PIX box capable of?
I'd rather tweak a whitebox than pay through the nose for a PIX.
I also had to switch to OpenBSD as there was a fatal crash with the bridge
device in FreeBSD when used with my paticular OpenVPN/CARP/pf combination.
AFAIK pf/forwarding only takes place on one core and wouldn't take advantage
of the other 3 cores, correct?
Correct. There has been some great speed and efficiency improvements
in pf and other networking parts of OpenBSD; though from anecdotal
evidence, 10GbE is not ready for 'primetime' (for certain definitions
of 'primetime').
actually I'll just skip making an ass out of myself and hope henning@
chimes in, since I believe he reads NANOG as well.
Shh... It's a secret and hasn't been released yet. We have have a few NPE-40Gs with four 10G XFP interfaces. Nah... I'm just wishing...
-Robert
Tellurian Networks - Global Hosting Solutions Since 1995 http://www.tellurian.com | 888-TELLURIAN | 973-300-9211
"Well done is better than well said." - Benjamin Franklin
Very interesting study I had not seen, and a bummer. That really puts a cramp in my advocation of our CARP+pf load balancers/firewalls/gateways. Than again, what's a PIX box capable of?
Well, you get what you pay for. If you're willing to blow $10k on a
firewall, maybe you'll be willing to blow $10k on a *BSD developer
to work on improving forwarding performance.
It'd only take ten or so people to make donations or sponsor work
of that size for the benefits to appear.
I also had to switch to OpenBSD as there was a fatal crash with the bridge device in FreeBSD when used with my paticular OpenVPN/CARP/pf combination.
Did you log a bug?
AFAIK pf/forwarding only takes place on one core and wouldn't take advantage of the other 3 cores, correct?
Uhm, its not quite that simple. ithreads on FreeBSD at least will run on
one CPU at a time (unless you're running some hacked up russian-driven
intel gige driver, which runs multiple ithreads for the device to improve
performance under certain circumstances!) and these classes of cards and
busses wouldn't benefit from >1 core contending for one card/bus.
If you're running >1 card then you may find the ithreads run on different
CPUs, each doing lookups and forwarding, but I haven't sat down and looked
at that sort of forwarding performance under FreeBSD. My focus at the moment
is "tcp proxy on a stick" throughput with one interfaces and >1 core doing
userland processing.
How sweet is a sub-$1k router that can do multiple gig-e's at 1.5mpps?
Sounds like a dynamite platform for high-end datacenter CPEs that are
soft
on dynamic routing...and even the open-source dynamic routing is
reasonably solid these days...
I can't believe I am about to ask this on a public mailing list, but..
Has anyone tested this in even a remotely production environment, while
running any sort of MPLS LDP as a LSR?
Nah... I'm just wishing...