We need true full routes and more CPU horsepower for crunching BGP
(+12 smaller peers + ISIS). OC3 interfaces are going to be mandatory,
one each at two locations. Oh yeah, we're still a larger startup
without endless pockets. Power, rack space, and SmartNet are not
concerns at any location (on-site cold spares). We may need an
upstream OC12 in the future but that's a ways out and not a concern
here.
Our engineering team has settled on three $20k/node options:
- Sup720-3BXLs with PS and fan upgrades
- Sup2s as switches + ISIS + statics and no BGP, push BGP edge routing
off to NPE-G2s across a 2-3Gbps port-channel
- Sup2s as switches + ISIS + statics and no BGP, push BGP edge routing
off to a 12008 with E3 engines across a 2-3Gbps port-channel.
Ideas and constructive opinions welcome, especially software and
stability-related.
For about $6k all in, you could pickup a monster dual Xeon server with a few
10GE PCI line cards and run a subscription service of the Vyatta open source
router. With high end machine specs, we've been able to run 5 full tables
and a solid amount of peers with about 6.5Gbps sustained to the net without
any stress. For access, we just trunk one of the PCI cards down to a 6506
or a 3750 and it runs nice and clean. The only downside to this setup is
the lack of cisco proprietary software features which it sounds like you
might need. If anything you might be able to keep your existing setup and
uplink everything to one of these routers as an edge device.
Adam
So I figure a summary is an order, with a whole array of choices
pitched so far...
- Sup720-3BXL works for light-duty premium ISP services, decent CPU
for BGP and an Ethernet hardware throughput monster. Decent enough for
our deployment scenario at least. No obvious solution for the
FlexWAN/OC3 but could easily be re-integrated with a stronger MSFC CPU
to back it up, assuming the IOS-of-the-week doesn't have issues. The
pesky OC3 could be pawned off to a dedicated G1/G2 router too along
with any oddball <=OC3 stuff our sales guys dream up.
- RSP720-3CXL is the best of all worlds option, if we had double the
budget to work with. Meh.
- ASR1002 is a hardware-assisted overhaul to the 7200/G2. Telco
interface options are much better than 7200s, good for OC12s and
OC48s. Using GoogleFu product pricing... a ASR1002 router with a SPA
OC3, 5Gbps ESP, and base software runs in the $28-30k range +
SmartNet. Beware the modular licensing model in addition to IOS
editions. Maybe a bit early yet as a core router as some of the
software is still getting bugs ironed out.
- Vyatta was proposed as an alternative system, probably best
architected out of the mainstream traffic flows (no hardware
forwarding), say a BGP route reflector or GBE edge router, similar
argument to a 7200/G[1|2]. I can't say I'm familiar with the software,
but the cost savings of premium x86/x64 hardware and 8x PCI-x serving
a few 10GBE interfaces + built-in GBEs is intriguing, especially
paired against our budget and relative Cisco costs. A spec'd out 1U
Dell box with dual power, 8x cores, 4GB, RAID1 SATA, and 2x 10GBE
XFP+2x GBE built-in came in under $7k with CPU headroom to burn.
Vyatta doesn't support ISIS though, best I can tell, but may not have
to... Maybe yet-another Linux router distro doomed to fail? Worth a
lab test internally on some demo hardware.
- Mixed thoughts about 7304 hardware. Hardware forwarding quality vs.
software and interface selection.
- Lots of fans for the 12000 series. Stick with the E3 (~2.5Gbps) and
E5 (~10Gbps) line cards for compatibility with XR software and best
line card performance. Our team liked the variety of SONET options
available too for our central office deployments, even though the
systems are power and space hungry. ...and if you can afford them (the
12008/GRP-B being the relative exception).
- 7200/G2s are great for <1Gbps throughput. Premium services cut into
the performance dramatically, being a fully software-based forwarding
platform. Don't bond interfaces looking for more throughput,
architecture limitations actually decrease throughput.
- Juniper MX series? A budget wildcard but indeed a worthy platform
engineering-wise.
You could break this list into "routers" and "switches", which in
itself spurs the philosophical/pragmatic architecture discussion that
got us the impasse to start with. Many thanks to all who've responded
with real-life successes, battle wounds, and horror stories. All very
helpful.
-Dave
Did you check PCI bus bandwidth? That's probably going to be the biggest
constraint on "a few 10GBE interfaces" if they all get going full blast.
Remember that each packet is going to burn bandwidth twice - once in and
once out...
PCIe, x8 or x16, which is serial point to point.
http://www.csc.kth.se/~olofh/10G_OSR/10Gbps.pdf
25 Gb/sec across 4x10G ports on higher end but far from topped out
hardware.
New architectures might be helpful to achieve such throughput e.g.
Myricom pci-e Gen2 10GE cards on new Intel Nehalem based servers.
-Azher
Leo Bicknell wrote:
ASR is embedded linux solution with Quantum Processor architect if I
remember correctly.
So it uses IOS-XE, which is a little bit different from standard IOS.
If you have some room for budget, you can check Foundry MLX/XMR series
router.
It is more geared toward Ethernet Service Router.
But if you need OC3/12/48, you can have those with additional license fee.
Foundry router price is a lot lower than Juniper MX series router.
Alex
David Storandt wrote:
further illustrating the point - 10gige ~linerate load balancing on a single core2 e8200 using haproxy + myricom 10gige cards:
http://haproxy.1wt.eu/10g.html
-jba