Hi,
Can somebody educate me on (or pass some pointers) what differentiates
a router operating and optimized for data centers versus, say a router
work in the metro ethernet space? What is it thats required for
routers operating in data centers? High throughput, what else?
Thanks, Venkatesh
There's corporate data centers and there's colo data centers. The two are
sufficiently different that the requirements are divergent. For starters,
in a colo, the guy on blade 3 port 5 is quite possibly a competitor of
the guy who's got blade 2 port 17. In the corporate data center, we
maintain the polite fiction that those two are working together for
a common goal. This has implications for security features, billing,
bandwidth engineering, and almost every other feature on a router.
The biggest difference that I see is that you generally use different resources in a Datacenter. (Colo Datacenter).
For example, I run out of HSRP groups on a 6500 long before I run out of ports or capacity. I don't need to worry about QoS much but a less complex rate limit command (As opposed to Policing) is very useful. Also, Front to back cooling is optimal in a Datacenter and often not available.
James
Hi,
Can somebody educate me on (or pass some pointers) what differentiates
a router operating and optimized for data centers versus, say a router
work in the metro ethernet space? What is it thats required for
routers operating in data centers? High throughput, what else?Thanks, Venkatesh
Well, they generally have to be rack mountable. Besides that, I have seen everything from
tiny Linux boxes to big refrigerator sized units (of course, the latter
may be on the floor). I don't think you are going to find much commonality there, so you
need to refine what it is you want to do. (For example, to move 10 Mbps or 100 Gbps or... ?
Run BGP or NAT or ... ?)
Regards
Marshall
Hi,
Can somebody educate me on (or pass some pointers) what differentiates
a router operating and optimized for data centers versus, say a router
work in the metro ethernet space? What is it thats required for
routers operating in data centers? High throughput, what else?
While this question has many dimensions and there is no real definition of either I suspect that what many people mean when they talk about a DC routers is:
Primarily Ethernet interfaces
High port density
Designed to deal with things like VRRP / VLAN / ethernet type features.
Possibly CAM based, possibly smaller buffers.
Less likely to be taking full routes.
This is very similar to the religious debate about "What's the difference between a 'real' router and a L3 switch?"
Just my 2 cents.
W
the power/cooling budget for a rack full of router vs a rack
full of cores might be distinction to make. I know that
historically, the data center operator made no distinction
and a client decided to "push past the envelope" and replaced
their kit with space heaters. most data centers now are fairly
restrictive on the power/cooling budget for a given footprint.
--bill
Historically, you would find that routers designed for long-haul transport (Cisco GSR/CRS, Juniper M-series, etc) generally had deeper buffers per-port and more robust QoS capabilities than datacenter routers that were effectively switches with Layer 3 logic bolted on (*coughMSFCcough*). That line has blurred quite a bit lately, however - Cisco's ES line cards are an example.
That said, there's plenty of debate as to whether or not these features actually make for a better long-haul router or not - I've seen more metro and national backbones built with Cat6500^H^H^H^H7600s than you'd think.
-C
While this question has many dimensions and there is no real
A "datacenter router" is a box which falls into a particular market
segment, characterized by extremely low cost, low latency, and high
density ethernet-centric boxes, at the expense of "advanced" features
typically found in more traditional routers. For example, these boxes
tend to lack any support for non-ethernet interfaces, MPLS, advanced
VLAN tag manipulation, advanced packet filters, and many have limited
FIB sizes. These days it also tends to mean you'll be getting a box with
only (or mostly) SFP+ interfaces, which are cheaper and easier to do
high density 10GE with, but at the expense of "long reach" optic
availability.
A "metro ethernet" box also implies a particular market segment,
typically a smaller box (1-2U) that has certain advanced features which
are typically not found in other "small" boxes. Specifically, you're
likely to see advanced VLAN tag manipulation and stacking capabilities,
MPLS support for doing pseudowire/vpn PE termination, etc, that you
might normally only expect to see on a large carrier-class router.
Also, an interesting side-effect of the quest for high density 10GE at
low prices is that modern datacenter routers are largely built on third
party "commodity" silicon rather than the traditional in-house ASIC
designs. Many of the major router vendors (Cisco, Juniper, Foundry,
Force10, etc) are currently producing "datacenter routers" which are
actually just their software (or worse, someone else's software with a
little search and replace action on a few strings) wrapped around third
party ASICs (EZchip, Marvell, Broadcom, Fulcrum, etc). These boxes can
definitely offer some excellent price/performance numbers, but one
unfortunate side effect is that many (actually, most) of these chips
have not been fully baked by the years of experience the more
traditional router vendors have developed. Many of them have some very
VERY serious design flaws, causing everything from preventing them from
fully implementing some of the features you would normally except from a
quality rouer (multi-label stack MPLS, routed vlan interface counters,
proper control-plane DoS filter/policing capabilities, etc), or worse
(in some cases, much, much worse). YYMV, but the 30 second summary is
that many vendors consider "datacenter" users and/or use cases to be
unsophisticated, and they're hoping you won't notice or care about some
of these serious design flaws, just the price per port. Depending on
your application, that may or may not be true. 
Cisco uses their own ASICS is their higher end flag ship devices.
Devices such as the Catalyst 6500 series or the 2960 switches. You
pretty much singled out all the major players, including those who have
been bought out (Foundry by HP) and claimed they do not provide their
own, yet 3rd party flawed ASICS. I am actually surprised you didn't
mention HP, Linksys or Dell as they are the most guilty of using 3rd
party ASICS and shotty software. If you are buying data center grade
equipment from these vendors, it will be quality hardware backed by
their support (if purchased) such as Cisco's SmartNet agreements.
Moral of the story, do your research on the devices you plan to
implement and ask for data sheets on how the features you need are
handled (in software or hardware). I know Juniper and Cisco provide such
documentation for their devices. Quality hardware, however more
expensive, will give you less trouble in the long run. You truly get
what you pay for in the networking industry.
Cisco uses their own ASICS is their higher end flag ship devices.
Devices such as the Catalyst 6500 series or the 2960 switches. You
pretty much singled out all the major players, including those who
have been bought out (Foundry by HP) and claimed they do not provide
their own, yet 3rd party flawed ASICS. I am actually surprised you
didn't mention HP, Linksys or Dell as they are the most guilty of
using 3rd party ASICS and shotty software. If you are buying data
center grade equipment from these vendors, it will be quality hardware
backed by their support (if purchased) such as Cisco's SmartNet
agreements.
My point was that every major vendor, even the ones who normally make
their own in-house ASICs, are also actively selling third party silicon
(or in some cases complete third party boxes) in order to compete in the
"cheap" "datacenter optimized" space. Folks like HP and Dell were never
in the business of making real routers to begin with, so them selling a
Broadcom reference design with 30 seconds of search and replace action
on the bundled software is not much of a shocker. The guys who do a
better job of it, like Foundry (who was bought by Brocade, not HP), at
least manage to use their own OS as a wrapper around the third party
hardware. But my other major point was that almost all of these third
party ASICs are sub-par in some way compared to the more traditional
in-house hardware. Many of them have critical design flaws that will
limit them greatly, and many of these design flaws are only just now
being discovered by the router vendors who are selling them.
BTW, Cisco is actually the exception to the "datacenter optimized" boxes
being third party, as their Nexus 7K is an evolution of the 6500/7600
EARL ASICs, and their third party hw boxes are EZchip based ASR9k's. Of
course their Nexus software roadmap looks surprisingly similar to other
vendors doing it with third party hw, go figure. 
Moral of the story, do your research on the devices you plan to
implement and ask for data sheets on how the features you need are
handled (in software or hardware). I know Juniper and Cisco provide
such documentation for their devices. Quality hardware, however more
expensive, will give you less trouble in the long run. You truly get
what you pay for in the networking industry.
It takes a pretty significant amount of experience and inside knowledge
to know who is producing the hardware and what the particular issues
are, which is probably well beyond most people. The vendors aren't going
to come out and tell you "Oh woops we can't actually install a full
routing table in our FIB like we said we could", or "Oh btw this box
can't filter control-plane traffic and any packet kiddie with a T1 can
take you down", or "FYI you won't be able to bill your customers 'cause
the vlan counters don't work", or "just so you know, this box can't load
balance for shit, and L2 netflow won't work", or "yeah sorry you'll
never be able to do a double stack MPLS VPN". The devil is in the
caveats, and the commodity silicon that's all over the datacenter space
right now is certainly full of them.
Well, if you look at the hardware it's dead obvious: airflow goes across
the linecards. Nexus 7k 10-slot has front bottom to back top airflow
because it uses vertically oriented cards.
~Seth
Cisco uses their own ASICS is their higher end flag ship devices.
Devices such as the Catalyst 6500 series or the 2960 switches. You
pretty much singled out all the major players, including those who
have been bought out (Foundry by HP) and claimed they do not provide
their own, yet 3rd party flawed ASICS. I am actually surprised you
didn't mention HP, Linksys or Dell as they are the most guilty of
using 3rd party ASICS and shotty software. If you are buying data
center grade equipment from these vendors, it will be quality hardware
backed by their support (if purchased) such as Cisco's SmartNet
agreements.My point was that every major vendor, even the ones who normally make
their own in-house ASICs, are also actively selling third party silicon
(or in some cases complete third party boxes) in order to compete in the
"cheap" "datacenter optimized" space. Folks like HP and Dell were never
in the business of making real routers to begin with, so them selling a
Broadcom reference design with 30 seconds of search and replace action
on the bundled software is not much of a shocker. The guys who do a
better job of it, like Foundry (who was bought by Brocade, not HP), at
least manage to use their own OS as a wrapper around the third party
hardware. But my other major point was that almost all of these third
party ASICs are sub-par in some way compared to the more traditional
in-house hardware. Many of them have critical design flaws that will
limit them greatly, and many of these design flaws are only just now
being discovered by the router vendors who are selling them.BTW, Cisco is actually the exception to the "datacenter optimized" boxes
being third party, as their Nexus 7K is an evolution of the 6500/7600
EARL ASICs, and their third party hw boxes are EZchip based ASR9k's. Of
course their Nexus software roadmap looks surprisingly similar to other
vendors doing it with third party hw, go figure.
Cisco definitely is doing some interesting things with the Nexus. Have
you seen the virtualized version?
Moral of the story, do your research on the devices you plan to
implement and ask for data sheets on how the features you need are
handled (in software or hardware). I know Juniper and Cisco provide
such documentation for their devices. Quality hardware, however more
expensive, will give you less trouble in the long run. You truly get
what you pay for in the networking industry.It takes a pretty significant amount of experience and inside knowledge
to know who is producing the hardware and what the particular issues
are, which is probably well beyond most people. The vendors aren't going
to come out and tell you "Oh woops we can't actually install a full
routing table in our FIB like we said we could", or "Oh btw this box
can't filter control-plane traffic and any packet kiddie with a T1 can
take you down", or "FYI you won't be able to bill your customers 'cause
the vlan counters don't work", or "just so you know, this box can't load
balance for shit, and L2 netflow won't work", or "yeah sorry you'll
never be able to do a double stack MPLS VPN". The devil is in the
caveats, and the commodity silicon that's all over the datacenter space
right now is certainly full of them.
I agree it takes a significant amount of experience to know that
informatin off the top of your head, but I am able to find block
diagrams, and part information for 98% of Cisco's hardware. Old or new.
One needs to do their research on the device to know if it meets their
needs. The caveats are everywhere I agree, even some of the experienced
network guys get tripped up with them if they aren't careful. Planning
is the key to overcoming these problems.
While this question has many dimensions and there is no real
definition of either I suspect that what many people mean when they
talk about a DC routers is:From the datacenter operator prospective, it would be nice if some of these vendors would acknowledge the need for front-to-back cooling. I mean, it is 2010.
Bakplanes make direct front to back cooling hard. non-modular platforms can do it just fine however.
Depending upon the specific requirements of the scenario at each type of site, the optimal devices could be either identical, or completely different.
adam.
Once upon a time, Joel Jaeggli <joelja@bogus.com> said:
>>> From the datacenter operator prospective, it would be nice if some of these vendors would acknowledge the need for front-to-back cooling. I mean, it is 2010.
Bakplanes make direct front to back cooling hard. non-modular platforms can do it just fine however.
There are servers and storage arrays that have a front that is nothing
but hot-swap hard drive bays (plugged into backplanes), and they've been
doing front-to-back cooling since day one. Maybe the router vendors
need to buy a Dell, open the case, and take a look.
The server vendors also somehow manage to make an empty case that costs
less than $10,000 (they'll even fill it up with useful stuff for less
than that).
Once upon a time, Joel Jaeggli <joelja@bogus.com> said:
From the datacenter operator prospective, it would be nice if some of these vendors would acknowledge the need for front-to-back cooling. I mean, it is 2010.
Bakplanes make direct front to back cooling hard. non-modular platforms can do it just fine however.
There are servers and storage arrays that have a front that is nothing
but hot-swap hard drive bays (plugged into backplanes), and they've been
doing front-to-back cooling since day one. Maybe the router vendors
need to buy a Dell, open the case, and take a look.
The backplane for a sata disk array is 8 wires per drive plus a common power bus.
The server vendors also somehow manage to make an empty case that costs
less than $10,000 (they'll even fill it up with useful stuff for less
than that).
Unit volume is little higher, and the margins kind of suck. There's a reason why hp would rather sell you a blade server chassis than 16 1us.
Equating servers and routers is like equating bouncy castle prices with renting an oil platform.
Joel's widget number 2
Once upon a time, Joel Jaeggli <joelja@bogus.com> said:
There are servers and storage arrays that have a front that is nothing
but hot-swap hard drive bays (plugged into backplanes), and they've been
doing front-to-back cooling since day one. Maybe the router vendors
need to buy a Dell, open the case, and take a look.The backplane for a sata disk array is 8 wires per drive plus a common power bus.
Server vendors managed cooling just fine for years with 80 pin SCA
connectors. Hard drives are also harder to cool, as they are a solid
block, filling the space, unlike a card of chips.
It's the same 80 wires on every single drive in the string.
There are fewer conductors embedded in 12 drive sca backplane as there are in a 12 drive sata backplane, in both cases they are generally two layer pcbs. Compared to what 10+ layer pcbs that are a approaching 1/4" thick on the router.
Hard drives are 6-12w each, a processor complex that's north of 200w per card is a rather different cooling exercise.
Aw come on, that's no reason you can't just drill it full of holes. I
mean, it is 2010. It should be wireless by now.
~Seth