Layer 2 vs. Layer 3 to TOR

Just to let you know - the Juniper EX4200 series only support a single label
stack, and RSVP not LDP - plus they have a restricted BGP table size, so
VPLS is out of the question.

Matthew Walster

i have seen no mention of arista as a tos switch/router, yet folk tell
me it is one of the hottest on the block today. is there anyone who is
actuallly using it who would care to report?


Good point about Arista - Doug Gourlay, of [ex-]Cisco fame, is probably the
person to ask all possible questions about those solutions.

Cisco UCS is missing, also - looking at the Nexus deployment as ToR solution
(2K + 5K, even 1KV, considering the needs for virtualization, also) with all
benefits of both traditional ToR and E/MoR will definitely shed some light
in the debate on whether L3 in ToR makes any sense at all (e..g how would
you VMotion across racks?!? - how you you sync SANs across L3 in the DC
(tunnel?!?), etc.).

Here are some interesting articles associated with technologies in new DC
designs, for example, allowing some rethinking of the L3 question: - search for ToR and VMotion articles
(actually poke arond the whole blog - it is very good) (start from 1, of
course) ...etc.


I've been using Arista's 7124S in a ToR deployment for a new build out for a high frequency trading client I've been engaged with. For the aggregation layer I went with Cisco 4900m's and have had much success with this deployment especially with the Arista's.

From a colleague here at NASA (high-performance computing area):

"We are currently using our three Arista switches as
an extremely economical way to get a 10G non-blocking
testbed for our various test areas. We have every
intention of looking at them as an option for
their routing capabilities, but have been buried
with setup and testing of our testbed equipment and
getting ready for Super Computing 2009. They seem
to have a number of very promising possibilities and
have so far proven to be very capable switches.

Paul Lang"


Randy Bush <>
Matthew Walster <>
nanog list <>
11/13/2009 08:34 AM
Re: Layer 2 vs. Layer 3 to TOR

i have seen no mention of arista as a tos switch/router, yet folk tell
me it is one of the hottest on the block today. is there anyone who is
actuallly using it who would care to report?


If you wanted something to do this, it's called an MX series. The ex is a switch... l3, but still a switch.

Disagree, the EX is a very capable L3 router for LANs.

....and in a similar vein, ATAoE ; either Coraid stuff or the the free
one in the Linux kernel. Its heavily used in some shops that use virtual
farms with SANS as it's cheap/free and works over existing hardware but
only at layer 2.

I even run it at home (!) - and it's a surprisingly easy way to have a
shelf of storage hanging off the back of a server, with 4GB of cache for
each set of 4 disks per box. Stand too close can feel the wind from it,
especially if RAIDed.

Depends if there's much call for VM-ing in your shop in the future?


Tore Anderson writes:

* Jonathan Lassoff

Are there any applications that absolutely *have* to sit on the same
LAN/broadcast domain and can't be configured to use unicast or multicast

FCoE comes to mind.

Doesn't FCoE need even more than that, i.e. "lossless" Ethernet with
end-to-end flow control, such as IEEE DCB? As far as I understand,
traditional switched Ethernets don't fit the bill anyway.

On the other hand iSCSI should be fine with routed IP paths; though
Malte's mail suggests that there are (broken?) implementations that aren't.

Juniper claims their switches can do clustering using ethernet
cabling, yet a cluster behaves as a single-system-image
configuration-wise. Should allow for very flexible cabling and
operations-wise for TOR switches. I have never tried it however.


The Ex4200 can be stacked by the ethernet expansion ports, either 4 x
1G or 2 x 10G.
And yes, it behaves as single switch with multiple line cards.

Yes, up to 10 EX4200 switches can be interconnected into a "Virtual
Chassis" using either the rear Virtual Chassis Ports (32 Gbps ingress
+ 32 Gbps egress for each of the 2 ports) with up to 5-meter VCP
cables, or using SFP, XFP or SFP+ fiber links (not sure if it works
with copper SFP, but might). You can mix/match each type of
interconnection within the same VC.