Recommended L2 switches for a new IXP

Mailman List Mirror (Read Only)
1

Dear Nanog community

We are trying to build a new IXP in some US Metro areas where we have
multiple POPs and I was wondering what do you recommend for L2 switches. I
know that some IXPs use Nexus, Brocade, Force10 but I don't personally have
experience with these switches. It would be great if you can share your
experience and recommendations. There are so many options that I don't know
if it makes sense to start with a modular switch (usually expensive because
the backplane, dual dc, dual CPU, etc) or start with a 1RU high density
switch that support new protocols like Trill and that supposedly allow you
to create Ethernet Fabric/Clusters. The requirements are simple, 1G/10G
ports for exchange participants, 40G/100G for uplinks between switches and
flow support for statistics and traffic analysis.

Thank you and have a great day.

Regards

2

I look forward to this thread.

I think one important thing is who is your addressable market size? I'm working with a startup IXP and there's only 20 carriers in the building. A chassis based switch would be silly as there would never be that many people present. 2x 1U switches would be more than plenty in their environment.

3

We used to use Brocade FastIrons until we needed more 10G port density. We moved to Brocade SX's.

Originally, when it was 2 or 3 peers, we used an old Netgear switch. :slight_smile:

Aaron

4

Like Mike says, it depends on your market. Are these markets where there are existing exchanges?

Cost per port is what we always look at. If we are going into a market where there won’t be much growth we look at Cisco and Force 10. Their cost per port is usually cheaper for smaller 10 Gig switches. You need something that is fairly robust.

Reliability in an exchange is a key component. If you go with a non-chassis switch make sure you have redundancy in your design. We like Chassis based switches because they tend to be more robust. But thats just my take on it.

Justin

5

For a startup IXP, it would probably not be sensible to use chassis based
kit due to cost / real estate issues.

Some personal opinions:

- I have a strong preference for using only open bridging protocols. This
excludes out vendor proprietary fabrics (VDX, OTV, etc). This is important
for when you do fabric upgrades on multi-site IXPs.

- You will probably want a product which supports sflow, as peer-to-peer
traffic graphs are massively useful. Most vendors support sflow on most of
their products with the notable exception of Cisco where only the Nexus 3K
team were enlightened enough to shim it in. I haven't yet come across a L2
netflow implementation which works well enough to be an adequate
substitute, but ymmv.

- VPLS based fabrics may be important if you have an interesting topology.
If it is important to you, then you will need a VPLS implementation which
will do proper load balancing over multiple links. Most don't and this is
a very hard problem to handle on smaller kit.

- There is no excuse for vendor transceiver locking or transceiver
crippling (e.g. refusing to show DDM values) and vendors who do this need
to be made aware that it's not an acceptable business proposition.

- you need kit which will support Layer 2 ACLs and Layer 3 ACLs on layer 2
interfaces.

- you should get in with the open-ix crowd and chat to people over pizza or
peanuts. You will learn a lot from in an afternoon of immersion with peers.

Nick

6

Substantial amounts of hive mind went into this topic in the formation of
Open-IX and particularly around optimizing costs and maximizing traffic.
See http://bit.ly/N-OIX1 for a reference.

Best,

-M<

7

We see a lot of IXPs being formed or upgrading with Cisco Nexus 3524 switches, which have 48 1G-10G SFP/SFP+ physical ports, license-limited to 24 active, upgradeable to 48 active.

FWIW, 83% of IXPs have 48 or fewer participants, and 70% of IXPs have 24 or fewer participants. And the failure rate of chassis-based switches is _way_ higher than that of stand-alone switches. So we never recommend that an IXP buy a switch larger than necessary to accommodate 18 months reasonably-projectable growth.

                                -Bill

8

[ clip, good stuff ]

- you should get in with the open-ix crowd and chat to people over pizza or

peanuts. You will learn a lot from in an afternoon of immersion with
peers.

And you can find that crowd here
http://mailman.open-ix.org/mailman/listinfo/public if interested.

Best,

-M<

9

Would tend to agree with this approach, and the above.

Multi-rate (i.e., 1Gbps/10Gbps SFP/SFP+) standalone 1U
switches are reasonable these days. The issue you'll
probably run into with them is limited support for features
you find being implemented by larger exchange points (VPLS,
Sflow, e.t.c.), and quirks with the hardware that could
impact things like Layer 2 or Layer 3 filtering (especially
if they are using off-the-self silicon), e.t.c.

Test before you buy, in as far as you can anticipate your
(growth) needs.

Mark.

10

People seem to be avoiding recommending actual devices, well I would
recommend the Juniper EX4600 -

http://www.juniper.net/us/en/products-services/switching/ex-series/ex4600/

They are affordable, highly scalable, stackable and run JunOS.

cheers

11

That's what I had recommended him directly :wink:

Mehmet

12

(and you can't do anything worthwhile for acls to protect that device
from the world/ix-users)

13

We've been quite happy with the EX4550, but the EX4600 is
good too, particularly if you're coming from its younger
brother.

Mark.

14

Is there any particular reason you prefer EX4600 over QFX5100 ? Not
counting obvious differences like ports and upgrade options.

It's the same chipset after all, and with all upgrades they have the
same 10G density (with breakouts). Is that because you can have more 40G
ports with EX4600 ?

I'm still trying to find out if there are any noticeable software or
feature differences.

15

QFX5100 is SDN ready.

16

We love our 5100s here.

I have 4 48S, and 2 24q¹s.

Super fast, TISSU when it works is awesome as well... like, really awesome.

Stephen Carter | IT Systems Administrator | Gun Lake Tribal Gaming
Commission
1123 129th Avenue, Wayland, MI 49348
Phone 269.792.1773

17

Manuel Marín writes:

Dear Nanog community
[...] There are so many options that I don't know if it makes sense to
start with a modular switch (usually expensive because the backplane,
dual dc, dual CPU, etc) or start with a 1RU high density switch that
support new protocols like Trill and that supposedly allow you to
create Ethernet Fabric/Clusters. The requirements are simple, 1G/10G
ports for exchange participants, 40G/100G for uplinks between switches
and flow support for statistics and traffic analysis.

Stupid thought from someone who has never built an IXP,
but has been looking at recent trends in data center networks:

There are these "white-box" switches mostly designed for top-of-rack or
spine (as in leaf-spine/fat-tree datacenter networks) applications.
They have all the necessary port speeds - well 100G seems to be a few
months off. I'm thinking of brands such as Edge-Core, Quanta etc.

You can get them as "bare-metal" versions with no switch OS on them,
just a bootloader according to the "ONIE" standard. Equipment cost
seems to be on the order of $100 per SFP+ port w/o optics for a
second-to-last generation (Trident-based) 48*10GE+4*40GE ToR switch.

Now, for the limited and somewhat special L2 needs of an IXP, couldn't
"someone" hack together a suitable switch OS based on Open Network Linux
(ONL) or something like that?

You wouldn't even need MAC address learning or most types of flooding,
because at an IXP this often hurts rather than helps. For building
larger fabrics you might be using something other (waves hands) than
TRILL; maybe you could get away without slightly complex "multi-chassis
multi-channel" mechanisms, and so on.

"Flow support" sounds somewhat tough, but full netflow support that
would get Roland Dobbins' "usable telemetry" seal of approval is
probably out of reach anyway - it's a high-end feature with classical
gear. With white-box switches, you could try to use the given 5-tuple
flow hardware capabilities - which might not scale that well -, or use
packet sampling, or try to use the built-in flow and counter mechanisms
in an application-specific way. (Except *that's* a lot of work on the
software side, and a usably efficient implementation requires slightly
sophisticated hardware/software interfaces.)

Instead of a Linux-based switch OS, one could also build an IXP
"application" using OpenFlow and some kind of central controller.
(Not to be confused with "SDX: Software Defined Internet Exchange".)

Has anybody looked into the feasibility of this?

The software could be done as an open-source community project to make
setting up regional IXPs easier/cheaper.

Large IXPs could sponsor this so they get better scalability - although
I'm not sure how well something like the leaf-spine/fat-tree design maps
to these IXPs, which are typically distributed over several locations.
Maybe they could use something like Facebook's new design, treating each
IXP location as a "pod".

18

What does it mean - to be SDN ready?

Cheers,
Jeff

19

it means "fully buzzword compliant".

Nick

20

AhhhŠ vertically integrated horizontal API¹s

Cheers,
Jeff