Peering VLANs and MAC addresses

Mailman List Mirror (Read Only)
1

Hi ,

We are unable to resolve a problem with our peering exchange connection and would like any assistance. Our peering setup is a follows:

- Our peering exchange connection goes into switch A
- Switch A has a dark fibre connection to switch B, which is in a different PoP
- Our peering router is connected to switch B

We use spanning tree across our network to allow the VLANs connectivity across our network.

The peering exchange has an MoU that only 1 MAC address should be visible on their switch. However they see 2 MAC addresses on our port.

- MAC address of Peering router
- MAC address of the port they are connected to on switch A

Is there any way to prevent switch A from presenting the interface MAC address? Or is this a symptom of spanning tree that cannot be stopped?

Your input will be most welcome.

The config on switch A is as follows:

interface GigabitEthernet0/5
description Peering Link
switchport access vlan 148
switchport mode access
speed nonegotiate
storm-control broadcast level 5.00
no cdp enable
spanning-tree portfast
spanning-tree bpdufilter enable
spanning-tree guard root

Regards

Simon Brilus

2

The peering exchange has an MoU that only 1 MAC address should be visible on
their switch. However they see 2 MAC addresses on our port.

- MAC address of Peering router
- MAC address of the port they are connected to on switch A

Is there any way to prevent switch A from presenting the interface MAC
address? Or is this a symptom of spanning tree that cannot be stopped?

Has it been confirmed that the violating frames are in fact STP BPDUs?
Could be all kinds of other autoconfig crap (CDP, VTP, etc.).

Don't know your VLAN setup and I certainly don't know that much about
Cisco L2 features. Is it possible to have topology groups with a master
VLAN (the one that does STP) and member VLANs that don't speak STP? If
so, that may be a way to keep STP traffic from coming out of your IX
port (barring vendor bugs).

Your input will be most welcome.

Is it absolutely necessary to run STP on the edge of the IX?

-- Steven

3

Hi Simon,

so you have:

IX---SwitchA---SwitchB---Router

why not disable spanning tree? There is no redundancy here anyway so disable it
in that particular VLAN.

Steve

4

IX---SwitchA---SwitchB---Router

ok, i gotta ask. you folk really do this on exchanges? i guess
so. well, if you're gonna shoot people for carrying backpacks,
i guess shooting yourselves and eachother in the foot is small
change, even if the coins are larger.

randy

5

I seem to think I've seen people doing this at most exchanges ISC has installed an F-root node at. The motivation is usually the avoidance of either expense ports on exchange switches, or the avoidance of expensive ports on routers (trunking the IX in as a vlan to a router on an existing port).

I'm not saying that the practice is good, or recommended, or without peril. But it's certainly not isolated to the UK.

Joe

6

I'm not saying that the practice is good, or recommended,
or without peril. But it's certainly not isolated to the
UK.

perhaps it should be :slight_smile:

as folk from all over read this list, i just could not let
discussion of how to do something that is generally broken
and quite ill-advised go without raising a red flag.

i would recommend all my competitors do this, except, as it
is being done to exchange meshes, it is damaging to all.

randy

7

What is the problem with this for the IXP, assuming proper
safeguards are in place which are best practice anyway (BPDU
filters, port security, ...)?

Which rule would you suggest for the IXP? The naive "connect
only routers" wouldn't do of course in nowaday's world of
hybrids.

Robert

8

What is the problem with this for the IXP, assuming proper
safeguards are in place which are best practice anyway (BPDU
filters, port security, ...)?

Hello Robert :slight_smile:

Which rule would you suggest for the IXP? The naive "connect
only routers" wouldn't do of course in nowaday's world of
hybrids.

I think the 'connect only routers' adage is probably a good conservative
motto to stick to. There are situations where connecting switches and
hybrids to IXPs is certainly more efficient and better suited, but only if
you know what you're doing or have a good reason for it. As I understand it,
most IXPs are pretty well protected against guff coming from switches these
days anyway, but it still doesn't make sense in my mind to have a free for
all on what people can connect. At least this adage might make someone who
might not be experienced in what they're doing think twice and ask someone
who knows better before doing it (as indeed it seems to have done in this
case).

Robert

Cheers,
Chris.

9

I've been following this with interest:

* How do you differentiate between a switch/router and a router? A lot of
people are deploying C76xx as peering routers these days because of the
cheapness of the ports (including 10G) on that particular problem.

* A lot of people with vendor J kit take a dot1q encaps GigE to a switch
and break out 100M ports, because 100M ports aren't very cost effective on
the M-series platform.

The rule on LINX always used to be "no switches", until the switch/router
convergence dilemma.

There was also an issue of keeping business (especially business we wanted
to keep!) on the IX, that wanted to connect through LAN extensions and
various other more colourful methods, so we had to find a way of
permitting this while mitigating the risks.

There's a fairly competitive IX environment in London, and the other
exchanges didn't seem to be too bothered by switches, especially if it
helped them attract participants from the larger exchange ;).

So, the rule at LINX is now "one port, one source MAC, any more and we
shut you down".

This deals with all the usual evils, such as ad-hoc extensions (or resold
connections) to the peering platform (a prime concern), incoming
STP/CDP/VTP, etc., and broken routers/switches (which end up spewing crap
- C75xx used to melt spectacularly in this way). At the same time it
permits switch/routers, and various LAN/Ethernet extension services, as
long as they are sensibly managed and configured.

We used to police this policy semi-manually, but now the switch vendors do
decent hardware-based port-security/mac-locking functionality, so that
does it for us, and actually does it pretty well.

- The switch learns the first address received on the interface, which
should be the first ingress frame (usually an ARP generated by the router
sending a BGP Open), and remembers it (with a 3 minute ageing time).

- This has the affect of applying an acl to the port (in hardware), which
permits traffic from the "good" address, and drops frames from other
addresses.

- Should more than 100 different source MACs be learned (99 of which will
be filtered and dropped) on the interface, the port will then log a
critical violation and shut the port down.

It works pretty well, it prevents all the usual badness we'd normally
associate with switches on the IXP.

So at the end of the day, it looks like we've been able to find a happy
medium, maintaining decent "hygiene", while being able to let people
indulge in deploying switches if they so choose.

Cheers,
Mike

10

There is no technical reason why you can't hook up as many switches as you
need, is there any real difference between a L3 switch and a L3 router
(except for its internal architecture and maybe a couple of 0's at the end
of the price :P). There are only good products, and bad products, smart
people, and stupid people. Stupid people running bad products will find a
way to leak stupid stuff to the IX and screw things up royally, regardless
of the type of product connected. Smart people running good products
USUALLY won't, no matter how many layer 2 and 3 switches stand between a
router and an IX port.

Of course I think part of the qualification for being considered a smart
person involves being able to connect switches to IX's without blowing
anything up, so those results might be a little biased. "Only connecting
routers" is really just attempting to mitigate the effects of stupid
people by forcing them to run configurations so simple "even a monkey
could pull it off".

11

yick hybrids...

12

* How do you differentiate between a switch/router and a router?

Ooh, that's easy: just look at the crap they spew towards the peering
fabric. :slight_smile:

A lot of people are deploying C76xx as peering routers ...

<rant>
... which should be prohibited by law. Actually, C76xx should be
prohibited by law.
</rant>

... At the same time it permits switch/routers, and various
LAN/Ethernet extension services, as long as they are sensibly managed
and configured.

Right. It does give complications wrt. trouble-shooting connectivity
though. Switch port up but no traffic (L2 black-holing), a single L2
provider with a loop or misconfiguration causing port security
violations on all of their customers' ports, communication channels when
those problems come up, etc. Having said that, so far it's still pretty
manageable.

-- Steven

13

Steven Bakker wrote:

A lot of people are deploying C76xx as peering routers ...

<rant>
... which should be prohibited by law. Actually, C76xx should be
prohibited by law.
</rant>

i know the current sport de jour in nanog is vendor bashing - but what specifically do you see as faults in the c6500/7600?

off-list or on-list - i don't mind - although don't want to violate any nanog policies.

cheers,

lincoln.

14

[ the voice of experience speaks ]

We used to police this policy semi-manually, but now the switch vendors do
decent hardware-based port-security/mac-locking functionality, so that
does it for us, and actually does it pretty well.

- The switch learns the first address received on the interface, which
should be the first ingress frame (usually an ARP generated by the router
sending a BGP Open), and remembers it (with a 3 minute ageing time).

- This has the affect of applying an acl to the port (in hardware), which
permits traffic from the "good" address, and drops frames from other
addresses.

- Should more than 100 different source MACs be learned (99 of which will
be filtered and dropped) on the interface, the port will then log a
critical violation and shut the port down.

It works pretty well, it prevents all the usual badness we'd normally
associate with switches on the IXP.

So at the end of the day, it looks like we've been able to find a happy
medium, maintaining decent "hygiene", while being able to let people
indulge in deploying switches if they so choose.

thanks! this approaches reassuring. why does it tolerate 100
macs? at first blush, i would think three or four would be a
bad enough sign.

randy

15

> A lot of people are deploying C76xx as peering routers ...

<rant>
... which should be prohibited by law. Actually, C76xx should be
prohibited by law.
</rant>

I've done my share of Cisco bashing in the past - but I have to say
that 6500/7600 worked pretty well as peering routers at my previous
employer. Great capacity, hardware forwarding, hardware ACLs and
policing. Handled Gbps sized DDoS attacks just fine.

6500/7600 as MPLS PE routers is another story altogether...

Steinar Haug, Nethelp consulting, sthaug@nethelp.no

16

It's a balance to avoid unduly penalising a genuine mistake, or being too
severe against some poor guy with a router which is still forwarding but
has an interface in it's death throes (and is occasionally generating
bursts of crap frames), and making his problems even worse.

In our experience, you either see a handful of macs caused by there being
a shakily configured switch-router attached, a slightly larger number of
macs caused by something being broken, or a couple of hundred due to
either a physical loop being applied or leaking other vlans (true
badness).

It's also a relatively sensible default when you apply the "restrict"
behaviour.

Cheers,
Mike

17

Mike, All,

I know the changes the LINX has implemented, and I am
curious... and this might affect other folk as well.

What is better - the LINX approach (blocking the port,
trying again in x minutes when too many MACs were seen)
or the Equinix approach (we hardcode your MAC per VLAN/
per port if untagged, all else we just drop)?

Alexander

18

Much of a muchness really.

With the former approach, it's easier for the participants to effect
changes to their IX equipment without having to ask the IX operator to
clear the locking/reconfigure the static MAC.

The protection against badness is pretty equal, whatever you do.

Cheers,
Mike

19

I like watching some of those hybrids boot though. It is fun to watch all the different layers of processors send their little boot messages to the console. It is so very reassuring to know that it takes an 8 layered cake to make my router function.

I had a few of 7609s at one network and I was, actually, quite happy with the performance. We had one failure of an OC12 card and that was it as I recall. Of course this tells me very little about performance in "bad" situations. If we had had a network meltdown it would have been interesting to observe the behavior of those switches in comparison to the other equipment.

It felt very weird building VLANs inside a router (and I made sure we only did it once so I would not wake up at night and scream).

We used RPR+ which was pretty nice. And you pretty much can't shake a stick at an interface card without it popping up with an Ethernet interface.

I should probably summarize a bit...

a. I still don't really like hybrids and feel they are too disjointed in their approach to survive in the long term.
b. The 7609 hybrid was not nearly as bad as I feared.
c. I would likely NOT purchase a hybrid if I could avoid it.
d. You will probably find that a hybrid is the most cost effective option and will have to fight your finance folks who know a thing or two about routers and switches! Well, I should say they know whatever the vendor is telling them behind your back<grin>.
e. You may not have to fight with finance as you may personally realize that you want another toy and purchasing the hybrid leaves you with enough capital to purchase the other toy and take the potential issues as a risk.

20

[ this seems to have been in an edit buffer for a while ]

fantasies about using 1918 space without leaking. and folk never
leaking igp<->bgp, and pigs flying, and cash falling from the sky.

Of course I think part of the qualification for being considered
a smart person involves being able to connect switches to IX's
without blowing anything up

i don't give a <bleep> if they're considered smart, green, or
gardenia. people who think they're smart and l33t probably do more
damage than those of us who are 0ld, st00pid, and 1azy. cleverness
is how disasters are made. testosterone poisoning is often fatal,
but it's impolite to take your neighbors with you.

in a world where over 90% of ix peers' net configs are not rigidly
automated, <bleep> is gonna happen. and it has, time and again.
the only stuff that makes me feel at all safe is what mike hughes
of linx described, or something even stricter, but i bow to mike's
experience.

and folk wonder why the grown-ups use pnis for anything important.

randy