Hello,
I have several Catalyst 6500 (Supervisor 32) aggregation switches with WS-X6148A-GE-TX and WS-X6148-GE-TX line cards.
These line cards do not support storm-control/broadcast suppression. This impacted us badly during a recent spanning tree event.
As it stands, we are at risk of overwhelming control planes with excess broadcast or multicast traffic, and I need to find alternative ways to protect these switches.
I have been researching STP enhancements, and control-plane policing in the following documents, and would appreciate advice from engineers who may have had to implement similar workarounds for storm-control in a service provider setting.
* Configuring Denial of Service Protection
http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/configuration/guide/dos.pdf
* Configuring Control Plane Policing
http://www.cisco.com/en/US/docs/switches/lan/catalyst4500/12.2/31sga/configuration/guide/cntl_pln.pdf
* Configuring Optional STP Features
http://www.cisco.com/en/US/docs/switches/lan/catalyst4500/12.2/31sga/configuration/guide/stp_enha.pdf
So, if we can't mitigate against STP events using storm-control or broadcast suppression, what might be the best combination of STP enhancements and control-plane policing?
For example, is it possible to rate-limit broadcast/multicast, STP and ARP on a per VLAN basis? If so, how?
Many thanks,
P
Peter,
This question would be better directed at cisco-nsp, but...
I have several Catalyst 6500 (Supervisor 32) aggregation switches with
WS-X6148A-GE-TX and WS-X6148-GE-TX line cards.
These line cards do not support storm-control/broadcast suppression.
This impacted us badly during a recent spanning tree event.
Not surprised. The 61xx cards are not service provider suitable line cards and they have proved this very clearly. Sorry to hear about these storms - they really are devastating, aren't they? But if you're running L2 customer facing services, particularly shared L2 domain access, there are two things you care about: storm control and port security (mac address counting). The 61xx cards don't do storm control.
For example, is it possible to rate-limit broadcast/multicast, STP and
ARP on a per VLAN basis? If so, how?
Yes, you replace your 61xx cards with 67xx cards. You can't do this sort of thing with qos or copp.
Nick
Chopping up the layer-2 broadcast domain for a given VLAN into smaller pieces via pVLANs can't hurt, either, as long as the hosts have no need to talk to one another - and it has other benefits, as well.
Roland Dobbins wrote:
Chopping up the layer-2 broadcast domain for a given VLAN into smaller pieces via pVLANs can't hurt, either, as long as the hosts have no need to talk to one another - and it has other benefits, as well.
Or you hit the extreme DSL concentrator end where you crank out q-in-q with roughly 1 vlan per customer (some equipment perhaps handling 1 to many with other built in security features) and let the router proxyarp between them.
Unnumbered vlans and RBE saved parts of my network from pending doom. Even fixed issues with dslams that overran the arp caches causing unicast broadcast storms, but the arp cache was irrelevant when it was 1 vlan per port.
I'm still waiting for other vendors to tell me how they can match that particular Cisco functionality.
Jack
Unless your broadcast storm happens on an untagged vlan. Or unless you're running VTP and also have ipv6 hosts connected to the switch on .1q tagged ports, and consequently have to disable VTP pruning in order to get said ipv6 .1q hosts to be able to talk to each other, and then if you have a broadcast storm on any vlan, it could hose your entire l2 network, because you've disabled vtp pruning.
Anyway, the point is: storm control on customer facing equipment is a basic requirement.
Nick
Yes, but this is evil and dangerous in a customer-facing environment; transparent mode is the preferred option, in most circumstances.
It is very evil, yes. SXH and later support VTPv3 which allows you to disable VTP on a per port basis. But as you say, not suitable for customer facing networks.
Nick
Yes, you replace your 61xx cards with 67xx cards. You can't do this sort
of thing with qos or copp.
The 67xx series cards aren't supported by the sup32, though. Would 65xx
line cards do the trick?
Andrew
The 67xx series cards aren't supported by the sup32, though. Would 65xx
line cards do the trick?
unfortunately not:
Catalyst 6500 Release 12.2SXF and Rebuilds Software Configuration Guide - Traffic-Storm Control [Cisco Catalyst 6500 Series Switches] - Cisco
� The following LAN switching modules do not support traffic storm control:
� WS-X6148A-GE-45AF
� WS-X6148A-GE-TX
� WS-X6148-GE-45AF
� WS-X6148-GE-TX
� WS-X6148V-GE-TX
� WS-X6548-GE-45AF
� WS-X6548-GE-TX
� WS-X6548V-GE-TX
Hmmm, expensive proposition to upgrade then - even though sup720 + ws-x67xx cards make a nice l2 platform for gig ethernet.
Nick
I understand why hosts need to send broadcasts. In a close/single customer environment, broadcasts could be useful. I hope most future protocol designers now think of using multicast or other
discovery mechanisms besides broadcast.
But in a service provider network (or any managed network), is there any reason why a customer needs to hear other customer's broadcasts? In practice, are there any useful broadcast messages in a multi-customer environment that can't/shouldn't be proxied by the network operator or handled other ways.
Not that I know of, ISPs have successfully done L2 isolation of customers for 10 years and I haven't heard of any problems with it.
Only bad part really is that if the customer is allowed several IPs and you use local-proxy-arp then traffic between customer computers will go via the ISP, which is one of the reasons I advocate the use of a home CPE router for IPv6, it's just a cleaner handoff.
In my opinion the Sup32 platform has some limitations when the
technology is considered for high data rate, deterministic carrier
customer-facing scenarios. Cisco sells the Sup32 as a wiring closet
aggregation switch the main purpose of which is to connect desktop users
to central core switches. In addition to the lack of
storm-control/broadcast suppression mentioned below, the 61XX line cards
also have a limit of, I believe, 2 ports in an Etherchannel.
Additionally, and perhaps most significantly for deterministic network
design, the copper cards share input hardware buffers for every 8 ports.
Running one port of the 8 at wire speed will cause input drops on the
other 7 ports. Also, the cards connect to the older 32 Gbps shared bus.
In my view, with high data rates, it is difficult, if not impossible, to
build a deterministic network with Sup32s. Cisco's solution for
designing a deterministic network is the sup720 which has a 720 Gbps
crossbar bus. The 67XX 48 port copper line cards have 2 20 Gbps
connectors to the 720 Gbps bus, the 24 port fiber sfp line cards have a
single 20 Gbps connector to the crossbar bus, and the 10 GiG 67XX line
cards have 2 20 Gbps bus connectors. The crossbar bus connects line card
connectors to each other in a point-to-point fashion. 67XX line cards
are adequately provisioned with input and output buffers. There is still
40/48 (1 GiGE copper), 20/24 (1 GiGE sfp), and 40/160 (10 GiGE X2)
oversubscription of ports to switch fabric connectors, however. Sup720
routing table lookups via Ternary Content Addressable Memory (TCAM)
still use the 32 Gbps bus to access the TCAM to search for next hop for
destination IP network.
The hardware-based FPM is interesting.
Additionally, and perhaps most significantly for deterministic network
design, the copper cards share input hardware buffers for every 8 ports.
Running one port of the 8 at wire speed will cause input drops on the
other 7 ports. Also, the cards connect to the older 32 Gbps shared bus.
IMO, a more serious problem with the 6148tx and 6548tx cards is the internal architecture, which is effectively six internal managed gigabit ethernet hubs (i.e. shared bus) with a 1M buffer per hub, and each hub connected with a single 1G uplink to a 32 gig backplane. Ref:
Troubleshoot Hardware and Issues on Catalyst 6500/6000 Switches - Cisco
In Cisco's own words: "These line cards are oversubscription cards that are designed to extend gigabit to the desktop and might not be ideal for server farm connectivity". In other words, these cards are fine in their place, but they are not designed or suitable for data centre usage.
I don't want to sound like I'm damning this card beyond redemption - it has a useful place in this world - but at the expense of reliability, manageability and configuration control, you will get useful features (including broadcast/unicast flood control) and in many situations very significantly better performance from a recent SRW 48-port linksys gig switch than from one of these cards.
Nick
We experienced the joy of using the X6148 cards with a SAN/ESX cluster.
Lots of performance issues! A fairly inexpensive solution was to switch to
the X6148A card instead, which does not suffer the the 8:1
oversubscription. It also supports MTU's larger than 1500, which was
another shortcoming of the older card.
Mike
Actually, the "A" variant of the x6148 is still 8:1 oversubscribed. The significant difference between the x6148 and x6148a is the buffer size. The original card had 1.4MB of buffer per port group (8 ports) while the "A" upgrade supports 5.5MB per port. Oh, that and support for 9k jumbo frames.
It's still a classic bus card, it still has the same QoS queues, and is still 8:1 over subscribed.
David
...