Just out of curiosity, are folks just applying the Cisco patch or do you go through some sort of testing/validation process to ensure that the patch doesn't cause any other problems? Given typical change management procedures how long is taking you to get clearance to apply the patch?
I'm trying here to gauge the length of time before this vulnerability is closed out.
irwin
I don't see a lot of choice with this exploit. It's easy and quick. I
knocked down a 7206 in less than a minute. Expedite where you can.
BobG
most providers can easily go from (for example)
12.0(21)S3 to 12.0(21)S7 with less testing than from 12.0(21)S to 12.0(25)S
The hurdles are still there to maintain the necessary
customer notifications, etc.. but aside from that, I think the
press is doing their job (good or bad) in that most customers are
aware that there's something bad going on and people are moving
to protect the internet infrastructure.
- jared
I'm trying here to gauge the length of time before this vulnerability is closed out.
The core routers have been bouncing as they upgrade all this week. A lot of places
will be putting the fixes in place during windows this weekend.
And some 30% of the routers won't ever get upgraded. We're going to be finding
vulnerable routers on eBay for years.
See Eric Rescorla's paper on how fast the OpenSSL vulnerabilities of a while ago
were actually fixed:
12.0(21)S* (at least S5 and above) have broken SNMP interface counters
and Cisco refuses to fix the bug in 12.0(21)S*, so people who don't
want to lose money (accounting) are forced to upgrade to 12.0(25)S*.
I guess they want to force all "conservative" ISPs to jump over
the 12.0(22)S "barrier".
Some things won't be forgotten (see also recent discussion about the
new non-"Cisco"-GBIC blocking). Voting with pockets takes place.
Regards,
Daniel
> most providers can easily go from (for example)
> 12.0(21)S3 to 12.0(21)S7 with less testing than from 12.0(21)S to 12.0(25)S12.0(21)S* (at least S5 and above) have broken SNMP interface counters
and Cisco refuses to fix the bug in 12.0(21)S*, so people who don't
Do you have a DDTS I can reference?
want to lose money (accounting) are forced to upgrade to 12.0(25)S*.
I guess they want to force all "conservative" ISPs to jump over
the 12.0(22)S "barrier".
I agree that Cisco should actually take more serious ownership
of these issues within a customers network. They're selling us
these software/hw and claiming that we can obtain a particular SLA
level. Yet they can't seem to add in some code that says
if (ifc->in_bps > ifc->phy_speed || ifc->out_bps > ifc->phy_speed)
{
crash_router();
}
If they added this code, they'd find these bugs in their
labs instead of in our networks.
- jared
Just out of curiosity, are folks just applying the Cisco patch or do
you go through some sort of testing/validation process to ensure that
the patch doesn't cause any other problems? Given typical change
management procedures how long is taking you to get clearance to apply
the patch?
I think a lot of providers are doing the minimum testing required
(upgrade, does it boot? can I ping?) and then rolling it out... I
guess it's more of an implicit trust type deal rather than having
routers running with an increased load because of ACL's and still having
the possibility of a vulnerable router because something was
overlooked..
Going forward, if you go the ACL route, every time you add a new
interface you need to be sure to either apply the "any any" acl to it,
or add the new ip's to the "big" block by ip acl ...
I'm trying here to gauge the length of time before this vulnerability
is closed out.
I don't think it will ever truly go away.. there are lots of "older"
routers that won't be able to support the newer code, albeit small
routers like the 2500's, but they'll exist..
I don't think it will ever truly go away.. there are lots of "older"
routers that won't be able to support the newer code, albeit small
routers like the 2500's, but they'll exist..
Yes I have some old routers (2500s) for which no code exists which is patched
and small enough to sit in the flash/memory... acl acl !
Steve
> I don't think it will ever truly go away.. there are lots of "older"
> routers that won't be able to support the newer code, albeit small
> routers like the 2500's, but they'll exist..Yes I have some old routers (2500s) for which no code exists which is patched
and small enough to sit in the flash/memory... acl acl !
And given the low processing power of those routers, acl's hurt ... 
Not handy, but from cisco-nsp Archives I've found CSCea35259 and
CSCdy30984, and a reference to CSCea63754 which I can't take a look
at in BugToolkit.
Symptom: SNMP output octet counter stops counting traffic (except
some control plane traffic it seems), with every few days jumping
by weird amounts producing such funny things like 150mbps spikes on
a FE interface.
I've seen a box with a nicely loaded FE (30-70mbps) which took
(reproducably) just about 48 hours to have this interface stop counting.
If this would have been a customer interface, it would have meant
"reload router every two nights or lose money".
This bug is supposed to be (finally) fixed in 12.0(25)S1.
Given that you a) don't want to lose money and b) don't want to
do two whole-network upgrades within a short time, going to 12.0(21)S7
to fix the vulnerabilty is no real option, so people are more or less
forced to put their networks on bigger risk by going from 12.0(21)S*
to (25)S1.
Regards,
Daniel
if (ifc->in_bps > ifc->phy_speed || ifc->out_bps > ifc->phy_speed)
{
crash_router();
}If they added this code, they'd find these bugs in their
labs instead of in our networks.
I remember seeing an article claiming that Cisco�s automated regression
testing does "more than 250000" tests before they release a piece of code.
However, questions about the nature of these tests and if any tests sent
more traffic than a random scripted ping went unanswered.
Pete
I'm running 12.0(25.2)S, and it has the bug REALLY squashed.
LER
This has me wondering if there are any BCPs that touch on the whole idea
of filtering traffic destined to your router, or what the advisory called
"infrastructure filtering". All in all, it seems like a good idea to
block any direct access to router interfaces. But as some have probably
found already, it's a big pain in the arse.
If I recall correctly, Rob's Secure IOS Template touches on filtering
known services (the BGP listener, snmp), but what are people's feelings on
maintaining filters on all interfaces *after* loading a fixed IOS?
Thanks,
Charles
Some high-end boxes already have thing called "receive filter" which
helps this a lot. Hope we see more of that or better yet router vendors
stop processing packets they shouldn�t be processing anyway much
earlier in the code path. "Be liberal what you accept" should not apply here.
Pete
Hi Charles,
* spork@inch.com (Charles Sprickman) [Fri 18 Jul 2003, 22:21 CEST]:
If I recall correctly, Rob's Secure IOS Template touches on filtering
known services (the BGP listener, snmp), but what are people's feelings
on maintaining filters on all interfaces *after* loading a fixed IOS?
You'll have to weigh the benefits to the downsides.
Benefits to filtering IP packets with those four protocols:
- You're protecting your network, and possibly others, from failure
- You can see pretty quickly when someone's trying to attack you or
other networks
Downsides:
- You're hampering the use of several technologies
- Possible impact on the load / forwarding capacity of your router
(dependent on its architecture)
Personally I'd try to filter packets destined for known router
interfaces and let the rest pass through. And of course not run
known-buggy software (famous last words...).
-- Niels.
This has me wondering if there are any BCPs that touch on the whole idea
of filtering traffic destined to your router, or what the advisory called
"infrastructure filtering". All in all, it seems like a good idea to
block any direct access to router interfaces. But as some have probably
found already, it's a big pain in the arse.If I recall correctly, Rob's Secure IOS Template touches on filtering
known services (the BGP listener, snmp), but what are people's feelings on
maintaining filters on all interfaces *after* loading a fixed IOS?
It shouldn't be done. transit internet providers should not
be the edges firewalls. The edge? They can filter what they
want, but you should not filter things for people that they
don't know is being filtered. I can see a few clear cases where this
is acceptable, and ms-sql was one of them.
Something that might be of interest and of possible value for
something like this in the future would be this:
draft-marques-idr-flow-spec-00.txt
Take a look at it. It would allow access-list/firewall-filters
to be able to be deployed across your network in the matter of
a few minutes instead of having to login to every router. Plus
you could count the packets too via snmp. (well, assuming that works
right 
)
- Jared
* jared@puck.Nether.net (Jared Mauch) [Fri 18 Jul 2003, 23:23 CEST]:
If I recall correctly, Rob's Secure IOS Template touches on filtering
known services (the BGP listener, snmp), but what are people's feelings
on maintaining filters on all interfaces *after* loading a fixed IOS?It shouldn't be done. transit internet providers should not
be the edges firewalls. The edge? They can filter what they
want, but you should not filter things for people that they
don't know is being filtered. I can see a few clear cases where this
is acceptable, and ms-sql was one of them.
Good point. Still, transit networks' ingress routers could filter on
destination addresses of nodes known not to run IP protocols
53/55/77/103 in order to protect them.
I suppose most networks have a limited number of ranges they use for
assigning space to loopback and point-to-point interfaces so this
needn't be an extreme amount of administration.
Regards,
-- Niels.
* jared@puck.Nether.net (Jared Mauch) [Fri 18 Jul 2003, 23:23 CEST]:
>> If I recall correctly, Rob's Secure IOS Template touches on filtering
>> known services (the BGP listener, snmp), but what are people's feelings
>> on maintaining filters on all interfaces *after* loading a fixed IOS?
> It shouldn't be done. transit internet providers should not
> be the edges firewalls. The edge? They can filter what they
> want, but you should not filter things for people that they
> don't know is being filtered. I can see a few clear cases where this
> is acceptable, and ms-sql was one of them.Good point. Still, transit networks' ingress routers could filter on
destination addresses of nodes known not to run IP protocols
53/55/77/103 in order to protect them.
hrm, what nodes don't run 55/53/77/103? What do? Do you have a list? Could
we have it?
Seriously though... the edge networks (as Jared pointed out) should be
able to decide what they want to filter and what they don't... perhaps
some large ISP would decide you don't want any traffic from 212/8 or
perhaps all porn? Or all religious material? You don't want someone
deciding what you do and don't get... unless that someone is you 
I suppose most networks have a limited number of ranges they use for
assigning space to loopback and point-to-point interfaces so this
needn't be an extreme amount of administration.
yes... inside my network I know what my loopbacks and links are, inside
yours?? No idea... or Jared's or Tim Battles or...
* chris@UU.NET (Christopher L. Morrow) [Sat 19 Jul 2003, 01:03 CEST]:
hrm, what nodes don't run 55/53/77/103? What do? Do you have a list? Could
we have it?
I'm sure you know what devices in your network run Mobile IP or Sun ND
(to paraphrase Randy Bush, you can probably count them on the fingers
of your nose).
Router#conf t
Router(config)#ip receive-acl 10 no-idiocy
Seriously though... the edge networks (as Jared pointed out) should be
able to decide what they want to filter and what they don't... perhaps
some large ISP would decide you don't want any traffic from 212/8 or
perhaps all porn? Or all religious material? You don't want someone
deciding what you do and don't get... unless that someone is you
That's why I said that transit networks could filter only towards their
own infrastructure.
yes... inside my network I know what my loopbacks and links are, inside
yours?? No idea... or Jared's or Tim Battles or...
Luckily it's not your responsibility to protect them (only to intervene
when advised they're under attack, which I've heard you're doing a very
good job at - but that aside).
Regards,
-- Niels.
As mentioned before, Receive Path ACL (rACL) is already in 12.0(21)S2 (and
forward) for the GSR and 12.0(24)S for the 7500. This is one way of doing
infrastructure filtering without packet filtering the data plane (customer
traffic). The second phase of Receive Path ACL (rACL) is going everywhere.
The marketing name is Control Plane Protocol (CPP) ... but it also takes
care of any packet punted to the receive path (i.e. packet with destination
address = to the router). It is MQC based (ACL + rate-limiting). Think of it
as a "TCP wrapper" for the receive path - but with the rate-limiting. The
rate limiting part is important.
It will first show up in 12.2S (and forward) and then cross-port/back-port
through customer pressure (talk to your Cisco Account Teams). You'll see it
on everything for the small boxes (26XX) to switches (CAT6Ks) to the high
end (GSRs).
Personally, I see this "TCP Wrapper with Rate-Limit" around a router as
something that is going to be a requirement for all vendors on the Net.