Cisco and the tobacco industry

I think there is also a LOT concern about all the unpatched routers that
remain unpatched simply because the admins don't feel like spending a week
running the cisco gauntlet to get patches when you don't have a support
contract with cisco. Its like cisco doesn't want you to patch or they would
make it easy.

Geo.

This is oh so true - contracts in order to patch your equipment. Normally
I would never mention the need for an authority to intervene on things
related to the Internet but how long will it be before the term "Digital
Pearl Harbor" is a reality.

Maybe it is time an authority figure steps in and makes some form of rules
for vendors to distribute fixes under some form of law. If this flaw of
Cisco's could lead to the kind of severe damage as Mr. Lynn claims,
shouldn't it fall on the shoulders of Cisco to get their act together and
provide a fix as opposed to sending in the hounds (legal shmoes via
lawsuit) to quash their problems.

I'm sort of taking a look at it from the tobacco company lawsuit stance
where the tobacco bigwigs would bury the truth in legal trash as opposed
to making things right. It's rather irresponsible behaviour on the part of
Cisco to avoid coming clean on this issue.

On matters of a public exploit and or the skill level necessary to create
an attack via whatever flaw Mr. Lynn spoke of: It is only a matter of time
before something is out there, so for some to criticize Mr. Lynn for being
a whistleblower, shame on you. I think he did a courageous thing.

* J. Oquendo:

Maybe it is time an authority figure steps in and makes some form of rules
for vendors to distribute fixes under some form of law. If this flaw of
Cisco's could lead to the kind of severe damage as Mr. Lynn claims,
shouldn't it fall on the shoulders of Cisco to get their act together and
provide a fix as opposed to sending in the hounds (legal shmoes via
lawsuit) to quash their problems.

But it looks as if Cisco actually did this, and you (and Geo) just
weren't part of the elite circle of operators whose networks are
considered U.S. national critical infrastructure.

Cisco always has provided free upgrades to non-contract holders
for security bugs.

  eg:

http://www.cisco.com/en/US/products/products_security_advisory09186a008042d51b.shtml

-- snip --
Please have your product serial number available and give the URL of this notice as evidence of your entitlement to a free upgrade. Free upgrades for non-contract customers must be requested through the TAC.
-- snip --

  Now the fact that there has been no advisory (yet) means
no free upgrade (yet?).

  This is much kinder than other companies have done where you
can't get squat.

  Now, for the doomsdayers, yes, it's likely we'll have something
nasty happen to the internet at some point. Yes, it'll disrupt 911 and
other critical services (finance, health care, etc..) but without people
taking active responsibility to the equipment they own and operate, the
question is who will get hurt and how bad.

  We do security testing on our IOS images and have found
bugs that have been reported to PSIRT and fixed "quietly". They've
been fairly good at solving the issues. I think with anytime I deal
with a vendor, promptness is always an issue, I'd always like a fix in a
few days, they never seem to move as fast as one would want.

  If you don't do testing of your images, I suggest you create
a plan and add it to your qualification procedures. Even if you don't
have a current contract, you can get free upgrades if you find a PSIRT
bug, perhaps that should make everyone *want* to help Cisco.

  Then again, there have been issues for years where this happens,
I encourage everyone to beat on their routers (in the lab) and work with
your vendors to solve the problems and not run around creating massive
amount of chaos, we've all seen what that does.

  - jared

Jared,

Have you ever actually tried to get the updates using this method? It really
does take the better part of a week and no less than half a dozen emails or
phone calls and then there is the begging...

Geo.

George Roettger
Netlink Services

Cisco always has provided free upgrades to non-contract holders
for security bugs.

eg:

http://www.cisco.com/en/US/products/products_security_advisory09186a008042d

51b.shtml

-- snip --
Please have your product serial number available and give the URL of this

notice as evidence of your entitlement to a free upgrade. Free upgrades for
non-contract customers must be requested through the TAC.
-- snip --

The point is you did get the update, right? It's better than
no update. As far as what happens, I've found the TAC underperform
my expectations in every possible situation, what you say above
doesn't shock me.

  - jared

I have, on at least two occasions I remember, and I don't recall it being
that big a deal, fill out the form, I don't recall if I even had to speak to
anyone, and I received the link to the image.

Jared,

Have you ever actually tried to get the updates using this method? It really
does take the better part of a week and no less than half a dozen emails or
phone calls and then there is the begging...

if it's critical to your business you'd think you'd have a support
contract for it, eh? (or you decided that the 'better part of a week' and
associated risk was an acceptable cost to your business)

('you' in the royal sense, not 'you geo')

No, the point is if you want the internet to be patched then you can't
torture people when they come to you for the patches.

Cisco routers are being sold to every company who connects to the internet,
it's one step up from consumer products. You can't expect every company who
owns a cisco router to buy an expensive contract or be willing to go thru
the gauntlet to get the patches.

Cisco needs to come up with a better way.

If your point is simply that it's possible to get the patches, well it's
possible to code them yourself too if you know assembler.

Geo.

George Roettger
Netlink Services

Have you ever actually tried to get the updates using this method? It

really

does take the better part of a week and no less than half a dozen emails

or

phone calls and then there is the begging...

  The point is you did get the update, right? It's better than
no update. As far as what happens, I've found the TAC underperform
my expectations in every possible situation, what you say above
doesn't shock me.

  - jared

In a message written on Thu, Jul 28, 2005 at 04:51:18PM -0400, Geo. wrote:

Cisco routers are being sold to every company who connects to the internet,
it's one step up from consumer products. You can't expect every company who
owns a cisco router to buy an expensive contract or be willing to go thru
the gauntlet to get the patches.

Cisco needs to come up with a better way.

In a message written on Thu, Jul 28, 2005 at 08:29:38PM +0000, Christopher L. Morrow wrote:

if it's critical to your business you'd think you'd have a support
contract for it, eh? (or you decided that the 'better part of a week' and
associated risk was an acceptable cost to your business)

Unfortunately Chris, that doesn't match how (small) business works.
I had to hold up Microsoft as an example of being a good corporate
citizen, but here it goes. If a 10 person company buys Windows XP
and runs it in their office they get free Windows Updates patches
for the "life" of the product (typically around 5-7 years). There
is no TAC or other system to go through, you just tell the box to
update and it does it.

Now, I'm not suggesting a large ISP would go with this model, but
Cisco has moved out of the core and into small edge and SOHO routers,
VOIP phones, and all sorts of other gizmos being bought by home
office users and small companies who don't buy support for their
other technology items, but get updates. Heck, even digital camera
makers and such put free firmware updates on their web site.

Expecting all of these users to buy a support contract that costs,
what, $350/year for a $2500 box is absurd. Even full tilt talk to
a real person with on-site service dell support is only around
$120/year.

There is a reason all of these boxes are running around unpatched.
Look at the percentage of windows boxes, which have auto-update
software, and free updates that are patched. Now think about the
routers out there, where there is no update software, and no free
updates. It should surprise no one that there are thousands of
routers on the ends of T1's and DS-3's running code 2-6 years (or
more) old, vulnerable to any number of things.

Why is Cisco so scared of this one? Well, before now hacking them
was low value. You could DDOS a 5 person company off the air, maybe
reboot their router with a vulnerability -- which frankly many of
them wouldn't notice. However, now they can be added to the zombie
army of your choice. From being able to simply trigger a flood
ping remotely to being able to upload a remote controllable module
it's all possible now.

Cisco knows a lot of these small offices don't have support. They
don't have someone who knows how to upgrade code on a Cisco. For
Cisco to actually upgrade a lot of these boxes (assuming people are
informed, and know to demand an upgrade) under their current system
means tens of thousands of tac calls from people who've never logged
into a router before needing to be walked through downloading code
and upgrading a router. Millions, if not tens of millions in support
costs.

Will all of these people demand it? Who knows. The popular press
picking up the issue is a huge step to alerting joe random with a
small office and a 2501 in the corner he should pay attention, but
it's probably not enough. If a hacker manages to take over twenty
or thirty thousand routers though....I suspect a flood of calls
Cisco's direction.

Software has bugs. Deal with it. Sometimes you have to pay for updates to
fix those bugs. If you don't like it, find another vendor. Except - all
vendors do that, don't they? Well, I guess if your business model isn't
compatible with purchasing support contracts on vital gear, you may not have
a viable business. YMMV.

Cisco's conduct in this case may or may not be improper - we'll have to wait
for a little more information. From a PR point of view, they probably should
have let things ride and allowed the Blackhat talk to occur. They look like
bullies now, which is never good. Hindsight is 20/20, though.

That being said, their policy of offering free updates for certain bug fixes
to those who don't pay them for support is generous. See that hand feeding
you? Don't bite it.

No, the point is if you want the internet to be patched then you can't
torture people when they come to you for the patches.

Cisco routers are being sold to every company who connects to the internet,
it's one step up from consumer products. You can't expect every company who
owns a cisco router to buy an expensive contract or be willing to go thru
the gauntlet to get the patches.

Sorry, but its a traditional part of the product model for
telecommunications equipment. PBX's, routers, pretty much everything -
support contract required. Sure, you could have it a different way, but you
would have to be willing to pay significantly more up front to pay for that
ongoing support. Its not like the vendors are deceiving anyone here - a
support contract is listed on the quote for pretty much every new piece of
gear you buy from a vendor.

Take it from Ice-T - "don't hate the player, hate the game". Words to live
by.

[snip]

Geo.

George Roettger
Netlink Services

Daniel Golding

Sorry, but its a traditional part of the product model for
telecommunications equipment. PBX's, routers, pretty much everything -
support contract required. Sure, you could have it a different way, but

you

would have to be willing to pay significantly more up front to pay for

that

ongoing support.

What ongoing support, just put the fixes on an ftp site. Cisco's problem is
they aren't patches, they are full versions. If they created an exe file
that attached via tcp/ip to the router and just changed the bits that needed
changing instead of requiring a whole new build be loaded it wouldn't be
such an issue to just leave the patches out there on cisco.com so anyone
with a router could get them without costing cisco anything but a bit of
bandwidth.

Look, it's up to Cisco how they do this but if DHS wants this country's
infrastructure to be secure then Cisco is going to need to realize that a
whole lot of people are not going to be willing to pay to fix product
defects and they're not going to be willing to spend days trying to get
those fixes for free.

Perhaps after a few router worms it will make more sense. Oh and I don't
know about you but if I buy a PBX and a flaw in it allows any remote caller
to make outbound calls at my expense, you can bet money that I'm going to
expect a flaw like that to be fixed free of charge, contract or not.

Geo.

What ongoing support, just put the fixes on an ftp site. Cisco's problem is
they aren't patches, they are full versions. If they created an exe file
that attached via tcp/ip to the router and just changed the bits that needed

The ability to connect to the router and push a software change? Let's think
this through a bit, shall we? :wink:

Perhaps after a few router worms it will make more sense.

Your mail header says:

X-mailer: Microsoft Outlook Express 6.00.2800.1506

Now, what were you saying about a few worms causing *ANY* change in behavior? :wink:

The ability to connect to the router and push a software change? Let's

think
this through a bit, shall we? ;)<<

Who said push? I said cisco's whole patch method is to move people to a new
version of IOS instead of patching the old version. Cisco charges for new
versions so it's not in their financial interest to make new versions
available for free like the patches need to be. So I suggest they employ a
different patch method, you download an exe from their ftp site, it takes
your current build which is stored on your computer, patches it, and uploads
it to your router or you then upload it to your router. Since this would
require you already have the image they could continue to manage their image
distributions as they do now. I mean your issue is not impossible to work
around.

X-mailer: Microsoft Outlook Express 6.00.2800.1506

Now, what were you saying about a few worms causing *ANY* change in

behavior? :wink:

it's amazing how safe software can be when used by a professional, isn't it?
Everyone here knows you have a woodie for OE by the format of your posts
which appear as attachments instead of normal text in OE. I notice that
behavior hasn't changed either <g>. Nuff said?

Geo.

George Roettger
Netlink Services

available for free like the patches need to be. So I suggest they employ a
different patch method, you download an exe from their ftp site, it takes
your current build which is stored on your computer, patches it, and uploads
it to your router or you then upload it to your router.

Your original suggestion was that it push it to the router. Security-wise, this
is very different from the router pulling it. (Hint - consider the authentication
issues, not only for a correctly set up machine, but for likely misconfigurations
actually seen out in the field).

Everyone here knows you have a woodie for OE by the format of your posts
which appear as attachments instead of normal text in OE. I notice that
behavior hasn't changed either <g>. Nuff said?

My behavior hasn't changed because my MUA has been able to understand the
formats originally defined in RFC1847 and RFC2015, as updated by RFC3156, for
over a decade now. If you don't like it, complain to your vendor, or find
a vendor who can follow the RFCs. Or you can fix it yourself by visiting
http://www.openpgp.org/resources/downloads.shtml and finding a plugin for your
MUA. A number of them are listed at http://www.gnupg.org/(en)/related_software/frontends.html#win

Curse the dark, or light a match. You decide, it's your dark.

available for free like the patches need to be. So I suggest they employ
a different patch method, you download an exe from their ftp site, it
takes your current build which is stored on your computer, patches it,
and uploads it to your router or you then upload it to your router.

Your original suggestion was that it push it to the router.
Security-wise, this is very different from the router pulling it. (Hint -
consider the authentication issues, not only for a correctly set up
machine, but for likely misconfigurations actually seen out in the field).

In any case, making router software updates dependent on Windows is a _REALLY_
bad idea. Why should I be locked into Micr0$0ft just because I bought a
piece of backbone hardware from Cisco (or any other vendor).

In general, I try to avoid any vendor that requires me to have stuff from
any specific other vendor. If you can't comply with open standards for
interoperability, your hardware doesn't belong in my network.

My behavior hasn't changed because my MUA has been able to understand the
formats originally defined in RFC1847 and RFC2015, as updated by RFC3156,
for over a decade now. If you don't like it, complain to your vendor, or
find a vendor who can follow the RFCs. Or you can fix it yourself by
visiting http://www.openpgp.org/resources/downloads.shtml and finding a
plugin for your MUA. A number of them are listed at
http://www.gnupg.org/(en)/related_software/frontends.html#win

Well said... It's really tiresome that so many users think the world
should comply with and accommodate their errors.

Curse the dark, or light a match. You decide, it's your dark.

I like that... I will probably plagiarize it. :slight_smile:

Owen

Your original suggestion was that it push it to the router.<<

Ok I guess it could be read that way but I was more suggesting they look for
a way to patch not upgrade to a new version. I've been around the industry
long enough to have seen Autodesk use the exe patch routine to patch
existing files right on disk so I know this is nothing new. My original
suggestion was to take that one step further and patch right in memory on
the router but if that's a security issue then fine patch the image on disk
and upload it like normal, makes no difference to my point.

My behavior hasn't changed because my MUA has been able to understand the

formats originally defined in RFC1847 and RFC2015, as updated by RFC3156,
for
over a decade now.<<

Yeah yeah, I've had this discussion several times, it's a bug in my software
and you couldn't give a darn if you are doing something that is incompatible
with what 90% of the world uses for email because you are right and everyone
else is wrong. Such is the spirit of the internet huh? (you picked on my use
of OE first, I was just responding)

Geo.

George Roettger
Netlink Services

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Folks.

All that is needed is for cisco to put an "upgrade" command into their
router. The "upgrade" command determines the routers version (and
current patch level) and requests the download of a version specific
patch file.

The command takes as arguments the on-disk (flash) version of the core
image and the beginning of a URL where to find the file. The filename
itself can be constructed based on the current version. The upgrade file
itself contains the checksum of the image it should be applied against
as well as the checksum of the final image. Of course it is digitally
signed by cisco (so Cisco will need a public key installed in its images).

The upgrade command then determines if sufficient flash exists to
perform the change and performs the upgrade. It might even be able to
patch in the in-core image (presumably this can be done via code that is
included in the patch itself, I leave this as an exercise for cisco).

The actual patch file can be located in a server at the customer's site
and Cisco can distribute them via BitTorrent :slight_smile:

Important points:

* Upgrade is initiated by the user. If the necessary arguments are
stored in the system configuration, perhaps the upgrade can be triggered
by SNMP even (yeah right).
* All patches are signed.
* Patches know what version they apply to and are careful to ensure they
are being applied to the right version (even if the customer improperly
names the files on their server).

This isn't trivial to do, but it isn't rocket science either!

      -Jeff

- --

This isn't trivial to do, but it isn't rocket science either!

True, but you ARE suggesting that Cisco produce a binary patch, to a
possibly compressed image.

I think you should really think long and hard before you conclude that
you really want that. IMHO, the risk/reward ratio as compared to just
downloading a full image is all wrong.

Tony

Just because 90% of the people in the world are stupid, does that mean that we all have to be stupid as well? If nine out of ten people jumped off a bridge, should the other guy be forced to do the same?