Networking Pearl Harbor in the Making

< CRAPAGANDA >

Which operating system, embedded in more than 80% of enterprise IT
environments, represents one of the fastest-growing hacker targets and
potentially the most-devastating information-security vulnerability? Hint:
It ain't Windows. Cisco Systems' Internetwork Operating System now sits at
the center of the information security vortex. Because IOS controls the
routers that underpin most business networks as well as the Internet,
anyone exploiting its flaws stands to wreak havoc on those networks and
maybe even reach into the computer systems and databases connected to
them. IOS is a highly sophisticated piece of software, but--as with
Microsoft's Windows--that's a double-edged proposition. Software
complexity can be a hacker's best friend.

http://www.informationweek.com/story/showArticle.jhtml?articleID=173402976&tid=5979%2C5989

EOC

I think in general this is an argument against converged networks,
the added complexity and outages may not be worth the gains..

  - jared

It is an argument for proper patching policy and procedures. There is no zero day exploit for this exploit and to my knowledge, there hasn't been one yet which came out at the same time as the advisory for ANY major vendor although the window is shrinking. All worms and other exploits which have achieved press coverage and caused major network disruption would have been avoided by proper patching. All of our network is now patched for the latest Cisco advisory. We were already running fixed code on a few routers when the advisory came out so we knew the code was stable and moved to it on all other boxes. I understand that not everyone can act as quickly as we do, but to delay patching indefinitely until the problem occurs - for "stability" reasons is not the solution either. Better code is part of the solution and teaching and enforcing proper programming techniques to create secure code in the first place are just part of the solution. Getting people to install (so far) secure code is another bigger problem which can be solved today. I think all the major vendors are aware of the extent of the problem and are making their systems more secure by auditing their existing code more thoroughly as well as teaching their programmers to code securely in the first place.

-Robert

Tellurian Networks - The Ultimate Internet Connection
http://www.tellurian.com | 888-TELLURIAN | 973-300-9211
"Well done is better than well said." - Benjamin Franklin

Robert,

All of our network is now patched for the latest Cisco advisory. We were
already running fixed code on a few routers when the advisory came
out so we knew the code was stable and moved to it on all other
boxes.

I'm not exactly "in the know" on this one, but the heap-overflow advisory
that we've seen indicates that the IOS updates Cisco put out are not patches
for this problem:

  "Cisco has devised counter-measures by implementing extra checks to
  enforce the proper integrity of system timers. This extra validation
  should reduce the possibility of heap-based overflow attack vectors
  achieving remote code execution."
from Networking, Cloud, and Cybersecurity Solutions - Cisco

We've asked Cisco for a better explanation - namely, are their recommended
updates "patches" to the problem (i.e. repairs) or simply mitigating
updates that make is harder to exploit. The wording of their advisory seems
to indicate the latter. This latter case is what worries me since it implies
that there is a fundamental problem in IOS, the problem still exists even after
patching, and that Cisco can't readily repair it. Unfortunately, so far we've
gotten the run-around and haven't been able to get a better answer, again
leading me to believe the worst.

Eric :slight_smile:

Looks like vendor J is going to benefit from the issues laid out for
Vendor C.

http://www.networkworld.com/news/2005/110405-juniper-cisco-hacker.html

Cisco, Juniper, or vendor "X". We all benefit by having "genetic diversity" in our routing/switching systems. I have been bit hard, as many of us on this thread have been bit, by bugs in vendor software/hardware. Support your IETF! Don't use proprietary protocols and insist on interoperability. If you have the wherewithal install at least two different vendors for your critical services. Then make them play nice together!

There, now I feel much better... glad to get that off my chest.

And yes, I actually have put my money where my mouth is and built a stable and efficient dual core with Cisco and Juniper running MPLS together. To be fair I was a bit wimpy about installing some of the latest greatest tricks though <grin>.

Regards,

Blaine

How do the operators/engineers explain to 'management', or whomever asks,
the 'training issues' that always crop up when more than one vendor are
proposed? Has anyone had good luck with this arguement? (my answer is sort
of along the lines of: "Its just a router, no matter the vendor and they
all have command-line help" but that's not always recieved well :slight_smile: )

Just curious as I'm sure there are folks stuck in an all vendor X shop who
look over the electronic fence and see vendor Y with 'so much better' or
'so much faster' or 'so much more blinkly lighty'... and try to have their
management agree to purchasing new devices :slight_smile:

http://www.networkworld.com/news/2005/110405-juniper-cisco-hacker.html

Cisco, Juniper, or vendor "X". We all benefit by having "genetic
diversity" in our routing/switching systems. I have been bit hard,
as many of us on this thread have been bit, by bugs in vendor
software/hardware. Support your IETF! Don't use proprietary
protocols and insist on interoperability. If you have the
wherewithal install at least two different vendors for your critical
services. Then make them play nice together!

How do the operators/engineers explain to 'management', or whomever asks,
the 'training issues' that always crop up when more than one vendor are
proposed? Has anyone had good luck with this arguement? (my answer is sort
of along the lines of: "Its just a router, no matter the vendor and they
all have command-line help" but that's not always recieved well :slight_smile: )

Just curious as I'm sure there are folks stuck in an all vendor X shop who
look over the electronic fence and see vendor Y with 'so much better' or
'so much faster' or 'so much more blinkly lighty'... and try to have their
management agree to purchasing new devices :slight_smile:

Well, the last time I just whined a lot ? <grin>

Seriously, we actually put together a presentation that described a series of major events that have occurred through the use of monoculture networks/systems and stated that for many financial/security reasons it is best to maintain at least two vendors.

We covered the following

o Security Implications: How monoculture networks/operating systems are prone to attack.
o Financial Impact: How managing multiple vendors can reduce long term expense.
o Stability: How monoculture networks/systems are prone to network/system wide failures.
o Viability: How existing technology is capable of interop and how we, the engineering team, were capable of making it happen.
o Customer demand: How our customers actually "felt better" about our service as a result of it's genetic diversity.

Regards,

Blaine

I would postulate that this is a fundamental problem in how machines represent and execute code.

How do the operators/engineers explain to 'management', or whomever asks,
the 'training issues' that always crop up when more than one vendor are
proposed? Has anyone had good luck with this arguement? (my answer is sort
of along the lines of: "Its just a router, no matter the vendor and they
all have command-line help" but that's not always recieved well :slight_smile: )

Just curious as I'm sure there are folks stuck in an all vendor X shop who
look over the electronic fence and see vendor Y with 'so much better' or
'so much faster' or 'so much more blinkly lighty'... and try to have their
management agree to purchasing new devices :slight_smile:

We have that exact problem, if one considers it to be a problem. We have only vendor X, and we've often wanted to try out others. However, the management arguments and opinions are often difficult to sway.

For one, we have very few problems that are ever seen by customers or management. So, convincing them that diversity could buy us better reliability is near impossible. Then the additional cost of training and spare equipment. We also have the customer opinions/perception to deal with (accompanied by marketing), where they think having a "Cisco Powered Network" would automatically mean it's the best.

Despite not having service impacting problems, we do have a number of "bugs", however would those issues get better or worse when having to deal with multiple vendors, various platforms per vendor, and inter-operability?

Well, the last time I just whined a lot ? <grin>

Seriously, we actually put together a presentation that described a series of major events that have occurred through the use of monoculture networks/systems and stated that for many financial/ security reasons it is best to maintain at least two vendors.

We covered the following

o Security Implications: How monoculture networks/operating systems are prone to attack.
o Financial Impact: How managing multiple vendors can reduce long term expense.
o Stability: How monoculture networks/systems are prone to network/ system wide failures.
o Viability: How existing technology is capable of interop and how we, the engineering team, were capable of making it happen.
o Customer demand: How our customers actually "felt better" about our service as a result of it's genetic diversity.

We covered only a couple of these areas, and maybe addressing some of the others might make a better case.

We have that exact problem, if one considers it to be a problem. We have only vendor X, and we've often wanted to try out others. However, the management arguments and opinions are often difficult to sway.

Something that often makes management understand the benefit is explaining how much more attentive your current vendor will become once you start implementing a competing vendor. I have had vendor that would barely give me the time of day suddenly give me 2-3 full time onsite SEs, a large pile of onsite spares, an additional 5-7% blanket discount, free training credits, T-shirts and and around $750k of "free" hardware to try and stop me moving to their competitor (as tempting as it may be, don't just cry wolf to get free stuff though!)

For one, we have very few problems that are ever seen by customers or management.

Well, in that case you should make sure that you are keeping management informed as to the deficiencies of the current vendor - as long as you know that your replacement vendor will not have similar issues :slight_smile:

So, convincing them that diversity could buy us better reliability is near impossible. Then the additional cost of training and spare equipment.

Talk to the new vendor - most vendors will be more than happy to give you free training credits to get you as a new customer, same goes for spares. Play up the free training to management - explain the importance of ongoing education, how eager the network engineers are to learn more, etc. This way they can give everyone training with no out of pocket expense!

We also have the customer opinions/perception to deal with (accompanied by marketing), where they think having a "Cisco Powered Network" would automatically mean it's the best.

I think that you might be pleasantly surprised by the increasing clue factor of your customers - they will understand that best-of-breed means that you get to choose the best device for the purpose. Your customers are involved in technology - while some of them like Linux and some like Windows, they all understand that different technologies have different strengths and weaknesses (ok, so maybe a true Linux zealot won't agreee that Windows is useful for anything - maybe emacs vs vi... hmmm, maybe not that either...)

Despite not having service impacting problems, we do have a number of "bugs", however would those issues get better or worse when having to deal with multiple vendors, various platforms per vendor, and inter-operability?

With multiple vendors you get to choose the device that best fits the requirements - if devices from multiple vendors fit equally well, you get to choose the one with the fewest bugs. You suddenly also have a lot more pull with your vendor to get your current "bugs" addressed.

I personally haven't found interoperability to be a major issue (providing you are not relying on any proprietary protocols) - the ability to choose the best device for the task more than pays for any interoperability concerns - you also get to call up the vendors and say "Fix this or I'm replacing it with a $other_vendor box".

There are also some things that certain vendors just don't do well - being limited to just choosing devices from a single vendor limits what you can do.

Warren

How do the operators/engineers explain to 'management', or whomever

asks,

the 'training issues' that always crop up when more than one vendor are
proposed? Has anyone had good luck with this arguement? (my answer is

sort

of along the lines of: "Its just a router, no matter the vendor and they
all have command-line help" but that's not always recieved well :slight_smile: )

It's a variation of the classic insource/outsource question. Do you
do the training in-house because it is mission critical to the business
or do you outsource it to a vendor focused training organization.
If you outsource training then you hire CCIEs to run a Cisco network
and JNCIEs to run a Juniper network. But it is not the only way.

If you choose to run a dual-vendor network such as Juniper and Cisco
then you could still outsource training but since the cost of training
is based on a model of one *CIE per person, it probably is not
cost-effective for each person to get both the CCIE and JNCIE.
Someone who considers training to be an inhouse responsibility
would not pay for employees to get their *CIE but would instead
pay for an internal training program.

I suspect that this inhouse training model will work best in small
and midsize companies but that larger networks simply can't afford
the complexity because they are less able to hire and maintain
clueful managers.

Just curious as I'm sure there are folks stuck in an all vendor X shop

who

look over the electronic fence and see vendor Y with 'so much better' or
'so much faster' or 'so much more blinkly lighty'... and try to have

their

management agree to purchasing new devices :slight_smile:

Management has probably made the vendor decision based on high-level
business reasons that involve strategy, stock price, and marketing
in addition to the low-level reasons of fitness for purpose and cost.

I don't expect that any large company will change unless some
crisis hits them that forces them to review their decisions.

--Michael Dillon

It really depends on your management. Good management will either
understand your arguments and support them or be non-technical enough that
they'll let you make the technical decision because that's what they're
paying you a lot of money to do.

Back in 1999, I was the director of IT Engineering for Macromedia. We were
putting up a new colo location at Exodus (the story of our relationship
with them is worthy of another rant), and had purchased something on the
order of $500K of Cisco equipment to provision it -- two Cisco 7513s and
two Cisco 6009s. Forgive me, but I can't reliably tell you why we chose
those specific models (six years ago!) -- but we needed to be able to
sustaing gigabit bandwidth and our Cisco VAR, in combination with the Cisco
district sales droids, said "get this!"

The provisioning took a while, during which time the equipment gathered a
bit of dust. Getting close to the installation date, our technical staff
(including me) took Exodus' technical staff to lunch during which they said
"Cisco ?! Ha! They're a piece of crap. You should look at Foundry." We
talked about it in more details, and left lunch persuaded we should look at
Foundry.

The next 2-3 weeks were interesting. Amusingly, our salesrep at a Cisco
VAR had left earlier to go join Foundry, so we already knew someone there.
We had him come in and discuss their product line, got some references from
him and talked to them (they were glowing), and priced out the competing
platform. We then told Cisco that we were goig to go with Foundry.
Amusingly, A) They did not deny their equipment could actually not sustain
gigabit speeds; B) An Exoduds SVP called my CIO to let him know that the
engineer I spoke with had 'over-communicated', that Exodus and Cisco had a
strong partnership, and that in no way would they actually recommend going
with someone else. Yeah, OK.

We spent about 25%-40%, I believe, of the Cisco price on the two BigIron
8Ks we got. We had to deal with some minor training issues, obviously, and
there were some things we were counting on being able to do that they
weren't able to do at the time -- I still remember the meeting where the
Foundry sales rep came in and said "I talked to my CEO about this feature
you need; we can't do it right now, but it's scheduled to be put in in
about 8 weeks. Meanwhile, we've gotten an OK to loan you this other
equipment that will let you do what you need." In the end, though, the
transition was remarkably smooth on a technical basis.

Up until this time we were a Cisco-only shop for any purchased equipment
(I'm making this subtle distinction because we still had to manage a bunch
of shared-media 10-baseT hubs in some of our older locations. Don't get me
started). Making the decision to go with Foundry required me to go to my
boss and have a grand total of one informal meeting with him that largely
consisted of "hey, I'm thinking of going with Foundry for the new colo. If
they work, I might start considering them for Corp use also. They're
cheaper, faster, and promise better service, and so far I see them
delivering on this."

Of course, I also promised that we'd be able to return the Cisco 7513s (we
repurposed the 6009s) -- and that's when Cisco started pissing us off with
an insane "no, wait, you've got to return it to the VAR ... But the VAR
needs our approval!" game. It was about 6 weeks of conference calls with
Cisco and our VAR and some meetings; apparently, the big issue was that we
had unpacked the routers and, much like Best Buy, Cisco required us to
return the equipment with all the original packaging. I suggested that
maybe, since we were looking at about $150K of equipment, we could actually
pay for replacement packaging, but no deal. Note that at the time we were
OK with just a credit -- we knew we were going to spend more money on Cisco
(at the time) anyway.

Long story short, one day my Foundry salesrep and his SE were visiting and
noticed I was pissed off. I told them about this issue I was having with
Cisco, and on the spot he offered to do a trade-in -- he'd give me my full
spent cost on the Ciscos for the 7513s as credit toward any Foundry
equipment. The day after, I rented a van and took the two 7513s to
Foundry's facilities, and dropped them off at shipping and receiving, where
I got some curious looks. That evening, when we called the Cisco district
manager and told him "don't worry about it -- we gave them to Foundry for a
credit," both my boss and I enjoyed the resulting shock and dismay.

So sometimes, moving away from being a one-vendor shop can be relatively
painless. Other than Cisco trying desperately to hold on to their
exclusivity in this case, we didn't really have too many problems. The key
was mutual trust within my organization, and the ability of each layer in
it -- my network engineers, me, my boss, my CIO -- to trust the other
layers and let them do their job*.

-roy

* No IT story is complete without an unhappy ending. A management shakeup
resulted in the replacement of the CIO, who ended up replacing management
with his own people. My replacement was a Cisco guy and they ended up
ripping out perfectly functioning Foundry equipment to put Cisco back in
there. Of course by then it wasn't my problem anymore, but I got to hear
the grumblings from my guys over beers.