NSA able to compromise Cisco, Juniper, Huawei switches

Hi Folks -

Clay Kossmeyer here from the Cisco PSIRT.

We've published the following document in response to the original (Dec. 29) Der Spiegel article:

http://tools.cisco.com/security/center/content/CiscoSecurityResponse/cisco-sr-20131229-der-spiegel

and are investing the claims in the Dec. 30 Der Spiegel article referencing 'persistent implants' for the PIX and ASA product lines under case number PSIRT-1384943056.

Any vulnerabilities we discover will be disclosed via our standard vulnerability handling process documented here:

http://www.cisco.com/web/about/security/psirt/security_vulnerability_policy.html

I'm not currently subscribed to NANOG, so if you have a reply you'd like me to see, please copy me directly.

Regards,

Clay

Clay Kossmeyer here from the Cisco PSIRT.

shoveling kitty litter as fast as you can, eh?

http://tools.cisco.com/security/center/content/CiscoSecurityResponse/cisco-sr-20131229-der-spiegel

"The article does not discuss or disclose any Cisco product vulnerabilities."

this is disengenuous at best. from the nsa document copied in der
spiegel and now many other places:

  "JETPLOW is a firmware persistence implant for Cisco PIX series and
   ASA firewalls ..."

so in cisco kitty litter lingo, what would be "discuss[ing] or
disclos[ing] any Cisco product vulnerabilities? the exploit code
itself?

randy

What is the vulnerability in Cisco product Randy?
That a 3rd party can replace the firmware in your firewall?
There isn't enough information to determine if this is a software
vulnerability triggered with exploit code or wholesale firmware
replacement. The document refers to an implant but not how it gets there.

* Randy Bush:

Clay Kossmeyer here from the Cisco PSIRT.

shoveling kitty litter as fast as you can, eh?

http://tools.cisco.com/security/center/content/CiscoSecurityResponse/cisco-sr-20131229-der-spiegel

"The article does not discuss or disclose any Cisco product vulnerabilities."

this is disengenuous at best. from the nsa document copied in der
spiegel and now many other places:

  "JETPLOW is a firmware persistence implant for Cisco PIX series and
   ASA firewalls ..."

There's a limit to what can reasonably be called a *product*
vulnerability. If you physically plant a bug in a phone, does it
exploit a vulnerability in the phone? I don't think so.
Theoretically, the manufacturer could have filled it completely with
glue. But the next step up is drilling out some of that to place the
bug, and then you're looking at tamper evidence, and that's an
extremely difficult matter.

Routers are expected to be modular, so it's difficult to avoid that
they have exposed buses with something that approaches DMA capability.
On-site debugging hooks through JTAG ports or similar might be
essential to reduce downtime in case of severe problems, so I doubt
one can get rid of them. Same for firmware downgrade and recovery
options.

In the end, the defense has to be political, not technical. "We don't
want to do this because it's wrong", and not "we can't do this because
it's impossible". After all, what's possible can change very quickly.
Appeasement in the form of lawful intercept turned out to be failure:
even if you comply, it's likely that your own, domestic intelligence
agencies consider your infrastructure, you and your colleagues
legitimate targets.

There's a limit to what can reasonably be called a *product*
vulnerability.

right. if the product was wearing a low-cut blouse and a short skirt,
it's not.

it's weasel words (excuse the idiom). shoveling kitty litter over a big
steaming pile.

let me insert a second advert for jake's 30c3 preso,
https://www.youtube.com/watch?v=b0w36GAyZIA

randy

+1

NSA states very clearly this is baked in and ³widely deployed². Either
Cisco is not very happy with their government overlords today, or they are
having long meetings at those oversized conference tables trying to figure
out what to tell everyone. I¹m curious about the implications to the US
DoD STIG¹s that are put out, as I¹m fairly sure they do not mention there
is a backdoor that anyone who knows how to knock can access.

My other question is.. How are they identifying unique ASA and PIX? Is
there a fingerprint mechanism that tells it what¹s going on? I¹d think
there would be quite a few admins out there with really weird syslog
entries??

Randy is right here.. Cisco has some Œsplainin to do - we buy these
devices as ³security appliances², not NSA rootkit gateways. I hope the .cn
guys don¹t figure out what¹s going on here, I¹d imagine there are plenty
of ASA¹s in the .gov infrastructures.

//warren

PS - I mentioned .cn specifically because of the Huawei aspect, in
addition to the fact that it has been widely publicized we are in a ³cyber
war² with them.

Clayton is responding to the ability that he's allowed, and he's using words very precisely.

Here's Cisco's official responses, so far.

<http://blogs.cisco.com/news/comment-on-der-spiegel-articles-about-nsa-tao-organization/>

<http://tools.cisco.com/security/center/content/CiscoSecurityResponse/cisco-sr-20131229-der-spiegel>

I know both Clay and jns quite well, and they're both straight-shooters.

The best response I've seen to all this hype and I completely agree with
Scott:

"Do ya think that you wouldn't also notice a drastic increase in outbound
traffic to begin with? It's fun to watch all the hype and things like
that, but to truly sit down and think about what it would actually take
to make something like this happen, especially on a sustained and
"unnoticed" basis, is just asinine.

Perhaps more work should be spent maintaining ones own equipment and
network than debating the chances that the sky may actually be falling or
the NSA hunting your ass down. :wink: Just my two cents for the day!
Happy New Year!

Scott Morris, CCIEx4 (R&S/ISP-Dial/Security/Service Provider) #4713, CCDE
#2009::smiley:,

CCNP-Data Center, CCNP-Voice, JNCIE-SP #153, JNCIE-ENT #102, JNCIS-QFX,
CISSP, et al.

IPv6 Gold Certified Engineer, IPv6 Gold Certified Trainer

CCSI #21903, JNCI-SP, JNCI-ENT, JNCI-QFX

swm@emanon.com

Knowledge is power.

Power corrupts.

Study hard and be Eeeeviiiil......"

Jonathan

<http://blogs.cisco.com/news/comment-on-der-spiegel-articles-about-nsa-tao-organization/>

<http://tools.cisco.com/security/center/content/CiscoSecurityResponse/cisco-sr-20131229-der-spiegel>

* Randy Bush:

There's a limit to what can reasonably be called a *product*
vulnerability.

right. if the product was wearing a low-cut blouse and a short skirt,
it's not.

Uh-oh, is this an attempt at an argument based on a "blame the victim"
rape analogy?

Hopefully, this drives home the importance of all the various BCPs like iACLs, isolated jump-off boxes for interactive access, config-file management, and network telemetry - including visibility into DCN/OOB traffic.

There are open-source tools out there which can be used for these purposes. It doesn't require a lot of capex, mainly opex - i.e., elbow-grease.

it's weasel words (excuse the idiom). shoveling kitty litter over a
big steaming pile.

Clayton is responding to the ability that he's allowed, and he's using
words very precisely.

qed

The best response I've seen to all this hype and I completely agree with
Scott:

"Do ya think that you wouldn't also notice a drastic increase in outbound
traffic to begin with? It's fun to watch all the hype and things like
that, but to truly sit down and think about what it would actually take
to make something like this happen, especially on a sustained and
"unnoticed" basis, is just asinine.

A drastic increase, definitely. Smaller increases (say a couple of Mbps
on a link normally carrying 100 Mbps or more), doubtful.

It all depends on the volume of the information you're looking for.

Steinar Haug, Nethelp consulting, sthaug@nethelp.no

More than you know.

As someone who has seen firsthand, in real time, an adversary exfiltrate
documents and other data out of an organization which he has gained
unauthorized internal access -- real professionals know how to blend in
with the noise & fly under the radar successfully.

- - ferg

Explaining, not a denial written by their legal department. I find it
insanely difficult to believe cisco systems has a backdoor into some of
their product lines with no knowledge or participation. Given the fact
that RSA had a check cut for their participation (sell outs..), would it
be out of the realm of possibility cisco knowingly placed this into their
product line? And would it be their mistake to come out with a “we had no
idea!” rather than “guys with badges and court orders made us do it!”?

Google has some deniability, as their networks were compromised without
their knowledge. Placing code into a PC BIOS or IOS image is a far
different beast than asking a fiber provider to give a split to a
governmental agency. Secret squirrel wires with secret squirrel modulation
techniques isn’t a surprise to me, what is a surprise to me is the level
of acceptance the IT community has shown thus far on NANOG.

On a side note, I found it unbelievable the NSA was so pissed off about
aeronautical access being hard to capture. The initial article made it
seem like they had already gotten ahold of the data, which would have
really pissed me off. If it’s really that difficult, I have a NSA proof
satellite platform with capacity should anyone need it.. :wink:

//warren

* Warren Bailey:

Explaining, not a denial written by their legal department. I find it
insanely difficult to believe cisco systems has a backdoor into some of
their product lines with no knowledge or participation.

As far as I understand it, these are firmware tweaks or implants
sitting on a privileged bus (think PCI with busmaster DMA). Such
things can be added after the device has left the factory by a
sufficiently knowledgeable third party.

That's really interesting. Where are these Cisco devices manufactured?

- - ferg

China. :wink: lol

that RSA had a check cut for their participation (sell outs..), would it
be out of the realm of possibility cisco knowingly placed this into their
product line? And would it be their mistake to come out with a “we had no
idea!” rather than “guys with badges and court orders made us do it!”?

Is this legal? Can NSA walk in to US based company and legally coerce to
install such backdoor? If not, what is the incentive for private company to
cooperate?

If legal, consider risk to NSA. Official product ran inside company to add
requested feature, hundred of people aware of it. Seems both expensive to
order such feature and almost guaranteed to be exposed by some of the
employees.

Alternative method is to presume all software is insecure, hire 1 expert whose
day job is to search for vulnerabilities in IOS. Much cheaper, insignificant
risk.

Which method would you use?

techniques isn’t a surprise to me, what is a surprise to me is the level
of acceptance the IT community has shown thus far on NANOG.

This seems like generalization, majority opinion seems to be, government has
no business spying on us.

Someone contacted me yesterday, after reading how I'd love to see some of
these attacks dissected and analysed to gain higher quality data than
screenshot of PDF.
He told me, he and his employer are cooperating with their vendor right now
looking at attack done against router they operate and claimed they are aware
of other operators being targeted. Unfortunately he couldn't share any
specifics, so hopefully we'll soon have situation where someone can dissect
publicly any of the attacks.

If this is as widespread as claimed, and if we'll gain knowledge how to see if
you are affected, there are potentially repercussions on geopolitical scale,
as I'm sure many on these lists would go public and share information if
they'd find being targeted.

Warren Bailey <wbailey@satelliteintelligencegroup.com>

I find it insanely difficult to believe cisco systems has a backdoor
into some of their product lines with no knowledge or participation.

actually, i suspect a mix of both, the usg encouraging calea gone bad
(while committing to bad-mouth huawei), and the TAO crew developing
serious attacks based on unintended product vulnerabilities.

Google has some deniability, as their networks were compromised
without their knowledge.

i doubt we will ever learn the extent of surprise vs culpability of
google, apple, twitter, msoft, ...

Saku Ytti <saku@ytti.fi>

Is this legal?

ROFL

If this is as widespread as claimed, and if we'll gain knowledge how
to see if you are affected, there are potentially repercussions on
geopolitical scale, as I'm sure many on these lists would go public
and share information if they'd find being targeted.

we are dealing with a world in which there are attackers and victims and
very few white hats to be seen. exposure via journalism, thanks
@ioerror, wikileaks, ... and constructive hacking to make protocols and
products more resistant are the main paths available to us.

and if you want to be ambarrassed for our peers, see the ietf pissing
all over itself deciding whether they can make simple statements that
these things are attacks and the ietf needs to do something about its
protocols.