Come in this morning to find one failover pair of ASA's had the primary crash and failover, then a couple hours later, the secondary crash and failover, back to the primary.
Another pair running the same code had the primary crash and fail in the same time window.
So, three crashes in 4 hours in our environment.
Open a TAC case on one of these for post-mortem analysis, and they interpreted the crash dump to point at a DOS bug first published in Oct.
The very interesting thing; on the phone the TAC engineer said this was "the 10th one of these I've dealt with this morning".
Here's the bug they reference:
https://tools.cisco.com/bugsearch/bug/CSCul36176/?reffering_site=dumpcr
Anyone else have observations to add on this?
Mark Mayfield
City of Roseville - AS 54371
Network Systems Engineer
2660 Civic Center Drive
Roseville, MN 55113
651-792-7098 Office
Come in this morning to find one failover pair of ASA's had the primary crash and failover, then a couple hours later, the secondary crash and failover, back to the primary.
Another pair running the same code had the primary crash and fail in the same time window.
So, three crashes in 4 hours in our environment.
Open a TAC case on one of these for post-mortem analysis, and they interpreted the crash dump to point at a DOS bug first published in Oct.
The very interesting thing; on the phone the TAC engineer said this was "the 10th one of these I've dealt with this morning".
Here's the bug they reference:
https://tools.cisco.com/bugsearch/bug/CSCul36176/?reffering_site=dumpcr
Anyone else have observations to add on this?
Not sure about ASA-specific DoS and the bug you're pointing at, but we saw
some NTP reflection this morning. Then there's the WSJ, NYSE, and UAL from this morning as well. Rough day on the internets?
Not sure it’s related but I’ve read reports on FRNoG of ASAs crashing as well, seems related to a late leap second related issue.
Regards, Michel
Thank you sir. I read your presentation quite some time ago, probably one of the first times you posted to the list. It has definitely informed many of my design processes; particularly with regard to server publishing, and been a major part of my supporting documentation in arguments with others at my organization over the last few years.
Of course, these particular ASA implementations are for law enforcement applications, so we are mandated to implement in ways that auditors from the state and federal agencies approve of.
However, this makes me consider the need to more aggressively ACL inbound traffic at the router level before these particular firewalls, which I can do, and may help mitigate such events, so thank you for the reminder!
Mark Mayfield
City of Roseville - AS 54371
Network Systems Engineer
2660 Civic Center Drive
Roseville, MN 55113
651-792-7098 Office
This is pretty scary when you take into account that the NYSE is still down.
Kirill Klimakhin
Principal Consultant
120 Seventh Street
Suite 202
Garden City, NY 11530
(C) 631-707-3303
(F) 631-982-0174
Kirill.Klimakhin@corebts.com
www.corebts.com
NANOG members:
Hi there. This is Dario Ciccarone from the Cisco PSIRT - the Product
Security Incident Response Team. This is to acknowledge we're aware of
this issue, and we're working with all the appropriate parties.
Indeed, it seems the culprit is Cisco bug ID CSCul36176 - which was
released as part of the Cisco Security Advisory "Multiple
Vulnerabilities in Cisco ASA Software ", which was published on October
8th, 2014. The full advisory is available at the following URL:
http://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20141008-asa
As I said, the Cisco PSIRT is working with the Cisco Technical
Assistance Center on this matter, and we're analyzing the available
information. The advisory will be updated to reflect the fact we're
seeing active exploitation of this issue.
NANOG members are welcome to contact us at psirt@cisco.com if they
have any additional questions or concerns, or any information relevant
to this issue.
Thanks,
Dario
We call all relax. The Commander-in-Chief of the USA has declared this to be a technical glitch, and not a security breach or attack.
However, this makes me consider the need to more aggressively ACL inbound traffic at the router level before these particular firewalls, which I can do, and may help mitigate such events,
Spot-on - reduce the state-surface as much as possible.
so thank you for the reminder!
Sorry for the repeat, but glad the preso was helpful!
;>
Really just people not patching their software after warnings more than six months ago:
July-08 UPDATE: Cisco PSIRT is aware of disruption to some Cisco customers with Cisco ASA devices affected by CVE-2014-3383, the Cisco ASA VPN Denial of Service Vulnerability that was disclosed in this Security Advisory. Traffic causing the disruption was isolated to a specific source IPv4 address. Cisco has engaged the provider and owner of that device and determined that the traffic was sent with no malicious intent. Cisco strongly recommends that customers upgrade to a fixed Cisco ASA software release to remediate this issue.
Cisco has released free software updates that address these vulnerabilities. Workarounds that mitigate some of these vulnerabilities are available.
Jared Mauch
Hi Jared,
thanks for update
do you know provider/source ip of the source of the attack ?
Colin
My guess is a researcher.
We saw the same issue in the past with a Cisco microcode bug and people doing ping record route. When it went across a LC with a very specific set of software it would crash.
If you crashed just upgrade your code, don't hide behind blocking an IP as people now know what to send/do. It won't be long.
Jared Mauch
you would think a researcher would stop once he realised effect being caused ?
Colin
I’m sure they did. It could also have been any number of other things. I’m just guessing. It could have been someone trying to scan their enterprise too and went a bit rogue.
Not everyone reads NANOG believe it or not 
Either way, if you haven’t upgraded for a 9 month old security advisory, shame on you. I don’t care what your change management process looks like, it’s bordering on network malpractice IMHO.
- Jared
Really just people not patching their software after warnings more than six months ago:
A lot goes into "updates". Not the least of which is *knowing* about the issue. Then getting the patched code, then lab testing, then regulatory approval(s), then maintenance window(s)...
Cisco has released free software updates that address these vulnerabilities. Workarounds that mitigate some of these vulnerabilities are available.
"Free" if you have a support contract. (the clause 3 "contact TAC" method is all too often a serious pain in the ass.)
--Ricky
Really just people not patching their software after warnings more than six months ago:
A lot goes into "updates". Not the least of which is *knowing* about the issue. Then getting the patched code, then lab testing, then regulatory approval(s), then maintenance window(s)…
Not my first rodeo.
Once again, it’s been since October 2014. If you failed to pay your credit card bill from October 2014 you can’t expect it to work either.
Cisco has released free software updates that address these vulnerabilities. Workarounds that mitigate some of these vulnerabilities are available.
"Free" if you have a support contract. (the clause 3 "contact TAC" method is all too often a serious pain in the ass.)
I’ve never had issues getting them to open a case for this hardware. You can either operate responsibly or not.
I wouldn’t be surprised if the situation gets worse. Either way, upgrade/patch/silo as necessary.
- Jared
No, free-as-in-beer.
You register a guest CCO account, email tac@cisco.com, provide the device
serial number (or output of "show hardware") and the bugid + Cisco PSIRT
URL reference. Cisco TAC will then provide you with a download link with
fixed software, at no cost to you. It's not a pain in the ass - it works fine.
Nick
I wouldn't classify someone sending known malicious traffic towards someone else's network device attempting to crash it as a 'researcher'. Criminal is a better term.
Chuck
There are other terms for people who don’t maintain their equipment, it’s usually described as negligent.
If my hardware were rebooting, I would be red-faced first about not having done something and not blaming someone outside.
I don’t know if it was a researcher or something buggy sending packets or anything else. (I have no unique direct insight).
What I do know is the ASAs under my control and purview had no issues. Take the free upgrade and move on folks.
- Jared