First real-world SCADA attack in US

Mailman List Mirror (Read Only)
21

education / resource issue. The existing methods that have been used for
years with reasonable success in the IT industry can 'fix' this problem.

The "existing normal methods" used by much of the IT industry fail
way too often,
and therefore, some measure of regulation is in order, when the
matter is about critical
public infrastructure -- it's simply not in the public interest to
let agencies fail or use slipshod/
half measure techniques that are commonly practiced by some of the IT industry.

They should be required to engage in practices that can be proven to
mitigate risks
to a know controllable quantity.

The weakness of typical IT security is probably OK, when the only
danger of compromise
is that an intruder might get some sensitive information, or IT might
need to go to the tapes.

That just won't do, when the result of compromise is, industrial
equipment is forced outside
of safe parameters, resulting in deaths, or a city's water supply is
shut down, resulting in deaths.

Hard perimeter and mushy interior with OS updates just to address
known issues,
and malware scanners to "try and catch" things just won't do.

..."an OS patch introduces a serious crash bug" is also a type of
security issue.
Patching doesn't necessarily improve security; it only helps with
issues you know about,
and might introduce issues you don't know about.

Enumerating badness is simply not reliable, and patch patch patch is
simply an example
of that -- when security really matters, don't attach it to a
network, especially not one that
might eventually be internet connected -- indirect or not.

Connection to a management LAN that has any PC on it that is or was
ever internet connected
"counts" as an internet connection.

Industrial Controls systems are normally only replaced when they are so old
that parts can no longer be obtained. PC's started to be widely used as
operator interfaces about the time Windows 95 came out. A lot of those
Win95 boxes are still running and have been connected to the network over
the years.

The "Windows 95" part is fine.

The "connected to the network" part is not fine.

22

From: "Jimmy Hess" <mysidia@gmail.com>

> education / resource issue. The existing methods that have been used for
> years with reasonable success in the IT industry can 'fix' this
> problem.

Careful with the attribution; you're quoting Mark, not me.

The weakness of typical IT security is probably OK, when the only danger of compromise
is that an intruder might get some sensitive information, or IT might need to go to the tapes.

That just won't do, when the result of compromise is, industrial equipment
is forced outside
of safe parameters, resulting in deaths, or a city's water supply is shut down,
resulting in deaths.

(72 character hard wrap... please.)

Hard perimeter and mushy interior with OS updates just to address
known issues, and malware scanners to "try and catch" things just won't do.

Precisely. THe case in point example these days is traffic light controllers.

I know from traffic light controllers; when I was a kid, that was my dad's
beat for the City of Boston. Being a geeky kid, I drilled the guys in the
signal shop, the few times I got to go there (Saturdays, and such).

The old design for traffic signal controllers was that the relays that drove
each signal/group were electrically interlocked: the relay that made N/S able
to engage it's greens *got its power from* the relay that made E/W red; if there
wasn't a red there, you *couldn't* make the other direction green.

These days, I'm not sure that's still true: I can *see* the signal change
propagate across a row of 5 LED signals from one end to the other. Since I
don't think the speed of electricity is slow enough to do that (it's probably
on the order of 5ms light to light), I have to assume that it's processor delay
as the processor runs a display list to turn on output transistors that drive
the LED light heads.

That implies to me that it is *physically* possible to get opposing greens
(which we refer to, in technical terms as "traffic fatalities") out of the
controller box... in exactly the same way that it didn't used to be.

That's unsettling enough that I'm going to go hunt down a signal mechanic
and ask.

Cheers,
-- jra

23

Not necessarily. Microwave ovens have an interlock system that has 3
sequentially timed microswitches. The first two cut power to the oven,
and the third one shorts out the power supply in case the previous two
failed, blowing a fuse. The switches are operated by 2 "fingers" placed
on the door so that if the door is bent enough to not seal properly, the
switches will be activated in the wrong order causing the shorting
switch to operate. This can also happen if you slam the door closed too
hard.

This is all nice in theory, in practice the microswitches are so flimsy
nowadays that I'd not be too surprised if the shorting switch did not
succeed in blowing a fuse - and the other two will easily weld together
even in normal use (I have seen this happen. Swap the switches and fuse
and the oven works again.)

The traffic lights can also have some kind of fault-detection logic that
sees they are in an illegal state and latches them into a fault mode.

IMHO this is stupid extra complexity when relays are obviously 100%
correct and reliable for this function, but it seems to be all the rage
nowadays to use some kind of "proven correct" software system for safety
critical logic. It is so much sexier than mechanical or
electro-mechanical interlocks.

Anybody who has seen what kind of bizarre malfunctions failed
electrolytics cause in consumer electronics will probably not feel very
comfortable trusting traffic lights whose safety relies on software that
is proven correct. OTOH, the risk is astronomically small compared to
someone just running the red lights.

Jussi Peltola

24

I agree, it is mostly education and resources issue . But the
environment of control networks is slightly different from IT
industry, IMHO.

1) control network people have been living in a kind of isolation for
too long and haven't realized that their networks are connected to Big
Bad Internet (or at least intranet..) now so the threat model has
changed completely.
2) There aren't many published cases of successful (or even
unsuccessful) attacks on control networks. As a result, the risk of an
attack is considered to have large potential loss and but *very* low
probability of occurring and high cost of countermeasures =>
ignoring..
3) Interconnections between control networks and "normal" LANs are a
kind of grey area (especially taking into account that both types of
networks are run by different teams of engineers). It is very hard to
get any technical/security requirements etc - usually none of them
exist. And as the whole system as as secure as the weakest element....
the result is easily predictable.
4) any changes in control network are to be done in much more
conservative way. all those "apply the patch..oh, damn, it
crashed..rollback' doesn't work there. In addition (from my experience
which might not be statistically reliable) the testing/lab resources
are usually much more limited for control networks;
5) as the life cycle of hw&sw is much longer than in IT industry, it
is very hard to meet the security requirements w/o significant changes
to existing control network (inc. procedures/policies) - but see #4
above..

So there is a gap - those control networks are 10 (20?) years behind
internet in terms of security. This gap can be filled but not
immediately.

The good news that such stories as the one we are discussing could
help scary the decision makers..oops, sorry, I was going to say 'raise
the level of security awareness'

25

Beware of bugs in the above code; I have only proved it correct, not tried it.
                -- Donald Knuth

:slight_smile:

26

Actually, it should be a wake up call whether or not NSA had signals
information. However, it's pretty obvious that the entire SCADA segment is
pretty much bound and determined to keep hitting the snooze button as long as
possible - they've known they have an endemic security problem just about the
same number of years the telecom segment has known they will need to deploy
IPv6. :wink:

And let's think about this for a moment - given that there's *no* indication
that the attack was an organized effort from a known group, and could quite
possibly be just a bored 12 year old in Toledo Ohio, why should the NSA have
any signals info before the attack?

Let's think it through a bit more. Even if the NSA *did* have info beforehand
that pointed at a kid in Toledo, they can't easily release that info before the
fact, for several reasons: (a) they're not supposed to be surveillancing US
citizens, so having intel on a kid in Toledo would be embarassing at the least,
and (b) revealing they have the intel would almost certainly leak out the
details of where, when, and how they got said info - and the NSA would almost
certainly be willing to sacrifice somebody else's water pump rather than reveal
how they got the info.

Bottom line - the fact the NSA didn't say something beforehand means that they
either didn't know, or didn't wish to tell. So why are you bringing the NSA into it?

27

The typical implementation in a modern controller is to have a separate
conflict monitor unit that will detect when conflicting greens (for
example) are displayed, and trigger a (also separate) flasher unit that
will cause the signal to display a flashing red in all directions
(sometimes flashing yellow for one higher volume route).

So the controller would output conflicting greens if it failed or was
misprogrammed, but the conflict monitor would detect that and restore
the signal to a safe (albeit flashing, rather than normal operation)
state.

     -- Brett

28

"... assuming the *conflict monitor* hasn't itself failed."

There, FTFY.

Moron designers.

Cheers,
-- jra

29

Yes, but then you're two failures deep -- you need a controller
failure, in a manner that creates an unsafe condition, followed by a
failure of the conflict monitor. Lots of systems are vulnerable to
multiple failure conditions.

Relays can have interesting failure modes also. You can only protect
for so many failures deep.

     -- Brett

30

Indeed. All solid-state controllers, microprocessor or not, are required to have a completely independent conflict monitor that watches the actual HV outputs to the lamps and, in the event of a fault, uses electromechanical relays to disconnect the controller and connect the reds to a separate flasher circuit.

The people building these things and writing the requirements do understand the consequences of failure.

Matthew Kaufman

31

Here is the latest folks,

"DHS and the FBI have found no evidence of a cyber intrusion into the SCADA system in Springfield, Illinois."

http://jeffreycarr.blogspot.com/2011/11/latest-fbi-statement-on-alleged.html

Andrew

32

Steven Bellovin wrote:

Probably nowhere near that sophisticated. More like somebody owned the PC running Windows 98 being used as an operator
interface to the control system. Then they started poking buttons on the pretty screen.

Somewhere there is a terrified 12 year old.

Please don't think I am saying infrastructure security should not be improved - it really does need help. But I really doubt
this was anything truly interesting.

That's precisely the problem: it does appear to have been an easy attack.
(My thoughts are at SMBlog -- 18 November 2011)

--Steve Bellovin, Steven M. Bellovin

Umm hmm. And here's another one poking around:

"I'm not going to expose the details of the box. No damage was done to any of the machinery; I don't really like mindless vandalism. It's stupid and silly.
On the other hand, so is connecting interfaces to your SCADA machinery to the Internet. I wouldn't even call this a hack, either, just to say.
This required almost no skill and could be reproduced by a two year old with a basic knowledge of Simatic."

--Michael

33

If you mean "an independent conflict monitor which, *in the event there is
NO discernable fault*, *connects* the controller to the lamp outputs... so
that in the event the monitor itself fails, gravity or springs will return
those outputs to the flasher circuit", than I'll accept that latter assertion.

Cheers,
-- jra

34

That protects against a conflicting output from the controller at the
same time the conflict monitor completely dies (assuming its death is
in a manner that removes voltage from the relays). It doesn't protect
against the case of conflicting output from the controller which the
conflict monitor fails to detect. (Which is one of the cases you
seemed to be concerned about before.)

     -- Brett

35

andrew.wallace wrote:

Here is the latest folks,

"DHS and the FBI have found no evidence of a cyber intrusion into the SCADA system in Springfield, Illinois."

Latest FBI Statement On Alleged Illinois Water Company Attack

Andrew

And "In addition, DHS and FBI have concluded that there was no malicious traffic from Russia or any foreign entities, as previously reported."

I'd bet we'll soon be hearing more from this loldhs pr0f character in .ro.

--Michael

36

This might be of interest to those wishing to dive deeper into the subject.

Telecommunications Handbook for Transportation Professionals: The Basics of
Telecommunications by the Federal Highway Administration.

http://ops.fhwa.dot.gov/publications/telecomm_handbook/

I'm still digging through it to see what they say about network security.

37

It's interesting to read the rest of the text while doing some deconstruction:

"There is no evidence to support claims made in the initial Fusion Center
report ... that any credentials were stolen, or that the vendor was involved
in any malicious activity that led to a pump failure at the water plant."

Notice that they're carefully framing it as "no evidence that credentials were
stolen" - while carefully tap-dancing around the fact that you don't need to
steal credentials in order to totally pwn a box via an SQL injection or a PHP
security issue, or to log into a box that's still got the vendor-default
userid/passwords on them. You don't need to steal the admin password
if Google tells you the default login is "admin/admin" :wink:

"No evidence that the vendor was involved" - *HAH*. When is the vendor *EVER*
involved? The RSA-related hacks of RSA's customers are conspicuous by their
uniqueness.

And I've probably missed a few weasel words in there...

38

They do state categorically that "After detailed analysis, DHS and the
FBI have found no evidence of a cyber intrusion into the SCADA system of
the Curran-Gardner Public Water District in Springfield, Illinois."

I'm waiting to see Joe Weiss's response.

    --Steve Bellovin, https://www.cs.columbia.edu/~smb

39

See http://www.wired.com/threatlevel/2011/11/scada-hack-report-wrong/

    --Steve Bellovin, https://www.cs.columbia.edu/~smb

40

in a manner that removes voltage from the relays). It doesn't protect
against the case of conflicting output from the controller which the
conflict monitor fails to detect. (Which is one of the cases you
seemed to be concerned about before.)

Reliable systems have triple redundancy.
And indeed... hardwired safety is a lot better than relying on software.
But it's not like transistors/capacitors don't fail either, so
whether solid state or not, a measure of added protection is in order
beyond a single monitor.

There should be a "conflict monitor test path" that involves a third
circuit intentionally
creating a safe "test" conflict at pre-defined sub-millisecond
intervals, by generating a
conflict in a manner the monitor is supposed to detect but won't
actually produce current
through the light, and checking for absence of a test signal on
green; if the test fails, the
test circuit should intentionally blow a pair of fuses, breaking the
test circuit's connections to the
controller and conflict monitor.

In addition the 'test circuit' should generate a pair of clock
signals of its own, that is a side effect
and only possible with correct test outcomes and will be verified by
both the conflict monitor
and the controller; if the correct clock indicating successful test
outcomes is not
detected by either the conflict monitor or by the controller, both
systems should
independently force a fail, using different methods.

So you have 3 circuits, and any one circuit can detect the most
severe potential failure of any pair of the other circuits.