IoT security

Mailman List Mirror (Read Only)
1

This afternoon's panel about IoT's lack of security got me thinking...

On the issue of ISPs unable to act on insecure devices because they
can't detect the devices until they're compromised and then only have
the largest hammer (full account ban) to act...

What about some kind of requirement or convention that upon boot and
successful attachment to the network (and maybe once a month
thereafter), any IoT device must _by default_ emit a UDP packet to an
anycast address reserved for the purpose which identifies the device
model and software build. The ISP can capture traffic to that anycast
address, compare the data against a list of devices known to be
defective and, if desired, respond with a fail message. If the IoT
device receives the fail message, it must try to report the problem to
its owner and remove its default route so that it can only communicate
on the local lan. The user can override the fail and if desired
configure the device not to emit the init messages at all. But by
default the ISP is allowed to disable the device by responding to the
init message.

Would have to cryptographically sign the fail message and let the
device query the signer's reputation or something like that to avoid
the obvious security issue. Obvious privacy issues to consider.
Anyway, throwing it out there as a potential discussion starting
point.

The presentation on bandwidth policers...

Seems like we could use some form of ICMP message similar to
destination unreachable that provides some kind of arbitrary string
plus the initial part of the dropped packet. One of the potential
strings would be an explicit notice to the sender that packets were
dropped and the bandwidth available.

Yes, we already have ECN, but ECN tells the receiver about congestion,
not the sender. More to the point, ECN can only be flagged on packets
that are passed, not the packets that are dropped, so the policer
would have to be complicated enough to note on the next packet that
the prior packet was dropped. Also, ECN only advises that you're close
to the limit not any information about the policer's target limit.

This thought is not fully baked. Throwing it out for conversation purposes.

Regards,
Bill Herrin

2

Uh, yuck at many levels. Do you leak your cisco ios versions to the internet?

Do you really want the responsibility for the remote kill switch for IoT S&M gear?

And of course, you're depending on rfc 3514, right?

Mike

3

This afternoon's panel about IoT's lack of security got me thinking...

On the issue of ISPs unable to act on insecure devices because they
can't detect the devices until they're compromised and then only have
the largest hammer (full account ban) to act...

What about some kind of requirement or convention that upon boot and
successful attachment to the network (and maybe once a month
thereafter), any IoT device must _by default_ emit a UDP packet to an
anycast address reserved for the purpose which identifies the device
model and software build. The ISP can capture traffic to that anycast
address, compare the data against a list of devices known to be
defective and, if desired, respond with a fail message. If the IoT
device receives the fail message, it must try to report the problem to
its owner and remove its default route so that it can only communicate
on the local lan. The user can override the fail and if desired
configure the device not to emit the init messages at all. But by
default the ISP is allowed to disable the device by responding to the
init message.

self identification is privacy hostile and tantamount to indicating a
willingness to be subverted (this is why we disable lldp on external
interfaces) even if it would otherwise be rather useful. the use of
modified eui64 addresses as part of v6 address selection hash basically
gone away for similar reasons.

4

This afternoon's panel about IoT's lack of security got me thinking...

Hi Joel,

For clarification I was referring to this:

The long and short of the panel was: as an industry (device vendors
and service providers both) it behooves us to voluntarily get on top
of the IoT security problem before some catastrophic event requires
the government to dictate the precise manner in which we will get on
top of the problem.

What about some kind of requirement or convention that upon boot and
successful attachment to the network (and maybe once a month
thereafter), any IoT device must _by default_ emit a UDP packet to an
anycast address reserved for the purpose which identifies the device
model and software build.

self identification is privacy hostile and tantamount to indicating a
willingness to be subverted (this is why we disable lldp on external
interfaces) even if it would otherwise be rather useful. the use of
modified eui64 addresses as part of v6 address selection hash basically
gone away for similar reasons.

I'm not sure how we get on top of the problem without offering an
effective network kill switch to the nearest security-competent
person. I think I'd prefer a user-disableable kill-switch used on a
single piece of equipment to a kill switch for my entire Internet
connection.

The IPv6 SLAAC address suffers a rather worse case of the privacy
problem since it allows the entire Internet to track your hardware,
not just your local ISP.

In any case, I thought "how do we fix this long term" could stand
discussion on the list. Because yes, the IoT device vendors mostly
produce trash and if (to borrow a phrase) it saves them a buck at
retail they will keep producing trash. But we're the ones letting that
trash cause nation-scale problems and when the regulatory hammer
crashes down it's gonna hit us all.

Uh, yuck at many levels. Do you leak your cisco ios versions to the
internet?

Hi Michael,

I'm not aware of any Cisco IOS devices that qualify as IoT. Some
lighter weight Cisco gear, yes. And no, I do not want to broadcast my
information. But I'm professional who customizes my gear when I plug
it in. I don't run with the defaults.

Do you really want the responsibility for the remote kill switch for IoT S&M
gear?

I already have the kill switch for the customer's entire S&M transit
link. I'd prefer to also have a smaller hammer whose use won't net me
a furious call from Sales.

And of course, you're depending on rfc 3514, right?

Nope. I'll decide what's evil and what's not (more likely I'll pay a
service to provide me a regularly updated database) and I depend only
on a high enough percentage of the devices offering themselves up for
that decision that it becomes impractical to construct another Mirai.

Regards,
Bill Herrin

5

I can think of at least four reasons why this idea must be killed
immediately and permanently. This is off the top of my head *before*
coffee, so I strongly suspect there are more.

1. An attacker who takes control of an IoT device can change the contents
of that packet, cause it to be emitted, suppress it from being emitted, etc.

2. This will allow ISPs to build a database of which customers have
which IOT devices. This is an appalling invasion of privacy.

3. This will allow ISPs to build a database of which customers have
which IOT devices. This will create one-stop shopping for attackers.

4. It won't take long for this to be used as a DDoS vector.

---rsk

6

What about some kind of requirement or convention that upon boot and
successful attachment to the network (and maybe once a month
thereafter), any IoT device must _by default_ emit a UDP packet to an
anycast address reserved for the purpose which identifies the device
model and software build.

I can think of at least four reasons why this idea must be killed
immediately and permanently. This is off the top of my head *before*
coffee, so I strongly suspect there are more.

1. An attacker who takes control of an IoT device can change the contents
of that packet, cause it to be emitted, suppress it from being emitted, etc.

Hi Rich,

Immaterial. The point is to catch vulnerable devices before they're
hacked. That can't always happen (even with customers and vendors
engaged in best practice patching), but it need only happen often
enough to limit the size of the resulting botnets.

2. This will allow ISPs to build a database of which customers have
which IOT devices. This is an appalling invasion of privacy.

As I envision it, it's an opt-out system. Anyone who cares and knows
enough to secure their network can turn it off for their devices. No
permission or action from the ISP required to turn it off. In fact,
you need only block packets to the well-known anycast address to
disable it for all devices on your network.

3. This will allow ISPs to build a database of which customers have
which IOT devices. This will create one-stop shopping for attackers.

Possible, though an ISP which retains the data opens itself to a class
action lawsuit if it can't keep control of it.

Nevertheless, let's not oversell the privacy implications: most of the
IoT devices we're talking about phone home anyway. They're tied to
this or that cloud service rather than accepting local access. An ISP
can't learn as much information by watching who they phone home to
(deep packet inspection), but they can learn an awful lot.

4. It won't take long for this to be used as a DDoS vector.

If the ISP can't keep control of its security head-end then sure, at
least until the ISP regains control.

Regardless, I encourage you to suggest alternate solutions which don't
run afoul of the problems you mention. The problem isn't going away.
Entirely cutting off customers doesn't work because there are too many
impacted customers and ISPs don't have the resources or the will to do
so. Demanding that trash vendors not produce trash won't get it done
either. When you eliminate responsible device hardening, cutting
customer connections and the kill switch, what's left?

Regards,
Bill Herrin

7

" any IoT device must _by default_ emit a UDP packet to an
anycast address reserved for the purpose which identifies the device
model and software build. "

Any semi-competent attacker will simply alter the way the network stack on
the device works to make it _not_ look like an IoT device for the purposes
of this check, rendering it moot.

"If the IoT
device receives the fail message, it must try to report the problem to
its owner and remove its default route so that it can only communicate
on the local lan."

If the vendor isn't thinking about security already, as evidenced by the
current issues with these devices, how are they to be convinced to add code
to make said device do something exceptionally specific?

I believe it's important to remember that network endpoints will never be
sure fully secure. They will never be fully in compliance with $standard .
They will never always follow best practices. This is not to say we should
do nothing (obligatory 'Perfect is the enemy of good' comment) but it's
something to think about when discussing possibilities.

8

I think the fundamental problem here is that these devices aren't good
network citizens in the first place. The odds of getting them to add
functionality to support a new protocol are even likely than getting them
to not have open services externally IMHO.

Couldn't a lot of this be caught by proactive vulnerability scanning and
working with customers to have an SPI firewall in place, or am I missing
something?

Historically residential ISP CPE options have been terrible. If you could
deliver something closer to user expectations you would likely see much
more adoption and less desire to rip and replace. Ideally a cloud-managed
device so that the config wouldn't need to be rebuilt in the event of a
hardware swap.

9

Immaterial. The point is to catch vulnerable devices before they're
hacked. That can't always happen (even with customers and vendors
engaged in best practice patching), but it need only happen often
enough to limit the size of the resulting botnets.

This won't work on the majority of devices: they're pre-compromised
at the factory. By the time you see the first packet from them,
IF you see the first packet from them, it will already be too late.

And a lot of them are deliberately designed and built to conduct attacks, e.g.:

  Vizio tracked and sold your TV viewing habits without consent
  Vizio tracked and sold your TV viewing habits without consent (updated)

What a nice favor to do for attackers: Vizio spent its time and money
putting this in place for them, so they didn't have to.

This is going to get worse -- MUCH worse -- as the few flimsy consumer
protections in place are systematically dismantled and the agencies
charged with enforcing them defunded.

As I envision it, it's an opt-out system.

No. In fact, HELL NO. Too many ISPs have already put absolutely clinching
proof on the table that they will silently opt-in customers to all manner
of tracking, surveillance, and security/privacy attacks. (I'm looking at
you, Verizon, at the moment.) Opt-out is inherently abusive.

Possible, though an ISP which retains the data opens itself to a class
action lawsuit if it can't keep control of it.

First, that's a meaningless remedy. Even if someone has the financial
resources to sue an ISP, they're going to wind up quietly settling years
later, the lawyers will get rich(er), the settlement will be sealed,
and they'll just keep right on doing it.

See the Vizio case above. Did anybody go to prison? No. Did it
cost them anything? Not really: the fine is just chump change.
Will Vizio do it again? OF COURSE they will, they'd be crazy not to.
All they have to do is wait a little while, rename it, shuffle things
around a bit, and they'll be fine.

Second, the most likely outcome here is that the ISP will use this data
to spy on its users and market to them. The next outcome is that someone
inside the ISP will figure out that this is a potential revenue source
and start selling it, under the argument that consumers didn't opt-out,
therefore they WANT their private data sold.

So let's not pretend that the delusional fiction of a successful class-action
lawsuit is a deterrent. It's just a belly laugh for the executives and a
windfall for the attorneys.

If the ISP can't keep control of its security head-end then sure, at
least until the ISP regains control.

I think you're severely underestimating the threat. And note that even
though you're just thinking about the problem from the perspective of ISPs,
they're not the only ones affected. There are a lot of attack vectors
available to anyone who's already on the inside.

Regardless, I encourage you to suggest alternate solutions which don't
run afoul of the problems you mention. The problem isn't going away.

I'm aware of that. But this is not the way. I've seen this movie
WAY too many times:

  1. We must do something
  2. This is something.
  3. Let's do this.
  4. We have done something.
  5. Congrats and awards all around, everybody.

---rsk

10

" any IoT device must _by default_ emit a UDP packet to an
anycast address reserved for the purpose which identifies the device
model and software build. "

Any semi-competent attacker will simply alter the way the network stack on
the device works to make it _not_ look like an IoT device for the purposes
of this check, rendering it moot.

Hi Tom,

Again, the idea is to remediate devices with insecure software
-before- they're breached. Obviously once breached the attacker can
present any profile he wants save that it has to emit packets
consistent with his desired attack.

"If the IoT
device receives the fail message, it must try to report the problem to
its owner and remove its default route so that it can only communicate
on the local lan."

If the vendor isn't thinking about security already, as evidenced by the
current issues with these devices, how are they to be convinced to add code
to make said device do something exceptionally specific?

Because it's consistent with the carelessly throw-together free open
source software process the trash vendors currently follow and when
the industry promotes its "look for the safe network logo" program
they won't want to have problems with their retailers over a trivially
added bit of free software.

I believe it's important to remember that network endpoints will never be
sure fully secure. They will never be fully in compliance with $standard .
They will never always follow best practices.

You can say that again.

I think the fundamental problem here is that these devices aren't good
network citizens in the first place. The odds of getting them to add
functionality to support a new protocol are even likely than getting them to
not have open services externally IMHO.

Hi Ray,

I can speak to that with authority, but again my working theory is
that adding the kill switch is consistent with the carelessly
throw-together free open source software process the trash vendors
currently follow.

Couldn't a lot of this be caught by proactive vulnerability scanning and
working with customers to have an SPI firewall in place, or am I missing
something?

One of us is missing something. The customer isn't scanning his
network (if he was, we wouldn't have the problem), and from the
outside of an SPI firewall how is the ISP supposed to do that?

Historically residential ISP CPE options have been terrible. If you could
deliver something closer to user expectations you would likely see much more
adoption and less desire to rip and replace.

The customer buys cool stuff he finds at CostCo. Seeking different
behavior seems unlikely to succeed to me.

Immaterial. The point is to catch vulnerable devices before they're
hacked.

This won't work on the majority of devices: they're pre-compromised
at the factory.

Well that's a silly misdirection Rich. Manufacturer's misbehavior is a
whole different class of problem than equipment breached by a third
party after deployment. Unless you claim a convenient way to merge
solutions, take it to your own thread.

If the ISP can't keep control of its security head-end then sure, at
least until the ISP regains control.

I think you're severely underestimating the threat.

I'm not estimating the threat at all, just stating the precondition
for a kill switch to be a denial of service vector.

Regardless, I encourage you to suggest alternate solutions which don't
run afoul of the problems you mention. The problem isn't going away.

I'm aware of that. But this is not the way. I've seen this movie
WAY too many times:

        1. We must do something
        2. This is something.
        3. Let's do this.
        4. We have done something.
        5. Congrats and awards all around, everybody.

And I've heard this song before too: If you choose not to decide you
still have made a choice.

Neither one of us will like the government solution, so if you have an
idea that hasn't already demonstrated its failure, let's hear it. When
there's no consensus on an optimal solution, we move forward by trying
ideas until something sticks. That's progress, including the attempts
which fail.

Regards,
Bill Herrin

11

you have a 30 second window there, maybe five minutes if you are lucky.

12

Looking at my logs from the past couple of months I think you are being generous by giving it thirty seconds.

     Richard

13

In a recent article (
https://www.schneier.com/blog/archives/2017/02/security_and_th.html), Bruce
Schneier sums up the IoT security mitigation issue quite nicely in this
paragraph:

"The market can't fix this because neither the buyer nor the seller cares.
The owners of the webcams and DVRs used in the denial-of-service attacks
don't care. Their devices were cheap to buy, they still work, and they
don't know any of the victims of the attacks. The sellers of those devices
don't care: They're now selling newer and better models, and the original
buyers only cared about price and features. There is no market solution,
because the insecurity is what economists call an externality: It's an
effect of the purchasing decision that affects other people. Think of it
kind of like invisible pollution."

- Ed Lopez

14

Hi Randy,

I'd expect a tattler kill switch to take maybe a tenth of that from
the anycast notification when the nic comes up to the ISPs response
that it is known to be vulnerable and should disconnect.

Regards,
Bill Herrin

15

assuming that it wasn't conveniently factory installed, cf usb sticks.

Mike

16

On Tue, Feb 07, 2017 at 10:01:29PM +0000, Ed Lopez quoted Bruce Schneier:

There is no market solution, because the insecurity is what economists
call an externality: It's an effect of the purchasing decision that
affects other people.

This is precisely correct. The only way to change this is to make
*our* problem *their* problem. Let me remind everyone of one of
very best things ever said on this mailing list:

  If you give people the means to hurt you, and they do it, and
  you take no action except to continue giving them the means to
  hurt you, and they take no action except to keep hurting you,
  then one of the ways you can describe the situation is "it isn't
  scaling well".
    --- Paul Vixie

This movie has been playing here 24x7 for the last few decades with
the spam problem (among others): most operations which emit it will
take absolutely no action of any kind until/unless it stops being *our*
problem and starts being *their* problem. Having observed and studied
that particular issue since it existed to be observed and studied,
I've concluded that the only thing that has ever worked effectively is
blacklisting: that is, the revocation of access/service privileges.

And that's because it makes *our* problem *their* problem. Everything
else is just avoidance and evasion. It's feel-good stuff that might,
on a good day, deal with the symptoms of the problem -- but that's
all it is. And dealing with symptoms isn't bad, per se: it makes the
patient feel better. But it should never be accepted as a substitute
for dealing with the underlying problem.

Now we can either spend another couple more decades trying to tapdance
around this or we can learn the lesson that's been taught to us thousands
of times over many years and just cut to the chase.

So how about if we save some time?

If IOT-driven attacks and abuse are coming from X, then that needs to be
made X's problem. Because right now X has no idea that this is happening,
and even if told, will take no action because it's not X's problem.

So make it X's problem.

And I don't just mean "X", the person who bought some badly-designed
poorly-engineered rushed-to-market never-tested piece of shiny
new cruft that was pre-compromised at the factory and hijacked by
attackers the moment it went live: I mean "X" the vendor who pulled that
stunt in order to make a quick buck and then dumped it on us.

We, for many values of "we" are not obligated to provide services to that
vendor. (I do recognize that some are, due to contractual agreements.)
We should cut them off until/unless they recall all those devices,
get them removed from service, and solve what is now *their* problem
to *our* satisfaction. I strongly suspect that it'll only take a
few pointed lessons in order for the message that this conduct is
unacceptable to be communicated in a language they understand.

I don't like this. In a better world, vendors would be far more
responsible, professional, and ethical. But we don't live in that
world. We live in one where they will happily dump toxic waste on
the Internet as fast as they can shovel it -- as long as it's not
their problem.

We need to make it their problem.

---rsk

17

How?

Regards,
Bill Herrin

18

The devices are trivially compromised (just log in with the default root
password). So here's a modest proposal: log in as root and brick the
device.

This will encourage the consumer to seek a solution. When 100k consumers
all discover their devices broke at the same time, they'll file a
class-action lawsuit against the manufacturer, or at least never buy from
them again. Market forces then solve the problem naturally, both for that
manufacturer and for others who don't want the same fate.

I realize there are drawbacks (including legal implications) to this method
(which is why I'm posting from a personal, not work, account). But I
challenge anyone to propose another solution that will work as well. Most
other proposals I've heard depend on individual ISPs to take action, or
governments to regulate devices and hope that foreign manufacturers care,
or ....

Damian

19

Okay, so within the confines of lawful activity, how?

'Cause I'm guessing that coordinated criminal activity is going to be
a community non-starter. At least when it's this unambiguous. :wink:

Regards,
Bill Herrin

20

I strongly suspect that when the problem gets bad *enough*, someone will
do exactly that. Yes, it is illegal in many places. Since when has the
fact that any particular act is illegal been sufficient to deter
*everyone*?

People still drive while drunk.