Broadband routers and botnets - being proactive

Mailman List Mirror (Read Only)
1

In this post I'd like to discuss the threat widely circulated insecure
broadband routers pose today. We have touched on it before.

Today, yet another public report of a vulnerable DSL modem type was posted
to bugtraq, this time about a potential WIRELESS flaw with broadband
routers being insecure at Deutsche Telekom. I haven't verified this one
myself but it refers to "Deutsche Telekom Speedport w700v broadband
router":
http://seclists.org/bugtraq/2007/May/0178.html

If you all remember, there was another report a few months ago about a UK
ISP named BeThere with their wireless router being accessible from the
Internet and exploitable, as another example:
http://blogs.securiteam.com/index.php/archives/826

Two issues here:
1. Illegitimate access to broadband routers via wireless communication.
2. Illegitimate access to broadband routers via the WAN.

I'd like to discuss #2.

Some ISPs which provide such devices (as in the example of #2 above) use
them as bridges only, preventing several attack vectors (although not
all). Many others don't. Most broadband ISPs have a vulnerable user-base
on some level.

Many broadband ISPs around the world distribute such devices to their
clients.

Although the general risk is well known, like with many other security
issues many of us remained mostly quiet in the hope of avoiding massive
exploitation. As usual, we only delayed the inevitable. I fear that the
lack of awareness among some ISPs for this "not yet widely exploited
threat" has resulted in us not being PROACTIVE and taking action to secure
the Internet in this regard. What else is new, we are all busy with
yesterday's fires to worry about tomorrow's.
Good people will REACT and solve the problem when it pops up in
wide-exploitation, but what we may potentially be facing is yet another
vector for massive infections and the creation of eventual bot armies on
yet another platform.

My opinion is, that with all these public disclosures and a ripe pool of
potential victims, us delaying massive exploitation of this threat may not
last. I believe there is currently a window of opportunity for service
providers to act and secure their user-base without rushing. Nothing in
security is ever perfect, but actions such as changing default passwords
and preventing connections from the WAN to these devices would be a good
step to consider if you haven't already.

My suggestion would be to take a look at your infrastructure and what your
users use, and if you haven't already, add some security there. You
probably have a remote login option for your tech support staff which you
may want to explore - and secure. That's if things were not left at their
defaults.

Then, I'd also suggest scanning your network for what types of broadband
routers your users make use of, and how many of your clients have port 23
or 80 open. Whether you provide with the devices or not, many will be
using different ones set to default which may pose a similar threat. Being
aware of the current map of vulnerable devices of this type in your
networks can't hurt.

It is not often that we can predict which of the numerous threats out
there that we do not address currently, is going to become exploited
next. If you can spare the effort, I'd strongly urge you to explore this
front and be proactive on your own networks.

The previous unaddressed threat which most of us chose to ignore was
spoofing. We all knew of it for a very long time, but some of us believed
it did not pose a threat to the Internet or their networks for no other
reason than "it is not currently being exploited" and "there are enough
bots out there for spoofing to not be necessary". I still remember the
bitter argument I had with Randy Bush over that one. This is a rare
opportunity, let's not waste it.

We are all busy, but I hope some of you will have the time to look into
this.

I am aware of and have assisted several ISPs, who spent some time and
effort exploring this threat and in some cases acting on it. If anyone can
share their experience on dealing with securing their infrastructure in
this regard publicly, it would be much appreciated.

Thanks.

Gadi Evron.

2

Gadi,

I and numerous others (including some whom any reasonable NANOG-L poster would respect and listen to) have asked you repeatedly to stop trolling NANOG-L with this botnet crap. It is off-topic here. The last time you pulled this (starting a 4-day troll-fest about a nonexistent "INNURNET EMERGENCY") I asked you to stop it, and not one of the legions of supporters you talk about spoke up to say "Wait, I want to see botnet crap on NANOG-L." Even if all 6 of your botnet-loving supporters spoke up, it would not change the fact that your botnet posts are off topic, unwanted, and disruptive. It's time for you to stop it. Please.

3

As frequent as Gadi is with his botnet posts, insecure and wide open
CPE getting deployed across a large provider is definitely
operational.

srs

4

Gadi Evron wrote:

>[snip]

The previous unaddressed threat which most of us chose to ignore was
spoofing. We all knew of it for a very long time, but some of us believed
it did not pose a threat to the Internet or their networks for no other
reason than "it is not currently being exploited" and "there are enough
bots out there for spoofing to not be necessary". I still remember the
bitter argument I had with Randy Bush over that one. This is a rare
opportunity, let's not waste it.

We are all busy, but I hope some of you will have the time to look into
this.

I am aware of and have assisted several ISPs, who spent some time and
effort exploring this threat and in some cases acting on it. If anyone can
share their experience on dealing with securing their infrastructure in
this regard publicly, it would be much appreciated.

I don't know who the "us" is who you are referring to. One of the first things I did when I took over the management of the network at $DAYJOB was to tighten up the packet filtering at the edge of my network. That included fixing up the inbound and outbound filters:

   * blocking most "small services" inbound
   * blocking ports inbound used in widespread attacks
   * blocking multi-cast IP addresses inbound
   * blocking BOGON and RFC1918 source IP addresses inbound
   * blocking non-owned IP source addresses, including RFC1918, outbound
   * null-routing RFC1918 target addresses outbound

(Under consideration but not yet implemented: null-routing BOGON target addresses)

In my research into my new job, I got the impression that the above was considered one of the Best Current Practices for router configuration.

I currently have a customer who is getting DDoSed by someone spoofing the source IP address in a TCP SYN flood. The problem us bad enough that I'm building a level-2 firewall (using a Linux box) to rate-limit TCP SYN to port 80 on his two IP addresses, and to raise an alarm when the incoming rate exceeds a threshold.

When I ask my upstream where the SYN flood is entering *his* routers, the answer is "everywhere, I see these packets on every single upstream port I have."

The last time I was able to do a packet capture and analysis during the flood, I found the source IP address of the packets that got through were evenly distributed across the IP address spectrum, with obvious notches in BOGON, RFC1918, and multicast IP ranges. (For those of you who like to build tools, I found using a FFT of the source addresses to be an excellent tool for analyzing traffic patterns.)

So I don't have a problem sourcing such floods, because my ACLs block attempts to do so. I sure have problems sinking them.

5

* Suresh Ramasubramanian:

As frequent as Gadi is with his botnet posts, insecure and wide open
CPE getting deployed across a large provider is definitely
operational.

And if Gadi's examples are not scary enoug for you, there are far more
relevant vulnerabilities.

It seems that the organization that assembles most of the firmware on
those CPEs just takes the Sourceforge project with the smallest
footprint they can find to implement a particular task. Not even a
cursory code review takes place. As most of the software is GPLed,
not just the firmware provider, but also the hardware manufacturer and
the ISP itself could stop the deployment until the most egregious bugs
have been fixed. Of course, you could argue that if Microsoft and
Debian don't do this, why should ISPs? To me, the answer is that
shipping vulnerable software is state of the art, but only if there is
some kind of patch management appendix.

Fortunately, there is a simple solution to this kind of problem: ISPs
are very likely liable if they fail to alert customers about security
problems, and do not provide updates in a timely manner. After a few
painful incidents, the ISPs will learn, and either ship better
software (unlikely) or implement some kind of patch management. With
a bit of luck, the latter does not just shift back liability back to
the customer, but also helps to parly solve the problem (in the sense
that CPE attacks are less attractive).

6

It won't solve the problem. ISPs will simply stop distributing CPE, and
tell customers to buy CPE from their nearest electronics store (Best Buy, Radio Shack, or the equivilent in other countries). If you thought it
was hard getting ISPs to patch CPE, try getting electronics stores to
patch the CPE. Look at the ancient bugs in D-Link, Linksys, Netgear boxes
that consumers haven't figured out how to patch for years.

You really need to identify the sources and fix it there.

7

"Passing the buck! Buck passer!" (see below - skip to Dilbert link)

Not saying that you are wrong but... Ahh, these are out of our
control, nor will they do anything if we don't. Might as well tell users
not to patch their Windows systems as it's the responsibility of the store
who sold them the computer. Yes, it could help if the stores did
something.

There is little to no financial incentive for ISPs to do something about
this problem right now, even if it is currently under their direct
control. Later on, when it is a problem - it will cost more.

Today? Some will do someting, others won't. It surprises me how many do
invest in this.

Almost everything we do in Internet security operations has nothing to do
with identifying the problem and fixing it. It's usually just about
identifying the sympthoms and getting rid of them. It's like I sometimes
tell law enforcement: "we can't afford to wait, we need to maintain
our networks". We wait anyway and end up eating a sock.

As to your suggestion here (quoting a /. user who wrote it down):

Dilbert is in the Boss's office.
Dilbert: I discovered a hole in our internet security.
Boss: What?!!
Boss: Good grief, man! How could you put a hole in our internet?
Dilbert, angry: I didn't PUT it there, I FOUND it.. and it's not...
Boss: It's your job to fix that hole. I want you to work 24-7!
Dilbert: Actually, that's NOT my job. But I'll inform our network
management group.
Boss, yelling: PASSING THE BUCK!!! YOU'RE A BUCK PASSER!!!
Dilbert: Forget it! There's no hole! It got better!
Boss: That's more like it.
Last panel, the boss is sitting alone smiling.
Boss thinks: I fixed the internet.

I found it on Google images:
http://stderr.de/funstuff/dilbert_fixed_the_internet.jpg

8

Hi Gadi,
reading all the email re off topic etc is wrong.
If this issue is dealt with then transit bandwidth will be less, security will improve and the end user experience will be better.

Great dilbert cartoon

Colin

9

Sean Donelan wrote:

Fortunately, there is a simple solution to this kind of problem: ISPs
are very likely liable if they fail to alert customers about security
problems, and do not provide updates in a timely manner. After a few
painful incidents, the ISPs will learn, and either ship better
software (unlikely) or implement some kind of patch management. With
a bit of luck, the latter does not just shift back liability back to
the customer, but also helps to parly solve the problem (in the sense
that CPE attacks are less attractive).

It won't solve the problem. ISPs will simply stop distributing CPE, and
tell customers to buy CPE from their nearest electronics store (Best
Buy, Radio Shack, or the equivilent in other countries). If you thought it
was hard getting ISPs to patch CPE, try getting electronics stores to
patch the CPE. Look at the ancient bugs in D-Link, Linksys, Netgear boxes
that consumers haven't figured out how to patch for years.

You really need to identify the sources and fix it there.

When your cpe costs $50 (to the consumer) it's not worth anyone's time
(consumer, isp, manufacturer, store that sold it etc) to patch/upgrade
the thing. If it's broken enough they'll eventually buy another one. or
they'll buy another one because they decide they need some wazoo new
feature, (802.11n, gigabit ethernet, p2p support, hard-disk etc)... The
trick is insuring that when they do buy another one it;s tangibly better
than the old one.

Even if your cpe costs more (cisco 8xx) it's still not worth patching it
if that is going to require external support (first time you call the
tac you blow the profit on a cisco 800).

Just remember, very few of these cpe devices existed 5 years ago, the
probability that the same one's will be in use in 5 years seems pretty low.

Deliver a compelling new technology platform and the users will upgrade
en-masse (50mbit vdsl, ftfh, docsis 3 cable modems, fixed wimax, etc)

It's my opinion that access isp's need to get out of the business of
selling/delivering cpe because frankly the consumer will probably spend
more on features and so forth, than the isp will when they lease you
some crappy actiontec dsl router for 3-bucks a month. The isp's shoot
themselves in the foot by shoveling the cheapest cpe they can out the
door when the consumer would probably go out and pay for it if they felt
like they weren't getting jacked.

10

So, out of curiousity, could you define: "under their control" for me/us?
I'm certainly think that weak CPE security is a problem, but I'm not sure
I agree that the CPE which is purchased as part of your service and
managed by you (you == random-customer in this example) is anyone's
problem except yours.

If you think that the lobbying groups inside consumer ISP's are going to
let through some set of regulations that gets them on the hook without
substantial rate increases you really need to re-evaluate that assumption
:frowning:

Some of this comes back to making people understand that computer security
is no different than anyother form of security (house, car, boat, wallet)
the only thing that is different is it's age.

-Chris

11

> There is little to no financial incentive for ISPs to do something about
> this problem right now, even if it is currently under their direct
> control. Later on, when it is a problem - it will cost more.

So, out of curiousity, could you define: "under their control" for me/us?
I'm certainly think that weak CPE security is a problem, but I'm not sure
I agree that the CPE which is purchased as part of your service and
managed by you (you == random-customer in this example) is anyone's
problem except yours.

If you think that the lobbying groups inside consumer ISP's are going to
let through some set of regulations that gets them on the hook without
substantial rate increases you really need to re-evaluate that assumption
:frowning:

Some of this comes back to making people understand that computer security
is no different than anyother form of security (house, car, boat, wallet)
the only thing that is different is it's age.

No arguments here, thank you for the clarification.

12

"Passing the buck! Buck passer!" (see below - skip to Dilbert link)

I guess you missed my attempts 3 or 4 years ago at trying to establish some standards for CPE concerning security. I've been at this party for
a long time, I know how the song ends.

Not saying that you are wrong but... Ahh, these are out of our
control, nor will they do anything if we don't. Might as well tell users
not to patch their Windows systems as it's the responsibility of the store
who sold them the computer. Yes, it could help if the stores did
something.

I spent about a year of my life working with Microsoft on getting patched versions of Windows in the pipeline and getting OEMs to regularly update their manufacturing copy being pre-installed on machines as they leave
the factory. So yes, it did help to improve things in the pipeline.

13

Just a joke, Sean.
What would you consider from your experience, the best way to make these
third parties take responsibility?

14

First, you need to identify the ODM making the software used in the CPE.

15

Gadi,

I appreciate your well thought out email but I sit here and wonder
what exactly you are trying to accomplish with it? Are you just trying
to shame the two ISPs listed publicly or are you trying to spark a
discussion about something that many people here can't fix?

Many businesses today are focused on driving revenue and fixing old
CPE equipment doesn't generate revenue, it only ties up money and
resources that can be used elsewhere to drive revenue. If I were you I
would try to spin this problem in a way where you can show large ISPs
by fixing CPE's it will free up network resources and staff which can
be used elsewhere.

The people that can fix these problems are usually unaware of them so
try to educate those people. Write CEOs/CTOs/CSOs educating them and
push the security teams for these companies to escalate these issues
to their upper management (on that note I would say this type of
discussion would be better suited for a security mailing list for the
reason I stated before, many people here can't fix these problems).

Simply stating that there is a problem and shunning ISPs with this
problem isn't a fix for the problem, it just makes them ignore you and
the problem.

-Ross

16

Gadi,

I appreciate your well thought out email but I sit here and wonder
what exactly you are trying to accomplish with it? Are you just trying
to shame the two ISPs listed publicly or are you trying to spark a
discussion about something that many people here can't fix?

Many businesses today are focused on driving revenue and fixing old
CPE equipment doesn't generate revenue, it only ties up money and
resources that can be used elsewhere to drive revenue. If I were you I
would try to spin this problem in a way where you can show large ISPs
by fixing CPE's it will free up network resources and staff which can
be used elsewhere.

The people that can fix these problems are usually unaware of them so
try to educate those people. Write CEOs/CTOs/CSOs educating them and
push the security teams for these companies to escalate these issues
to their upper management (on that note I would say this type of
discussion would be better suited for a security mailing list for the
reason I stated before, many people here can't fix these problems).

Simply stating that there is a problem and shunning ISPs with this
problem isn't a fix for the problem, it just makes them ignore you and
the problem.

You are quite right. Thank you.

I found some ways of showing several issues to be revenue-tied, such as
blocking port 25, etc. This issue is something I am at a stage of
exploring, and like it or not.. network operators are the ones who deal
with this (on whatever level they do).

I am unsure of where else to go with this, and if some ISPs do something
for now, that is a step in the right direction until a better way shows
itself. Whichever way we discover, for now, raising awareness is all I can
think of.

On a sarcastic evil tone, we may just plan to release a "fix" worm to
harden all these devices world-wide. Right! Because that worked so well
for us before. :>

-Ross

  Gadi.

17

Ross Hosman wrote:

Gadi,

I appreciate your well thought out email but I sit here and wonder
what exactly you are trying to accomplish with it? Are you just trying
to shame the two ISPs listed publicly or are you trying to spark a
discussion about something that many people here can't fix?

Many businesses today are focused on driving revenue and fixing old
CPE equipment doesn't generate revenue, it only ties up money and
resources that can be used elsewhere to drive revenue. If I were you I
would try to spin this problem in a way where you can show large ISPs
by fixing CPE's it will free up network resources and staff which can
be used elsewhere.

The people that can fix these problems are usually unaware of them so
try to educate those people. Write CEOs/CTOs/CSOs educating them and
push the security teams for these companies to escalate these issues
to their upper management (on that note I would say this type of
discussion would be better suited for a security mailing list for the
reason I stated before, many people here can't fix these problems).

Simply stating that there is a problem and shunning ISPs with this
problem isn't a fix for the problem, it just makes them ignore you and
the problem.

-Ross

Hi Ross,

Gadi is talking about DTAG.de our biggest ISP in germany and quasi a
monopoly. Gadi has reached the ears of the Pirates Party, a political
party that fights monopolies.

The hardware is very likely a branded version from AVM. They have no
updates for the branded version, but you can unbrand it. Then you
have a hardware that accepts open source firmware.

Kind regards

Peter and Karin