RE: botnets: web servers, end-systems and Vint Cerf [LONG, sorry]

Mailman List Mirror (Read Only)
1

But suppose you put such a firewall in place. You'll need to
configure the firewall properly -- paying as much attention to
outbound rules as inbound.

Sounds like a good thing to document in a best practices document that
can be used to certify firewall implementations. When trying to solve a
social problem, techniques like the Good Housekeeping seal of approval
are quite effective. As recommended by the editors of...

You'll need to add anti-virus software. And anti-spyware software.
Then you need to make sure the "signature" databases for both of those
are updated early and often,

What if the guidelines state that subscription and database oriented
techniques for virus detection are not adequate and therefore not
compliant. Only heuristic, capability-based systems are acceptable.

And you'll need to de-install IE and Outlook,

Thus ensuring that Firefox/Thunderbird will be the main target of the
malware people. Is this necessarily any better? Note that Windows
provides an extensive series of hooks which can be used by an
application which wishes to subvert the normal operation of the OS. That
subversive application could be the security monitor which is required
by the ISP for Internet access because it is recommended in your
guidelines.

Something which requires this much work just to make it through its
first day online, while being used by J. Random Person, is hopelessly
inadequate. Which is why systems like this are routinely
compromised in
huge numbers. Which is why we have a large-scale problem on
our hands.

We live in a complex world. Computers are more complex than they were.
OSes are more complex. Apps are more complex. Networks are more complex.
And SOLUTIONS are more complex. But if the designers of computers, OSes,
apps and networks can deal with the complexity, why can't security folks
do likewise?

This left me with >1.5M observed hosts seen in a month.
They're all sending
spam. (How do I know? Because 100% of the mail traffic sent to that
server is spam.)

What you did sounds dumb except that you said this is an experiment.
Unfortunately, real live email servers do exactly the same, i.e. talk to
all comers, because the email architecture is flat like a pancake. Some
people consider this to be a Windows malware problem. I consider it to
be an email architecture problem. We all know that you need hierarchy to
scale networks and I submit that any email architecture without
hierarchy is broken by design and no amount of ill-thought-out bandaids
will fix it.

Pop quiz, bonus round: how much does it cost Comcast to defend its
mail servers from Verizon's spam, and vice versa? Heck, how much
does it cost Comcast to defend its mail servers from its own spam?

That actually sounds like an answerable question, if a company took it
seriously enough. If the senders and receiver are both on your network,
your finance department should be able to come up with some cost
figures.

--Michael Dillon

2

I look forward to your paper on "the end to end concept, and why it doesn't
apply to email" :wink:

I'm not convinced there is an email architecture problem of relevance to the
discussion. People mistake a security problem for its most visible symptoms.

The SMTP based email system has many faults, but it seems only mildly stressed
under the onslaught of millions of hosts attempting to subvert it. Most of
the attempts to "fix" the architecture problem so far have moved the problem
from blacklisting IP addresses, to blacklisting domains, or senders, or other
entities which occupy a larger potential space than the IPv4 addresses, which
one can use to effectively deal with most of the symptom. In comparison,
people controlling malware botnets, have demonstrated their ability to
completely DDoS significant chunks of network, suggesting perhaps that other
protocols are potentially more vulnerable than SMTP, or more approrpiate
layers to address the problem at.

We may need a trust system to deal with identity within the existing email
architecture, but I see no reason why that need be hierarchical, indeed
attempts to build such hierarchical systems have often failed to gather a
critical mass, but peer to peer trust systems have worked fine for decades
for highly sensitive types of data.

I simply don't believe the higher figures bandied about in the discussion for
compromised hosts. Certainly Microsoft's malware team report a high level of
trojans around, but they include things like the Jar files downloaded onto
many PCs, that attempt to exploit a vulnerability that most people patched
several years ago. Simply identifying your computer downloaded (as designed),
but didn't run (because it was malformed), malware, isn't an infection, or of
especial interest (other than indicating something about the frequency with
which webservers attempt to deliver malware).

3

The end-to-end principle has no bearing upon this discussion at all, unless you're referring to firewalls/NATs.

4

And you'll need to de-install IE and Outlook,
    

This will not happen. Not even remotely.

Thus ensuring that Firefox/Thunderbird will be the main target of the
malware people. Is this necessarily any better? Note that Windows
provides an extensive series of hooks which can be used by an
application which wishes to subvert the normal operation of the OS. That
subversive application could be the security monitor which is required
by the ISP for Internet access because it is recommended in your
guidelines.

I concur with ISP's looking for IE as some form of guideline. Stupid story... So I call Cox because for the 8mb down I am supposed to be getting, I was maxing out at 2mb, not a big deal.

TechGirl: Can you go to your start menu...
Me: No I don't use Windows
TechGirl: Please hold
TechGirl: (five minutes later) Are you using OSX?
Me: No. Using Solaris, what would you like me to do?
TechGirl: Please hold
TechGirl: (minutes later) We don't support Solaris
Me: What does an operating system have to do with lousy bandwidth...
TechGirl: Please hold
TechGirl: (minutes later) I have to escalate this to my manager
TechGirl: Please hold
Manager: Please go to your start menu...
Me: No. As stated I'm not on Windows nor OSX. I use Solaris and I AM CONNECTED the service is horrible
Manager: Well we only support Windows and OSX
Me: (*ponders what this has to do with cruddy connectivity) Forget it... (Plugs in Windows laptop to make things easier).

ISP's have come to rely on the bane of their client's issues. Asking someone to remove IE only to have their support group look for it is a nightmare in itself. Too many people have become so overdependent on Windows.

We live in a complex world. Computers are more complex than they were.
OSes are more complex. Apps are more complex. Networks are more complex.
And SOLUTIONS are more complex. But if the designers of computers, OSes,
apps and networks can deal with the complexity, why can't security folks
do likewise?

The issue of security folks dealing with complexities is, they shouldn't have to when it comes to 65% of the problems which lead to incidents. Why should an ISP have to deal with issues that have nothing to do with their networks. I get calls day and night from VoIP customers: "My service is down your service sucks...."

2007-02-19 00:23:36 '212XXX6428' at 212XXX6428@71.231.xxx.xxx:5060 for 3600
2007-02-19 07:59:43 '212XXX6428' at 212XXX6428@71.231.xxx.xxx:5060 for 3600
2007-02-19 10:58:44 '212XXX6428' at 212XXX6428@71.231.xxx.xxx:5060 for 3600
2007-02-19 12:58:05 '212XXX6428' at 212XXX6428@71.231.xxx.xxx:5060 for 3600

This client goes up and down like a see-saw at least 8 times a day. Their provider is horrible. Why should I spend resources trying to fix what has nothing to do with my company. Same applies to anyone in the security industry to a degree. A security engineer can only do so much given parameters most work with. "We're a Windows only shop!" touted the MCSE with glee as he wondered why he spent so much time rebooting.

That actually sounds like an answerable question, if a company took it
seriously enough. If the senders and receiver are both on your network,
your finance department should be able to come up with some cost
figures.
  
They won't because they haven't been pressed to do so, and it is rare that someone will take it upon themselves to do a good deed when it comes to situations like this.

Roland Dobbins wrote:

> NATting firewalls don't help at all with email-delivered malware, browser exploits, etc.

Antivirus and ad-aware like programs almost often do when used appropriately. It boils down to education which won't happen. If forced however it is a different story so again I will point to customer sandboxing.

And yes firewalls do help if configured properly on the business side of things. I use the same brute forcing script to create firewall rules to block IN AND OUT those offensive networks. So even if say a machine were to get infected, its only momentarily before I catch it, but this is my network(s) and those I manage/maintain. I have zero tolerance for junk and don't mind blocking a /8 if needed. People want to complain then I point out logfiles with information on why their entire class is blocked.

5

I don't understand why you don't believe those numbers. The estimates
that people are making are based on externally-observed known-hostile
behavior by the systems in question: they're sending spam, performing
SSH attacks, participating in botnets, controlling botnets, hosting
spamvertised web sites, handling phisher DNS, etc. They're not based
on things like mere downloads or similar. As Joe St. Sauver pointed
out to me, "a million compromised systems a day is quite reasonable,
actually (you can track it by rsync'ing copies of the CBL and cummulating
the dotted quads over time)".

So I'm genuinely baffled. I'd like someone to explain to me why this
seems implausible.

BTW #1: I'm not asserting that my little January experiment is the basis
for such an estimate. It's not. It wasn't intended to be, otherwise
I would have used a very different methodology.

BTW #2: All of this leaves open an important and likely-unanswerable
question: how many systems are compromised but not as yet manifesting
any external sign of it? Certainly any competent adversary would hold
a considerable fraction of its forces in reserve. (If it were me,
that fraction would be at least "the majority".)

---Rsk

6

Hi Rich,

<snip good stuff> thanks for your input, Rich. As always, quite
interesting.

BTW #2: All of this leaves open an important and likely-unanswerable
question: how many systems are compromised but not as yet manifesting
any external sign of it? Certainly any competent adversary would hold
a considerable fraction of its forces in reserve. (If it were me,
that fraction would be at least "the majority".)

I stopped really counting bots a while back. I insisted, along with many
friends, that counting botnets was what matters. When we reached thousands
we gave that up.

We often quoted anti nuclear weapons proliferation sentiments from the
cold war, such as: "why be able to destroy the world a thousand times
over if once is more than enough?" we often also changed it to say "3
times" as redudancy could be important. :>

Today, it is clear the bad guys can get their hands on as many bots as
they need, or in a more scary scenario, want. They don't need that many.

As a prime example, I believe that VeriSign made it public that only 200
bots were used in the DNS amplification attacks against them last
year. Even if they missed one, two or even three zeroes, it speaks quite a
bit as to our fragile infrastructure.

If brute force alone can acheive this, what of application
attacks, perhaps even 0days? :slight_smile:

Still, we keep surviving and we will still be here next year, too, with
bigger and bigger trucks and tubes to hold the Internet together, whether
for regular or malicious usage. eCommerce and online banking might not
survive in a few years if people such as us here don't keep doing what we
do, but that part of it is off topic to NANOG.

10 years ago, almost no one knew what botnets were. Counting and
measuring seemed to be very important 3 years ago, and to governments and
academics, and even a year ago. Today it is just what funding for botnet
research is based on ( :slight_smile: ), still, I don't really see the
relevance. Botnets are a serious issue, but they are only a sympthom
of the problem called the Internet.

Sitting on different networks and testing them for how many malicious
scans happen every second/minute/hour/day and then checking that against
how many machines with trivially exploited vulnerabilities exist on these
networks can fill in some of the puzzlea, but the delta from what we may
see if we consider email attachments and malicious web sites...

The factor may be quite big.

We will never be able to count how many bots exist. We can count limited
parts of that pool such as those seen in spam. Those are several millions
every day (which should be scary enough) but not quite the right number.

And this is before we get into the academic off-topic discussion of what a
bot actually is, which after almost 11 years of dealing with these I find
difficult to define. Is it an IP address? A computer? Perhaps an instance
of a bot (and every machine could have even hundreds).

Welcome to the realm of Internet security operations and the different
groups and folks involved (and now industry). It is about Internet
security rather than this or that network security or this and that sample
detection.

---Rsk

  Gadi.

7

If you can't measure a problem, its difficult to tell if you are
making things better or worse.

I don't understand why you don't believe those numbers. The estimates
that people are making are based on externally-observed known-hostile
behavior by the systems in question: they're sending spam, performing
SSH attacks, participating in botnets, controlling botnets, hosting
spamvertised web sites, handling phisher DNS, etc. They're not based
on things like mere downloads or similar. As Joe St. Sauver pointed
out to me, "a million compromised systems a day is quite reasonable,
actually (you can track it by rsync'ing copies of the CBL and cummulating
the dotted quads over time)".

Counting IP addresses tends to greatly overestimate and underestimate
the problem of compromised machines.

It tends to overestimate the problem in networks with large dynamic
pools of IP addresses as a few compromised machines re-appear across
multiple IP addresses. It tends to underestimate the problem in
networks with small NAT pools with multiple machines sharing a few IP
addresses. Differences between networks may reflect different address
pool management algorithms rather than different infection rates.

How do you measure if changes are actually making a difference?

8

NAT on the one end, DHCP on the other. Time-based calculations along with
OS/Client fingerprinting often seem to produce interesting results.

9

Yes, but (I think) we already knew that. If the goal is to provide
a minimum estimate, then we can ignore everything that might cause
an underestimate (such as NAT). In order to avoid an overestimate,
multiple techniques can be used. For example, observation from multiple
points over a period of time much shorter than the average IP address
lease time for dynamic pools, use of rDNS to identify static pools,
use of rDNS to identify separate dynamic pools (e.g., a system which
appears today inside hsd1.oh.comcast.net is highly unlike to show up
tomorrow inside hsd1.nj.comcast.net), classification by OS type (which,
BTW, is one way to detect multiple systems behind NAT), and so on.

I think Gadi makes a good point: in one sense, the number doesn't really
matter, because sufficiently clueful attackers can already lay their
hands on enough to mount attacks worth paying attention to.

On the other hand, I still think that it might be worth knowing, because
I think "the fix" (or probably more accurately "fixes") (and this is
optimistically assuming such exist) may well be very different if we
have 50M than if we have 300M on our hands.

---Rsk