RE: botnets: web servers, end-systems and Vint Cerf

I've concluded three things (by doing experiements like
that). (a) Where
there are Windows boxes, there are zombies. "Securing
Microsoft operating
systems adequately for use on the Internet" is not a solved problem in
computing.

I disagree. Since 1994 I have been in the habit of setting up MS Windows
boxes with Win98 and up, by installing from CD, connecting to the net
and installing various patches and updates from the Windows Update
service. I've never had a virus infection, a bot, a root kit or
whatever. The secret is simple. These machines never connected directly
to the Internet but went through a NAT box. Way back when it was a
FreeBSD machine running TIS Firewalls Toolkit. These days it is an
off-the-shelf Ethernet switch with DSL modem and NAT built-in.

Therefore, I assert that securing systems adequately for use on the
Internet is indeed a SOLVED PROBLEM in computing. However, it isn't yet
solved in a social or business sense. On the business side, I wonder why
PC's don't come with a built-in firewall/NAT device. It is cheap enough
to do these days. This means that a computer would have no Ethernet
ports on it. Instead, an internal Ethernet port would be directly
connected to a NAT/firewall device on the same circuit board (or via
PCI/PCMCIA/etc.). The external Ethernet port would belong to the
firewall/NAT device. On the social side, people don't realize that such
a solution is possible and therefore they aren't demanding computer
vendors to build it in. The box vendors only build what the OS vendors
want and the OS vendors are not interested in a piece of hardware that
runs its own OS, most likely FreeBSD or Linux.

In the UK, companies who sell TV services (cable and satellite) give
there customers a box to connect with. Why can't ISPs also sell their
services with a proper box included? By proper, I mean a NAT/firewall,
not a USB-connected DSL modem.

(c) Amusingly, it's possible
to detect new end-user allocations and service rollouts by noting when
spam starts to arrive from them. (e.g. the Verizon FIOS
deployment, if I
may use hostnames of the form *.fios.verizon.net as a guide, is going
well in NYC, Dallas, DC, Tampa, Philly, LA, Boston and
Newark, but lags
behind in Seattle, Pittsburgh, Buffalo and Syracuse.)

I wonder if Verizon is violating any SEC rules by not reporting this
information publicly? This is a good example of something that would not
be revealed if they provided a NAT/firewall box to every customer who
didn't already have one.

Has anyone implemented a tool that ISPs could use to detect whether or
not a NAT/firewall device is present? Perhaps based on OS
fingerprinting? Or even based on an agent that must be installed by the
customer? If such tools are available then an ISP could offer customers
a discount for being compliant with a NAT/firewall rule in their
contract.

--Michael Dillon

* michael.dillon@bt.com (michael.dillon@bt.com) [Fri 16 Feb 2007, 17:31 CET]:
[..]

Therefore, I assert that securing systems adequately for use on the Internet is indeed a SOLVED PROBLEM in computing.

A HUNDRED MILLION machines beg to differ.

  -- Niels.

Yes, p0f detects when hosts are behind a nat gateway.

I really don't want to get into an OS debate here, but this does
have major operational impact, so I will anyway but will be as
brief as possible. Please see second (whitespace-separated) section
for some sample hijacked system statistics which may or may not
reflect overall network population.

I disagree. [...]

Therefore, I assert that securing systems adequately for use on the
Internet is indeed a SOLVED PROBLEM in computing. However, it isn't yet
solved in a social or business sense.

I think I understand your point about the social and business sense of the
problem; if so, then we're probably in at least rough agreement on that.
People do stupid things with computers (like reading email with a web
browser, or replying to spam) and it's proven to be very difficult to
convince them to stop doing those things.

I'm reminded of Ranum's point (from
The Six Dumbest Ideas in Computer Security ) about
how if user education was going to work...it would have worked by now.
I think the ongoing success of phishing operations, including those run
by illiterate amateurs, in face of massive publicity via nearly every
communications channel society has to offer, illustrates it nicely.

But, and this may be where we disagree, it's not solved where Microsoft
operating systems are concerned -- and I don't accept the notion that
just putting such systems behind a firewall/NAT box is adequate.
(I'll also argue that any OS which *requires* an external firewall
to survive more than a few minutes' exposure is unsuitable for use
on the Internet. *Not good enough*.)

But suppose you put such a firewall in place. You'll need to
configure the firewall properly -- paying as much attention to
outbound rules as inbound. (And how many people ever do that? Even
on corporate networks, there are still people stunningly incompetent
enough to use default-permit policies on outbound traffic. And
controlling outbound traffic from these systems is arguably more
important than controlling inbound -- inbound likely only abuses
the owner, outbound abuses the entire Internet.)

You'll need to add anti-virus software. And anti-spyware software.
Then you need to make sure the "signature" databases for both of those
are updated early and often, keeping in mind that you have now elected
to play a game that you will inevitably lose the first time that new
malware propagates faster than the keepers of those databases can develop
and distribute signatures. Vegas lives for suckers like this.

And you'll need to de-install IE and Outlook, since
everything else you've done will be defeated as soon as the next
IE/Outlook-remotely-exploitable-and-leading-directly-to-
full-system-compromise-here's-a-working-demo is published on
full-disclosure, which should be, oh, about three hours from now.

And this is before we even get to the licensing and DRM backdoors
*designed into* Vista.

Something which requires this much work just to make it through its
first day online, while being used by J. Random Person, is hopelessly
inadequate. Which is why systems like this are routinely compromised in
huge numbers. Which is why we have a large-scale problem on our hands.

Which brings me to the second point, and that is skepticism over the
100M ballpark figure that's been bandied about. Personally, I wouldn't
even blink if someone produced convincing proof that the real number
was 300M. I think that's completely plausible -- "plausible" but still,
I very much hope, unrealistically high. So from my point of view, this
100M stuff is old news -- i.e., I'm telling you the ocean is wet.

A tiny example: some data (summarized below) from a small experiment last
month using a single test mail server. I threw away all the data blocked
outright by the firewall in front of it. I threw away all data that didn't
involve connections directed at port 25. I threw away all the data for
connecting hosts without rDNS. I threw away all the data for connecting hosts
with rDNS that looked even vaguely server-like. I threw away repeat visits.
All of which means that my sampling method is akin to waving a thimble in
a hurricane and will thus provide a gross (and likely skewed) underestimate.

This left me with >1.5M observed hosts seen in a month. They're all sending
spam. (How do I know? Because 100% of the mail traffic sent to that
server is spam.) And they're all running Windows, except for a handful
which aren't or which were indeterminate. Note that rDNS lookups were
from local long-lived cache, so rDNS may be well out-of-date in some cases.

Some random examples:

  41.241.32.87 dsl-241-32-87.telkomadsl.co.za
  89.28.3.133 89-28-3-133.starnet.md
  190.49.152.243 190-49-152-243.speedy.com.ar
  218.178.50.40 softbank218178050040.bbtec.net
  200.171.123.83 200-171-123-83.dsl.telesp.net.br
  74.132.179.31 74-132-179-31.dhcp.insightbb.com
  61.246.79.101 dsl-del-dynamic-101.79.246.61.airtelbroadband.in
  71.177.11.75 pool-71-177-11-75.lsanca.fios.verizon.net
  82.53.186.23 host23-186.pool8253.interbusiness.it
  200.181.9.48 200-181-9-48.gnace702.dsl.brasiltelecom.net.br
  190.64.51.225 r190-64-51-225.dialup.adsl.anteldata.net.uy
  91.124.188.160 160-188-124-91.pool.ukrtel.net
  201.212.228.70 201-212-228-70.cab.prima.net.ar
  81.192.49.178 adsl-178-49-192-81.adsl.iam.net.ma
  72.74.126.140 pool-72-74-126-140.bstnma.east.verizon.net
  83.110.220.148 dxb-b17260.alshamil.net.ae
  62.42.65.225 62.42.65.225.dyn.user.ono.com
  90.5.6.84 apoitiers-156-1-103-84.w90-5.abo.wanadoo.fr
  70.113.76.57 cpe-70-113-76-57.austin.res.rr.com
  70.224.195.25 ppp-70-224-195-25.dsl.applwi.ameritech.net
  24.17.158.50 c-24-17-158-50.hsd1.mn.comcast.net
  58.69.28.69 58.69.28.69.pldt.net
  61.230.54.51 61-230-54-51.dynamic.hinet.net
  190.76.27.171 190-76-27-171.dyn.movilnet.com.ve
  200.78.237.196 na-200-78-237-196.na.avantel.net.mx
  201.143.64.242 red-corp-201.143.64.242.telnor.net
  87.167.4.103 p57a70467.dip0.t-ipconnect.de
  24.158.153.152 24-158-153-152.dhcp.jcsn.tn.charter.com
  84.158.211.24 p549ed318.dip.t-dialin.net
  86.143.38.135 host86-143-38-135.range86-143.btcentralplus.com
  88.153.92.54 bzq-88-153-92-54.red.bezeqint.net
  83.27.13.15 auf15.neoplus.adsl.tpnet.pl
  64.234.17.88 host-64-234-17-88.nctv.com
  70.156.1.90 adsl-156-1-90.bna.bellsouth.net
  83.152.201.100 dyn-83-152-201-100.ppp.tiscali.fr
  189.136.208.229 dsl-189-136-208-229.prod-infinitum.com.mx
  12.210.197.239 12-210-197-239.client.mchsi.com
  195.14.207.192 xdsl-195-14-207-192.netcologne.de
  201.19.177.223 20119177223.user.veloxzone.com.br
  68.184.147.110 68-184-147-110.dhcp.stbr.ga.charter.com
  77.183.169.130 dsbg-4db7a982.pool.einsundeins.de
  83.40.159.62 62.red-83-40-159.dynamicip.rima-tde.net
  124.255.100.4 pppa509.e12.eacc.dti.ne.jp
  75.87.103.180 cpe-75-87-103-180.kc.res.rr.com
  200.247.145.36 145036.fln.virtua.com.br
  75.41.225.22 adsl-75-41-225-22.dsl.chcgil.sbcglobal.net
  24.71.79.23 s01060050ba8b5c7c.ok.shawcable.net

Some totals by north American ISP:

81828 Comcast
68794 Verizon
60716 Roadrunner
23165 Charter
23099 Pacbell
17981 Ameritech
15855 SWBell
14801 ATTBI
13212 Shaw
  9833 Adelphia
  9769 QWest
  7353 Bellsouth

By other ISP:

  73944 rima-tde.net
  60220 wanadoo.fr
  49730 t-dialin.net
  45285 tpnet.pl
  38587 proxad.net
  32800 hinet.net
  32169 telesp.net.br
  26630 telecomitalia.it
  24665 veloxzone.com.br
  20157 t-ipconnect.de
  19547 arcor-ip.net
  19504 interbusiness.it
  18363 bezeqint.net
  17071 ttnet.net.tr
  15330 prod-infinitum.com.mx
  15136 blueyonder.co.uk

By TLD:

729481 net
217643 com
120017 fr
  93806 br
  75782 pl
  75183 it
  61156 de
  42124 jp
  39376 ar
  34422 il

(.edu checks in with only 424, #59 on the list, by the way.)

Consider what a larger, distributed effort would reveal. *has* revealed.
And there is no reason to think the numbers are going down. There are
a lot of reasons to think the numbers are going up.

Pop quiz: how many of these operations tell their own customers
"we take the spam problem seriously" while at the same time
running networks that double as massive spam generation engines?

  Note: spam is just *one* of the many things that these systems
  are busily engaged in, and it's by far not the nastiest. It just
  happens to be one of the easier things to observe -- a tell-tale,
  if you will.

Pop quiz, bonus round: how much does it cost Comcast to defend its
mail servers from Verizon's spam, and vice versa? Heck, how much
does it cost Comcast to defend its mail servers from its own spam?

Pop quiz, extra super special round: can any of these defend their
networks from a concerted, clueful DDoS attack launched from thousands
of hosts that are *on their network*?

---Rsk

How much do they spend on abuse/customer security? Is it more or less
than they spend on their mail servers? Even if they shifted all the money they spent on the mail departments (opex and capex) to their abuse/customer security departments would it make a significant difference?

Getting money usually isn't that much of a problem, within reason. Figuring out what to spend it on that actually makes a difference is the problem.