The weak link? DNS

Watching the Iraqi Ururklink and Al Jazeera over the weekend what struck
me is how many different ways network administrators can mess up.
Although malicious actors have been trying (and succeeding) to exploit
vulnerabilities, the worst problems seem to be self-inflicted.

Administrators had used firewalls and locked down their web sites,
sometimes so well they couldn't handle the traffic load.

But the real weak link was their DNS servers.

For example, Al Jazeera had time-to-live set of their domain records set
to 15 minutes, making them even more vulnerable to increasing the load
on their systems. Of course, Al Jazeera had other problems too.

What even stranger about the Iraqi state provider Uruklink.net is the DNS
servers are now self-identifying with earlier (with known bugs) versions
of BIND. Last week the Uruklink name server 62.145.94.1 was running
8.2.2-P5, but now is running 8.1.2. Although the web site for
www.uruklink.net is up, DNS lookups for www.uruklink.net return various
other IP addresses (not in 62.145.94.0/24). Including some addresses
running web sites claiming the site is "owned." In reality, the site
isn't owned, you are being redirected to a unrelated web site.

sean@donelan.com (Sean Donelan) writes:

What even stranger about the Iraqi state provider Uruklink.net is the DNS
servers are now self-identifying with earlier (with known bugs) versions
of BIND. Last week the Uruklink name server 62.145.94.1 was running
8.2.2-P5, but now is running 8.1.2. ...

at http://www.isc.org/products/BIND/bind-security.html we see:

  Name: "BIND: Remote Execution of Code"
  [Added 11.12.2002]
  Versions affected: BIND 4.9.5 to 4.9.10
  BIND 8.1, 8.2 to 8.2.6, 8.3.0 to 8.3.3
  Severity: SERIOUS
  Exploitable: Remotely
  Type: Possibility to execute arbitrary code.
  Description:

  When constructing a response containing SIG records a incorrect
  space allows a write buffer overflow. It is then possible to
  execute code with the privileges of named.

the list goes on. i'm sure several folks will use this as an opportunity to
hawk their own alternative non-BIND DNS solution, i wish you well except plz
change the Subject: header on your reply since what i really want to talk
about is: how to get people to upgrade their software when defects are found.

sending out announcements through CERT and the bind-announce m/l isn't working.

so here's a proposal. we (speaking for ISC here) could add a config option
(default to OFF) to make bind send some kind of registration packet at boot
time, containing an e-mail address for a technical contact for that server,
and perhaps its hostname as well. the destination would be configurable, and
the format would be open, and we would include in the distribution a tool
capable of catching these. any campus/WAN admin who wanted to run their own
"BIND registration system" could do so. anyone who wanted to simply config
their server to send registration data to ISC could do so. for data received
at ISC, we'd (a) keep it completely private other than public statistics,
(b) clean it of obvious trash (some people will sent registration data for
president@whitehouse.gov just for fun; we know that), and (c) use the contact
information only in the event that a security defect discovered in that
version. remember, the default would be OFF.

given such a feature, whose default was OFF, would anyone here who uses
BIND stop using it out of protest? if so plz answer publically (on nanog).

given such a feature, would anyone here create their own registration system
so they had their own database of local BIND instances on their campus/WAN,
or would anyone here config their servers to send registration data to ISC?
if so plz answer privately (i'll summarize to the list.)

Personaly I'v not looked favorably at given my email to various lists,
although its probably way too late and everyone by now has it...

1. I have another idea though, during setup of the server ask for email
address of list administrator, but keep that on the server itself.
2. Setup some dns server that provides dns record if there are necessary
updates (here is one example in reverse dns notation...:
1.2.9.bind.updates.isc.org and set it to particular ip or
particular MX or whatever to show if update is needed).
Possibly have this special update dns recorb be HINFO to url where more
information on update is available.
3. When bind starts if the record in #2 exists and shows that update is
necessary, have bind server email to address entered in 1 and with
abscense of that have it email to postmaster@computername.com and if HINFO
exists, it can take that and enter this as custome email.

The above also makes it unnecessary for ISC to maintain that huge email
database or email everybody (and probably get bunch of people angry at you
for spamming...). Anyway just a thought...

[snip]

so here's a proposal. we (speaking for ISC here) could add a config option
(default to OFF) to make bind send some kind of registration packet at boot
time, containing an e-mail address for a technical contact for that server,
and perhaps its hostname as well. the destination would be configurable, and
the format would be open, and we would include in the distribution a tool
capable of catching these. any campus/WAN admin who wanted to run their own
"BIND registration system" could do so. anyone who wanted to simply config
their server to send registration data to ISC could do so. for data received
at ISC, we'd (a) keep it completely private other than public statistics,
(b) clean it of obvious trash (some people will sent registration data for
president@whitehouse.gov just for fun; we know that), and (c) use the contact
information only in the event that a security defect discovered in that
version. remember, the default would be OFF.

I'm not sure this helps. The people who don't subscribe or pay
attention to CERT advisories are the same ones that won't turn this
option on. It is like the cache option in Apache; the people who
would get the most benefit, the ones with mainly static web pages, are
the same ones who do not know to turn it on.

-Dave

In a message written on Wed, Mar 26, 2003 at 04:09:06AM -0500, Sean Donelan wrote:

For example, Al Jazeera had time-to-live set of their domain records set
to 15 minutes, making them even more vulnerable to increasing the load
on their systems. Of course, Al Jazeera had other problems too.

This is very much a double edged sword. If they decided to add
more servers, or Akamize, or set up traditional mirrors a long TTL
would have prevented a large number of people from using them as
ISP's continued to serve up cached entries. Imagine not being able
to get new servers properly loaded for a week if you followed more
traditional TTL guidelines.

For web content, I would recomend a TTL approximately equal to the
longest you would expect a single user to view your web site.
Worst case, if every user had to ask your DNS servers directly,
would then be one query for every web visitor. Given that there
will be caching at larger ISP's and the like it will actually be
even less. If you can scale your web infrastructure to serve the
pages to that number of users, scaling DNS to answer one query for
each one of those users should be a trivial exercise.

In addition to making it easier to change the service on the fly and
have those changes take effect, it also makes it easier on smaller
companies that cache content. How many small ISP's or corporate caching
servers keep entries around for a week or more when one person spent
10 minutes at one site? The shorter TTL allows these boxes to get
rid of the junk entries much faster.

Depending on your web content I'd recommend 15 minute to 1 hour TTL
values.

I would not use such a feature, and I suspect that most people who would
use such a feature would not have a clue that it was there or how to
turn it on. What I would like to see is somewhat of the idea in
reverse. The ISC would host a zone that would contain TXT records with
security/bug advisories for every version:

$ORIGIN .

security-notice.bind IN SOA ns.isc.org. postmaster.isc.org. 1 7200 3600 604800 3600

$ORIGIN security-notice.bind.

8.3.3 IN TXT "Name: BIND: Multiple Denial of Service [yadda yadda yadda...]"
4.9.10 IN TXT "Name: LIBRESOLV: buffer overrun [yadda yadda yadda...]"

yadda yadda yadda...

Ideally the zone would be DNSSEC signed as well.

Then, by default, BIND would query the zone periodically (perhaps every
24 hours or so) for it's version. If any records are found it would log
a message and/or send email to root@localhost, which would be repeated
periodically (I'd log a message every time that a check was performed,
but I'd only email once a week). There would be config options so that
the clueful admin could customize/disable this behavior to his or her
liking.

This way no one would be collecting a central database of email
addresses, but everyone would get notified of security advisories in a
timely manner.

Jeff

I really like this solution. It seems clean and unobjectionable, while
getting the job done.

                                -Bill

Plus, it is expandable to other common services that might want this
service, like sendmail or OpenSSH.

* vixie@vix.com (Paul Vixie) [Wed 26 Mar 2003, 16:24 CET]:

so here's a proposal. we (speaking for ISC here) could add a config option
(default to OFF) to make bind send some kind of registration packet at boot
time, containing an e-mail address for a technical contact for that server,
and perhaps its hostname as well. the destination would be configurable, and
the format would be open, and we would include in the distribution a tool
capable of catching these. any campus/WAN admin who wanted to run their own
"BIND registration system" could do so. anyone who wanted to simply config
their server to send registration data to ISC could do so. for data received
at ISC, we'd (a) keep it completely private other than public statistics,
(b) clean it of obvious trash (some people will sent registration data for
president@whitehouse.gov just for fun; we know that), and (c) use the contact
information only in the event that a security defect discovered in that
version. remember, the default would be OFF.

given such a feature, whose default was OFF, would anyone here who uses
BIND stop using it out of protest? if so plz answer publically (on nanog).

So how much would this differ from `make install' running this shell
script?

Perhaps nameservers could periodically poll

  dig @?.root-servers.net 2.2.9.is-vuln.bind. txt chaos

or something similar; I suggest using roots because DNS queries
to them are far less likely to be filtered. If corresponding RR
is valid (see below), shut down BIND, thus forcing admins to look
into the problem.

Harsh? Yes. Sadly, "it runs, so it must be correct" is far more
common an attitude than "it must be correct before it's allowed
to run".

Worried about spoofing? Distribute the public key with BIND, and
let the TXT response be encoded.

Of course, the clueless generally don't compile from source. I
wonder how long it would be before some vendor circumvented such
controls so their userbase wouldn't be hassled with such
silliness as forced critical upgrades.

Eddy

I can not go into details, but suffice it to say DNS was just a symptom of
other events, not the problem itself. DNS TTL on the global load balancing
system was at 5 seconds and DNS load never rose above trivial.

Al Jazeera's main problem looks like its web site or upstream decided to
drop them as a customer.

i see that a lot of folks are responding publically. sorry to spawn a thread.

niels=nanog@bakker.net (Niels Bakker) writes:

So how much would this differ from `make install' running this shell script?

most bind installations are prefab -- the come with the operating system and
there's no "make install" done after the host has a name.

christian.kuhtz@BellSouth.com ("Kuhtz, Christian") writes:

Administrator inertia is the root cause. I don't see how an automatism such
as the one described changes human behavior. And unless you change that
inertia, no amount of notification, databases, registries, yada yada yada
will make any difference.

this argues for time bombs, where the software will stop working after it
detects some condition (too much time has passed, or an advertisement for
newer software is seen, or a vulnerability notice is seen). this would be
wildly unpopular, contrary to the open source philosophy, and never adopted.

roque@sbcglobal.net (Pedro R Marques) writes:

If you want to address this issue my suggestion would be to make BIND
automatically update itself... and option that needs to default to ON
and that can be disabled in managed systems where admins are expected
to read CERT and act upon it.

this solution implies a trust relationship between a server operator and the
software provider which in fact never exists in reality. even my microsoft
sysadmin friends carefully eyeball any "software update" patch before they'll
put it on production iron. then there's the local customization issue -- and
the binary problem, since many name server hosts do not have compilers. again
this would be contrary to the open source philosophy.

Date: 26 Mar 2003 21:22:59 +0000
From: Paul Vixie

having the server check for updates and issue local mail is

This assumes the configured notification email address remains
valid, and the recipient doesn't sort them into a "I'll get
around to it" folder.

appealing, but i'm more concerned about MIM when fetching
update information than i am with simply registering package
version numbers, hosts, and e-mail addresses.

Distribute BIND with public key. Updates are encrypted or signed
with its counterpart.

Eddy

But don't distributors already provide this service? Several Linux
distributions (at least Redhat and Debian) and Unix companies (Sun
at least) already provide [semi-]automatic updates of packages like bind.
Just look at the vendor list in the average CERT notice.

Someone who downloads, compiles and installs bind directly from the ISC
is already indicating that they want to go beyond the safe vendor supplied
version thats good-enough for 99% of people.

I'm also worried about any concept of trying to "force" people to upgrade,
even with bind I use some features (namely an external named-xfer program)
of bind v8 that arn't available in bind v9 . For the servers which I need
this on I run bind 8.3.3 (Vendor backported with the 8.3.4 fixes) of copy
the named-xfer program over to the bind 9 box.

Date: Thu, 27 Mar 2003 09:55:08 +1200 (NZST)
From: Simon Lyall

I'm also worried about any concept of trying to "force"
people to upgrade, even with bind I use some features (namely
an external named-xfer program) of bind v8 that arn't
available in bind v9 . For the servers which I need this on I

I'm curious... why not use "dig @wherever zone.example axfr"
instead?

run bind 8.3.3 (Vendor backported with the 8.3.4 fixes) of
copy the named-xfer program over to the bind 9 box.

Agreed re the hazards of being heavy-handed. However, I'm sure
there would be those who disabled the automated checkup... the
question is, would they be the crowd for whom the approach in
question was intended?

Eddy

so here's a proposal. we (speaking for ISC here) could add a config option
(default to OFF) to make bind send some kind of registration packet at boot
time, containing an e-mail address for a technical contact for that server,
and perhaps its hostname as well. the destination would be configurable, and
the format would be open, and we would include in the distribution a tool
capable of catching these. any campus/WAN admin who wanted to run their own
"BIND registration system" could do so. anyone who wanted to simply config
their server to send registration data to ISC could do so. for data received
at ISC, we'd (a) keep it completely private other than public statistics,
(b) clean it of obvious trash (some people will sent registration data for
president@whitehouse.gov just for fun; we know that), and (c) use the contact
information only in the event that a security defect discovered in that
version. remember, the default would be OFF.

Isn't the problem with this that in order to get the code out, people need to upgrade and you therefor risk ending up with only notifying the people that upgrade anyway?

- kurtis -

Thinking about it again, this would have additional advantage of
collecting statistics at where bind is being used (you get ips of the
servers) and what versions they are running. So even if they did not
update the software, you can still find out where they are by ip address
and if situation is very very serious, some kind of proactive contact
option would still be available.