Reasons why BIND isn't being upgraded

Apologies for the delay - but I read the list in digest and this is my
first post.

I have done a slightly bigger survey of BIND version strings as of
16:00GMT Thursday February 1st, taking the nameservers for Keynote40
Business to Consumer sites and a selection of the domainnames from the
front page of www.persiankitty.com.

Of 440 DNS servers surveyed;
35% are reporting strings that clearly indicate use of 8.2.3 or better
(Hardly anyone is revealed to be using 9.0 or 9.1).
35% are reporting strings that strongly suggest they are susceptable to
known BIND bugs (Some are rather old versions of BIND are out there).

These figures charitably assume that;
people who don't reveal version strings are not afflicted by BIND bugs.
Bind variants that have source code modifications aren't affected -
they probably are, but I said we were being charitable.

So in answer to the question - are people too busy to upgrade BIND -
clearly 35% of admins found time to do it already. Those who had
upgraded to 8.2.2-P7 to avoid the previous DoS bugs should have had
nothing more to do than replace the binaries and test at 8.2.3, so I
think the time excuse is a bit thin.

Interestingly the sexsites were only marginally behind the Keynote40
sites, in terms of DNS versions. So perhaps your credit card number is
nearly as safe as your share portfolio.

Of the strings returned the most humorous was "SIN" from a sexsite. The
most constructive administrator entry was the admin e-mail address (But
since this is in the SOA resource record and RFC2142 (you do have all
the RFC2142 mail addresses don't you?!) I'm not awarding any prizes).

The ISC.ORG web site recommends leaving the BIND version string
unchanged to assist in troubleshooting.

I remain unconvinced that showing the version string helps much.
Certainly any site with 7x24 hour technical support should consider
putting a relevant contact phone number IMHO - such that anyone trying
to diagnose a genuine problem can get help, whilst nosey crackers can
revert back to social engineering. (E-mail addresses may not be a good
choice if your DNS or their DNS is playing up?!).

Whilst you can add a "bind" zone of type Chaos and overwrite the default
behaviour, as given in an earlier post. Later versions of BIND let you
drop the version string in the options section, which is a lot less
typing, and a lot easier to understand. An extract of my BIND 9.1rc1
named.conf is given below, clearly my server is only listening to the
loopback address, but you'd be amazed how many times it has been queried
in the last couple of weeks whilst I got the scripts sorted.

options {
        directory "/etc/namedb"; // Working directory
        pid-file "named.pid"; // Put pid file in working dir
        listen-on { 127.0.0.1; }; // private server for
performance
        listen-on-v6 { none; } ; // No version 6 IP
        version "Contact +44(0)1395 232769"; // Don't release
version number
};

version ""; doesn't do what I expected at 9.1rc1.....

The survey mentioned above will form part of a bigger report, which was
inspired by the Microsoft debacle (Aren't those Akamai DNS servers
weird, oh and all Microsoft's e-mail relays have sequential IP addresses
- what planet are they on), so I will be checking which domains have all
their DNS servers (for which glue records exist) in the same network
(Based on 'NetName' - better ideas welcome from routing experts - can I
easily get IP to ASN?). If I have time I may include mail servers and
relays as well as DNS servers, of course if all your DNS servers are on
one network having off network mail relays won't help much when you muck
up the routing.

  HTH

  Simon

This is untrue. I expected this same thing. Then I ran into these gems of
bogosity while updating 8.2.2-P7 to 8.2.3:

(1) 8.2.3 Doesn't accept the "(" in the SOA string to be on the next line
    after the IN SOA. Our script-generated zonefiles, about 45000 of them,
    all had this.
(2) 8.2.3 Changed the meaning of the last field of the SOA record and
    needs a $TTL directive to cover the default TTL. This also affected
    all of our zones (86400 seconds timeout on negative caching is, you
    must agree, way over the top so not a value you want to propagate).
(3) 8.2.3 Is unforgiving against errors in zonefiles. Where previously
    individual records were rejected (or served as-is), bind now insists
    on dropping the entire zone if something went wrong. Needless to say
    in a reload of 45K domains it takes a bit of time to fish out the
    bad ones.

When downloading I expected a security upgrade, not a service pack. The
extra traffic that new serial numbers for this amount of domains generate
is probably well-measurable. The webpage, nor any of the obvious
documentation (README, CHANGES) mentions any of these problems and I've
been bitten by them. Yes we're running 8.2.3-REL fine now, but it took a
couple of _expensive_ reloads to get everything right. If ISC wants my
trust in the future of their codebase, they will have to work on seeing
the difference between an "architecture upgrade" and a "security patch".

Yes, the information was out there, but in the context it was presented
(update now or the internet will die) I think they should make a stronger
point out of the pitfalls. And you can certainly expect more people to run
into these problems, then shrug and roll back to 8.2.2P7 because they
bumped into the same wall.

Cheers,
Pi

Simon@wretched.demon.co.uk (Simon Waters) writes:

The ISC.ORG web site recommends leaving the BIND version string
unchanged to assist in troubleshooting.

I remain unconvinced that showing the version string helps much.

it helped you with your survey, didn't it?

hiding it doesn't help at all. people who want to know if you're vulnerable
and to what have tools to find out.

hiding it DOES however make it harder for people (including network owners)
to do surveys.

until, that is, somebody breaks into a server using some published hole and
then modifies the version string so the admin's periodic audit won't show it
as needing to be upgraded. so -- don't believe it if it says you're safe,
use indications of unsafety as reasons to prioritize those servers for a
closer audit.

pi@vuurwerk.nl (Pim van Riezen) writes:

bogosity while updating 8.2.2-P7 to 8.2.3:

(1) 8.2.3 Doesn't accept the "(" in the SOA string to be on the next line
    after the IN SOA. Our script-generated zonefiles, about 45000 of them,
    all had this.

Neither do the relevant RFC's, or any other DNS implementation. Pre-8.2.3
was simply _wrong_ to accept that syntax.

(2) 8.2.3 Changed the meaning of the last field of the SOA record and
    needs a $TTL directive to cover the default TTL. This also affected
    all of our zones (86400 seconds timeout on negative caching is, you
    must agree, way over the top so not a value you want to propagate).

This also is per several (recent) RFC's, and again, pre-8.2.3 was simply
_wrong_ in its use of the SOA.MINTTL as a default TTL for the whole zone.

(3) 8.2.3 Is unforgiving against errors in zonefiles. Where previously
    individual records were rejected (or served as-is), bind now insists
    on dropping the entire zone if something went wrong. Needless to say
    in a reload of 45K domains it takes a bit of time to fish out the
    bad ones.

A zone either has an identity or it doesn't. There's no such thing as a
best effort identity. If the file is not syntactically valid, it's not a
zone and ought not be served, since it has no specific identity for the
serial number to map to.

When downloading I expected a security upgrade, not a service pack.

You and a lot of other people. 8.2.2-P8 will be along shortly.

I always thought that it was regarded as generally good security practice to
give out as little information about your systems as possible, and none at
all if you can help it. The BIND version should at least only be accessible
from a set of defined IP addresses, defaulting to 127/8.

--Adam

Pim van Riezen (pi@vuurwerk.nl) said, on [010201 17:29]:

This is untrue. I expected this same thing. Then I ran into these gems of
bogosity while updating 8.2.2-P7 to 8.2.3:

(1) 8.2.3 Doesn't accept the "(" in the SOA string to be on the next line
    after the IN SOA. Our script-generated zonefiles, about 45000 of them,
    all had this.

Not accepting a bogus zone file is hardly classifiable as "bogosity".

documentation (README, CHANGES) mentions any of these problems and I've
been bitten by them. Yes we're running 8.2.3-REL fine now, but it took a
couple of _expensive_ reloads to get everything right. If ISC wants my
trust in the future of their codebase, they will have to work on seeing
the difference between an "architecture upgrade" and a "security patch".

So, you deployed a new version of bind to a non-trivial set of
production servers without doing any testing on development or QA
systems, and you're blaming your production problems on the isc? I'm
fairly certain that I'm glad you're not running my network,
thankyewverymuch.

-P.

Pim van Riezen (pi@vuurwerk.nl) said, on [010201 17:29]:
>
> This is untrue. I expected this same thing. Then I ran into these gems of
> bogosity while updating 8.2.2-P7 to 8.2.3:
>
> (1) 8.2.3 Doesn't accept the "(" in the SOA string to be on the next line
> after the IN SOA. Our script-generated zonefiles, about 45000 of them,
> all had this.

Not accepting a bogus zone file is hardly classifiable as "bogosity".

Parsing human input isn't hard, you know. Robustness doesn't come from
being anal. If there's a bogus entry, reject the entry not the entire
zone. The rejection as such doesn't even classify as bogosity, it's the
fact that this rejection is _introduced_ in a 0.0.1 upgrade that is
advertised as an Urgent Security Fix and is being discussed right here on
this list in wonder considering why some people haven't upgraded yet. I'm
telling you that if I run into these problems (and manage to eventually
fix them) others will too and it will be for these reasons.

I also seriously counter your claim that having this bracket on the next
line is in any way bogus. It's visually superior to the now enforced
option of having it on the same line. There is nothing in the parser not
to understand it. Spreading data across lines is commonly accepted in a
lot of configuration languages and bind has been among this in all
versions I previously ran. Why is that now suddenly bogus?

> documentation (README, CHANGES) mentions any of these problems and I've
> been bitten by them. Yes we're running 8.2.3-REL fine now, but it took a
> couple of _expensive_ reloads to get everything right. If ISC wants my
> trust in the future of their codebase, they will have to work on seeing
> the difference between an "architecture upgrade" and a "security patch".

So, you deployed a new version of bind to a non-trivial set of
production servers without doing any testing on development or QA
systems, and you're blaming your production problems on the isc? I'm
fairly certain that I'm glad you're not running my network,
thankyewverymuch.

I followed all-out instructions to immediately upgrade to the new bind
because of alleged gaping security holes. We are short on staff on a major
scale[1] and our secondary was coping just fine. My complaint is not that
"ISC broke my network", noone did. My complaint was that I had to spend a
lot of time figuring out what should be blisteringly obvious.

I thank you for your character judgement anyway. May I add that I'm glad
I'm not working on your network, too? Your noc-list must be really cozy.
Hope you do get some work done besides excercising quick wit and
situational prejudice on other people.

Cheers,
Pi

[1] I'm the head of development, that's how short we are on admin staff
    ok? And this situation is not unique. Welcome to the real world.
    There are more Wanters in this world than there are Makers.

pi@vuurwerk.nl (Pim van Riezen) writes:

When downloading I expected a security upgrade, not a service pack.

yeah, i'm pissed with isc and that vixie guy too.

after all, i paid them 0's and 0's of dollars to come up with a timely
fix to the security hole, and what do they do?

fixed a bunch of damned bugs!

what a bunch of bastards!

[ big 8^) for those who don't see the sarcasm ]

According to RFC1035, it's ALWAYS been bogus. It's just been a bug that
BIND's parser that accepted the bogus data.

It's funny that when I reported this as a bug when bind9 testing was going
on, ISC was like "it accepts paren on the next line?" ... since bind9 was
written from scratch, it didn't inherit the legacy bogosity of accepting
\n(, and crapped out, just as 8.2.3 did. It appears that my (and others,
I'm sure) bug report on 9.0.0beta-whatever caused the re-evaluation of the
8.2.x codebase to fix that bug in 8.x.

So I guess I'll apologize for opening my trap. :wink:

D

Pim van Riezen (pi@vuurwerk.nl) said, on [010201 18:58]:

Parsing human input isn't hard, you know. Robustness doesn't come from
being anal. If there's a bogus entry, reject the entry not the entire
zone. The rejection as such doesn't even classify as bogosity, it's the

I fail to understand this. You seem to suggest that a name server should
reject the SOA record, but accept and attempt to serve the zone.
Precisely how would that work?

I also seriously counter your claim that having this bracket on the next
line is in any way bogus. It's visually superior to the now enforced
option of having it on the same line. There is nothing in the parser not
to understand it. Spreading data across lines is commonly accepted in a
lot of configuration languages and bind has been among this in all
versions I previously ran. Why is that now suddenly bogus?

Because rfc1035 has always defined it as bogus. The parenthesis is, as
you are now no doubt aware, a line continuation character:

5.1. Format

The format of these files is a sequence of entries. Entries are
predominantly line-oriented, though parentheses can be used to continue
a list of items across a line boundary, and text literals can contain
CRLF within the text. Any combination of tabs and spaces act as a
delimiter between the separate items that make up an entry. The end of
any line in the master file can end with a comment. The comment starts
with a ";" (semicolon).

-P.

[ On Thursday, February 1, 2001 at 21:13:20 (-0500), Adam McKenna wrote: ]

Subject: Re: Reasons why BIND isn't being upgraded

I always thought that it was regarded as generally good security practice to
give out as little information about your systems as possible, and none at
all if you can help it. The BIND version should at least only be accessible
from a set of defined IP addresses, defaulting to 127/8.

Not necessarily.

As Paul has shown, and as I and others have explained in other forums,
hiding the version identifier in this case can obscure the presense of
an older buggy version that's in desparate need of upgrading.

Only the most simplistic and poorly designed exploits would trust the
version identifier anyway, *especially* after these kinds of discussions! :wink:

Never try to hide something that's plainly obvious on some other level.
It only makes people more curious, and I'm including those wearing grey
and black hats in "people" here.....

[ On Friday, February 2, 2001 at 02:22:56 (+0100), Pim van Riezen wrote: ]

Subject: Re: [NANOG] Re: Reasons why BIND isn't being upgraded

This is untrue. I expected this same thing. Then I ran into these gems of
bogosity while updating 8.2.2-P7 to 8.2.3:

(1) 8.2.3 Doesn't accept the "(" in the SOA string to be on the next line
    after the IN SOA. Our script-generated zonefiles, about 45000 of them,
    all had this.

I seem to recall having that problem with much older versions (maybe
even 4.9.x) and I've used this format for a very long time now:

@ IN SOA localhost. postmaster.localhost. (
        2000041800 ; Serial number (yyyymmddhh)
        8h ; Refresh
        2h ; Retry
        1w ; Expire
        3h ) ; Minimum TTL

(except of course for the more sensible units of time now allowed)

(2) 8.2.3 Changed the meaning of the last field of the SOA record and
    needs a $TTL directive to cover the default TTL. This also affected
    all of our zones (86400 seconds timeout on negative caching is, you
    must agree, way over the top so not a value you want to propagate).

The use of $TTL was well documented since at least 8.2.1, and indeed
recent versions also syslog'ed complaints if it was missing. This is
from doc/html/master.html:

    $TTL

   Syntax: $TTL <default-ttl> [<comment>]

   Set the default Time To Live (TTL) for subsequent records with undefined
   TTL's. Valid TTL's are of the range 0-2147483647.

   $TTL is defined in RFC 2308.

(any specification of units of time can be used in place of a plain
integer number of seconds). And of course RFC 2308 is from 1998, so not
very new. I've been slowly adding $TTL to manually edited zone files
for the past couple of years and didn't have really all that many to fix
in the past few days.

(3) 8.2.3 Is unforgiving against errors in zonefiles. Where previously
    individual records were rejected (or served as-is), bind now insists
    on dropping the entire zone if something went wrong. Needless to say
    in a reload of 45K domains it takes a bit of time to fish out the
    bad ones.

So far as I saw 8.2.2-P7 logged all of these errors and so could have
given you ample time to fix up all of the problems just like it did for
me.

Of course I've had "check-names master fail;" in my options clause for a
couple of years too, so I don't notice any difference with 8.2.3.

In any case it is far better to drop the entire zone than it is to load
a borked one, especially if you don't catch it in time before a half
dozen secondaries all tranfer the damage. First off it gives you a much
more direct indication of the presense of a problem (your server is
suddenly lame for the zone so even if you forget to read the log
messages after a reload you'll still find out about the problem).
Furthermore your secondaries will still continue to answer
authoritatively with the previous revision, at least until they expire,
thus preventing unexpected damage (the presense of the old version is
expected, at least for some limited amount of time, after all)

When downloading I expected a security upgrade, not a service pack.

As the version number indicated you installed a new release, not a patch
(though in the past even BIND patch releases with "-PN" suffixes on
their identifiers have often included more than just pure fixes). Any
presumptions about what a +0.0.1 version number increment means are
likely to be incorrect unless you have intimate knowledge of the project
producing the release.

Unfortunately there was no patch this time.... I suspect a patch could
have been possible, and it seems one may soon be on its way, but given
the relative history of the deployment of new releases of BIND, or lack
thereof, it does in some ways make sense to only make a new release
available. As a developer you no doubt know full well that developing
patch releases in conjunction with full releases more than doubles the
amount of effort necessary for SCM and QA tasks.

Unfortunately BIND has never come with the equivalent of a GNU "NEWS"
file to mention explicitly all of the user-visible differences and with
all new releases it sometimes a bit of an adventure to discover all the
new features and any incompatibilities.

Out of curiosity, why?

Okay, we'll accept that that's the way it is, and I'm not going to rail
about how it shouldn't be, but let's try to understand Mr. Mockapetris'
reasoning in this. I see this from the RFC (that lists P. Mockapetris as
author, for those who don't place the name):

The format of these files is a sequence of entries. Entries are
predominantly line-oriented, though parentheses can be used to continue
a list of items across a line boundary, and text literals can contain
CRLF within the text.

also:

( ) Parentheses are used to group data that crosses a line
                boundary. In effect, line terminations are not
                recognized within parentheses.

The only clue I can find to his reasoning is:
Because these files are text files several special encodings are
necessary to allow arbitrary data to be loaded.

...which is patently untrue, given that BIND didn't need those special
encodings; it worked just fine without the particular one that we're all
beyond bored of hearing about.

So: Why? The fact that pre-8.3 versions of BIND will accept
@ IN SOA
                    ( ns.whomever.com

and etc. does not seem like a huge issue to me, though fixing that in
thousands of issues does. What's the big deal with leaving it
non-compliant?

Matthew Devney

[ On Thursday, February 1, 2001 at 22:16:13 ( -0800), mdevney@teamsphere.com wrote: ]

Subject: Re: [NANOG] Re: Reasons why BIND isn't being upgraded

Out of curiosity, why?

Because (as you quoted from RFC 1035):

The format of these files is a sequence of entries. Entries are
predominantly line-oriented, though parentheses can be used to continue
a list of items across a line boundary, and text literals can contain
CRLF within the text.

I.e. I thought it was self-obvious.

Records are one per line unless an open parenthesis `(' or double quote
`"' is encountered in which all fields up to the closing `)' or `"' are
part of the current record even if there are intervening newlines.

What more justification do you need?

The parenthesis aren't part of the record -- they're just a way of
getting the additional values onto separate lines. If you put all those
numbers on the same line with the rest of the record then you don't need
the parenthesis at all. This was one of the very first lessons I
learned when I wrote my very first zone file and tried to do something
with parenthesis that didn't work like I naively expected it to.

Yes the master file format goes against the grain of modern data file
interpretations, but, well, nobody's written a new RFC and proposed that
it replace 1035 yet, so we're still stuck with it. Personally I'd like
to at least do away with the semi-colon as the comment character, but
I'm not going to write a new RFC and push it through the IETF just for
that! :wink:

The only clue I can find to his reasoning is:
Because these files are text files several special encodings are
necessary to allow arbitrary data to be loaded.

well you might want to look at the date on RFC 1035, and even more
particularly at the dates of the RFCs it supercedes. Back in 1983 the
tools available to deal with this kind of data were very different and
possibly much more restricted. All the world was not yet a VAX back
then, it was still a PDP-11! :slight_smile:

However the real clue is (again from RFC 1035):

    5.1. Format

    [[ ... ]]

    The following entries are defined:

    [[ ... ]]

        <domain-name><rr> [<comment>]

        <blank><rr> [<comment>]

    [[ ... ]]

    The last two forms represent RRs. If an entry for an RR begins with a
    blank, then the RR is assumed to be owned by the last stated owner. If
    an RR entry begins with a <domain-name>, then the owner name is reset.

I.e. if a line begins with whitespace then it is a new record that
defaults to having the same domain name (and class) as the previous
record.

So, a modified example like the one you've used for the SOA:

  @ IN SOA master contact
        ( 1 2 3 4 5 )

is interpreted as this:

  @ IN SOA master contact
  @ IN 1 2 3 4 5

Now what? Both records are broken, with missing fields for the SOA, and
an invalid type (`1') and probably too many fields for the next line!

...which is patently untrue, given that BIND didn't need those special
encodings; it worked just fine without the particular one that we're all
beyond bored of hearing about.

Sure, BIND was OK when handling this one very special case, but since
the master file format is intended to be portable to other
implementations, the question to as is whether some other conforming
implementation could mis-interpret this kind of error and cause the zone
to be rejected, and the answer is most certainly YES!

Sure BIND could do away with support for RFC 1035 standard master file
formats, but where would that get us with so many other tools generating
and parsing these files?

So: Why? The fact that pre-8.3 versions of BIND will accept
@ IN SOA
                    ( ns.whomever.com

I'm not so sure they all can. You might want to check the pre-8.x
versions before asserting this. IIRC this sloppyness is a relatively
recent abberation in BIND.

So far as I saw 8.2.2-P7 logged all of these errors and so could have
given you ample time to fix up all of the problems just like it did for
me.

In an ideal world, I agree with you. The base of all of this discussion
was how it can be that so many people take a while upgrading bind. I think
the situation I ran into is a realistic real world scenario. Understaffed
organizations will run into their own walls pretty easily when trying to
tackle the upgrade.

[snip serving borken zones is bad]

I am near to agreeing with you if it were about not picking up a
zone-change when the zonefile has turned bogus. However, the effect of a
zone no longer being authoritative on the primary is not really what I'd
define as fun either :).

Unfortunately BIND has never come with the equivalent of a GNU "NEWS"
file to mention explicitly all of the user-visible differences and with
all new releases it sometimes a bit of an adventure to discover all the
new features and any incompatibilities.

Voila, I think that this is what my problem was. Like I said, the
information _was_ out there, it was just not intuitively available. So the
upgrade will scare some people off, if they don't manage to find it.

Cheers,
Pi

Voila, I think that this is what my problem was. Like I said, the
information _was_ out there, it was just not intuitively available. So the
upgrade will scare some people off, if they don't manage to find it.

You are in an unenviable position. But in support of ISC, this problem is
listed in FAQ which is reachable through www.isc.org.

By the same token one might argue that atempting to hide vunerabilities
to those paying you for "early warnings" doesn't help at all.

Just something to consider.

Not at all... If you're trying to hide a vulnerability by lying about
your version number, that presupposes generally-held knowledge of an
association between a vulnerability and a version number.

"Early warning" is specifically a means of delaying the general
availability of knowledge of that association.

These are temporally sequential states. Not comparable strategies within
the same context.

                                -Bill

[ On Friday, February 2, 2001 at 12:48:50 (+0100), Pim van Riezen wrote: ]

Subject: Re: [NANOG] Re: Reasons why BIND isn't being upgraded

> [snip serving borken zones is bad]

I am near to agreeing with you if it were about not picking up a
zone-change when the zonefile has turned bogus. However, the effect of a
zone no longer being authoritative on the primary is not really what I'd
define as fun either :).

Well, strictly speaking not dropping the zone when any error is
encountered during its load is contrary to the requirements of RFC 1035.
(section 5.2, which gives very much the reasons I did, but without
mentioning zone transfers explicitly since of course any errant record,
or missing record, can be propogated for its TTL or negative TTL)

It might not be fun to have your primary be lame for one or zillions of
zones (even if it's an unadvertised primary), but it's not dangerous (at
least not unless you're already violating dozens of other DNS
requirements). The "non-fun" should merely be incentive to get you to
correct your procedures and process so that future errors are caught
before they're loaded. :slight_smile:

Which leaves those that have not been informed of such vunerabilities
acutely vunerable.

Script kiddies may be stupid, but the people writing the program that they
utilize generally aren't.

Without rehashing the whole "open-disclosure" vs. "non-disclosure"
arguments related to security issues in software, or the historically
extreme inadequacies of CERT in offering timely notification of ANY
security-related issues, it's very disappointing to see ISC resort to a
fee-based, non-public-disclosure-at-the-time-of-discovery, NDA'd and
"we'll update people via CERT" method of dealing with the community they
have served for so long.

I would have hoped by now that lists such as Bugtraq would have adequately
exhibited the folly of such methodologies.

Obviously that is not the case.