isc - a good business

Mailman List Mirror (Read Only)
1

greetings. i didn't notice this before, and i want to complete the record.

i'm paying more attention to the quoting this time, too.

> > Paul will be there to turn things off when
> > they no longer make money for his company.
>
> is the dns changer thingy making money for isc?

no. yes. depends on what you mean.

we're under contract to the department of justice to run the servers. the amount is
intended to be cost-recovery level. (doing stuff like this for nothing does not scale,
unless the business is successful enough elsewhere, and even then there'll be limits.)

From: bmanning at vacation.karoshi.com (bmanning at vacation.karoshi.com)
Date: Wed, 23 May 2012 21:00:27 +0000
Message-ID: <20120523210027.GA26231@vacation.karoshi.com.>

  pretty sure. a contract w/ the Feds, outsouring contracts w/ affected ISPs
  when the Fed deal runs out, development funding to code these kinds of fixes
  into future versions of software, any number of second and third order fallout.

i don't know of any outsourcing contracts that will come into force when the fed deal
runs out. there is no development funding because there are no code fixes for this kind
of problem. i am certainly hoping for second and third order fallout; we pay the people
who write BIND using money collected from other people who buy support for BIND, or
consulting, or training, or custom feature development. that's how we keep BIND free,
and that's how we use a BSD-like license rather than a GPL-like license -- noone who
derives their appliance or product from BIND has any obligation to anybody. (noting,
nlnetlabs for unbound and nsd also use a BSD-like license, so ISC is not alone here.)

  No telling how effective constent self-promotion is. One thing is clear, Paul
  is able to tell a great story.

thanks for your kind words. i've been trying to uplevel my storytelling capability since
i've long since lost my coding skills.

  but its all speculation from here. ISC is well positioned to extract value
  from both ends of the spectrum. They have a great business model. The optics
  look pretty odd from here, at lesat to me however - I am very glad for: )open
  source & )other vendors of DNS SW.

the women and men of ISC work their asses off to keep the world safer and saner. as you
puzzle over our business model please keep in mind that there is no "exit" possibility
here -- no IPO, no buyout. salaries at ISC are fair and reasonable, but nobody's getting
rich doing this work. so while i am likewise glad for open source and for other vendors
of open source DNS software, i'm struggling to grasp the intended suggestion. should ISC
stop giving away software? should we stop adding the features to our software that make it
relevant and desireable? should we stop selling the support and training and consulting
that make it possible to give away this software?

if you have a specific accusation of evil-doing, or a specific suggestion for how ISC can
become more morally pure, then please say exactly what you mean, and we can discuss that.

for more information, see: <http://www.isc.org/community/blog/201001/why-isc-not-profit&gt;\.
the short version is: ISC is a good business, we do good things, mostly for free, but also
for our customers. and we're totally unapologetic about what we do and who we are.

paul

2

fwiw, i think isc and isc staff are very well intentioned and do a lot
of good work for the community. i have doubts about isc's business
model, but definitely not that it makes too much money or is greedy.
maybe a bit too much layer ten for my taste. and i run and appreciate
the software.

randy

3

... maybe a bit too much layer ten for my taste. ...

on that, we're trying to improve. for example, we used to forego
features that some of us found repugnant, such as nxdomain remapping /
ad insertion. since the result was that our software was less relevant
but that there was no reduction in nxdomain remapping as a result of
BIND not providing it. so we dropped some of the layer ten stuff and
moved more in the direction of providing features the community of
interest found relevant. some say we sold out. maybe that's what manning
is on about, i can't tell. the software is free, and isc cherishes our
relevance.

if you catch us doing wierd layer ten stuff that bugs you, give a shout.
maybe we don't really mean it.

... and i run and appreciate the software.

that's why we're here.

paul

4

Thanks for the clarification.

-chris
(another bind user...)

5

To clarify that a bit...

You're saying you used to decline to include in BIND the capability to break
the Internet by returning things other than NXDOMAIN for names which do not
exist...

but now you're *ok* with breaking the internet, and BIND now does that?

If that's what you mean, I'll explain to you why that's a bad layer 10 call.

*Now*, you see, we no longer have a canonical Good Engineering Example to
which we can point when yelling at people (and software vendors) which
*do* permit that, to say "see? You shouldn't be doing that; it's bad."

"The Web Is Not The Internet."

Cheers,
-- jra

6

It's past given that large entities that can forge the use of BIND; at that point, engineering aside, Paul's point that the market and code have spoken is hard to deny.

Sucks when it works against us...

George William Herbert

7

(all caught up after this.)

Jay Ashworth <jra@baylink.com> writes:

From: "paul vixie" <vixie@isc.org>

> ... maybe a bit too much layer ten for my taste. ...

on that, we're trying to improve. for example, we used to forego
features that some of us found repugnant, such as nxdomain remapping /
ad insertion. since the result was that our software was less relevant
but that there was no reduction in nxdomain remapping as a result of
BIND not providing it.

To clarify that a bit...

let's keep trying.

You're saying you used to decline to include in BIND the capability to
break the Internet by returning things other than NXDOMAIN for names
which do not exist...

no, that's not what i'm saying.

but now you're *ok* with breaking the internet, and BIND now does that?

no, that's also not what i'm saying.

If that's what you mean, I'll explain to you why that's a bad layer 10 call.

it's not, but i'm listening.

*Now*, you see, we no longer have a canonical Good Engineering Example to
which we can point when yelling at people (and software vendors) which
*do* permit that, to say "see? You shouldn't be doing that; it's bad."

"The Web Is Not The Internet."

i see what you mean, and i'm sad that this arrow is no longer in your
quiver. perhaps you can still refer to nlnetlabs unbound for this purpose.

if i thought there was even one isp anywhere who wanted to use nxdomain
remapping but didn't because bind didn't have that feature, i'd be ready to
argue the point. but all isc did by not supporting this feature was force
some isp's to not use bind, and: isc is not in the "sour grapes" business.

meanwhile isc continues to push for ubiquitous dnssec, through to the stub,
to take this issue off the table for all people and all time. (that's "the
real fix" for nxdomain remapping.)

8

From: "Paul Vixie" <vixie@isc.org>

> *Now*, you see, we no longer have a canonical Good Engineering
> Example to
> which we can point when yelling at people (and software vendors)
> which
> *do* permit that, to say "see? You shouldn't be doing that; it's
> bad."
>
> "The Web Is Not The Internet."

i see what you mean, and i'm sad that this arrow is no longer in your
quiver. perhaps you can still refer to nlnetlabs unbound for this
purpose.

if i thought there was even one isp anywhere who wanted to use nxdomain
remapping but didn't because bind didn't have that feature, i'd be ready to
argue the point. but all isc did by not supporting this feature was force
some isp's to not use bind, and: isc is not in the "sour grapes"
business.

Well, I disagree on that, but I am not widely travelled, and perhaps
the obvious argument I see wasn't ever actually used.

This is the "do I put cigarette burn preventers on the toilet paper
dispensers in my 'no smoking' restroom" problem, pretty much exactly.

meanwhile isc continues to push for ubiquitous dnssec, through to the stub,
to take this issue off the table for all people and all time. (that's "the
real fix" for nxdomain remapping.)

You really believe that the outcome of that will be "we can't make some
extra revenue off NXDOMAIN remapping because of DNSSEC? Well, the hell
with DNSSEC, then"?

Cheers,
-- jra

9

[snip]

if i thought there was even one isp anywhere who wanted to use nxdomain
remapping but didn't because bind didn't have that feature, i'd be ready to
argue the point. but all isc did by not supporting this feature was force

Maybe they would think twice, if they used BIND, and BIND not only
excluded the feature,
or if activating the feature required loading an "Unsupported"
plugin, external module, patch, that would periodically generate
syslog warnings about dangerous NXDOMAIN mapping settings, but
documentation also contained a FAQ explaining why the feature "global
wildcards" or "nxdomain remapping" are not officially supported, and
explained some of the serious problems the concept of NXDOMAIN
remapping causes..

There is no point in "denying a feature" that is in demand, because
that simply means there will then be demand to fork the product, and
provide the users the features they want, that the developer is
denying them for "political reasons".

Also, the cats out of the bag once the feature is provided --
introducing a breaking change that would completely remove an
in-demand existing feature from the userbase would be ill-advised.

some isp's to not use bind, and: isc is not in the "sour grapes" business.

Other implementations mimic BIND; the BIND implementation is kind of a leader.
By including the feature, other implementations are influenced to
include the feature.
It would be best for the BIND developers' position on the usage of
the feature to be clear.

"Don't turn this on just because it's there... here's why.. oh, by
the way, you need to do this, this, and this other thing, before you
can turn it on"

meanwhile isc continues to push for ubiquitous dnssec, through to the stub,
to take this issue off the table for all people and all time. (that's "the
real fix" for nxdomain remapping.)

It's a good fix, but the code bits to implement it on major OSes
don't exist today,
and if they are fully implemented by all vendors, it's still
approximately 3 years before
they make it into a new OS' release cycle, and then another 10 years before
the functionality gets deployed and enabled by default, and older
systems get retired.

Discouraging the breakage is a good interim fix, when ubiqutous
DNSSEC to the stub
is optimistically 15 years out.

10

The code is DNSSEC aware, it doesn't perform redirection if the
client can detect that redirection has occured. So sign your zones
and use a validating client (or just one that sets DO=1).

Mark

11

> From: "Paul Vixie" <vixie@isc.org>

> > *Now*, you see, we no longer have a canonical Good Engineering
> > Example to
> > which we can point when yelling at people (and software vendors)
> > which
> > *do* permit that, to say "see? You shouldn't be doing that; it's
> > bad."
> >
> > "The Web Is Not The Internet."
>
> i see what you mean, and i'm sad that this arrow is no longer in your
> quiver. perhaps you can still refer to nlnetlabs unbound for this
> purpose.
>
> if i thought there was even one isp anywhere who wanted to use nxdomain
> remapping but didn't because bind didn't have that feature, i'd be ready to
> argue the point. but all isc did by not supporting this feature was force
> some isp's to not use bind, and: isc is not in the "sour grapes"
> business.

Well, I disagree on that, but I am not widely travelled, and perhaps
the obvious argument I see wasn't ever actually used.

This is the "do I put cigarette burn preventers on the toilet paper
dispensers in my 'no smoking' restroom" problem, pretty much exactly.

> meanwhile isc continues to push for ubiquitous dnssec, through to the stub,
> to take this issue off the table for all people and all time. (that's "the
> real fix" for nxdomain remapping.)

You really believe that the outcome of that will be "we can't make some
extra revenue off NXDOMAIN remapping because of DNSSEC? Well, the hell
with DNSSEC, then"?

People will route around ISP that do stupid things. They do so
today. When your browers supports DANE there will be more incentive
to ensure that DNSSEC does not break and more incentive to route
around ISP's that do break DNSSEC.

Even a ISP that is redirecting on NXDOMAIN wants to be sure that
it is a real NXDOMAIN not one that is spoofed do the path to the
ISP's resolver will be DNSSEC clean and they will be validating.

Until stub resolvers set DO=1 pretty much ubiquitously this won't
be a problem for ISP's that want to do nxdomain redirection. There
still plenty of crappy DNS proxies in CPE routers to be replaced
before you can just set DO=1 as a default without worrying about
breaking DNS lookups. Even setting EDNS as a default is a issue.

That said we are starting down the long path to making it EDNS a
default. DiG in BIND 9 defaults to using EDNS and "dig +trace"
turns set DO=1 as well. You don't get things fixed if the breakage
is not visible.

Mark

12

From: "Mark Andrews" <marka@isc.org>

[ vix: ]

> > meanwhile isc continues to push for ubiquitous dnssec, through to
> > the stub,
> > to take this issue off the table for all people and all time.
> > (that's "the
> > real fix" for nxdomain remapping.)
>
> You really believe that the outcome of that will be "we can't make
> some
> extra revenue off NXDOMAIN remapping because of DNSSEC? Well, the
> hell
> with DNSSEC, then"?

People will route around ISP that do stupid things. They do so
today. When your browers supports DANE there will be more incentive
to ensure that DNSSEC does not break and more incentive to route
around ISP's that do break DNSSEC.

My personal reaction to that, Mark, is to say that you *badly* overestimate
the average Internet end-user (who make up, roughly, 80% of the endpoints,
in my jackleg estimation).

Even a ISP that is redirecting on NXDOMAIN wants to be sure that
it is a real NXDOMAIN not one that is spoofed do the path to the
ISP's resolver will be DNSSEC clean and they will be validating.

I'm not sure I understood that...

Until stub resolvers set DO=1 pretty much ubiquitously this won't
be a problem for ISP's that want to do nxdomain redirection. There
still plenty of crappy DNS proxies in CPE routers to be replaced
before you can just set DO=1 as a default without worrying about
breaking DNS lookups. Even setting EDNS as a default is a issue.

...but that's probably because I don't understand DNSSEC well enough.

That said we are starting down the long path to making it EDNS a
default. DiG in BIND 9 defaults to using EDNS and "dig +trace"
turns set DO=1 as well. You don't get things fixed if the breakage
is not visible.

We may be talking about different breakage here...

Cheers,
-- jra

13

Jay Ashworth writes:

please do not feed the troll

When your browers supports DANE

and a billion home nats support dnssec :frowning:

randy

14

Until stub resolvers set DO=1 pretty much ubiquitously this won't
be a problem for ISP's that want to do nxdomain redirection. There

Yeah.............
Right now current _server_ implementations don't even have it right,
for properly implementing DNSSEC validation down to that level, let
alone the stubs below the server.

Many SME LANs utilize Windows-based endpoints, and commonly have
Windows-based DNS servers on the LAN, used by endpoints, where the
Windows DNS servers are set to forward queries to ISP recursive
servers. Current Windows' DNS server in 2008 "supports" DNSSEC.
When Windows DNS server is configured to forward DNS queries to a
list of reasonable recursive DNS servers, the server sets CD (Check
disabled) bit set, and the DO bit set for all queries it sends;
there is no option to "support dnssec queries from smart stubs but
don't send queries from dumb stubs with CD".

Also, there are by default no trust anchors, and _Every_ response is
treated as valid. (In other words, CD bit and DO bits are set in
forwarded queries, but no validation occurs).
It's kind of like having a SSL implementation that treats ALL SSL
certificates as valid, until and unless you take manual steps to
configure a CA list.

If you don't like this default and want to configure Windows DNS with
the root trust anchor.... Sorry, Algorithm #5 RSA/SHA-1 is the only
supported key type, and the current root signing key is not of a
compatible format.

Your "smart" stub can send a query to this broken DNSSEC
implementation with the DO bit set, for sure.

still plenty of crappy DNS proxies in CPE routers to be replaced
before you can just set DO=1 as a default without worrying about
breaking DNS lookups. Even setting EDNS as a default is a issue.

[snip]

I'm all for smart stubs as long as (1) They can get the data to them
(2) They can get the proper logging/reporting to them, E.g. No
"silent" upstream validate/discard; No upstream silent "ignore
the stub's requested CD bit and validate/discard anyways"
and

(3) The validation path is intact for _dumb_ non-validating stubs too.

15

In message <23491623.6382.1338256344974.JavaMail.root@benjamin.baylink.com>, Jay Ashworth writ
es:

> From: "Mark Andrews" <marka@isc.org>

[ vix: ]
> > > meanwhile isc continues to push for ubiquitous dnssec, through to
> > > the stub,
> > > to take this issue off the table for all people and all time.
> > > (that's "the
> > > real fix" for nxdomain remapping.)
> >
> > You really believe that the outcome of that will be "we can't make
> > some
> > extra revenue off NXDOMAIN remapping because of DNSSEC? Well, the
> > hell
> > with DNSSEC, then"?
>
> People will route around ISP that do stupid things. They do so
> today. When your browers supports DANE there will be more incentive
> to ensure that DNSSEC does not break and more incentive to route
> around ISP's that do break DNSSEC.

My personal reaction to that, Mark, is to say that you *badly* overestimate
the average Internet end-user (who make up, roughly, 80% of the endpoints,
in my jackleg estimation).

Google's recursive nameservers see plenty of traffic.

> Even a ISP that is redirecting on NXDOMAIN wants to be sure that
> it is a real NXDOMAIN not one that is spoofed do the path to the
> ISP's resolver will be DNSSEC clean and they will be validating.

I'm not sure I understood that...

        Authoritative server
                ^
       secure
         (DO=1 queries)
    v
  ISP's validating recursive server
                ^
     insecure, redirect possible
         (DO=0 queries)
                v
           Stub clients.

Putting it another way, the ISP doesn't want to be fooled even if
it is fooling its customers. The ISP can't allow it's recursive
servers to get the wrong answers for irs.gov, picking a signed
domain, as they would look silly for not validating when there is
a simple way for them to be sure that they are not being spoofed.

> Until stub resolvers set DO=1 pretty much ubiquitously this won't
> be a problem for ISP's that want to do nxdomain redirection. There
> still plenty of crappy DNS proxies in CPE routers to be replaced
> before you can just set DO=1 as a default without worrying about
> breaking DNS lookups. Even setting EDNS as a default is a issue.

...but that's probably because I don't understand DNSSEC well enough.

The ISP <-> stub client path often has a additional piece in the
path that is often a heap of steaming !@$! that doesn't do basic
DNS let alone EDNS and DNSSEC. For example the Netgear router at
home doesn't support DNS over TCP which is basic DNS (I'm using it
as a access point not a router because of this). It's this sort
of breakage that is stopping OS vendors changing defaults. This
can often be bypassed by forcing the resolver to use the ISP's
nameservers directly but you need to know to do that.

     ISP's validating recursive server
                ^
                v
            CPE Router (broken DNS proxy code)
                ^
                v
           Stub clients.

You can also replace CPE Router with a broken firewall that is a
steaming heap of !@#% when it comes to DNS packet inspection. These
are harder to bypass and require someone to fix the configuration
to replace/upgrade the box.

16

> Until stub resolvers set DO=1 pretty much ubiquitously this won't
> be a problem for ISP's that want to do nxdomain redirection. There

Yeah.............
Right now current _server_ implementations don't even have it right,
for properly implementing DNSSEC validation down to that level, let
alone the stubs below the server.

Many SME LANs utilize Windows-based endpoints, and commonly have
Windows-based DNS servers on the LAN, used by endpoints, where the
Windows DNS servers are set to forward queries to ISP recursive
servers. Current Windows' DNS server in 2008 "supports" DNSSEC.
When Windows DNS server is configured to forward DNS queries to a
list of reasonable recursive DNS servers, the server sets CD (Check
disabled) bit set, and the DO bit set for all queries it sends;
there is no option to "support dnssec queries from smart stubs but
don't send queries from dumb stubs with CD".

Well I'm trying to get this fixed at the protocol level for other
reasons as explained in

draft-ietf-dnsext-dnssec-bis-updates-18.txt is still in IETF last
call and if you think always setting CD=1 when forwarding is bad for
whatever reason I could do with some support.

17

don't lie to us, but we lie to our customers.

  and you don't see a problem with this?

/bill

18

I didn't say I like it. It may even be illegal in some juristictions
for a ISP to do it without properly informing the customer and
allowing them to opt in/out. Doing it to yourself however can't
be illegal. In the end it is a tool and the method of deployment
is often as important as whether you deploy it or not.

It's a little like direct marketing via email. If you have consent
of the party being emailed it isn't spam. If you don't have consent
it is spam at least here in Australia. Other juristictions have
looser guidelines.

Mark

19

We tried saying no, we tried walking customers away, we tried not adding the feature, we tried shame, someone reputedly had dolls with pins in them.

We lost.

There is an endgame around that; when it's ready....

George William Herbert

20

I expect there will be a depressingly large amount of DNS-over-TLS in the
future in order to bypass broken ALGs.

Tony.