Let's hope some very large service providers get their act together
real soon now.
http://www.hackerfactor.com/blog/index.php?/archives/204-Poor-DNS.html
- - ferg
There is always a tension between discovery, changing, testing and finally deployment.
DNS vendors learned about the vulnerability on March 31 (or possibly earlier). DNS vendors waited over 3 months to publically release their
patches, even though they knew their customers and users were vulnerable.
It probably took the vendors some time to change their code, test their changes, work on beta releases in various deployments because programmers are human and sometimes patches have bugs too. Then they announced their patches to the world, and the world (and ISPs, etc) has much less time
to regression test and verify the systems still work. Vendors have
released bugging patches in the past. Patching a large ISP infrastructure under ordinary circumstances can be challanging. If it takes software
vendors 90+ days to fix something, is it a surpise it may take a large ISP more than 14 days?
If they move to quickly and crash the resolvers because of a bug the human programmers may have not forseen in the ISPs DNS architecture, the Internet is effectively "down" for a large number of users. Result: Bad press, angry customers, lawsuits, etc.
If they don't move quickly enough and the vulnerability is exploited by
a human bad guy, the Internet is effectively "corrupted" for a large
number of users. Result: Bad press, angry customers, lawsuits, etc.
Damned if they do, damned if they don't. Or in this case: Damned if
they are too fast, damned if they are too slow.
I don't think there really is a correct answer. People are going
to say they suck no matter what. Anyone who has ever been in the position
of scheduling security patches across a large ISP knows they aren't going
to get much thanks.
Although I didn't know the right answer, I did try to always patch production network first and the corporate network last; so if we didn't get everything finished before the exploit hit I could tell customers
we did try to put the customer first. Although internal MIS folks would sometimes get mad at me for waiting to tell them. Some people
think you should patch the corporate network first, and the production
network later.
So it brings up the ancient question about the schedule of vulnerability
announcements and whether some providers of some core infrastructure
should have an early start to patch their systems; because everyone
else will be depending on them functioning to obtain the patches when
the vulnerability is widely disclosed. How do you decide how early,
who, what, how, ...
Or do not play favorites, and announce everything to everyone at
the exact same time; and its off to the races.
Or something in between.
It isn't going to happen without BIG political pressure, either from users, or
governments, and other bodies.
I checked last night, and noticed TLD servers for .VA and .MUSEUM are still
offering recursion amongst a load of less popular top level domains.
Indeed just under 10% of the authoritative name servers mentioned in the root
zone file still offer recursion.
I didn't check IPv6 servers, but these IPv4 servers are potentially vulnerable
to this (and other) poisoning attacks. Hard to pin down numbers as some have
been patched, and some have unusual behaviour on recursion, but I fancy my
chances of owning more than a handful of TLDs if I had the time to try (and
immunity from prosecution).
The advice NOT to allow recursion on TLD servers is well over a decade old. So
who thinks the current fashionable problem will be patched widely in a
month - given it is much less critical in nature?
The .MUSEUM server that is offering recursion is hosted by the Getty
Foundation, so I assume money isn't the issue. The Vatican ought to be able
to find someone in its billion adherents prepared to help configure a couple
of name servers.
I also noticed that one of the ".US" servers doesn't exist in the DNS proper,
glue exists but not the record in the zone. I'm guessing absence of a name
servers name record in its proper zone makes certain spoofing attacks easier
(since you are only competing with glue records), although I can't
specifically demonstrate that one for blackhat 2008 - it suggests a certain
lack of attention on the part of the domain's administrators.
I was tempted to write a mock RFC, proposing dropping all top level domain
names which still have recursion enabled in one or more of their name
servers - due to "lack of maintanence". A little humour might help make the
point, slashdot might go for it.
While not ideal, at least most resolvers will not go asking those
servers for anything other than what they are authoritative for unless
an attacker for some reason wanted to setup a long chain of poisons. The
large, shared caching servers and all those open CPE devices are a
much larger concern I think.
John
Indeed--you won't hear arguments from me on other threats, especially not CPE devices which I fought to bring recognition to.
But sticking to the point, TLD servers should (under most circumstances) be recursive. Thing is, those that are, are likely to stay that way.
I personally know several folks from within and wayyy from outside the DNS world who discovered this very out there and obvious issue and worked hard to try and contact the operators. Those that haven't fixed it yet, likely won't if all thing remain even.
Others I am not aware of likely did the same, withs imilar results. I guess we could try again.
Gadi.
Should NEVER, oops.
I personally know several folks from within and wayyy from outside the
DNS
world who discovered this very out there and obvious issue and worked
hard
to try and contact the operators. Those that haven't fixed it yet,
likely
won't if all thing remain even.
I don't know that a failure to act immediately is indicative of ignoring
the problem. Not to defend AT&T or any other provider, but it's not as
simple as rolling out a patch.
-M<
I personally know several folks from within and wayyy from outside the
DNS
world who discovered this very out there and obvious issue and worked
hard
to try and contact the operators. Those that haven't fixed it yet,
likely
won't if all thing remain even.I don't know that a failure to act immediately is indicative of ignoring
the problem. Not to defend AT&T or any other provider, but it's not as
simple as rolling out a patch.
Marty, are we talking of the same problem? I am talking about recursion enabled in bind?
Right. What scares me is all of the semi-managed hotspots with
software that's rarely updated.
--Steve Bellovin, http://www.cs.columbia.edu/~smb
simonw@zynet.net (Simon Waters) writes:
The advice NOT to allow recursion on TLD servers is well over a decade old.
it's not just advice, really. on the mailing list that's a little like this
one except that unlike this one it's meant for DNS operations issues, i said
<http://lists.oarci.net/pipermail/dns-operations/2008-July/002994.html>\.
subscription info at <http://lists.oarci.net/mailman/listinfo/dns-operations>\.
From: Gadi Evron [mailto:ge@linuxbox.org]
Sent: Thursday, July 24, 2008 11:52 AM
To: Martin Hannigan
Cc: nanog@nanog.org
Subject: RE: TLD servers with recursion was Re: Exploit for DNS
CachePoisoning- RELEASED>
>
>>
>> I personally know several folks from within and wayyy from outside
the
>> DNS
>> world who discovered this very out there and obvious issue and
worked
>> hard
>> to try and contact the operators. Those that haven't fixed it yet,
>> likely
>> won't if all thing remain even.
>>
>
>
> I don't know that a failure to act immediately is indicative of
ignoring
> the problem. Not to defend AT&T or any other provider, but it's not
as
> simple as rolling out a patch.Marty, are we talking of the same problem? I am talking about
recursion
enabled in bind?
I'm reading this as a complaint that people aren't fixing an obvious
problem that has a high impact on the network. You're making sense in
that respect, but my impression that the angst is in the speed of the
fix, not in the need.
If that observation is off, sorry.
-M<
Gadi Evron wrote:
I personally know several folks from within and wayyy from outside the
DNS
world who discovered this very out there and obvious issue and worked
hard
to try and contact the operators. Those that haven't fixed it yet,
likely
won't if all thing remain even.I don't know that a failure to act immediately is indicative of ignoring
the problem. Not to defend AT&T or any other provider, but it's not as
simple as rolling out a patch.Marty, are we talking of the same problem? I am talking about recursion enabled in bind?
I'm confused by the last sentence. I don't understand if you are asking a question, or stating that recursion should be disabled.
If it is a statement, then you must mean that ops should disable recursion, and enable forwarding for name resolution, correct? In this case, its been proven that having an upstream forward that is 'broken' will have the exact same effect as having a broken recursive server.
My apologies if I've misunderstood your comment.
Steve
We are talking about ccTLD NS.
Gadi.