Paul Vixie: Re: [dns-operations] DNS issue accidentally leaked?

downplay this all you want, we can infect a name server in 11 seconds now,
which was never true before. i've been tracking this area since 1995. don't
try to tell me, or anybody, that dan's work isn't absolutely groundbreaking.

i am sick and bloody tired of hearing from the people who aren't impressed.

Well, Paul, I'm not *too* impressed, and so far, I'm not seeing what is
groundbreaking, except that threats discussed long ago have become more
practical due to the growth of network and processing speeds, which was
a hazard that ... was actually ALSO predicted.

And you know what? I'll give you the *NEXT* evolutionary steps, which
could make you want to cry.

If the old code system could result in an infected name server in 11
seconds, this "fix" looks to me to be at best a dangerous and risky
exercise at merely reducing the odds. Some criminal enterprise will
figure out that you've only reduced the odds to 1/64000 of what they
were, but the facts are that if you can redirect some major ISP's
resolver to punt www.chase.com or www.citibank.com at your $evil server,
and you have large botnets on non-BCP38 networks, you can be pumping
out large numbers of answers at the ISP's server without a major
commitment in bandwidth locally... and sooner or later, you'll still
get a hit. You don't need to win in 11 secs, or even frequently. It
can be "jackpot" monthly and you still win.

Which is half the problem here. Bandwidth is cheap, and DNS packets are
essentially not getting any bigger, so back in the day, maybe this wasn't
practical over a 56k or T1 line, but now it is trivial to find a colo with
no BCP38 and a gigabit into the Internet. The flip side is all those nice
multicore CPU's mean that systems aren't flooded by the responses, and they
are instead successfully sorting through all the forged responses, which
may work in the attacker's advantage (doh!)

This problem really is not practically solvable through the technique that
has been applied. Give it another few years, and we'll be to the point
where both the QID *and* the source port are simply flooded, and it only
takes 11 seconds, thanks to the wonder of ever-faster networks and servers.
Whee, ain't progress wonderful.

Worse, this patch could be seen as *encouraging* the flooding of DNS
servers with fake responses, and this is particularly worrying, since
some servers might have trouble with this.

So, if we want to continue to ignore proper deployment of DNSSEC or
equivalent, there are some things we can do:

* Detect and alarm on cache overwrite attempts (kind of a meta-RFC 2181
  thing). This could be problematic for environments without consistent
  DNS data (and yes, I know your opinion of that).

* Detect and alarm on mismatched QID attacks (probably at some low
  threshold level).
  
But the problem is, even detected and alerted, what do you do? Alarming
might be handy for the large consumer ISP's, but isn't going to be all
that helpful for the universities or fortune 500's that don't have 24/7
staff sitting on top of the nameserver deployment.

So, look at other options:

* Widen the query space by using multiple IP addresses as source. This,
  of course, has all the problems with NAT gw's that the port solution
  did, except worse.

  This makes using your ISP's "properly designed" resolver even more
  attractive, rather than running a local recurser on your company's
  /28 of public IP space, but has the unintended consequence of making
  those ISP recursers even more valuable targets.

Makes you wish for wide deployment of IPv6, eh.

every time some blogger says "this isn't new", another five universities
and ten fortune 500 companies and three ISP's all decide not to patch.
that means we'll have to wait for them to be actively exploited before they
will understand the nature of the emergency.

While I applaud the reduction in the attack success rate that the recent
patch results in, I am going to take a moment to be critical, and note that
I feel you (and the other vendors) squandered a fantastic chance. Just
like the Boy Who Cried Wolf, you have a limited number of times that you can
cry "vulnerability" and have people mostly all stand up and pay attention in
the way that they did.

Hardly the first (but possibly one of the most noteworthy), RIPE called
for signing of the root zone a year ago.

I note with some annoyance that this would have been a great opportunity
for someone with a lot of DNS credibility to stand up and say "we need
the root signed NOW," and to force the issue. This would have been a lot
of work, certainly, but a lot of *worthwhile* work, at various levels.
The end result would have been a secure DNS system for those who chose to
upgrade and update appropriately.

Instead, it looks to me as though the opportunity is past, people are
falsely led to believe that their band-aid-patched servers are now "not
vulnerable" (oh, I love that term, since it isn't really true!) and the
next time we need to cry "fire," fewer people will be interested in
changing.

The only real fix I see is to deploy DNSSEC.

I've tried to keep this message bearably short, so please forgive me if
I've glossed over anything or omitted anything.

... JG

So, look at other options:

* Widen the query space by using multiple IP addresses as
source. This,
  of course, has all the problems with NAT gw's that the port solution
  did, except worse.

  This makes using your ISP's "properly designed" resolver even more
  attractive, rather than running a local recurser on your company's
  /28 of public IP space, but has the unintended consequence of making
  those ISP recursers even more valuable targets.

Makes you wish for wide deployment of IPv6, eh.

The only real fix I see is to deploy DNSSEC.

You seem to be saying, above, that IPv6 is also a real fix, presumably
because it allows for the 64-bit host id portion of an IP address to
"fast flux". Or have I misunderstood?

It would be nice for someone to explain how (or if) IPv6 changes this
situation since many networks are already well into the planning stages
for IPv6 deployment within the next two to three years.

--Michael Dillon

> i am sick and bloody tired of hearing from the people who aren't impressed.

Well, Paul, I'm not *too* impressed, and so far, I'm not seeing what is
groundbreaking, except that threats discussed long ago have become more
practical due to the growth of network and processing speeds, which was
a hazard that ... was actually ALSO predicted.

11 seconds.

and at&t refuses to patch.

and all iphones use those name servers.

your move.

This caught my attention, and so I tossed the AT&T wireless card in my laptop and ran the test:

[rogue:~] steve% dig +short porttest.dns-oarc.net TXT
z.y.x.w.v.u.t.s.r.q.p.o.n.m.l.k.j.i.h.g.f.e.d.c.b.a.pt.dns-oarc.net.
"209.183.54.151 is GOOD: 26 queries in 0.8 seconds from 26 ports with std dev 22831.70"
[rogue:~] steve% host 209.183.54.151
151.54.183.209.in-addr.arpa domain name pointer bthinetdns.mycingular.net.

doxpara.com tests to lock up my iPhone, or I would use that checker to verify the iPhone DNS. Anyone have a link to a decent test that I could run on the iPhone?

Thanks,
Steve

downplay this all you want, we can infect a name server in 11 seconds now,
which was never true before. i've been tracking this area since 1995. don't
try to tell me, or anybody, that dan's work isn't absolutely groundbreaking.

i am sick and bloody tired of hearing from the people who aren't impressed.

Well, Paul, I'm not *too* impressed, and so far, I'm not seeing what is
groundbreaking, except that threats discussed long ago have become more
practical due to the growth of network and processing speeds, which was

Joe, you should be well aware most of what we face today and will face tomorrow is based on concepts of old, and/or has been thought of/seen before in a different format.

But as you well know, threats are still real, especially where the practical vulnerability is hight. Further, this is especially true in the operators communities where unless there is a power point presentation about it, the threat is thought a pipe-dream.

a hazard that ... was actually ALSO predicted.

Well, it's here now, and I am happy there is something to be done about it, pro-actively.

   Gadi.

Has at&t told you they are refusing to patch? Or are you just spreading
FUD about at&t and don't actually have any information about their plans?

I believe it is a hypothetical situation being presented...

William

Joe,

Early attacks were based on returning out-of-scope data in the
Additional section of the response. This was an implementation error
in the servers: they should never have accepted out of scope data.

Later attacks were based on forging responses to a query. The resolver
sends a query packet and the attacker has a few tens of milliseconds
in which to throw maybe a few tens of guesses about correct ID at the
resolver before the real answer arrives from the the real server.
These were mitigated because:

a. You had maybe a 1 in 1000 chance of guessing right during the
window of opportunity.
b. If you guessed wrong, you had to wait until the TTL expired to try
again, maybe as much as 24 hours later.

So, it could take months or years to poison a resolver just once, far
below the patience threshold for your run-of-the-mill script kiddie.

What's new about this attack is that it removes mitigator B. You can
guess again and again, back to back, until you hit that 1 in 1000.

Paul tells us this can happen in about 11 seconds, well inside the
tolerance of your normal script kiddie and long before you'll notice
the log messages about invalid responses.

Anyway, it shouldn't be hard to convert this from a poisoning
vulnerability to a less troubling DOS vulnerability by rejecting
responses for a particular query (even if valid) when received near a
bad-id response. From there it just takes some iterative improvements
to mitigate the DOS.

In the mean time, randomizing the query port makes the attack more
than four orders of magnitude less effective and causes it to require
more than four orders of magnitude of additional resources on the
attacker's part.

Regards,
Bill Herrin

> > 11 seconds.
> >
> > and at&t refuses to patch.
> >
> > and all iphones use those name servers.
>
> Has at&t told you they are refusing to patch? Or are you just spreading
> FUD about at&t and don't actually have any information about their plans?

I believe it is a hypothetical situation being presented...

so, noone else has had multiple copies of the following fwd'd to them with
the heading, "WTF,O?" note that it's full of factual errors but does seem
to represent AT&T's position on CERT VU# 800113. that someone inside AT&T
just assumed that this was the same problem as described in CERT VU#252735
and didn't bother to call CERT, or kaminsky, or me, to verify, ASTOUNDS me.

(if someone from AT&T's DNS division has time to chat, my phone numbers are
in `whois -h whois.arin.net pv15-arin`.)

That doesn't sound like "refuses to patch."

It sounds like at&t is testing the patch and will deploy it as soon as its testing is finished.

"Refuses to patch" sounds likes FUD.

"Refuses to patch" sounds likes FUD.

go ask 'em, and let us all know what they say.

kaminsky tried to get everybody a month, but because of ptacek's sloppiness
it ended up being 13 days. if any dns engineer at any internet carrier goes
home to sleep or see their families before they patch, then they're insane.

yes, i know the dangers of rolling patches out too quickly. better than most
folks, since i've been on the sending side of patches that caused problems,
and i've learned caution from the pain i've inadvertantly caused in that way.

in spite of that caution i am telling you all, patch, and patch now. if you
have firewall or NAT configs that prevent it, then redo your topology -- NOW.
and make sure your NAT isn't derandomizing your port numbers on the way out.

and if you have time after that, write a letter to your congressman about the
importance of DNSSEC, which sucks green weenies, and is a decade late, and
which has no business model, but which the internet absolutely dearly needs.

Give this one a try:

http://entropy.dns-oarc.net/test/

In this test, my iPhone reports:

209.183.33.23 Source Port Randomness: GREAT
209.183.33.23 Transaction ID Randomness: GREAT

I encourage anyone else concerned with their providers to actually test them instead of taking anyone's word for it.

Steve

I believe at&t has already said they are testing the patch and will deploy it as soon as their testing is completed. Other than you, I have not heard anyone in at&t say they are refusing to patch.

Doing my own tests, at&t appears to be testing the patch on DNS
servers in Tulsa and St. Louis. They may be testing on other DNS
servers in other regions too.

The at&t anycast DNS ip addresses go to different servers in different
locations, so you may get different results using the same IP address
in your DNS client.

But if you want to continue spreading the FUD that at&t is refusing to
patch, I can't stop you.

> go ask 'em, and let us all know what they say.

I believe at&t has already said they are testing the patch and will deploy
it as soon as their testing is completed. Other than you, I have not
heard anyone in at&t say they are refusing to patch.

i read at&t write that this was a rehash of a previously known issue.

i heard at&t tell a customer that they were in no hurry to patch.

Doing my own tests, at&t appears to be testing the patch on DNS
servers in Tulsa and St. Louis. They may be testing on other DNS
servers in other regions too.

that's good news.

The at&t anycast DNS ip addresses go to different servers in different
locations, so you may get different results using the same IP address
in your DNS client.

But if you want to continue spreading the FUD that at&t is refusing to
patch, I can't stop you.

hopefully at&t will issue a clarifying statement, indicating that they know
now that this is not a rehash of last year's update, and that they will have
those iphones covered by july 25 even though they were noticed before july 8.

by the way we just found an abuse@ mailbox protected by challenge-response
(there's a notification effort underway to let folks know which of their open
resolvers are unpatched. i don't know what this means anymore.)

Paul Vixie wrote:

"Refuses to patch" sounds likes FUD.

go ask 'em, and let us all know what they say.

AT&T dsl line.

#dig +short porttest.dns-oarc.net TXT @68.94.157.1
z.y.x.w.v.u.t.s.r.q.p.o.n.m.l.k.j.i.h.g.f.e.d.c.b.a.pt.dns-oarc.net.
"65.68.49.31 is POOR: 26 queries in 1.4 seconds from 1 ports with std dev 0.00"

Ken

Steve Tornio wrote:

xpara.com tests to lock up my iPhone, or I would use that checker to verify the iPhone DNS. Anyone have a link to a decent test that I could run on the iPhone?

Give this one a try:

http://entropy.dns-oarc.net/test/

In this test, my iPhone reports:

209.183.33.23 Source Port Randomness: GREAT
209.183.33.23 Transaction ID Randomness: GREAT

I encourage anyone else concerned with their providers to actually test them instead of taking anyone's word for it.

Steve

on AT&T you might want to run it more than once.. Mine shows POOR 1 out of 5 times. :frowning:
Hope they finish patching sooooon!
Ken

Is it just me or is the test page below down now?

Or maybe some poisoned the NS record for dns-oarc.net and sent it to
nowhere to stop testing! (J/K since I can get to the rest of the page
fine).

  -Scott

If the customer believes AT&T reputation is more reliable than Paul Vixie's reputation (not saying they are right or wrong, just if);
point out at&t said they are testing the patch and plan to deploy
it as soon as their testing is finished.

If the customer believes at&t, the customer should also at least be
testing the patch NOW and should deploy it WHEN they finish testing it.

That is not the same as "refusing to patch." Test Patch->Deploy Patch.

Suggestion - add to the bottom of the results page a link to the CERT page:
http://www.kb.cert.org/vuls/id/800113

-Hank