.biz DNSSEC borked

Seems the entire .biz tld is failing DNSSEC validation now.
All of my DNSSEC validating resolvers are tossing all domains in .biz. The non-signed domains too of course because trust of the tld itself cannot be established.

http://dnssec-debugger.verisignlabs.com/nic.biz

confirmed

None of the 5 DNSKEY records could be validated by any of the 2 DS records
The DNSKEY RRset was not signed by any keys in the chain-of-trust

biz has SOA record a.gtld.biz. hostmaster.neustar.biz. 12161960 900 900
604800 86400 (BOGUS (security failure)) validation failure <biz. SOA IN>:
no keys have a DS from 156.154.127.65 for key BIZ. while building chain of
trust

tcp: biz has SOA record a.gtld.biz. hostmaster.neustar.biz. 12161960 900
900 604800 86400 (BOGUS (security failure)) validation failure <biz. SOA

: no keys have a DS from 156.154.127.65 for key BIZ. while building

chain of trust

Seems the entire .biz tld is failing DNSSEC validation now.
All of my DNSSEC validating resolvers are tossing all domains in .biz.

The non-signed domains too of course because trust of the tld itself cannot
be established.

Looks like it is clearing up now. Flushing the caches of your resolvers might be wise.

I wonder who made the boo-boo

Another Internet reboot?

Can we find something better?

So which event caused more disruption? 50K .com's in a failed DDoS
mitigation, or every single .biz lookup by something that actually does
dnssec?

I think two different things happened here:

1) biz breakage reinforces the fact that validation can cause disruption.
   if it were .com and not fixed for a few hours, every major ISP would
   likely turn off validation for a year or more.

2) com issue shows some major "brands" they need to be more demanding from
   their providers.

   some really interesting data here, i ran a few domains through some
   dns server lists i have lying around, and saw stuff like this:

8.23.128.129/53/www.usps.com^IN^CNAME^www.usps.com.edgekey.net|www.usps.com.edgekey.net^IN^CNAME^usps.georedirector.usps.com.akadns.net|usps.georedirector.usps.com.akadns.net^IN^CNAME^e7154.dscb.akamaiedge.net|e7154.dscb.akamaiedge.net^IN^A^23.35.198.219//usps.com^IN^NS^ns1621.ztomy.com|usps.com^IN^NS^ns2621.ztomy.com

so, you see www.usps.com points at edgekey, but the authority for
usps.com was still held as ztomy for some time.

(I don't have it printing the TTLs, but could add that...)

This excludes DNS servers that are *very* broken, such as will replace
existing authority/delegation w/ stuff returned in an unrelated query
(as seen above) or other unsolicited data. (i get many servers that tell
me stuff I *really* didn't ask for)

(i queried for openresolverproject and got back something about betterbricks.com)

190.51.146.2/21528/betterbricks.com^IN^MX^30 betterbricks.com.s10b1.psmtp.com|betterbricks.com^IN^MX^40 betterbricks.com.s10b2.psmtp.com|betterbricks.com^IN^MX^10 betterbricks.com.s10a1.psmtp.com|betterbricks.com^IN^MX^20 betterbricks.com.s10a2.psmtp.com//

or this that seems to delegate root to some nipr.mil host.

214.4.226.2/53//con2.nipr.mil^IN^A^199.252.162.234|con1.nipr.mil^IN^A^207.132.116.25/.^IN^NS^con2.nipr.mil|.^IN^NS^con1.nipr.mil

  - Jared

I don't think we are trying to quantify which one was worst or point fingers at, but how do we remediate these type of issues in the future? I think these events will happen more and more often...

a TTL of 2 days seems rather long for NS and do I see 6 days TTL for DNSSEC records for .biz ?