We (at Kurchatov Insitute) still use 144.206.0.0/16, the legacy
block, and seeing the breakage rooted at ARIN since this night,
{{{
$ dig +trace -t soa 206.144.in-addr.arpa
; <<>> DiG 9.10.4-P6 <<>> +trace -t soa 206.144.in-addr.arpa
;; global options: +cmd
. 206351 IN NS a.root-servers.net.
. 206351 IN NS c.root-servers.net.
. 206351 IN NS d.root-servers.net.
. 206351 IN NS e.root-servers.net.
. 206351 IN NS l.root-servers.net.
. 206351 IN NS f.root-servers.net.
. 206351 IN NS g.root-servers.net.
. 206351 IN NS h.root-servers.net.
. 206351 IN NS j.root-servers.net.
. 206351 IN NS m.root-servers.net.
. 206351 IN NS k.root-servers.net.
. 206351 IN NS i.root-servers.net.
. 206351 IN NS b.root-servers.net.
. 514733 IN RRSIG NS 8 0 518400 20170329170000 20170316160000 61045 . OjsjoA6WCcThV3BqhAyMaXI3bU1m228Gl8nvaR074qBtem/RjaFJh1Oe r7LPI6W15jgbRGuCY7/GUNgDex4ZM43yvx2iY+2GpSk9b2/pKGbDaDIp X1Hd8418206eow1P/SgPqtT+2LzjM+lz/HKUvCyPoN4uX5/7GDExn8ir 2Q/vw81Za2nJ4Ji6oeGAmE8g6V3RJfTI0El/CP9Vl2/r0jWDZZ+wtYRA uwqvQYj+7VZc5+UJDJ3/Gne1LBnzXKnZRwnfJtZenhfZ7X20D9PiXvfJ M2s1ryqZvg8K5RqYt/r8pIiq1Udd1KWQBgdN7Q629+jNSngZQtH19R9H fM7h1w==
;; Received 1097 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms
in-addr.arpa. 172800 IN NS e.in-addr-servers.arpa.
in-addr.arpa. 172800 IN NS f.in-addr-servers.arpa.
in-addr.arpa. 172800 IN NS d.in-addr-servers.arpa.
in-addr.arpa. 172800 IN NS c.in-addr-servers.arpa.
in-addr.arpa. 172800 IN NS b.in-addr-servers.arpa.
in-addr.arpa. 172800 IN NS a.in-addr-servers.arpa.
in-addr.arpa. 86400 IN DS 53696 8 2 13E5501C56B20394DA921B51412D48B7089C5EB6957A7C58553C4D4D 424F04DF
in-addr.arpa. 86400 IN DS 63982 8 2 AAF4FB5D213EF25AE44679032EBE3514C487D7ABD99D7F5FEC3383D0 30733C73
in-addr.arpa. 86400 IN DS 47054 8 2 5CAFCCEC201D1933B4C9F6A9C8F51E51F3B39979058AC21B8DF1B1F2 81CBC6F2
in-addr.arpa. 86400 IN RRSIG DS 8 2 86400 20170330000000 20170316230000 33786 arpa. gqNN6MmLVxFtSG0oxdyoYVSXZlp6vY9yxpnJW89TYCHfTkv+ZmklM76O IHdTznEhLHXbyr4BaxjFCshsC3WacdH/YZYa/SZynJ9Q1N6/bogrIUTn qs61Y/YD4Sk5lI7YvUxVnPrF3lk1pY8dpoTkprTLZngoeQEsm552inqG ypU=
;; Received 733 bytes from 198.41.0.4#53(a.root-servers.net) in 64 ms
144.in-addr.arpa. 86400 IN NS r.arin.net.
144.in-addr.arpa. 86400 IN NS u.arin.net.
144.in-addr.arpa. 86400 IN NS x.arin.net.
144.in-addr.arpa. 86400 IN NS y.arin.net.
144.in-addr.arpa. 86400 IN NS z.arin.net.
144.in-addr.arpa. 86400 IN NS arin.authdns.ripe.net.
144.in-addr.arpa. 86400 IN DS 62856 5 1 484154736CF9D4DC4F1C2B807BDC06E5B1645AFF
144.in-addr.arpa. 86400 IN RRSIG DS 8 3 86400 20170328085556 20170307105911 4341 in-addr.arpa. X3nPKgjQbRef6BxK3msEXi/IcfpG1rylY2xeWwfNO6fJB5x/QT38ZCA5 XMs6CBeXb59tpQFbgjwZX+prqSpjGFtZ+phFuzsaGmuPOF0FFypMHj7F swykEFvaPY5z4d8mo9FKFJetF2gZfzYthwJWW0x2oo2H2BG0lEedVUCb 1aGs/HE=
;; Received 380 bytes from 193.0.9.1#53(f.in-addr-servers.arpa) in 48 ms
144.in-addr.arpa. 0 IN SOA z.arin.net. dns-ops.arin.net. 2017022432 1800 900 691200 10800
144.in-addr.arpa. 0 IN RRSIG SOA 5 3 86400 20170331083220 20170317073220 33345 144.in-addr.arpa. k2sZXuQQR3MyyRiB9Wd7fw45yZTbNokjQtFFNY+aCEa/85AnSyuomLlZ jZs4A0bO376/Z1v+bTHoeFqsyHr/xuWHX74r0QC/TCeP0amc+w8RqCvw Yox1TfoJxwhblGNRCgyLw52WszMYyr49EfWPfpHAKFU+G94gdilf3C6x GOM=
144.in-addr.arpa. 10800 IN NSEC 0.144.in-addr.arpa. NS SOA RRSIG NSEC DNSKEY
144.in-addr.arpa. 10800 IN RRSIG NSEC 5 3 10800 20170331083220 20170317073220 33345 144.in-addr.arpa. opvG8SC7+UjHSTtBBtekWNkhLFMIxFxOdejJ7sM7t8lyGNzM6sOABeSI ADXfo8q3p4YFBHgBU5fRCAhBdiQ/AiZC0dLMnHb4WdKEv5bDsBbq5Pt6 vabJaIv7Gizw7kBy1JZ/ZrC4Jmp4EX0HiYA+Ak+aQggD7gdI/6tFDNpl N7M=
205.144.in-addr.arpa. 10800 IN NSEC 207.144.in-addr.arpa. NS RRSIG NSEC
205.144.in-addr.arpa. 10800 IN RRSIG NSEC 5 4 10800 20170331083220 20170317073220 33345 144.in-addr.arpa. F1e5mA7C3Mx10YSaNtshzfzfER8pudDOQkUoe+jeDhHeDZeR8FM1FoqW dqrNh5X6JVNlTdWGio6evnSkxnaoJFYyeYJtc+tJ8dTd5APJxJCSOfhH VfRNy7VHs2PdElRo1rRwMw+5Zncc0u8uPdmFDv2ZG8Vv3zj5C6l7rMla 3SA=
;; Received 718 bytes from 199.212.0.63#53(z.arin.net) in 128 ms
}}}
Seems like the other /16 from 144.in-addr.arpa are affected too
(at least).
ARIN tickets are engaged, but I am seeking some live person:
having the whole reverse zone being unavailable isn't the stuff
that is to be handled in 5x8 or alike.
If anybody knows what happened, can point me to my errors
in assertions or give an e-mail/phone of 24x7-like person
or service at ARIN -- will be tremendous.
Thanks!
a message of 71 lines which said:
Seems like the other /16 from 144.in-addr.arpa are affected too
(at least).
Also in 164.in-addr.arpa, it seems?
a message of 71 lines which said:
We (at Kurchatov Insitute) still use 144.206.0.0/16, the legacy
block, and seeing the breakage rooted at ARIN since this night,
{{{
$ dig +trace -t soa 206.144.in-addr.arpa
According to DNSDB, it arrived yesterday, around 1630 UTC.
Job, good day.
Fri, Mar 17, 2017 at 09:55:56AM +0000, Job Snijders wrote:
171 also seems affected.
Not the whole one, it seems:
{{{
$ dig +trace -t soa 1.171.in-addr.arpa
; <<>> DiG 9.10.4-P6 <<>> +trace -t soa 1.171.in-addr.arpa
;; global options: +cmd
. 202634 IN NS m.root-servers.net.
. 202634 IN NS i.root-servers.net.
. 202634 IN NS k.root-servers.net.
. 202634 IN NS c.root-servers.net.
. 202634 IN NS a.root-servers.net.
. 202634 IN NS d.root-servers.net.
. 202634 IN NS l.root-servers.net.
. 202634 IN NS e.root-servers.net.
. 202634 IN NS g.root-servers.net.
. 202634 IN NS b.root-servers.net.
. 202634 IN NS j.root-servers.net.
. 202634 IN NS h.root-servers.net.
. 202634 IN NS f.root-servers.net.
. 511016 IN RRSIG NS 8 0 518400 20170329170000 20170316160000 61045 . OjsjoA6WCcThV3BqhAyMaXI3bU1m228Gl8nvaR074qBtem/RjaFJh1Oe r7LPI6W15jgbRGuCY7/GUNgDex4ZM43yvx2iY+2GpSk9b2/pKGbDaDIp X1Hd8418206eow1P/SgPqtT+2LzjM+lz/HKUvCyPoN4uX5/7GDExn8ir 2Q/vw81Za2nJ4Ji6oeGAmE8g6V3RJfTI0El/CP9Vl2/r0jWDZZ+wtYRA uwqvQYj+7VZc5+UJDJ3/Gne1LBnzXKnZRwnfJtZenhfZ7X20D9PiXvfJ M2s1ryqZvg8K5RqYt/r8pIiq1Udd1KWQBgdN7Q629+jNSngZQtH19R9H fM7h1w==
;; Received 1097 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms
in-addr.arpa. 172800 IN NS a.in-addr-servers.arpa.
in-addr.arpa. 172800 IN NS b.in-addr-servers.arpa.
in-addr.arpa. 172800 IN NS c.in-addr-servers.arpa.
in-addr.arpa. 172800 IN NS d.in-addr-servers.arpa.
in-addr.arpa. 172800 IN NS e.in-addr-servers.arpa.
in-addr.arpa. 172800 IN NS f.in-addr-servers.arpa.
in-addr.arpa. 86400 IN DS 47054 8 2 5CAFCCEC201D1933B4C9F6A9C8F51E51F3B39979058AC21B8DF1B1F2 81CBC6F2
in-addr.arpa. 86400 IN DS 53696 8 2 13E5501C56B20394DA921B51412D48B7089C5EB6957A7C58553C4D4D 424F04DF
in-addr.arpa. 86400 IN DS 63982 8 2 AAF4FB5D213EF25AE44679032EBE3514C487D7ABD99D7F5FEC3383D0 30733C73
in-addr.arpa. 86400 IN RRSIG DS 8 2 86400 20170330000000 20170316230000 33786 arpa. gqNN6MmLVxFtSG0oxdyoYVSXZlp6vY9yxpnJW89TYCHfTkv+ZmklM76O IHdTznEhLHXbyr4BaxjFCshsC3WacdH/YZYa/SZynJ9Q1N6/bogrIUTn qs61Y/YD4Sk5lI7YvUxVnPrF3lk1pY8dpoTkprTLZngoeQEsm552inqG ypU=
;; Received 731 bytes from 193.0.14.129#53(k.root-servers.net) in 41 ms
171.in-addr.arpa. 86400 IN NS ns1.apnic.net.
171.in-addr.arpa. 86400 IN NS ns2.lacnic.net.
171.in-addr.arpa. 86400 IN NS ns3.apnic.net.
171.in-addr.arpa. 86400 IN NS ns4.apnic.net.
171.in-addr.arpa. 86400 IN NS apnic.authdns.ripe.net.
171.in-addr.arpa. 86400 IN NS apnic1.dnsnode.net.
171.in-addr.arpa. 86400 IN NS tinnie.arin.net.
171.in-addr.arpa. 86400 IN DS 49699 5 1 0240801C96480900CC6D93130AF45CFE86EDE940
171.in-addr.arpa. 86400 IN DS 49699 5 2 6E0BA96F05B4D39C1668979731E040213BB6130DE33E86E063B5F7F7 5C465C88
171.in-addr.arpa. 86400 IN RRSIG DS 8 3 86400 20170330042932 20170309125911 4341 in-addr.arpa. RKcPZJdG1MBrwpfa1mIK7ikIU585i1Jv+UkEBHxuM3BxxAbD+ht0W04C ljboyXJfYK8ly/YTDqNUkWKZLwDHlkq/rgNwTG63sPT8iK8qya+2qUVB iTXVsD2HaV1V/KwgJjlWHRNnlwkK0YZ5Q7tevXaq4yyLT2Q2mu5dkhFC evNQOyI=
;; Received 482 bytes from 199.253.183.183#53(b.in-addr-servers.arpa) in 69 ms
171.in-addr.arpa. 0 IN SOA ns1.apnic.net. read-txt-record-of-zone-first-dns-admin.apnic.net. 5276 7200 1800 604800 172800
171.in-addr.arpa. 0 IN RRSIG SOA 5 3 172800 20170416100102 20170317090102 7858 171.in-addr.arpa. dH+uuGq9pgMc9TEMkTwrf9NGMj5zFnbZt3kjlNSF0kPzGk3WUr5g84jv 0SN88Y26eP/Op4Vv8JewyVu2IT1POnmiNpDYELkJxrTWnPPc6CJ2wugR TrFLOq6eIsqtrCcWbatQ7XSXXj1UglqVVydlX1ExoIh9MdP+1eBneD1I 4OU=
171.in-addr.arpa. 172800 IN NSEC 10.171.in-addr.arpa. NS SOA TXT RRSIG NSEC DNSKEY
171.in-addr.arpa. 172800 IN RRSIG NSEC 5 3 172800 20170416100102 20170317090102 7858 171.in-addr.arpa. QieHZ0W7jx3QCgLuvqtKtFoLPYkxn5R9nRO1oMbSSXkdF+S4iQQkVlX7 DyQvoL69hKA+S9idfernf6DwNTQgeWFfCdjUyOjMCa3Ag/S8ARZs7K4F N9DxB/cBCZm7KTzwAzXFe3cJ/d1Wo45Tx9jf6Drx551oqOw1gmDIL4y6 +k4=
;; Received 530 bytes from 202.12.31.140#53(ns4.apnic.net) in 347 ms
}}}
since it is APNIC who respond to queries about it. Any more specific
reverse records that are cross ARIN and die there from 171.x.y.z?
Me too.
We have a 164.138.208.0/22 and we have issues with dns reverse too
We sent an email to ripe support, but no answer yet
I just receive an answer from RIPE and now its working again:
Dear Alberto,
Our apologies for this. There was a bug in our DNS provisioning system, and it temporarily caused some delegations (where the parent zone is operated by ARIN) to be removed. The bug has been fixed, and the correct data is now being published on the RIPE NCC FTP server. We expect ARIN's DNS provisioning system to pick it up shortly, and the delegations will be restored.
Regards,
Anand Buddhdev
RIPE NCC
Eygene -
We are aware there’s an issue and working on it presently with RIPE.
Expect additional updates shortly.
/John
John Curran
President and CEO
ARIN
You can find a detailed announcement from the RIPE NCC here:
https://www.ripe.net/ripe/mail/archives/dns-wg/2017-March/003394.html
-Alex Band
Dear all,
RIPE NCC have issued a statement about the issue here:
https://www.ripe.net/ripe/mail/archives/dns-wg/2017-March/003394.html
Our apologies for the inconvenience caused.
Kind regards,
Romeo Zwart
RIPE NCC
John, Alex, Romeo,
Fri, Mar 17, 2017 at 12:31:06PM +0100, John Curran wrote:
We are aware there’s an issue and working on it presently with RIPE.
Expect additional updates shortly.
Fri, Mar 17, 2017 at 12:50:48PM +0100, Alex Band wrote:
You can find a detailed announcement from the RIPE NCC here:
https://www.ripe.net/ripe/mail/archives/dns-wg/2017-March/003394.html
Fri, Mar 17, 2017 at 12:52:10PM +0100, Romeo Zwart wrote:
RIPE NCC have issued a statement about the issue here:
https://www.ripe.net/ripe/mail/archives/dns-wg/2017-March/003394.html
Our apologies for the inconvenience caused.
Thanks for your work: the issue seem to be fixed. I am trying to
verify if everything works as expected, there are some neats,
{{{
206.144.in-addr.arpa. 172800 IN NS ns1.rrcki.ru.
206.144.in-addr.arpa. 172800 IN NS ns.kiae.ru.
206.144.in-addr.arpa. 172800 IN NS ns3.rrcki.ru.
206.144.in-addr.arpa. 172800 IN NS ns2.grid.kiae.ru.
206.144.in-addr.arpa. 172800 IN NS ns.grid.kiae.ru.
206.144.in-addr.arpa. 10800 IN NSEC 207.144.in-addr.arpa. NS RRSIG NSEC
206.144.in-addr.arpa. 10800 IN RRSIG NSEC 5 4 10800 20170331110430 20170317100430 33345 144.in-addr.arpa. vbFwaKdRa7Jd70aAbJ
5mC37BsTzMg3nWVI5gqQLLOqSaCZfH0XUez+Uk MbTNvepziCRzH+HgSLabuvRSo4nIUP1SjOd2WX0wySSdb/blqhfmjw3l n8laqOxy/lj8TDiIuxOdw2JhM1v5x/DH4aDnwdGFfUEOdgzCU5k8LdAT oyA= ;; Received 373 bytes from 199.180.180.63#53(r.arin.net) in 198 ms
;; Question section mismatch: got 206.144.0.0.in-addr.arpa/PTR/IN
;; Question section mismatch: got 206.144.0.0.in-addr.arpa/PTR/IN
206.144.in-addr.arpa. 3600 IN SOA ns.kiae.ru. noc.rrcki.ru. 2017020803 10800 3600 1800000 3600
;; Received 105 bytes from 144.206.239.1#53(ns.grid.kiae.ru) in 0 ms
}}}
about question mismatch, though my guts feel that this is the local
issue at rrcki.ru.
Thanks again! And for a nicely-written summary -- too.
Folks -
RIPE NCC has backed out the change at issue, and we are again processing
zonelet files received from them. They’ve posted more details here -
<https://www.ripe.net/ripe/mail/archives/dns-wg/2017-March/003394.html>
FYI,
/John
John Curran
President and CEO
ARIN
Hmm. That sounds like an ARIN-side bug too. ARIN's code responded to
corrupted data by zeroing out the data instead of using the last known good
data. That's awfully brittle for such a critical service.
Regards,
Bill Herrin
William Herrin <bill@herrin.us> writes:
or proper whitelisting of your own infrastructure 
- Jared
If you don't have a chaos monkey, the Internet will provide one free of charge.
Hmm. That sounds like an ARIN-side bug too. ARIN's code responded to
corrupted data by zeroing out the data instead of using the last known good
data. That's awfully brittle for such a critical service.
Regards,
Bill Herrin
Hi Bill,
The analysis was not yet complete when the notice went out from RIPE. After doing a post-mortum, there were no bugs in ARIN’s software in regards to this issue. We followed exactly what RIPE told us to do. When we noticed an issue with RIPE’s updates yesterday, we notified them as well.
Two zones managed by APNIC that received delegations from RIPE were also affected by RIPE’s bug – it was more than just a RIPE/ARIN thing.
The three impacted RIR’s have been working closely together to get RIPE’s DNS entries correctly added into our respective authoritative DNS systems.
Thanks,
Mark
ARIN CTO
Or to a service partner or underlying infrastructure you rely on.
Isn't complexity grand...
-george
On 3/17/17, 12:26 PM, "NANOG on behalf of William Herrin" <
Hmm. That sounds like an ARIN-side bug too. ARIN's code responded to
corrupted data by zeroing out the data instead of using the last
known good
there were no bugs in ARIN’s software in regards to this issue. We
followed exactly what RIPE told us to do.
Hi Mark,
That shot my eyebrow up. You misspoke here, right? There's no bug -solely
because- you did what the design said to do? The design calls for some
self-check information and it's not a critical design bug to zero-out the
publish if the self-check fails?
Regards,
Bill Herrin
Hmm. That sounds like an ARIN-side bug too. ARIN's code responded to corrupted data by zeroing out the data instead of using the last known good data. That's awfully brittle for such a critical service.
Agreed in principle - receiving incorrect data (improperly formatted, corrupted, or not properly signed)
should result in appropriate notice and no change to the running system. This is actually the case with
ARIN’s systems.
However, we received a properly formatted and signed zonelet file, albeit one which contained zero
records. APNIC also received similar correctly formatted/signed zonelet files as a record of the RIPE
bug, and the three RIRs have been working closely together to get the correct RIPE data loaded back
into our authoritative DNS systems.
Thanks!
/John
John Curran
President and CEO
ARIN