ARIN down?

I haven’t been able to connect to http://arin.net for several hours, but was able to open a ticket this morning. I’ve tried from several different networks, all roads seem to lead to the same place, with packets dropping at the NTT interface 129.250.196.154. e.g.:

$ traceroute arin.net<http://arin.net>
traceroute: Warning: arin.net<http://arin.net> has multiple addresses; using 199.43.0.44
traceroute to arin.net<http://arin.net> (199.43.0.44), 64 hops max, 52 byte packets
1 l100.lsanca-vfttp-106.verizon-gni.net<http://l100.lsanca-vfttp-106.verizon-gni.net> (98.112.74.1) 5.992 ms 4.865 ms 4.943 ms
2 172.102.106.24 (172.102.106.24) 9.962 ms 9.723 ms 12.242 ms
3 ae2-0.lax01-bb-rtr2.verizon-gni.net<http://ae2-0.lax01-bb-rtr2.verizon-gni.net> (130.81.22.238) 29.982 ms *
    so-4-1-0-0.lax01-bb-rtr2.verizon-gni.net<http://so-4-1-0-0.lax01-bb-rtr2.verizon-gni.net> (130.81.151.248) 9.428 ms
4 0.ae6.br1.lax15.alter.net<http://ae6.br1.lax15.alter.net> (140.222.225.137) 9.806 ms * *
5 ae-7.r01.lsanca20.us.bb.gin.ntt.net<http://us.bb.gin.ntt.net> (129.250.8.85) 10.409 ms
    0.ae6.br1.lax15.alter.net<http://ae6.br1.lax15.alter.net> (140.222.225.137) 19.783 ms 9.757 ms
6 ae-7.r01.lsanca20.us.bb.gin.ntt.net<http://us.bb.gin.ntt.net> (129.250.8.85) 10.292 ms 9.357 ms 12.291 ms
7 ae-17.r01.lsanca07.us.bb.gin.ntt.net<http://us.bb.gin.ntt.net> (129.250.4.207) 22.541 ms
    ge-101-0-0-3.r06.asbnva02.us.bb.gin.ntt.net<http://us.bb.gin.ntt.net> (129.250.196.153) 72.412 ms
    ae-17.r01.lsanca07.us.bb.gin.ntt.net<http://us.bb.gin.ntt.net> (129.250.4.207) 22.167 ms
8 ge-101-0-0-3.r06.asbnva02.us.bb.gin.ntt.net<http://us.bb.gin.ntt.net> (129.250.196.153) 72.510 ms 74.590 ms 72.258 ms
9 ge-101-0-0-3.r06.asbnva02.us.ce.gin.ntt.net<http://us.ce.gin.ntt.net> (129.250.196.154) 69.960 ms * 70.930 ms
10 * * *
11 * * *

$ traceroute www.arin.net<http://www.arin.net>
traceroute: Warning: www.arin.net<http://www.arin.net> has multiple addresses; using 199.43.0.43
traceroute to www.arin.net<http://www.arin.net> (199.43.0.43), 64 hops max, 40 byte packets
1 router1.sb.becknet.com<http://router1.sb.becknet.com> (206.83.0.1) 1.010 ms 0.420 ms 0.536 ms
2 206-190-77-9.static.twtelecom.net<http://206-190-77-9.static.twtelecom.net> (206.190.77.9) 3.983 ms 0.732 ms 0.686 ms
3 64-129-238-182.static.twtelecom.net<http://64-129-238-182.static.twtelecom.net> (64.129.238.182) 2.760 ms lax2-pr2-xe-1-3-0-0.us.twtelecom.net<http://lax2-pr2-xe-1-3-0-0.us.twtelecom.net> (66.192.241.218) 2.816 ms 64-129-238-186.static.twtelecom.net<http://64-129-238-186.static.twtelecom.net> (64.129.238.186) 18.203 ms
4 4.68.71.137 (4.68.71.137) 3.245 ms 2.877 ms 2.889 ms
5 * * *
6 ae-28.r00.lsanca07.us.bb.gin.ntt.net<http://us.bb.gin.ntt.net> (129.250.9.93) 3.731 ms 3.483 ms 3.850 ms
7 ae-3.r01.lsanca07.us.bb.gin.ntt.net<http://us.bb.gin.ntt.net> (129.250.5.29) 3.517 ms 3.433 ms 3.458 ms
8 ge-101-0-0-3.r06.asbnva02.us.bb.gin.ntt.net<http://us.bb.gin.ntt.net> (129.250.196.153) 69.503 ms 68.021 ms 68.072 ms
9 ge-101-0-0-3.r06.asbnva02.us.ce.gin.ntt.net<http://us.ce.gin.ntt.net> (129.250.196.154) 67.075 ms 67.102 ms 67.122 ms
10 * * *
11 * * *

I recall ARIN had a DDoS attack a week or so ago. Does anybody know if this is a recurrence?

-mel

Yep, they're under another DDoS attack:

From: ARIN <info@arin.net>
Subject: [arin-announce] ARIN DDoS Attack
Date: March 25, 2016 at 1:31:34 PM PDT
To: arin-announce@arin.net

Starting at 3:55 PM EDT on Friday, 25 March, a DDoS attack began against ARIN. This was and continues to be a sustained attack against our provisioning services, email, and website. We initiated our DDoS mitigation plan and are in the process of mitigating various types of attack traffic patterns. All our other public-facing services (Whois, Whois-RWS, RDAP, DNS, IRR, and RPKI repository services) are not affected by this attack and are operating normally.

We will announce an all clear 24 hours after the attacks have stopped.

Regards,

Mark Kosters
Chief Technology Officer
American Registry for Internet Numbers (ARIN)
_______________________________________________

Regards,
-drc

You’d think with all the money they collect, they’d have permanent DDOS mitigation in place. Time for them to call BlackLotus :slight_smile:

-mel

An announcement went out on arin-announce yesterday (but you might not be able to follow the link if you can’t reach list.arin.net):

http://lists.arin.net/pipermail/arin-announce/2016-March/001963.html

tl;dr: Massive DDoS. Usual affair. Welcome to the Internet.

I haven’t been able to connect to http://arin.net for several hours
I recall ARIN had a DDoS attack a week or so ago. Does anybody know if this is a recurrence?

Yes, it is. I attach Mark’s notice about it from this afternoon.

                                -Bill

Yeah, lists.arin.net<http://lists.arin.net> is down too, despite being hosted elsewhere. The netvermin apparently are being thorough. Still, there are many ways to broadly distribute static content like mailing lists.

-mel

I’m sure we all sympathize with the workload a DDOS attack imposes, as most of us have been there. But I can’t understand why there is so little broadcast communication of the attack through multiple channels. lists.arin.net<http://lists.arin.net> is rather esoteric. Facebook and Twitter are obvious alternative channels that are hard to attack, yet both are silent on the subject:

https://www.facebook.com/TeamARIN/
https://twitter.com/teamarin

Google shows only four hits for “arin dos attack march 25 2016”, and those are only fragments of the lists.arin.net<http://lists.arin.net> announcement, all of which dead end at arin.net<http://arin.net> right now.

It’s creepy that a major chunk of Internet infrastructure can be down for so long with so little public notice.

-mel

Hi Mel,

They do. www.arin.net is accessible for me and most of the rest of the
Internet. Your traceroute didn't work because the UDP to random ports
that traceroute generates is likely among the packets the DDOS
mitigator filters out.

If you can't get to the web page with a browser, some things to consider:

1. Are you behind a NAT with anybody else? Anybody who might, say, be
unknowingly participating in a botnet?

2. How good a job does your ISP do scrubbing spoofed source addresses
originated by its clients?

Regards,
Bill Herrin

They do, and they’re using it.

                                -Bill

http://www.downforeveryoneorjustme.com/www.arin.net

William,

How did you determine that ARIN is accessible for “most of the rest of the Internet”?

I’ve tried accessing the web site from nine different networks: Cox, Comcast, Level3, Verizon, AT&T, CenturyLink, Frontier, Sprint and Cogent. None of them can reach it. I’ve used non-firewalled network monitors, as well as NAT’d devices. The DDoS attack seems to be blocking access from a large subset of U.S. ISPs. I am an ISP and we follow standard anti-IP spoofing practices, so at least my networks aren’t DDOS spoof sources.

-mel

I can reach it just fine via Level3 and NTT right now.

                                -Bill

Since they’re hosted at NTT, that you can reach it from their seems reasonable. But I’ve just tried again from my Level 3 rack in the Santa Barbara hub, and no access. So it’s intermittent at best.

Hopefully they’ll get clear soon. We had a turn-up today that got waylaid by the outage.

-mel

Anyone know how big really? One org's "Massive DDoS" is another's "oh, is someone sending us some extra DNS traffic again?"