AOL DNS - temporary resolution of problem

Mailman List Mirror (Read Only)
1

At about noon today NetworkTwo (formerly Autonet) noticed heavy usage of
our Internet links and DNS. When we investigated we discovered what you
already know ... someone pointed AOL's root server entry at us. We
contacted AOL about the same time they contacted us. AOL asked us to load
their primary zone file on our DNS, but it quickly became apparent that our
upstream pipe and our DNS server could not handle the load. We (AOL and
N2) contacted NetworkTwo's upstream provider MichNet (aka Merit of
nanog@merit.edu fame). Merit loaned us their new, not yet in service, DNS
server. This was loaded with both the AOL and Autonet primary zones.
Merit then hijacked the 206.88.0.x network and redirected it to their
server, where AOL and Autonet are currently resolving. Some of my clients
are affected, but most have been pointed to other name servers.

The InterNIC folks predict it will take 18 hours for the root servers to
be up to date. We will monitor the situation throughout the weekend, and
take apart this hack when the number of queries drops off.

On behalf of NetworkTwo, I'd like to thank the on call staff at Merit and
AOL, all of whom pitched in totally professional way with time and
equipment to solve this problem. Thanks also to Goodnet (spelling?), a
peer of AOL and MichNet, who offered equipment and bandwidth that we might
have needed, but didn't.

On a personal note, it's nice to find out that people can still work
together in a crisis. Now if we can only get NSI to secure the domain
update process ...

With hopes for a calmer weekend,

Dave Hares

2

Wow. I thought originally that this was a hijack; good to see that it
wasn't.

The question that I have remaining is, "How'd this happen?"

How did the primary DNS mysteriously change?

At about noon today NetworkTwo (formerly Autonet) noticed heavy usage of
our Internet links and DNS. When we investigated we discovered what you
already know ... someone pointed AOL's root server entry at us. We
contacted AOL about the same time they contacted us. AOL asked us to load
their primary zone file on our DNS, but it quickly became apparent that our
upstream pipe and our DNS server could not handle the load. We (AOL and
N2) contacted NetworkTwo's upstream provider MichNet (aka Merit of
nanog@merit.edu fame). Merit loaned us their new, not yet in service, DNS
server. This was loaded with both the AOL and Autonet primary zones.
Merit then hijacked the 206.88.0.x network and redirected it to their
server, where AOL and Autonet are currently resolving. Some of my clients
are affected, but most have been pointed to other name servers.

The InterNIC folks predict it will take 18 hours for the root servers to
be up to date. We will monitor the situation throughout the weekend, and
take apart this hack when the number of queries drops off.

On behalf of NetworkTwo, I'd like to thank the on call staff at Merit and
AOL, all of whom pitched in totally professional way with time and
equipment to solve this problem. Thanks also to Goodnet (spelling?), a
peer of AOL and MichNet, who offered equipment and bandwidth that we might
have needed, but didn't.

On a personal note, it's nice to find out that people can still work
together in a crisis. Now if we can only get NSI to secure the domain
update process ...

With hopes for a calmer weekend,

Dave Hares

--
David L. Hares, Director of Network Engineering
NetworkTwo Communications Group Phone: (313) 995-6539
175 Jackson Plaza FAX : (313) 995-6458
Ann Arbor, MI 48106 (USA) Email: dhares@networktwo.net

-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
   ISPF, The Forum for ISPs by ISPs. October 26-28, 1998, Atlanta, GA.
    Three days of clues, news, and views from the industry's best and
    brightest. http://www.ispf.com/ for information and registration.

     Atheism is a non-prophet organization. I route, therefore I am.
       Alex Rubenstein, alex@nac.net, KC2BUO, ISP/C Charter Member
               Father of the Network and Head Bottle-Washer
     Net Access Corporation, 9 Mt. Pleasant Tpk., Denville, NJ 07834
Don't choose a spineless ISP; we have more backbone! http://www.nac.net
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --

3

Wow. I thought originally that this was a hijack; good to see that it
wasn't.

It was a hijak, but not by the admins at AutoNet (or NetworkTwo). Take a
look at the follow URL, the third paragraph down:

http://www.news.com/News/Item/0,4,27655,00.html?st.ne.fd.gif.d

AOL was just using the MAIL-FROM auth. By setting this who ever was
listed as the Technical or administrative contact could alter the
domain. Internic just checks to see if the from address is a valid
one and if so the ACK is not required (I can tell you about this from
an experience we had). Therefore even a crude forgery can change the
domain servers if the auth is MAIL-FROM.

The strange thing is that the contacts listed for AOL (i.e. the previous
contacts if they were changed) received the piece of email that the
change was going through and did nothing about it until it was too
late. When this happened to us we jumped right on things and noone
was the wiser on the internet (although I guess AutoNet couldn't handle
the DNS traffic which is generated for AOL's web servers so that
would be a problem, even if things were caught).

bye,
ken emery