I don't know that this is the case, I suspect it's
resource management. If the database is getting
slaughtered by applications on uncontrolled auto pilot,
it's unusable for the rest of us.
-M<
From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of
Paul G
Sent: Tuesday, March 01, 2005 5:03 AM
To: nanog@merit.edu
Subject: Re: High volume WHOIS queries
--- snip ---
point - they're trying to restrict the practicality of
attempting to harvest
the data and an open to the public whois server with no
access restrictions
would defeat that.
I don't know that this is the case, I suspect it's
resource management. If the database is getting
slaughtered by applications on uncontrolled auto pilot,
it's unusable for the rest of us.
well, the OP quoted a portion of the aup that requires bulk
whois data recipients to take measures to prevent harvesting,
so i presume that arin does care about that and, in fact, that
consideration is likely the reason they declined to permit the
OP to run *his own* whoisd off of his *local* copy of the data.
-p
If memory serves, that restriction didn't appear until spam became a problem. The verbiage in the AUP is there to give ARIN recourse in the event that some spammer, and it has happened, runs a harvest against domain names or serialized NIC handles to seed a spam source.
- billn
Understood.
So why not make it easy -- both for yourselves and for everyone else?
Just publish all WHOIS data on static web pages -- not even
marked up with HTML, just plain ASCII text -- whose URLs are
easy to construct, a la
www.verisign.com/foo/bar/blah/example1.com
www.verisign.com/foo/bar/blah/example2.net
and refresh them from backing store whenever the "real" data changes.
(And yes, I realize I'm using an example based on domains, not
networks, but I trust it's still applicable.)
This makes the load on the servers about as small as it's
going to get. (Heck, they could be served from a cut-down web
server designed to serve static content only.) It also makes
it trivially easy for people to look things up without worrying
about rate-limiting. Heck, once the search engines indexed it,
it'd be even easier.
As to "...then the spammers will mass-harvest it...": they already HAVE.
They're busy selling it to each other on CD/DVD and via other means.
This has been going on for years, and however-they're-doing-it, they're
doing it well enough to acquire recently-modified data.
So that toothpaste is completely out of the tube and there's no way
to put it back in. I don't think any substantive purpose is served
by pretending/wishing that it's otherwise: there's a demand for this
data, and plenty of money to be made by those who will supply it,
therefore it's going to be acquired and sold.
But the people who *can't* access the data -- not without taking measures
to evade the rate-blocking that's in place -- are abuse victims who are
trying to track down those responsible.
So I view the problem of overload on WHOIS servers as self-inflicted
damage, easily fixed by giving up the pretense that restricting access
to the data has any real value for anyone. (Well, it *does* benefit
those selling it, but I trust that ensuring their profits isn't a goal
that anyone's particularly worried about. 
)
---Rsk