Hi
This was posted on arin-announce this morning as many of you
may be interested:
ARIN is pleased to announce that it plans to deploy an improved Whois
service called Whois-RWS on 26 June 2010. Included in the deployment are
the following services that provide the general public with access to
ARIN's registration data.
* a RESTful Web Service (RWS)
* a NICNAME/WHOIS port 43 service
* a user-friendly web site (http://whois.arin.net)
A demo of this service has been available since October 2009. The
demonstration service will be available at
http://whoisrws-demo.arin.net until the production service is deployed
on 26 June 2010.
When using Whois-RWS you will notice some differences in behavior for
certain queries and corresponding result sets on the NICNAME/WHOIS port
43 service. ARIN will make a separate announcement on 11 June when it
publishes detailed documentation on these differences along with the
demonstration service update.
ARIN continues to welcome community participation on the Whois-RWS
mailing list, and we invite you to subscribe and share your thoughts and
suggestions at:
http://lists.arin.net/mailman/listinfo/arin-whoisrws
More detailed information on these changes and other future features
that may impact the community at ARIN is available at:
https://www.arin.net/features/
Regards,
Mark Kosters
Chief Technical Officer
American Registry for Internet Numbers (ARIN)
I just found out that with the move to this new service that the bulk
access FTP is going to be phased out. By design, there will be no way
to automate the bulk download of this data.
Is anyone else using the data in an environment that will be seriously
impacted by this change?
Apparently we're supposed to be going all Web 2.0 now.
~Seth
Nothing wrong with having a nicer interface, but hopefully not at the
expense of bulk data. If it's a huge issue to support FTP data transfers,
they could at least provide a means through the web service to get bulk data
intelligently.
Apparently we're supposed to be going all Web 2.0 now.
Web 2.0 can handle bulk transfers of data just fine.
I wonder if this is somehow related to privacy and data protection laws.
Just recently, RIPE announced that they were going to block bulk
transfers as a result of data protection laws, presumably because some
law has just changed. Obviously ARIN is under a different legal regime
than RIPE, however data protection has recently been a hot button
issue in the USA and it is possible that something similar will
happen. Given the importance of case law in the USA, as opposed to
legislation, I wouldn't be surprised if there was some sort of legal
review going on.
But again, as far as technology goes, HTTP is a superior file transfer
protocol to FTP, so the move to Web 2.0 RESTful transactions over HTTP
does not give any technical reason to stop bulk transfers. In fact, it
may just be an oversight so you should really ask them Clearly, if
nobody bothers to ask about bulk transfers, then nobody uses them and
nobody cares, so shutting them down is the right thing to do.
--Michael Dillon
It's very clear. I went back and forth with support, asking how to
automate my bulk transfer with the new system.
Me: Is the bulk data download going to be available for automated
download. I can currently download the data daily from the ftp via a
script. The new web page doesn't seem to support that.
Support: No, there is no automation by design.
I'm ok with whatever system they provide if the functionality stays
the same. I don't understand what they gain by making a human login
and download the file.
I'm ok with whatever system they provide if the functionality stays
the same. I don't understand what they gain by making a human login
and download the file.
Accountability. If versions X and Y of database got abused (breach of
ToS), and only user U has downloaded such versions, gotcha.
Using honeytokens on the downloaded file can be interesting to quickly
connect the dots: if one of the handles on the list is
comeonspammer32767@wannahaveapieceofme.com, dynamically generated to
match a download session, and suddenly this account starts to get
spam...
Rubens
well... yes.. doesn't help much if the token being abused is the
admin POC's phone number, however. A session-based generated
token alone would not be a very robust form of accountability; it
is only as good as the strength of the verification required to get an
account
(and the confidence that multiple accounts do not collude).
A user might simply sign up twice or more using fake signup details,
they can compare their different downloads, and screen out any records
that changed between the several sessions.
e.g. grab 3 copies of thesame file (that were obtained using 3
different logins, from 3 different countries), run a 3-way diff,
strip out any lines that changed.
Any session-specific token would be excluded...
That is, if obtaining such a listing of e-mail addresses is even is
worth it to them. Maybe it is not.
Maybe the more common abuse is manual solicitation by a human being,
trying to sell some high-margin product targeted at enterprises in
the directory, who can easily recognize "comeonspammer" and stay
away.
I doubt the average POC is going to be duped by the pill salesmen,
latest money making scam, too-good-to-be-true offer, go phish
attempt, or other standardized junk mail.
Hi
ARIN is making significant improvements to our systems and services. ARIN
encourages the community to look for upcoming features as details are
available at: https://www.arin.net/features.
I would like to clear up the confusion about the changes to access to
Bulk Whois that have been discussed in this thread. The next release of
ARIN Online on 26 June 2010 will include an easy way of automating bulk
Whois reports. ARIN sent an announcement about this change to all current
Bulk Whois recipients on 1 June 2010. The current legacy ftp service for
Bulk Whois recipients will continue to operate until 31 August 2010. This
should allow enough time to make the changes required to your scripts to
migrate to the new solution.
Regards,
Mark Kosters
ARIN CTO
It's very clear. I went back and forth with support, asking how to
automate my bulk transfer with the new system.
Me: Is the bulk data download going to be available for automated
download. I can currently download the data daily from the ftp via a
script. The new web page doesn't seem to support that.
Support: No, there is no automation by design.
I'm ok with whatever system they provide if the functionality stays
the same. I don't understand what they gain by making a human login
and download the file.
Jason -
My apologies for the confusion over this when you called in;
while we had briefed the support team on RESTful WHOIS, we
hadn't covered the updated Bulk Whois interface as it is a
bit of a specialized item and coming out on the next release
of ARIN Online due to its need for "API key" support. The
26 June release of ARIN Online will allow you to create and
manage these keys, which in turn may be used in RESTful calls
(and email templates!) for authentication. A brief overview
of this feature was provided at the ARIN Toronto meeting and
is available here:
https://www.arin.net/participate/meetings/reports/ARIN_XXV/PDF/Tuesday/Newton-REST-and-Relax.pdf
We will rollout the API key functionality and Bulk Whois via the
RESTful interface with this next release of ARIN Online on 26 June,
and this will allow the Bulk Whois data to be downloaded directly
without logging into ARIN Online by using a RESTful HTTP request
containing your "API key". As Mark Kosters noted in his message,
we did contact current Bulk Whois users ahead of time about these
changes, but if you were missed or have any questions about the
change, please don't hesitate to contact myself or Mark directly.
Thanks!
/John
John Curran
President and CEO
ARIN
My apologies for the confusion over this when you called in;
while we had briefed the support team on RESTful WHOIS, we
hadn't covered the updated Bulk Whois interface as it is a
bit of a specialized item and coming out on the next release
of ARIN Online due to its need for "API key" support. The
26 June release of ARIN Online will allow you to create and
manage these keys, which in turn may be used in RESTful calls
(and email templates!) for authentication. A brief overview
of this feature was provided at the ARIN Toronto meeting and
is available here:
https://www.arin.net/participate/meetings/reports/ARIN_XXV/PDF/Tuesday/Newton-REST-and-Relax.pdf
We will rollout the API key functionality and Bulk Whois via the
RESTful interface with this next release of ARIN Online on 26 June,
and this will allow the Bulk Whois data to be downloaded directly
without logging into ARIN Online by using a RESTful HTTP request
containing your "API key". As Mark Kosters noted in his message,
we did contact current Bulk Whois users ahead of time about these
changes, but if you were missed or have any questions about the
change, please don't hesitate to contact myself or Mark directly.
john,
today, a research batch script running periodic bulk whois work has a
line something like
ncftpget ftp://user:pass@ftp.arin.net/arin_db.txt.gz
well, it can actually be simpler.
for the web 9.3 impaired of us, could you describe the simple batch
script line under the new improved system?
thanks!
randy
john,
today, a research batch script running periodic bulk whois work has a
line something like
ncftpget ftp://user:pass@ftp.arin.net/arin_db.txt.gz
well, it can actually be simpler.
for the web 9.3 impaired of us, could you describe the simple batch
script line under the new improved system?
Randy - You're going to have to get on ARIN Online at least
once to generate an key (this means after June 26), but then
accessing the data should be just as simple for a batch script
(i.e. use curl or wget for this purpose). I've extracted the
relevant draft info from the June 26 release documents and
attached below. This is obviously subject to change until
the release actually comes out...
/John
---- DOWNLOADING USING AN API KEY ----
The report can be downloaded directly without logging into
ARIN Online using a RESTful HTTP request containing your
API key. The URL must look like:
https://www.arin.net/public/rest/downloads/nvpr?apikey=YOUR-API-KEY
There are a variety of ways to automate the retrieval of
this report. For example, on a Linux system, where your
API key is API-1111-2222-3333-4444, you can use the
following 'curl' command to download the report file:
curl ARIN - American Registry for Internet Numbers > arin_nvpr.zip
You can manage your API keys on the when logged into your
ARIN Online account.
You're going to have to get on ARIN Online at least once to generate
an key
i can probably survive this experience. is there a tee shirt? 
The report can be downloaded directly without logging into
ARIN Online using a RESTful HTTP request containing your
API key. The URL must look like:
ARIN - American Registry for Internet Numbers
this looks quite doable.
thank you!
randy
You're going to have to get on ARIN Online at least once to generate
an key
i can probably survive this experience. is there a tee shirt?
Your request has been noted... 
The report can be downloaded directly without logging into
ARIN Online using a RESTful HTTP request containing your
API key. The URL must look like:
ARIN - American Registry for Internet Numbers
this looks quite doable.
thank you!
Thank you (and Jason Lewis!) for pointing out the lack of
actionable information regarding this announcement and its
impact on Bulk Whois. I've chatted with Mark and Nate on
the timing of this service change, and going forward we'll
make sure to have replacement services fully deployed and
verifiable by the community before announcing an end date
for the current service.
Thanks again!
/John
John Curran
President and CEO
ARIN