Correct inclusion of rwhois info in WHOIS server output?

I've been talking to ARIN about the rwhois setup on our SWIPped blocks, and
there appears to be a problem with the standard output from whois.arin.net. The
two rwhois clients I've tried are rwhois and jwhois. The rwhois client behavior
is something like this:

1. Query whois.arin.net.
2a. If the response contains the name of an rwhois server, query that server and
return its output.
2b. If the response doesn't contain the name of an rwhois server, follow the
links. Query every rwhois server you find and return all of the output.

The jwhois client behavior is something like this:

1. Query whois.arin.net.
2a. If the response contains the name of an rwhois server, query that server and
return its output.
2b. If the response doesn't contain the name of an rwhois server, return the SWIP.

On blocks which are owned by CoreNAP, that works fine. For example, if I type:

whois -h whois.arin.net 66.219.44.0

The whois server returns our complete SWIP record including:

ReferralServer: rwhois://rwhois.corenap.com:4321/

So this block works fine with both jwhois and rwhois:

bash-2.05$ jwhois 66.219.44.0
[Querying whois.arin.net]
[Redirected to rwhois.corenap.com:4321]
[Querying rwhois.corenap.com]
[rwhois.corenap.com]
%rwhois V-1.5:003fff:00 cache02.ns.corenap.com (by Network Solutions, Inc.
V-1.5.7.3)
network:Auth-Area:66.219.32.0/19
...

On blocks which are SWIPped to CoreNAP by an upstream provider, the response
from whois.arin.net does not include an rwhois record. For example, if I type:

whois -h whois.arin.net 65.59.252.0

The whois server returns this:

Level 3 Communications, Inc. LC-ORG-ARIN-BLK2 (NET-65-56-0-0-1)
                                   65.56.0.0 - 65.59.255.255
Core NAP, L.P. LVLT-CORENAP-NETBLOCK-03 (NET-65-59-252-0-1)
                                   65.59.252.0 - 65.59.252.255
VC Sterling, Inc. NET-65-59-252-0-1 (NET-65-59-252-0-2)
                                   65.59.252.0 - 65.59.252.255

Since there is no rwhois server listed here, rwhois clients don't necessarily
manage to find the referral. rwhois apparently follows both links and returns
results from every rwhois server it finds, but jwhois doesn't follow either
link; it just returns the SWIP info. I believe that the correct response to this query would be:

Level 3 Communications, Inc. LC-ORG-ARIN-BLK2 (NET-65-56-0-0-1)
                                   65.56.0.0 - 65.59.255.255
ReferralServer: rwhois://rwhois.level3.net:4321
Core NAP, L.P. LVLT-CORENAP-NETBLOCK-03 (NET-65-59-252-0-1)
                                   65.59.252.0 - 65.59.252.255
ReferralServer: rwhois://rwhois.corenap.com:4321/
VC Sterling, Inc. NET-65-59-252-0-1 (NET-65-59-252-0-2)
                                   65.59.252.0 - 65.59.252.255

I've read through the apparently relevant RFCs (812, 954, 1714, 1834, 1835, 1913, 1914, 2050, 2167, 3912) but did not find a clear specification of correct WHOIS server output. The people I talked to at ARIN say that the configuration of whois.arin.net can be changed based on "significant community consensus" but they suggested that the problem could be fixed by rewriting the jwhois client (and any other client that doesn't follow links to search for an rwhois server). I spent a fair amount of time looking through the (apparently non-searchable) mailing list archive at http://lists.arin.net/pipermail/dbwg/ and saw some discussion of rwhois issues but I didn't manage to find information showing how the previous change was initiated. Questions:

1. Does anyone agree that the present lack of rwhois server information in the initial WHOIS response for SWIPped blocks is a problem?

2. Can anyone think of a compelling reason why rwhois server information should not be included in the initial response to a standard whois query for all IP blocks, including SWIPped blocks, besides the fact that it is not included now?

3. Would this change (adding rwhois server information to the initial response to a standard whois query for SWIPped blocks) break your scripts that parse WHOIS output?

4. How disruptive was the change when rwhois server information was initially added to WHOIS output?

5. Was the issue fully thought through at that time, and the rwhois server information intentionally left out of the initial response for SWIPped blocks, or did this happen by accident?

6. Does anyone know where that change process was documented?

I think the real problem here is that jwhois is preferring ARIN's whois
service rather than their rwhois service, because the human-readable whois
output looks nicer. Oh, and another possibility is that for a very long
time, ARIN's rwhois server was badly borken and just redirected back to IANA
(oops!). (It appears to be working now as of this writing.)

If jwhois instead understood rwhois protocol (it does not currently), it
would have no problem, because ARIN has referrals all the way down to you
for the network you posted (65.59.252.x). That is, of course, the whole
point of using the structured rwhois protocol in the first place. :slight_smile:

I've been talking to ARIN about the rwhois setup on our SWIPped blocks, and
there appears to be a problem with the standard output from whois.arin.net.

Be carefull about using word "standard", there is no standard output for
whois plus both for ARIN and other RIRs output depends on the options used
when queries. In this case, I think you wanted to say that the problem you see is with their default output.

The two rwhois clients I've tried are rwhois and jwhois.

jwhois is not an rwhois client. It does whois query on port 4321 but does
not actually interpret that data as real rwhois client would have.

The rwhois client behavior is something like this:
1. Query whois.arin.net.
2a. If the response contains the name of an rwhois server, query that server and
return its output.
2b. If the response doesn't contain the name of an rwhois server, follow the
links. Query every rwhois server you find and return all of the output.

I have not tried rwhois as a client for a while, but I believe you're not
describing it properly. Rwhois would do rwhois (port 4321) query on root.rwhois.net and would follow rwhois system referrals from there (which on the technical level is not the same as whois listed referral line). As of about year ago ARIN finally (after 4 or 5 years!) fixed those rwhois referrals (previously it would only work for very few blocks) so that rwhois server would follow to your own rwhois system.

I do want to note though that rwhois output at arin is not alwas insync
with their whois data (I think they generate it and once/day or less,
where as whois is updated twice/day) and does not list any contact
data - only organization name.

The jwhois client behavior is something like this:

1. Query whois.arin.net.
2a. If the response contains the name of an rwhois server, query that server and
return its output.
2b. If the response doesn't contain the name of an rwhois server, return the SWIP.

That seems reasonable for simple whois system which does not want to go
into all these complexities and rare cases of mixed SWIP and rwhois usage.

BTW - Completewhois behavior is to:
  1. Output swip data including every swip for multiple levels
  2. Look at if there is rwhois referral ReferralServer in the swip
     (AND also look at if rwhois server is listed by name in the
     comments and try to use that as well even if ReferralServer
     is not present) and if so do rwhois lookup (using rwhois
     library but without following referrals) to listed server.

On blocks which are SWIPped to CoreNAP by an upstream provider, the response
from whois.arin.net does not include an rwhois record. For example, if I type:

whois -h whois.arin.net 65.59.252.0

You might want to try whois -h whois.arin.net "+ 65.59.252.0"

The whois server returns this:

Level 3 Communications, Inc. LC-ORG-ARIN-BLK2 (NET-65-56-0-0-1)
                                 65.56.0.0 - 65.59.255.255
Core NAP, L.P. LVLT-CORENAP-NETBLOCK-03 (NET-65-59-252-0-1)
                                 65.59.252.0 - 65.59.252.255
VC Sterling, Inc. NET-65-59-252-0-1 (NET-65-59-252-0-2)
                                 65.59.252.0 - 65.59.252.255

Since there is no rwhois server listed here, rwhois clients don't necessarily
manage to find the referral. rwhois apparently follows both links and returns
results from every rwhois server it finds, but jwhois doesn't follow either
link; it just returns the SWIP info. I believe that the correct response to this query would be:

That is a problem with jwhois and not with default arin output which is
very simplistic and lists only list of swips without detail on each swip.
Jwhois should query the detail or use extended query ("+") to find all
the relevent information.

BTW, compare this to the output given by completewhois:
  whois -h whois.completewhois.com 65.59.252.1
(saved in http://www.completewhois.com/cgi-bin/whois.cgi?query=R33857865)

As you notice it tried to do query to both rwhois.level3.net and to
rwhois.corenap.com as well as displayed SWIP data. While admitedly
its too much information, that is how the system works - it provides
as much data as possible and lets user decide on what is relevent
to him/her.

But by automated means this really is a good example of how to confuse
the ip whois client, because you have 2 rwhois servers listed (for
top-most ip block and 2nd level deligation block) as well as SWIPs all
the way to the assignment to customer. If you want automated systems to use rwhois, then query to your own server would never have been made (the referral to follow would be rwhois.level3.net), actually let me correct myself - if rwhois.level3.net were to run rwhois server accessible to public, the correct rwhois-only setup would be for L3 to enter rwhois referral to rwhois.corenap.com on their server (like ARIN does on root.rwhois.net). In this case client would find top-most rwhois server as rwhois.level3.net and follow from there and get to the lowest rwhois server available and give the results. But if you don't want automated systems to use rwhois data as primary, then the correct is to output the lowest swip level data in detail, i.e.
  whois -h whois.arin.net NET-65-59-252-0-1

Level 3 Communications, Inc. LC-ORG-ARIN-BLK2 (NET-65-56-0-0-1)
                                 65.56.0.0 - 65.59.255.255
ReferralServer: rwhois://rwhois.level3.net:4321
Core NAP, L.P. LVLT-CORENAP-NETBLOCK-03 (NET-65-59-252-0-1)
                                 65.59.252.0 - 65.59.252.255
ReferralServer: rwhois://rwhois.corenap.com:4321/
VC Sterling, Inc. NET-65-59-252-0-1 (NET-65-59-252-0-2)
                                 65.59.252.0 - 65.59.252.255

I'd be against any change like this because it will severely break a lot
of currently whois client software that know about this ARIN list data
and this would not be compatible with it.

I've read through the apparently relevant RFCs (812, 954, 1714, 1834, 1835, 1913, 1914, 2050, 2167, 3912) but did not find a clear specification of correct WHOIS server output.

You're correct. Whois protocol has no standards for output format.
There were however several attempts to do that on top of whois with "whois+" and "rwhois" as well as RPSL.

The people I talked to at ARIN say that the configuration of whois.arin.net can be changed based on "significant community consensus" but they suggested that the problem could be fixed by rewriting the jwhois client (and any other client that doesn't follow links to search for an rwhois server).

I agree with ARIN's assessment. But if I were doing coding for jwhois
I would not do it as you suggest and would insted write code that can
output the lowest level ARIN deligation (follow the lowest swip
in the list when its returned as above) and then look at that data and
see if that lowest swip has listed ReferralServer and follow that.

In your case it would not cause lookup to rwhois.corenap.com, but as
I said that is very very rare case that causes a lot of confusion for
pretty much every whois client except completewhois.

I spent a fair amount of time looking through the (apparently non-searchable) mailing list archive at http://lists.arin.net/pipermail/dbwg/ and saw some discussion of rwhois issues but I didn't manage to find information showing how the previous change was initiated.

ARIN formed subgroup within DBWG to discuss rwhois issues about 3 years ago.

Questions:
1. Does anyone agree that the present lack of rwhois server information in the initial WHOIS response for SWIPped blocks is a problem?

My general feeling is that ARIN is wrong in providing default list response
rather then detailed data. But as far as rwhois not being listed in default
list response, I do not believe it to be appropriate either.

2. Can anyone think of a compelling reason why rwhois server information should not be included in the initial response to a standard whois query for all IP blocks, including SWIPped blocks, besides the fact that it is not included now?

Yes, I can - see above.

3. Would this change (adding rwhois server information to the initial response to a standard whois query for SWIPped blocks) break your scripts that parse WHOIS output?

Yes, it would.

4. How disruptive was the change when rwhois server information was initially added to WHOIS output?

Not disruptive because it was done to detailed data output and everyone
expected data there to be multiple "Field: data" lines and that new fields could be added in the future.

5. Was the issue fully thought through at that time, and the rwhois server information intentionally left out of the initial response for SWIPped blocks, or did this happen by accident?

I can't speak for ARIN, but I think they fully realized that they are
adding ReferralServer line only for detailed output and not list/summary.

6. Does anyone know where that change process was documented?

Ask ARIN, I don't think it is documented though.

It would be SUCH a good thing if there was some move to standardize these

There is, of course, CRISP ..

> Be carefull about using word "standard", there is no standard output

for

> whois plus both for ARIN and other RIRs output depends on the options

used

> when queries. In this case, I think you wanted to say that the problem

you

> see is with their default output.
>

It would be SUCH a good thing if there was some move to standardize

these

There is, of course, CRISP ..

CRISP is merely smoke and mirrors at present.

There is the whois format used by all the other RIRs which is,
I believe, based on RPSL. RIPE even makes their server freely
available. Assuming that there is some kind of database behind
SWIP and whois, it should not be that difficult for ARIN to
switch to using the RIPE format for displaying answers to
whois queries.

--Michael Dillon

There is a long and occasionally sordid tale behind the history of various WHOIS server output formats. However, for ARIN's server specifically, assuming you want to change anything, you want to be making comments on ARIN's dbwg mailing list; see www.arin.net .

But in brief, RIPE NCC started up, developed their own. APNIC started up, lacked resources, used the RIPE NCC-developed server. ARIN started up some years later, inherited NSI's server + database, and kept on using it. LACNIC started up, and modified the .BR registry's WHOIS server to provide RPSL-like output. AFRINIC started up, and used the RIPE NCC-developed server. No-one was really interested in rwhois for ages :wink:

Here's a repost of a message I sent recently to the ARIN ppml on this subject:

Thanks to everyone who replied on and off-list. I'm concluding that there is a problem with WHOIS server output, caused mostly by a lack of standards, but people with more influence than me are already working on fixing that. In the meantime I'll see if I talk to the gnu maintainer about making jwhois more rwhois-friendly.

Marco d'Itri's whois client does quite a lot of what you want,
including automatically following an rwhois referral when it finds one

Of course not much can be done when the rwhois server to which you're
redirected just doesn't respond

suresh@etherkiller 11:33:29 <~> $ telnet rwhois.level3.net 4321
Trying 209.244.1.179...

srs