Reducing Usenet Bandwidth

Date: Fri, 08 Feb 2002 22:09:12 -0800
From: Stephen Stuart <>
Subject: Re: Reducing Usenet Bandwidth
To: David Schwartz <>


Some of those problems in that vein that appeared insurmountable ten years
ago might have solutions in current "peer-to-peer" networking technologies
(thus the off-hand comment about Napster).

What you're really asking for is a NNTP-over-FastTrack (Kazaa/Morpheus) feed
object. This is sort of/really an odd idea, but if you think about it, all
the required parts are pretty easy to articulate in skeleton form:

1) Each file dumped into FastTrack/Kazaa/Morpheus would get as its "filename"
the article's message-ID. A site that wanted to retrieve an article would
issue (via FastTrack/Kazaa/Morpheus) a search request for that message-ID,
retrieve the article from a FastTrack/Kazaa/Morpheus server that offered that
file, and then add it to the local news news server spool (and/or, optionally,
a FastTrack/Kazaa/Morpheus shared directory on the local server).

2) How would servers know what article IDs to ask for? Well, Usenet overview
data for each group could also be published by news servers via
FastTrack/Kazaa/Morpheus with filenames of the format

The pathtag would be required because obviously different news servers will
know about different articles, and you will want to disambiguate which
server's overview data you want (not because there's any expectation that
any news server will in fact provide public access to any articles, but
simply because one might expect that N servers might all simultaneously offer
comparable but non-identical overview data for any given group, and you might
want to choose the view from server foo, or bar, or baz over other

The component of the overview file name simply
allows folks to identify the overview data file they'd want.

The date/time stamp (GMT) provides a means of selecting the most recently
available overview data from a list of matching responses.

Overview data would obviously need to be cryptographically signed with a
key tied to the pathtag to avoid problems with people providing forged
overview data (wouldn't want people to be fed lists of bogus/inappropriate
lists of message ID's as a DOS/spam attack, right?).

Overview files could be generated at some server-determined periodicity, and
to avoid inspiring any sort of synchronization effect, I'll omit mentioning
a value here except to say that I've got something in the double-digits-of-
minutes in mind here. (Besides, if you're going to publish overview data for
some tens of thousands of groups, it will take a while to walk that tree).

3) Client servers interested in retrieving articles via
FastTrack/Kazaa/Morpheus could routinely retrieve overview records for
groups of interest, retrieving all/some of the articles as a "prefetch"
for popular groups routinely read by their users, or the server could wait
until articles have been requested, then retrieve an overview file, then
retrieve articles.

Alternatively, individual users with "News over FastTrack/Kazaa/Morpheus"
clients could be retrieving articles directly from Kazaa, removing any need
for an intervening news server whatsoever (although hopefully a local
"regular" news server would always be faster than a distributed
FastTrack/Morpheus/Kazaa-based news spool).



P.S. Needless to say, this would really change the FastTrack/Kazaa/Morpheus
content base, and given the volumes Usenet is currently transferring, would
be sort of an interesting experiment if we're talking about a full feed...