Usenet and Usenet Software FAQ [1 of 2 parts]

Path: news.compuserve.com!news.compuserve.com!newsfeed.internetmci.com!tank.news.pipex.net!pipex!in2.uu.net!deshaw.com!do-not-use-path-to-reply
Date: Tue, 12 Sep 1995 08:01:08 GMT
Supersedes:
Expires: Thu, 12 Oct 1995 08:01:08 GMT
Message-ID:
From: netannounce@deshaw.com (Mark Moraes)
Subject: Usenet Software: History and Sources
Newsgroups: news.admin.misc,news.announce.newusers,news.software.readers,news.software.b,news.answers
Followup-To: news.admin.misc
Approved: netannounce@deshaw.com (Mark Moraes)
Lines: 608
Xref: news.compuserve.com news.admin.misc:43242 news.announce.newusers:1919 news.software.readers:22063 news.software.b:18701 news.answers:49933
Status: N
Archive-name: usenet/software/part1
Original-from: spaf@cs.purdue.edu (Gene Spafford)
Comment: edited until 5/93 by spaf@cs.purdue.edu (Gene Spafford)
Last-change: 11 Sep 1995 by netannounce@deshaw.com (Mark Moraes)
Changes-posted-to: news.admin.misc,news.misc,news.software.readers,news.software.b,news.answers

Currently, Usenet readers interact with the news using a number of software packages and programs. This article mentions the important ones and a little of their history, gives pointers where you can look for more information and ends with some special notes about "foreign" and "obsolete" software. At the very end is a list of sites from which current versions of the Usenet software may be obtained.

Note that the number of software packages available to run news, especially on PCs, is increasing. This article lists only a few of the many news packages available, and the presence or absence of any particular software package should not be construed as indicating anything about its suitability usefulness.

The material contained in this post is probably not 100% up-to-date. Many of the software packages described in this posting are undergoing constant development, and it is not always possible to know when new releases have been made. Should you discover that information in this post has been superseded by a new release, please send mail to the poster of this article with the corrected information.

While the "official" ftp archive sites for packages are listed, note that most large archive sites carry news software; please try the one nearest to you before you use up expensive bandwidth on a trans-continental network link. You can check the official site for the current version number, if you want to make sure you're getting the latest version. In general, a good place to get recent versions of the more popular news transport and reading software by anonymous ftp is ftp.uu.net:/networking/news/ directory.

History

Usenet came into being in late 1979, shortly after the release of V7 Unix with UUCP. Two Duke University grad students in North Carolina, Tom Truscott and Jim Ellis, thought of hooking computers together to exchange information with the Unix community. Steve Bellovin, a grad student at the University of North Carolina, put together the first version of the news software using shell scripts and installed it on the first two sites: "unc" and "duke." At the beginning of 1980 the network consisted of those two sites and "phs" (another machine at Duke), and was described at the January Usenix conference. Steve Bellovin later rewrote the scripts into C programs, but they were never released beyond "unc" and "duke." Shortly thereafter, Steve Daniel did another implementation in C for public distribution. Tom Truscott made further modifications, and this became the "A" news release.

In 1981 at U. C. Berkeley, grad student Mark Horton and high school student Matt Glickman rewrote the news software to add functionality and to cope with the ever increasing volume of news -- "A" News was intended for only a few articles per group per day. This rewrite was the "B" News version. The first public release was version 2.1 in 1982; the 1.* versions were all beta test. As the net grew, the news software was expanded and modified. The last version maintained and released primarily by Mark was 2.10.1.

Rick Adams, at the Center for Seismic Studies, took over coordination of the maintenance and enhancement of the B News software with the 2.10.2 release in 1984. By this time, the increasing volume of news was becoming a concern, and the mechanism for moderated groups was added to the software at 2.10.2. Moderated groups were inspired by ARPA mailing lists and experience with other bulletin board systems. In late 1986, version 2.11 of B News was released, including a number of changes to support a new naming structure for newsgroups, enhanced batching and compression, enhanced ihave/sendme control messages, and other features.

The current release of B News is 2.11, patchlevel 19. Article format is specified in RFC 1036 (see below). B News has been declared "dead" by a number of people, including Rick Adams, and is unlikely to be upgraded further; most new UUCP sites are using C News or INN (see next paragraphs).

In March 1986 a package was released implementing news transmission, posting, and reading using the Network News Transfer Protocol (NNTP) (as specified in RFC 977). This protocol allows hosts to exchange articles via TCP/IP connections rather than using the traditional uucp. It also permits users to read and post news (using a modified news user agent) from machines which cannot or choose not to install the Usenet news software. Reading and posting are done using TCP/IP messages to a server host which does run the Usenet software. Sites which have many workstations like the Sun and SGI, and HP products find this a convenient way to allow workstation users to read news without having to store articles on each system. Many of the Usenet hosts that are also on the Internet exchange news articles using NNTP because the load impact of NNTP is much lower than uucp (and NNTP ensures much faster propagation).

NNTP grew out of independent work in 1984-1985 by Brian Kantor at U. C. San Diego and Phil Lapsley at U. C. Berkeley. Primary development was done at U. C. Berkeley by by Phil Lapsley with help from Erik Fair, Steven Grady, and Mike Meyer, among others. The NNTP package (now called the reference implementation) was distributed on the 4.3BSD release tape (although that was version 1.2a and out-of-date) and is also available on many major hosts by anonymous FTP. The current version is 1.5.12.1. It includes NOV support and runs on a wide variety of systems. It is available from ftp.academ.com:/pub/nntp1.5/nntp.1.5.12.1.tar.Z. For those with access to the World-Wide Web on the Internet, the WWW page http://www.academ.com/academ/nntp.html contains a description and news about NNTP. A different variant, called nntp-t5, implements many of the extensions provided by INN (most notably News Overview NOV support -- see below). It is available from:

     ftp.uu.net:/networking/news/nntp/nntp-t5.tar.gz.
One new version of news, known as C News, was developed at the University of Toronto by Geoff Collyer and Henry Spencer. This version is a rewrite of the lowest levels of news to increase article processing speed, decrease article expiration processing and improve the reliability of the news system through better locking, etc. The package was released to the net in the autumn of 1987. For more information, see the paper "News Need Not Be Slow," published in The Winter 1987 Usenix Technical Conference proceedings. This paper is also available from ftp.cs.toronto.edu in doc/programming/c-news.*, and is recommended reading for all news software programmers. The most recent version of C News is the Sept 1994 "Cleanup Release." C News can be obtained by anonymous ftp from its official archive site,
     ftp.cs.toronto.edu:pub/c-news/c-news.tar.Z.

Another Usenet system, known as InterNetNews, or INN, was written by Rich Salz . INN is designed to run on Unix hosts that have a socket interface. It is optimized for larger hosts where most traffic uses NNTP, but it does provide full UUCP support. INN is very fast, and since it integrates NNTP many people find it easier to administer only one package. The package was publicly released on August 20, 1992. For more information, see the paper "InterNetNews: Usenet Transport for Internet Sites" published in the June 1992 Usenix Technical Conference Proceedings. INN can be obtained from many places, including the 4.4BSD tape; its official archive site is ftp.uu.net in the directory /networking/news/nntp/inn. The current version is 1.4sec, last release 22-dec-1993.

Towards the end of 1992, Geoff Collyer implemented NOV (News Overview): a database that stores the important headers of all news articles as they arrive. This is intended for use by the implementors of news readers to provide fast article presentation by sorting and "threading" the article headers. (Before NOV, newsreaders like trn, tin and nn came with their own daemons and databases that used a nontrivial amount of system resources). NOV is fully supported by C News, INN and NNTP-t5. Most modern news readers use NOV to get information for their threading and article menu presentation; use of NOV by a newsreader is fairly easy, since NOV comes with sample client-side threading code.

ANU-NEWS is news package written by Geoff Huston of Australia for VMS systems. ANU-NEWS is a complete news system that allows reading, posting, direct replies, moderated newsgroups, etc. in a fashion closely related to regular news. The implementation includes the RFC 1036 news propagation algorithms and integrated use of the NNTP protocols (see below) to support remote news servers, implemented as a VAX/VMS Decnet object. An RFC 977 server implemented as a Decnet object is also included. ANU-NEWS currently includes support for the following TCP/IP protocols: MultiNet, CMU/TEK, Wollongong WIN/TCP, UCX (TCP/IP Services for OpenVMS), EXOS, and TCPware. The ANU-NEWS interface is similar to standard DEC screen oriented systems. The license for the software is free, and there are no restrictions on the re-distribution. For more info, contact gih900@fac.anu.oz.au (Geoff Huston). ANU-NEWS is available for FTP from kuhub.cc.ukans.edu. Contact SLOANE@KUHUB.CC.UKANS.EDU for more info.

A screen-oriented news client for VMS that works with CMU/tek, EXOS, MultiNet, UCX, Wollongong and DECnet is also available via ftp from iraun1.ira.uka.de, info.rz.uni-ulm.de, and ftp.spc.edu (contact Bernd Onasch for details).

Reader NNTP clients for VMS are also available, including VMS/VNEWS (current release 1.4) and an upcoming reader only version of ANU-NEWS. VMS/VNEWS is available via anonymous ftp from arizona.edu (contact jms@arizona.edu for more information) or at any site which archives vmsnet.source. Although the current release of ANU-NEWS is usable as a reader it can be difficult when used with a UNIX server.

FNEWS is a fast news reader, for VAX/VMS and UNIX. It is basically a mixture of NEWSRDR and ANU-NEWS, (a bit like 'nn' in how it works) giving a nice (but different) full-screen interface and fast response to all 3000 groups without heavily loading your local machine. It works by caching the news indexes from a UNIX news system (CNEWS or INN), and then dynamically loading the items when the user wants to read them. Indexes are only cached for groups which are actually read, so the load and disk usage can be very small. FNEWS Versions are available for VMS, ALPHA-VMS and UNIX via anonymous ftp from grunt.marc.cri.nz in pub/fnews. Contact chrisp@marc.cri.nz for more information.

A port of C News for the Commodore Amiga under AmigaDOS (NOT Unix), is available. The port was done by Frank J. Edwards , and available from Larry Rosenman . Also, Matt Dillon , has greatly improved the UUCP clone for AmigaDOS, currently V1.16D, available for ftp from ftp.uu.net in /systems/amiga/dillon. The package also includes a newsreader very loosely like the real rn. Dillon also has a "vn" port provided by Eric Lee Green. This software is also available on Bix, and for ftp from wuarchive.wustl.edu (many other Amiga newsreaders are also available on theis ftp site).

Several popular screen-oriented news reading interfaces have been developed in the last few years to replace the traditional "readnews" interface. The first of these was "vnews" and it was written by Kenneth Almquist. "vnews" provides a "readnews"-like command interface, but displays articles using direct screen positioning. It appears to have been inspired, to some extent, by the "notes" system (described below). "vnews" is currently distributed with the standard 2.11 news source.

A second, more versatile interface, "rn", was developed by Larry Wall (the author of Perl) and released in 1984. This interface also uses full-screen display with direct positioning, but it includes many other useful features and has been very popular with many regular net readers. The interface includes reading, discarding, and/or processing of articles based on user-definable patterns, and the ability of the user to develop customized macros for display and keyboard interaction. "rn" is currently at release 4.4.4. It is being maintained by Stan Barber . "rn" is not provided with the standard news software release, but is very widely available because of its popularity. The software can be obtained from its official archive site, ftp.academ.com, using FTP. A description and some news about it can be found on the WWW page:

     http://www.academ.com/academ/rn.html.

Wayne Davison's "trn" is a superset of "rn". Trn adds the ability to follow "threads of discussions" in newsgroups; its latest version 3.6 is based on rn 4.4. It uses a Reference-line database to allow the user to take advantage of the "discussion tree" formed by an article and its replies. This results in a true reply-ordered reading of the articles, complete with a small ascii representation of the current article's position in the discussion tree. Trn is also capable of "menu-based" selection of articles, allows one to do useful things to operate on a set of selected newsgroups. Trn can be obtained from ftp.uu.net in the /networking/news/readers/trn directory, and from many other archive servers world-wide.

xrn is an X11-based interface to NNTP that was written by Rick Spickelmier and Ellen Sentovich (UC Berkeley) and is currently maintained by Jonathan Kamens (OpenVision Technologies, Inc.). The current version is 7.00, available by anonymous ftp from ftp.x.org in /contrib/applications/xrn. xrn supports many features, including sorting by subject, user-settable key bindings, graceful handling of NNTP server crashes, and many of the features of rn (including KILL files and key bindings similar to rn).

Another X11-based newsreader is xvnews, written by Dan Currie, currently maintained by Hans de Graaff . xvnews is an OPENLook newsreader written primarily for Sun workstations running OpenWindows, but it will run on any X workstation which has the XView libraries. It works with NNTP only, and is compatible with rn style commands. The current version is 2.2.1 and is available from its archive site

     ftp.twi.tudelft.nl in the /pub/news directory.

There are two popular macro packages named "GNUS" and "Gnews" that can be used with the GNU Emacs text editor. These allow reading, replying, and posting interaction with the news from inside the Emacs text editor. Client code exists to get the articles using NNTP rather than from a local disk. Copies can be found on most archive sites that carry the GNU archives eg. ftp.cs.titech.ac.jp, archie.au, archive.eu.net, gatekeeper.dec.com, prep.ai.mit.edu.

"nn" is yet another reader interface, developed by Kim F. Storm of Texas Instruments A/S, Denmark, and released in 1989. nn differs from the traditional readnews and vnews by presenting a menu of article subject and sender-name lines, allowing you to preselect articles to read. nn is also a very fast newsreader, as it keeps a database of article headers on-line. (I.e. it trades space for time. A good rule of thumb is that the nn database size is 5%-10% of your news spool. So up to 110% of your news spool is the amount of space news and the nn database will take.) nn is now maintained by Peter Wemm , and the "offical" ftp location is the /pub/nn directory on uniwa.uwa.edu.au. The current version of nn is 6.4.18. Non-Australian sites should request the sources from their nearest backbone site.

Yet another newsreader is the "tin" reader. It operates with threads, uses NOV-style index files if available, has different article organization methods, and is full-screen oriented. tin works on a local news spool or over an NNTP connection. It has been posted to alt.sources; further information is available from Iain Lea (iain.lea@erlm.siemens.de). The current release of tin is 1.22. Tin is based more on the Notes and tass systems than "rn". There is an extensive list of features, including interfaces to batch modes and auto unpacking mechanisms. The official ftp site for tin is ftp.germany.eu.net, the Unix version can be found in /pub/news/newsreader/unix/tin and the OS/2 version in /pub/news/newsreader/os2/tin

Pine(tm) --a Program for Internet News & Email-- is a tool for reading, sending, and managing electronic messages. It was designed specifically with novice computer users in mind, but can be tailored to accommodate the needs of "power users" as well. Pine uses Internet news and mail message protocols and runs on Unix and PCs. Pine is copyrighted, but freely available. The latest version, including source code, can be found on the Internet host "ftp.cac.washington.edu" in the file "pine/pine.tar.Z" (accessible via anonymous FTP). To try Pine out from the Internet, you may telnet to "demo.cac.washington.edu" and login as "pinedemo". There is also a Pine-specific Internet news group (comp.mail.pine). For further information, send e-mail to pine@cac.washington.edu. Pine was originally based on Elm, but there is little if any Elm code left. Pine is the work of Mike Seibel, Steve Hubert, Mark Crispin, Sheryl Erez, David Miller and Laurence Lundblade* at the University of Washington Office of Computing and Communications. Pine and Pico are trademarks of the University of Washington. (* Laurence is now at Virginia Tech.)

An NNTP newsreader for Macintoshs is available called HyperNews. It is implemented as a HyperCard stack and depends on MacTCP. It is available from many Mac archives, including ftp.apple.com and sumex-aim.stanford.edu.

A newsreader preferred by many Macintosh users is NewsWatcher by by j-norstad@nwu.edu (John Norstad). The current version is 2.0b8 and is available in ftp.acns.nwu.edu:/pub/newswatcher.

Nuntius is another newsreader for the Mac, written by Peter Speck . It can be obtained from frederik.ruc.dk, or the Cornell mirror site ftp.cit.cornell.edu in /pub/mac/comm/test.

There is also an NNTP-based netnews reader for Symbolics Lisp Machines (under Genera 7) available for anonymous FTP from ucbvax.berkeley.edu [128.32.133.1] in pub/nntp-clients/lispm written by Ian Connolly and maintained by Richard Welty . In addition, another NNTP-based news browser is available running under Genera 7 and Genera 8. It provides mouse driven hierarchic browsing of newsgroups and articles, with support for X11 servers on remote machines. It is available for anonymous FTP on flash.bellcore.com [128.96.32.20] in the directory pub/lispm/news-reader/. It is written and maintained by Peter Clitherow

A TOPS-20 reader was developed by Dave Edwards of SRI , but current availability is unknown. An NNTP reader suite for PC's running MS-DOS and having Excelan boards is available for ftp from ames.arc.nasa.gov; get the pcrrn files. There are two MS-DOS news readers that can be obtained from bcm.tmc.edu in the "nfs" directory. They both require PC-NFS (from Sun) to work. They will both work under PC/TCP from FTP Software early this year. Source will be provided at that time.

"trumpet" is a NNTP based news reader for DOS and Windows. There is Lan Workplace version which is also available. It runs over packet drivers, which can work side-by-side with a Novell Network. For information on the Crynwr Packet Driver Collection, send mail to or send a FAX to +1-315-268-9201. Trumpet offers a very intuitive interface with most of the basic facilities required in a newsreader (but without some of the 'bells and whistles found in something like rn). It has facilities for using SMTP to forward/reply etc. The latest version is 1.07 and is shareware available at most main ftp sites.

     trumpet       ftp.trumpet.com.au:/trumpet/
     wintrumpet    ftp.trumpet.com.au:/wintrump/

"WinVN" is a public domain NNTP newsreader for Microsoft Windows and Windows NT. There are versions available for WINSOCK, Novell LWP, and DEC Pathworks/LanMan. It supports the XOVER extension, and can display articles in thread trees. SMTP and MAPI outgoing mail are supported. The latest version is 0.99.2. Sources and binaries are always available from:

ftp.ksc.nasa.gov:[.pub.win3.winvn].
ripem.msu.edu:/pub/pc/win/winvn/titan.ksc.nasa.gov (mirror).

Details on several newsreaders for systems running "Waffle" may be found in the FAQ posted to the comp.bbs.waffle newsgroup on a regular basis. At least 8 different readers are available, and all can be obtained via ftp and mailserver from ftp.halcyon.com (look in /pub/waffle/news).

Details on many other mail and news readers for MSDOS, Windows and OS/2 systems can be found in the FAQ posted to the comp.os.msdos.mail-news.

At least one IBM VM/SP (CMS) version of the Usenet software is available. It is known as PSU NetNews, and it is maintained by Linda Littleton (lrl@psuvm.bitnet/psuvm.psu.edu). Version 2.4 of the software is available from LISTSERV@PSUVM. PSU NetNews supports only 3270 terminals, and uses XEDIT as its screen driver. Most major VM sites appear to use this package. NetNews supports locally-stored news, not NNTP reading.

Since January 1993, a complete NNTP server is available for VM systems. It provides news reading, posting and feed processing compatible to the Unix NNTP implementations. The code is written in IBM's VM Rexx. It assumes that you already have installed PSU VM NETNEWS. The VM NNTP package, written by Kris Van Hees (gutest6@cc1.kuleuven.ac.be) who also takes care of the maintenance. The current version is 1.0.1 and requires IBM's FAL TCP/IP and Arty Ecock's RXSOCKET which is available from the IBMTCP FIELLIST on listserv@pucc. The VM NNTP package can be obtained from the NNTP PACKAGE on listserv@blekul11.

There is NNTP support for PSU NetNews for bulk news receipt: NNTPXFER will poll another site for news, and NNTPRCVR will receive news sent from a Unix NNTPXMIT process. Either program sends the news articles to NetNews for processing. Contact Andy Hooper (hooper@qucdn.bitnet or qucdn.queensu.ca) for more information, or obtain them from listserv@qucdn in PUBLIC FILELIST. These programs are provided with source, and require IBM's FAL TCP/IP and Pascal. An NNTPXMIT sender that works in cooperation with PSU NETNEWS is available from Herman Van Uytven (SYSTHVU@cc1.kuleuven.ac.be).

There is at least one NNTP news-reader for VM using XEDIT as its screen manager: NNR. Contact Paul Campbell (pc@mbunix.mitre.org) for information. The program requires IBM's FAL TCP/IP. The software is available for anonymous ftp from ftp.uni-stuttgart.de in the directory pub/comm/news/beginner/software/nnr/*.

An NNTP news reader is available for TSO/ISPF, called NNMVS. NNMVS is written by Steve Bacher at Draper Laboratory. It requires C/370 V1R2 or SAS/C; ISPF V2; and TCP/IP for MVS (either IBM's "FAL" or SNS). It is now available via anonymous ftp at ftp.uni-stuttgart.de under the directory /pub/comm/news/beginner/software/nnmvs and from ftp.mic.ucla.edu, directory /pub/mvs/netnews. The current version is Version 3 Release 2. There's also an object-code-only distribution for folks without C compilers, but that's an at-your-own-risk distribution, and requires the IBM C/370 run-time library. The source code distribution can be compiled with either C/370 or SAS/C.

Newsfeed management software

Gup, the Group Update Program is a Unix mail-server program that lets a remote site change their newsgroups subscription on their news feed without requiring the intervention of the news administrator at the feed site. Gup operates with the INN (and likely the C News) batching mechanisms. The news administrators at the remote sites simply mail commands to gup to make changes to their own site's subscription list. The mail/interface is password protected. Gup checks the requests for valid newsgroup names, patterns that have no effect and so on. Gup's authors are Mark Delany and Andrew Herbert . Its official ftp location is bushwire.mira.net.au:/pub/external/gup-0.4.tar.gz, but since that's not as well connected as uunet, people are strongly advised to obtain it from a mirror site. eg. ftp.uu.net:/networking/news/misc/gup-0.4.tar.gz

dynafeed is a package from Looking Glass Software Limited that maintains a ..newsrc for every remote site and generates the batches for them. Remote sites can use uucp or run a program to change their .newsrc dynamically. It comes with a program that the remote site can run to monitor readership in newsgroups and dynamically update the feed list to match reader interest. The goal of this is to get a feed that sends only exactly the groups currently being read. dynafeed can be obtained from ftp.clarinet.com as sources/dynafeed.tar.Z.

News processing software

Software also exists to automatically archive Usenet newsgroups. The package rkive, written by Kent Landfield can be configured to archive news automatically based on different headers -- Archive-Name, Volume-Issue, Chronological, Subject and External-Command to name a few. It can be run in batch mode from the command line or from cron. It can also be installed in the sys/newsfeeds file to process articles as they are received. rkive supports local spool directories as well as NNTP based access. rkive is available via ftp from ftp.sterling.com in the directory /rkive.

Newsclip is a programming language for writing news filtering programs, from Looking Glass Software Limited, marketed by ClariNet Communications Corp. It is C-like, and translates to C, so a C compiler is required. It has data-types to represent the kinds of things found in article headers and bodies. It can maintain databases of users, message-ids, patterns, subjects, etc. These can be used to decide whether to ignore or select an article. Newsclip can either operate as a standalone program or as part of rn. It is free for non-commercial use and is available from ftp.clarinet.com as sources/nc.tar.Z. Contact clari-info@clarinet.com with a subject line of "newsclip" for more info.

Special note on "notes" and old versions of news

Many years ago, there was another distributed "news" system called "notes". The "notes" software package used a different internal organization of articles, and a different interchange format than that of the standard Usenet software. It was inspired by the notesfiles available in the PLATO system and was developed independently from the Usenet news. Eventually, the "notes" network and Usenet were joined via gateways doing (sometimes imperfect) protocol translation. "notes" was written in 1980-1981 by Ray Essick and Rob Kolstad, (then) grad students at the University of Illinois at Urbana-Champaign. The first public release of "notes" was at the January 1982 Usenix conference. The last release of notes was version 1.7; it is no longer being actively maintained and the newsgroup for discussing it (news.software.notes) was removed in April 1995.

"B" news software is currently considered obsolete. Unix sites joining the Usenet should install C news or INN to ensure proper behavior and good performance. Most old B news software had compiled-in limits on the number of newsgroups and the number of articles per newsgroup; the increasing volume of news means that B news software cannot reliably cope with a moderately-full newsfeed.

Go to Part 2