View source for How to Use Checkgroups and Control Messages

The purpose of this page is to provide information to News Service Providers about how to configure their news server to automatically create or remove newsgroups, thereby keeping their lists of Big-8 newsgroups in compliance with the Checkgroups message posted monthly in news.announce.newgroups.

__TOC__

== What is a Control Message? ==

* "Control messages are specially-formatted messages that tell news servers to take various actions" ([http://www.eyrie.org/~eagle/software/inn/docs-2.4/install.html#S14|Allbery]).  
* Control messages for the Big-8 hierarchies are PGP-signed and are, for the most part, used to add or subtract newsgroups from [[Checkgroups]].
* From [http://www.eyrie.org/~eagle/faqs/usenet-hier.html#S3 Allbery's Usenet Hierarchy Administration FAQ]:

Most hierarchies now PGP-sign all control messages. The PGP signature is visible in the X-PGP-Sig header of the control message. 

Background information and (now somewhat outdated) instructions are at:

[ftp://ftp.isc.org/pub/pgpcontrol/README.html ftp://ftp.isc.org/pub/pgpcontrol/README.html]

and the exact format of the signature is at:

[ftp://ftp.isc.org/pub/pgpcontrol/FORMAT ftp://ftp.isc.org/pub/pgpcontrol/FORMAT]

== What is Checkgroups? ==

* It is the [[Checkgroups|canonical list]] of groups that constitute the [[Big-8]].  

* "The 'application/news-checkgroups' body part contains a complete list of all the newsgroups in a (sub)hierarchy, their newsgroup-descriptions and their moderation status" ([http://www.landfield.com/usefor/drafts/draft-ietf-usefor-article-13.01.unpaged News Article Format and Transmission draft 13.01]).

* Since 1996, when tale (David C. Lawrence) created the PGP key used by [[news.announce.newgroups]] moderators, there has been a reliable mechanism for identifying the newsgroups that belong in the Big-8 namespace.  The Big-8 [[Technical Team]] updates the list of valid Big-8 groups whenever a group is added or removed, and that list is sent out monthly both as a checkgroups control message and as a posting to [[news.announce.newgroups]].  This trustworthy list is also available from [ftp://ftp.isc.org/pub/usenet/news.announce.newgroups/Group_Lists/ the ISC ftp server.]

== Why should I care about following control messages or Checkgroups? ==

* In brief: because your customers will be best served if you provide them with access to groups that are well-propagated on news servers run by other NSPs.

* One of the beauties of Usenet is that it is sustained by the free cooperation of news administrators throughout the world who agree to carry the same newsgroups as their competitors.  When news administrators consistently follow the Big-8 control messages and/or use Checkgroups to align their server with other news servers, a post made on their own system will be broadcast to other compliant news servers around the world and their server, in turn, will receive the posts that are initiated on other news servers.  It is a win/win situation for all involved, both news administrators and their customers.

* Customers are best served by having a list that is reliable and well-propagated.
** Reliable: groups on the list are alive, not dead; bogus groups are weeded out and missing groups are supplied.
** Well-propagated: carried on news servers outside the orbit of their own NSP.

* Automated compliance will help make sure that posts to moderated groups get inserted into the right stream to be approved and circulated worldwide.  When a moderated group is misconfigured, the posts do not show up on properly configured servers elsewhere in the world.  Customers do not get to participate in the group by having their posts appear where all other participants in the group can see them. 

* News administrators who verify the control messages themselves or who use Checkgroups keep their server in sync with thousands of other news servers around the world, thus providing their customers with the largest possible audience for their Big-8 posts and with the largest number of posts to read.  

* Your fellow news administrators at other NSPs or in non-profit organizations may, in one sense, be competitors for your customers, but in a larger sense are your colleagues.  All of you benefit when there is a reliable method for carrying the same set of Big-8 newsgroups.

== How Do I Verify PGP Signatures on Control Messages? ==

=== General ===

As of 28 June 2021, the control message signing key for the Big 8 hierarchies is [https://www.eyrie.org/~eagle/big-8/key.asc FAFE7B550C18C8B7].  It is available from popular PGP key servers or from https://www.eyrie.org/~eagle/big-8/.

The old control message signing key, which was in use for many years, was [https://www.eyrie.org/~eagle/big-8/old-key.asc C25D3AD3B88DA9C1]. Duplicate control messages are still issued using this key and will be for the indefinite future, but this key is old, weak, could probably be broken with sufficient effort, and cannot be used with current versions of GnuPG. Please use the above key by preference and update your configuration if you have not already. 

A good starting point for verifying control messages is 'pgpcontrol', which comes with INN, or separately at http://archives.eyrie.org/software/pgpcontrol/

For an explanation of PGP, try this document: [http://www.rossde.com/PGP/index.html "Pretty Good Privacy" by David E. Ross]

=== INN ===

As noted, above, software to handle PGP control messages comes with INN.  More information is available as part of the INN documentation - [http://www.eyrie.org/~eagle/software/inn/docs-2.4/install.html#S14 Configuring INN to Verify Control Messages]

A control.ctl entry for the Big 8 groups is as follows:

<pre>
## COMP, HUMANITIES, MISC, NEWS, REC, SCI, SOC, TALK (The Big Eight)
# Contact: board@big-8.org
# URL: https://www.big-8.org/
# Admin group: news.announce.newgroups
# Key fingerprint: 8E18 1BEF 0BE6 3896 2C1A  5C98 FAFE 7B55 0C18 C8B7
# *PGP*   See comment at top of file.
newgroup:*:comp.*|humanities.*|misc.*|news.*|rec.*|sci.*|soc.*|talk.*:drop
rmgroup:*:comp.*|humanities.*|misc.*|news.*|rec.*|sci.*|soc.*|talk.*:drop
checkgroups:group-admin@isc.org:comp.*|humanities.*|misc.*|news.*|rec.*|sci.*|soc.*|talk.*:verify-news.announce.newgroups
newgroup:group-admin@isc.org:comp.*|humanities.*|misc.*|news.*|rec.*|sci.*|soc.*|talk.*:verify-news.announce.newgroups
rmgroup:group-admin@isc.org:comp.*|humanities.*|misc.*|news.*|rec.*|sci.*|soc.*|talk.*:verify-news.announce.newgroups
</pre>

=== Diablo ===

From http://www.openusenet.org/diablo/faq/faq.html#Utilities0 :

  dsyncgroups - Synchronise the active file with a another news server

=== High Winds ===

From http://contrib.highwinds-software.com/support/contributed_details.aspx?contributeID=16 :
  
  control.pl is a perl script that should be run from cron periodically to
  process newgroup and rmgroup control messages.  It uses the same
  control.ctl file format as INN, making for a smooth transition from INN.
  It also supports the pgpverify control message authentication mechanism.

=== Other Software Packages / By Hand ===

* Download pgpverify at ftp://ftp.isc.org/pub/pgpcontrol/
* Configure pgpverify to use gpgv, not to use syslog, and to not look for INN shell variables (all should be fairly obvious modifications to the script, but must be custom to your local setting
* Run 'gpg --recv-keys 0xB88DA9C1' to download the appropriate Big-8 signing key from the public key servers.  (We use subkeys.pgp.net.)
* Run 'pgpverify < msg' on the control messages you want to verify.  ('pgpverify -test < msg' will give you a lot of useful debugging information as well.)

While we cannot offer a simple drop-in solution for custom news servers (they are custom, after all), this should offer a good starting point for writing your own scripts to automatically verify control messages.  Set up your server to pass control messages to this script, and if the control message passes, call another script to modify your group list accordingly.

== How do I automate compliance with control messages? ==

* One of the easiest methods is to sync with the newsgroups list published in the [ftp://ftp.isc.org/pub/usenet/news.announce.newgroups/group_lists/ ISC ftp index].  This list is updated hourly.  News admins can pull down the newsgroups file, extract the hierarchies they care about, and then use that to update their group list.  That's a lot easier than setting up pgp signature validation.
* If administrators are using a major usenet news package like inn, diablo, dnews, etc., then of course the capability and configuration generally comes with the package, but that doesn't help with custom software.

== How do I verify a control message by hand? ==

* Download pgpverify at ftp://ftp.isc.org/pub/pgpcontrol/
* Configure pgpverify to use gpgv, not to use syslog, and to not look for inn shell variables (all should be fairly obvious modifications to the script, but must be custom to your local setting
* Run 'gpg --recv-keys 0xb88da9c1' to download the appropriate big-8 signing key from the public key servers.  (we use subkeys.pgp.net.)
* Run 'pgpverify < msg' on the control messages you want to verify.

== How can I check checkgroups regularly? ==
Doug Mclaren has written a script to show the differences between checkgroups and your nntp master list:
<pre>
# This script is designed to compare the active file on your NNTP
# server vs. the master list kept at ftp.isc.org to see which groups
# should (could) be added, removed or changed.  It does not require
# any special privileges to run beyond access to a NNTP server and to
# the Internet, but of course you will need admin access to the NNTP
# server itself to make any changes.
#
# It will not actually make any changes by itself, but can instead
# give you a list of commands that can be run (if you're running innd)
# to make your server match the master list.  (If you're not running
# innd, send me the equivilent commands for your server and I'll add
# support for it.)
</pre>
* The current version of this script is available from [http://panic.frenzied.us/~dougmc/usenet/check-active-file-vs-isc-list Doug's web site].
* Sample output:
** [http://panic.frenzied.us/~dougmc/usenet/check-active-file-easynews.txt Easynews]
** [http://panic.frenzied.us/~dougmc/usenet/check-active-file-news-server.austin.rr.com.txt austin.rr.com]

Very temporary FAQ maintainer: [mailto:moleski@canisius.edu moleski@canisius.edu]

[[Category:FAQs|Checkgroups]]