Usenet administration administration FAQ
Last-modified: 2018-07-16 Posted-by: postfaq 1.17 (Perl 5.28.1) Archive-name: usenet/hierarchy-admin URL: https://www.eyrie.org/~eagle/faqs/usenet-hier.html Posting-frequency: monthly
This FAQ attempts to provide help to Usenet hierarchy administrators, the people who try to maintain the canonical lists of newsgroups in managed hierarchies. It is aimed at the hierarchy administrators rather than at news admins and tries to summarize the issues to consider in making it easy for news admins to carry the hierarchy.
If you're reading this on Usenet, this FAQ is formatted as a minimal digest, so if your news or mail reader has digest handling capabilities you can use them to navigate between sections. In rn variants, you can use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break each section into a separate article.
Please send any comments, suggestions, or updates to eagle@eyrie.org. Bear in mind when sending me e-mail that I receive upwards of 800 mail messages a day and sometimes have a large backlog of personal e-mail.
This FAQ is posted monthly to news.admin.hierarchies, and is available on the web at <https://www.eyrie.org/~eagle/faqs/usenet-hier.html>.
Subject: Contents
1. Introduction and Terms 2. Basic Hierarchy Administration 3. PGP-Signing Control Messages 4. Maintaining Moderated Groups 5. About the ftp.isc.org Newsgroup Lists 6. Other Resources
Subject: 1. Introduction and Terms
This FAQ assumes a basic familiarity with Usenet (there are other documents that explain the fundamentals better), but there are a few additional concepts that are specifically important for Usenet hierarchy administrators.
A Usenet hierarchy is, reduced to its essence, a set of Usenet newsgroups that share a common naming prefix, such as all groups starting with "comp." or all groups starting with "de.". The names of Usenet newsgroups define a hierarchy of names, with "." used as the separator between the levels of the hierarchy, like host names. Unlike host names, the most significant part of the name is given first. The first component of the name is special and more significant than the rest of the name, since it defines the top-level Usenet hierarchy to which that group belongs.
Generally, every top-level hierarchy is completely independent of the others (although there are a few exceptions where multiple hierarchies share the same management procedures). How the list of newsgroups in that hierarchy should be maintained varies very widely between hierarchies, from the complete anarchy of alt.* to the highly formal system used by comp.*, or simply by fiat of the organization running the hierarchy as with microsoft.*. Which maintenance methods you should use for your hierarchy is out of scope for this FAQ; this document is about how to publish the results of those methods, assuming that you want to have a single canonical list of newsgroups that everyone carrying that hierarchy can agree on. If you don't want that (if your hierarchy is like alt.*, for example), most of this document will not apply.
Usenet newsgroups are created and removed via specially formatted messages called control messages that tell news servers to do something. A hierarchy should have a hierarchy administrator who is responsible for following whatever procedures were agreed on for changing the list of groups in that hierarchy and then publishing the results using control messages. The structure of control messages is explained in the Usenet news standards and in many FAQs and web pages, so it won't be explained here. (Some sites don't use control messages for various reasons, and it's therefore best to also publish the results via other methods, as will be explained.)
Besides newgroup (create a newsgroup) and rmgroup (remove a newsgroup) control messages, there is also a control message called a checkgroups which provides a complete list of newsgroups in a hierarchy complete with short descriptions. A "checkgroups" is therefore also used to mean a complete list of newsgroups in a hierarchy. The checkgroups format is a list of groups, one per line, with the name of the group, a tab, and a short description. If the newsgroup is moderated, the description must end in the literal text " (Moderated)". (Sadly, you can't translate the word to another language. It's an ugly wart on the protocol.)
Originally, control messages were authenticated only by the (easily forged) address of the sender of the message, which worked when Usenet was small but broke down badly as it got larger. As a result, most hierarchies now sign their control messages using PGP, an open standard for public key cryptography, allowing receiving sites to verify that the control message was issued by the person it claims to be from.
Subject: 2. Basic Hierarchy Administration
A hierarchy administrator has three separate audiences who should be kept in mind when publishing information about a Usenet hierarchy: users who may be interested in reading or posting to the hierarchy, news administrators who are not currently carrying the hierarchy on their servers and want to, and news administrators who are already carrying the hierarchy and want to keep current with changes to it.
The most important audience is probably the users, but that's also the audience that's the hardest to make general statements about, since how you communicate with potential users varies quite a bit by hierarchy. The users will primarily be interested in a description of what the hierarchy is for, a list of groups in the hierarchy, any hierarchy-wide policies, the charters of the newsgroups, and the newsgroup creation procedure. They'll generally speak the same language as the hierarchy (since if they don't, they probably won't be interested in reading it).
News administrators who aren't currently carrying the hierarchy likely won't be as interested in detail like particuliar policies or group charters, but will also be interested in the overall purpose for the hierarchy and any policies that specifically affect sites carrying the groups. They will specifically need the list of newsgroups in the hierarchy in checkgroups format, however, and preferrably as a plain-text file that they can download and feed into their news software. They'll also need instructions for processing control messages to pick up changes to the hierarchy, generally as an INN control.ctl fragment. Finally, news administrators may not speak the language of your hierarchy (they may be running a news server for a large international ISP, for example), so you may want to provide instructions specifically for news administrators in English if the language of your hierarchy isn't English.
News administrators who are already carrying the hierarchy are mostly interested in being notified of changes to it (usually via control messages). They also want ways of checking their group list against the current one so that they can get back into sync if they've missed some updates.
So, given that, here are a few specific recommendations:
* Have a web site. Try to make sure that the URL for your hierarchy web site is stable and doesn't change, since the URL makes it into various FAQs and configuration files that live for years. Put all of the hierarchy information on the web site, and make sure that the web site stays up to date. <http://www.news-admin.org/> may be available as a hosting site for your web site.
* Every time you create a group, remove a group, rename a group (which can be done with a creation and removal), or change the moderation status of a group, send the appropriate control message. By looking for it in <ftp://ftp.isc.org/pub/usenet/control/> under your hierarchy, you can check that the control message has propagated. Control messages should have, in the Newsgroups header, the group being created or removed (even for newgroup messages; they'll propagate correctly even though the group hasn't been created yet because newgroup messages have special propagation rules). The exception is checkgroups messages, which should normally be posted to the administrative group of your hierarchy and possibly crossposted to news.admin.hierarchies. Don't put a Distribution header in your control messages unless you really intend to limit the availability of the groups to sites configured to accept that distribution (which mostly doesn't work anyway).
* Send a checkgroups control message periodically. If you do this, you don't really need to send duplicate control messages for changes; if someone misses a change, they'll catch it with the next checkgroups message. In the checkgroups control message (and ideally in your other control messages as well), include an X-* header pointing to your hierarchy web site so that people can get more information about the hierarchy. Do not put a Supersedes header on your checkgroups messages; some sites will filter out any message that contains both a Control header and a Supersedes header.
* Put the *current* checkgroups on your web site as a plain-text file (ending the file name with .txt will give this hint to most web servers) so that it can be easily downloaded by news administrators. Make sure that this file is kept current. Having a separate pretty HTMLified list of groups is often useful for users, but please be sure to provide the plain text checkgroups message as well since the HTML version is nearly useless for news administrators trying to add your hierarchy.
* Write a guide for news administrators on how to carry the hierarchy, including an overall description. Make sure that it clearly indicates where to find the checkgroups for the hierarchy. Put this guide on your web site, and it doesn't hurt to put a URL for it in the headers of your control messages as well. If your hierarchy uses a language other than English, write this guide both in that language and in English, since English seems to be the de facto international language for news administration right now. (At the least, try to use the literal word "checkgroups" on the web site on the top page for a link to the current checkgroups; that's the main thing that people will be looking for, and that way news admins can find it even if they can't read the rest of the site.)
* Write a control.ctl entry for your hierarchy and publish it on your web site in the section for news admins and in the guide. For a hierarchy that doesn't PGP-sign control messages, this will look like:
## EXAMPLE (Example Hierarchy, Olympus Mons)
# Contact: news@example.org # URL: http://www.usenet.example.org/
checkgroups:news@example.com:example.*:doit
newgroup:news@example.com:example.*:doit
rmgroup:news@example.com:example.*:doit
The first line contains your hierarchy name in all caps and then in parens a description of your hierarchy, including the applicable location if the hierarchy is regional. Then comes information about who to contact about the hierarchy and the URL for your web site. Finally, the uncommented lines specify the type of control message, the sender (which should match the From and Sender headers of your control message), the pattern matching the groups in your hierarchy, and the word "doit" saying to act on those control messages.
News administrators can then just cut and paste this entry into their news server configuration files and start honoring your control messages. If you're using PGP, see the section on that below.
* Use PGP to sign all your control messages. This isn't necessary for small hierarchies, but if there end up being any fights over your hierarchy or it draws hostile attention for some reason, people can easily cause a lot of confusion if your control messages aren't signed. There's more information later about this.
* Make sure the information on your web site stays up to date! In particular, be *sure* that the checkgroups on the web site is accurate, since that's the list most people will start with. It's also sometimes more convenient for someone to process a checkgroups from a web site than process a control message (one can use tools like actsync to do that, for example), so if it's not accurate, a news administrator may undo all your control messages by getting information from the web site instead.
* Check <http://usenet.trigofacile.com/hierarchies/> for your hierarchy and make sure the information is accurate. The information here is based on the information kept at <ftp://ftp.isc.org/pub/usenet/CONFIG/> and <ftp://ftp.isc.org/pub/usenet/control/> and provides a quick way to check on recent changes to your hierarchy and ensure that nothing is out of date.
Some hierarchies also have a publically available news server that carries that hierarchy (this is particularly common for hierarchies intended for support of the products of a particular company). If you do, make sure to point both users and news administrators at it. Users can use that server if their local server doesn't carry your hierarchy, and news administrators can use that server to get an up-to-date list of newsgroups in the hierarchy using tools like actsync. If you do have such a server, add the line:
# Syncable server: news.example.org
pointing to that server to your control.ctl entry.
Be careful about running open servers like this, as they have frequently been abused to damage the rest of Usenet. At the least, it's best to not allow users to crosspost to groups not carried by the server or add Supersedes or Control headers, and ideally you should have some sort of spam filtering or posting rate limits in place.
Subject: 3. PGP-Signing Control Messages
Most hierarchies now PGP-sign all control messages. The PGP signature is visible in the X-PGP-Sig header of the control message. Background information and (now somewhat outdated) instructions are at:
<ftp://ftp.isc.org/pub/pgpcontrol/README.html>
and the exact format of the signature is at:
<ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>
You don't need to understand the information in the last link, though, at least if you're using Unix and can use the existing tools for generating signed control messages.
In order to sign control messages, you'll need three things: A PGP key pair that you'll use for signing and verifying the messages, a PGP implementation you can use, and some software to generate the right type of signatures for Usenet control messages.
If you're not already familiar with public key cryptography, here's a brief primer: A PGP key pair consists of two keys, a public one and a private one. You sign messages with the private key, which you have to keep secret and protect since anyone with possession of it can pretend to be the hierarchy administrator. You give out the public key to anyone and everyone, and anyone with the public key can check a signature and verify that it was signed by the private key (without being able to sign messages themselves).
Most Usenet news sites that honor control messages are set up to verify messages signed with an algorithm called RSA, which was the algorithm used by the original PGP implementation. Unfortunately, this is now fairly obsolete. Current PGP implementations use a newer, more secure algorithm for generating signatures (although the additional security is probably overkill for Usenet control messages, at least for right now). While this doesn't pose a problem for signing messages (current PGP implementations can still use old RSA keys to sign things), it does cause problems if you're starting fresh, since the keys generated by current implementations will not work with old versions of PGP.
What all this means is that you have a few hard choices when it comes to choosing a PGP implementation and generating your initial key pair. You can use GnuPG <http://www.gnupg.org/>, which is probably the best available PGP implementation, and not bother with a RSA key at all. This will mean, unfortunately, that only sites that are also using GnuPG or another current PGP implementation will be able to verify your control messages. Or you can go to <http://www.pgpi.org/>, download an older version of PGP (something of the 2.6 vintage), and use it to sign your control messages, which will work with all versions of PGP but may be more of a pain. (You can also use an old version of PGP only to generate the initial key, and then import it into GnuPG and use GnuPG to sign control messages, but this is complex and not recommended for people who have never touched PGP before. There are some instructions on the GnuPG web site, though.)
Whatever choice you make, follow the documentation of your PGP implementation to generate a key pair. Pay careful attention to the user ID that you put on your key. In order to work with Usenet, that user ID must not contain any spaces. The two most common key IDs used for signing Usenet control messages are the name of the administrative group in the hierarchy (like example.config) or the sender of the control message (like news@example.org). The latter is better practice and is recommended for new hierarchies, although make sure that e-mail address is stable and is one that you will be able to use for decades to come.
If you're using GnuPG, in order to not get any spaces into the user ID, you need to use gpg --gen-key --allow-freeform-uid, enter the desired user ID as the name, and then press Enter when asked for an e-mail address or comment. The recommended user ID is the e-mail address of the sender, but it has to be entered as the name or GnuPG will not generate the right user ID. A later version of pgpverify will hopefully make this unnecessary, but older versions will be around for quite some time.
Resist the temptation to put any additional user IDs on your key. Your key should only have one user ID, the one that will be used in control.ctl entries. If it has any additional user IDs, this can confuse the pgpverify process with some PGP implementations and cause your control messages to be ignored.
If you're using a modern PGP implementation, it will automatically sign the public key (this is called a self-signature). If you're using an older PGP implementation, make sure that you do this, following the instructions for your software. Keys that aren't self-signed can be tampered with in various ways, and modern PGP implementations will refuse to import or honor them.
Now, you're ready to create and sign a control message. I'm aware of several major implementations of the glue software to do the signing:
signcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
This is the original implementation by David Lawrence of the signed control message protocol. The signcontrol script is a Perl script that works with pgp. It will require some editing to set information about your hierarchy. There is also a shell implementation in the same directory.
signcontrol <http://www.trigofacile.com/divers/usenet/clefs/signcontrol.htm>
A different implementation in Python that may be somewhat easier to use. It similarly will require some editing to add information for your particular hierarchy and requires that GnuPG be available.
News::Article <http://search.cpan.org/~agierth/>
A Perl module, and therefore mostly useful if you're writing your own software for your hierarchy in Perl. It makes fewer checks than signcontrol and in general does less (just the signing) which can be more convenient. It requires the PGP::Sign module from CPAN; see your Perl documentation for how to install it.
cmsg-tools <http://www.linux.it/~md/software/cmsg-tools.tgz>
A Perl script which uses News::Article and GnuPG (but not PGP::Sign) to automatically generate control messages. It can be used to manage multiple hierarchies.
Pointers to additional implementations, particularly instructions for Windows users, are very welcome.
Once you have a signed control message, verify it to make sure that it verifies properly. You can do that with the pgpverify program that comes with INN or is available in the same directory as signcontrol above.
If you got all of that working, you should put the PGP public key into your guide for news administrators. To see examples of PGP public keys for various hierarchies, see:
<ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>
You should also make the public key available for download on your web site, preferrably as a plain text file (ending the file name in .txt will give this hint to most web servers) that can be easily downloaded. It's best to also submit it to the keyservers (see <http://pgp.mit.edu/>). Also change your control.ctl entry that you have on your web site to look like:
## EXAMPLE (Example Hierarchy, Olympus Mons) # Contact: news@example.org # URL: http://www.usenet.example.org/ # Key URL: http://www.usenet.example.org/pgpkey.txt # Key fingerprint = G7 11 96 E8 34 32 7E 78 01 0D 69 99 A3 8F 34 B8 # *PGP* See comment at top of file. checkgroups:news@example.com:example.*:verify-news@example.org newgroup:news@example.com:example.*:verify-news@example.org rmgroup:news@example.com:example.*:verify-news@example.org
The key URL field is the URL of the plain-text PGP public key available from your web site. The fingerprint is the output of gpg --fingerprint or pgp -kvc (depending on what version of PGP that you're using) and is used as a check to be sure that the key downloaded is the right one. The *PGP* comment makes sense in the context of the standard control.ctl file, which has information about PGP-signed control messages at the top. And note that the "doit" string to act on the control messages has been replaced by "verify-" followed by the user ID of your PGP key (whatever that may be).
Now, just use your new signing procedure whenever you send a control message for your hierarchy. Oh, and if you're changing to signing control messages from not signing them, be sure to announce that in news.admin.hierarchies.
Subject: 4. Maintaining Moderated Groups
If there are moderated newsgroups in your hierarchy, this involves a bit of additional hassle over unmoderated groups. Rather than just checking whether the newsgroup is there are not, news administrators also have to make sure that the moderation status is set correctly so that users can post, and news servers need to know to which e-mail address to send posts so that they'll reach the moderator.
When managing moderated groups, be sure to always include the correct moderation flag in your newgroup control messages, and make sure that your checkgroups lines for moderated groups end in " (Moderated)". (Yes, literally. It can't be translated; news software actually looks for this.) It's sometimes useful to send a few duplicate control messages if you ever convert an unmoderated group to a moderated group or vice versa, since it's harder to get this change to take than a simple newsgroup creation. Expect to have a few frustrated users who's news providers just cannot get this right. You may want to put together a form message to send to those providers explaining who you are and that they have groups in your hierarchy configured incorrectly.
Finally, you have to deal with getting the posts to the moderator. There are two main ways of handling this, which can be combined. One is to keep the central moderators.isc.org relay systems up to date, and the other is to set up your own relay system that accepts mail addressed to the newsgroup name with all periods replaced by hyphens @ some system you control and then forwards that mail along to the real moderator.
Nearly all news servers that try to handle moderated groups default to forwarding any message posted to a moderated group to the name of the group with periods replaced by hyphens @moderators.isc.org. The relay systems behind that DNS record then forward those messages along to the real moderator. To have one of those addresses created or to change where it forwards to, mail moderators-request@isc.org. These changes have to be made manually and there's often a backlog; to save time, please identify yourself as the hierarchy administrator for the hierarchy and preferrably send the mail from your hierarchy contact address (and you may also want to PGP-sign the message if you can do that easily). Even still, be prepared for this to sometimes take a while.
It's possible to configure news servers to forward mail somewhere other than to moderators.isc.org, and easiest to do that as wildcard entries for hierarchies that forward all posts to moderated groups in that hierarchy to a particular host instead of moderators.isc.org (again with the period to hyphen change). Some hierarchies do this instead so that they don't have to wait for moderators.isc.org changes, but it makes things harder for news administrators who have to keep track of those special cases. If you do do this, you have to tell all news administrators to add a line like:
example.*:%s@example.org
to their moderators file, where the first part is the pattern that matches groups in your hierarchy and the second part is the e-mail address to send postings to (%s will be replaced by the group name with periods changed to hyphens). If that forwarding host ever goes away, or if you ever have to change this, it will be a major hassle and it will take a very long time to catch all the stragglers.
What many hierarchy administrators do is combine these two approaches. They set up their own forwarding system that they have direct control over, so that they can change the moderators for an existing group without having to contact moderators-request@isc.org, and they ask the moderators.isc.org sites to just forward to the appropriate address at that host. Then, the only things they have to let moderators-request know about are new moderated groups or removal of moderated groups. This does mean that posts to moderated groups go through two hops before reaching the moderator instead of just one, though, which can make problems slightly more difficult to diagnose.
Unfortunately, due to the way that they're distributed and used, the alias lists for moderators.isc.org cannot contain wildcard entries, so you cannot ask them to just forward all posts to any moderated example.* group to example.org. Each group still has to be configured separately and each new moderated group still has to be sent to moderators-request@isc.org for manual processing. Hopefully in the future there will be some more easily automated system for handling this.
Subject: 5. About the ftp.isc.org Newsgroup Lists
Maintained at:
<ftp://ftp.isc.org/pub/usenet/CONFIG/>
is a copy of a control.ctl file with as many public hierarchies as possible with active maintainers listed in it, along with lists of newsgroups in active file and checkgroups format maintained by processing control messages using those control.ctl rules. Also maintained from the same information is a collection of hierarchy public keys for news administrators to get from one source, in:
<ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>
A unified view of this information, including logs of recent changes, is at:
<http://usenet.trigofacile.com/hierarchies/>
These can be valuable resources for you as a hierarchy administrator in a couple of different ways.
First, some users and some news sites use these lists of newsgroups to determine what newsgroups to carry, either using the list verbatim, checking the list to see if a newsgroup is listed before being willing to add it, or synchronizing particular hierarchies against that list. Having your hierarchy there may therefore increase the number of sites that carry your hierarchy and may make it easier for interested users to get the groups added at their local site.
Second, particularly if you're using PGP-signed control messages, this list can serve as a check that your control messages are working properly. The list updates once an hour, so if you send a control message and in a few hours the result isn't reflected in the ftp.isc.org lists, something may have gone wrong (maybe you forgot an Approved header, or the PGP signature wasn't valid for some reason).
Finally, many news administrators, even if they don't use the newsgroup lists, use the control.ctl file and PGPKEYS file to configure their own news servers. This control.ctl file is also included in INN releases.
To get your hierarchy listed in these files, send mail to usenet-config@isc.org. Please include your control.ctl entry, as described above, along with your PGP public key or a URL from which it can be obtained (preferrably the latter). Also provide a pointer to a current list of newsgroups in checkgroups format so that they can be added to the list at the same time. usenet-config@isc.org sometimes has a bit of a backlog, so please allow a couple of weeks for a response.
Subject: 6. Other Resources
Here are some other resources to be aware of if you're maintaining a hierarchy:
news.admin.hierarchies
Most discussion of new hierarchies and of hierarchy administration in general takes place here. There isn't a lot of discussion normally, just hierarchy FAQs, but a lot of people read the group and can provide help and suggestions if you post.
news.software.nntp
The technical discussion group for news software and the news protocols, the readers of this group may be able to help if you have questions about news server configuration or about the details of the PGP signature system for control messages.
Control message archive <ftp://ftp.isc.org/pub/usenet/control/>
An archive of all newgroup and rmgroup control messages, used by some sites to check to see whether a requested group had a valid control message and useful as a check to be sure that your control messages are getting out.
Newsgroup list archive <ftp://ftp.isc.org/pub/usenet/CONFIG/>
A collected control.ctl file (which is also included in INN) and lists of newsgroups maintained by those rules. Also includes a file describing the checkgroups format, providing some guidelines for newsgroup naming, and describing how to write newsgroup descriptions.
Usenet hierarchy information <http://usenet.trigofacile.com/hierarchies/>
A merged view of the information maintained at ftp.isc.org and the easiest way to view the information together and check it for accuracy.
pgpcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
Mentioned many times above, this is the central site for the reference implementations of the PGP control message signing protocol. It also has a list of public keys of hierarchy administrators and information about how to enable PGP verification of control messages.
news-admin.org <http://www.news-admin.org/>
A hosting service for hierarchy administrators and Usenet hierarchies that you may want to consider using, as well as a good collection of examples of web sites for various hierarchies.
