ListProc 8.0 Release Notes
Contents- New Features
- Multi-Threading
- Performance Enhancements
- Accounting Capabilities
- Improved Loop Detection
- The header Config File Directive
- General Changes
- Bug Fixes
ListProc 8.0 is fully multi-threaded, which allows it to process mail for several lists simultaneously.
- The "threads" config file directive
- A config file directive "threads" controls the number of
threads ListProc will create. Usage of the "threads" directive is
as follows:
threads numberwhere number is the number of threads to use. Valid numbers of threads are between 1 and 256, inclusive. A value of 1 results in the previous, unthreaded behavior.If no value of "threads" is specified, ListProc defaults to 1 thread.
- Synchronized access to data files
- Accesses to aliases and ignored files have been synchronized for
multiple threads. Synchronization is done via the system's
semaphore. In previous versions, this was only necessary when ILP
was enabled. The use of multiple threads in 8.0 requires the use of
semaphores at all times.
- Error mail processing
- As long as the config file defines more than one thread, ListProc
will handle error mail for all lists asynchronously with a single
thread, rather than dealing with it when processing the list mail.
This has the following implications:
- There is at most one thread processing error mail at any one time, regardless of the number of threads defined, and the number of threads still free.
- It is possible for ListProc to concurrently process list mail and error mail for a single list. In this case, the error mail is given lower priority.
- This will significantly speed up the processing of each list, since each list's thread will only need to handle regular list mail and digests.
- ListProc's flow of control
- Simultaneous processing is performed on as many lists as there are
threads defined in the config file. Hence, if your config file
contained "threads 10", ListProc would be able to concurrently
process mail from 10 different lists.
The spawning of threads is still subject to delays specified by the "frequency" directive in the config file. For example, suppose your config file contained the line "frequency 0 30". In ListProc 7.X, the flow of list processing would look like this:
process list 1 (regular mail, digests, error mail) process server mail process list 2 (regular mail, digests, error mail) process server mail ... process final list (regular mail, digests, error mail) process server mail wait 30 seconds repeatListProc 8.0 would instead do the following:spawn list 1 thread (regular mail, digests) spawn error thread if error mail exists and no error thread is already running. spawn server mail thread unless one already exists spawn list 2 thread (regular mail, digests) spawn error thread if error mail exists and no error thread is already running. spawn server mail thread unless one already exists ... ... spawn final list thread (regular mail, digests) spawn error thread if error mail exists and no error thread is already running. spawn server mail thread unless one already exists wait 30 seconds repeatNotice that in 8.0 ListProc does not need to wait for the processing of one list to finish before it goes spawns a thread to process the next list.Within each list, processing is still serialized: regular mail and digests are never handled concurrently for a given list. As noted above, the error processing has been taken out of the list processing loop. Error mail is processed by the first available thread after error mail arrives.
- Replacement of system() calls
- The multi-threaded code required the use of fork() and exec()
rather than system(). This also allows us to circumvent the
problems with the shell that's spawned by system(). Consequently,
this allows us to lift the restriction of no apostrophes in a user
address. (see the GENERAL CHANGES
section below) All binaries have
been converted to make use of the new sysexec() routine that
performs the fork(), exec() and wait() functions. All of these
required significant rewriting of signal handling routines in
serverd.c.
- Updated semset utility
- The semset utility included in the distribution has been updated to
show how many lists are delivering mail at the time of invocation.
Invoked with no arguments, semset returns basic information about
ListProc's interprocess communication (the output has been wrapped
to fit an 80 column display):
%semset IPC status fromWhen semset is called with the semaphore ID as an argument, it returns specific information about that semaphore's current state. The following flag information can be displayed:as of Thu Feb 8 12:22:48 1996 Message Queue facility not in system. Shared Memory facility not in system. T ID KEY MODE OWNER GROUP CREATOR Semaphores: s 131072 0x0x00000000 --ra------- server staff root CGROUP NSEMS OTIME CTIME staff 1 12:20:26 13:49:31 SEM_REQ_ID Requesting a tag ID, request #, etc. SEM_SYSFILES Accessing/Processing global server files SEM_LISTFILES Accessing/Processing list specific files SEM_ARCHIVES Accessing/Processing a list archive SEM_DLVR_MAIL(N) Delivering mail (N = number of lists currently delivering)The sample output below shows ListProc's processing on our server over the course of about 30 seconds:% semset 131072 Current value: 0x10; flags on: SEM_DLVR_MAIL(1x) % semset 131072 Current value: 0x10; flags on: SEM_DLVR_MAIL(1x) % semset 131072 Current value: 0x10; flags on: SEM_DLVR_MAIL(1x) % semset 131072 Current value: 0x14; flags on: SEM_LISTFILES SEM_DLVR_MAIL(1x) % semset 131072 Current value: 0x20; flags on: SEM_DLVR_MAIL(2x) % semset 131072 Current value: 0x20; flags on: SEM_DLVR_MAIL(2x) % semset 131072 Current value: 0x20; flags on: SEM_DLVR_MAIL(2x) % semset 131072 Current value: 0x24; flags on: SEM_LISTFILES SEM_DLVR_MAIL(2x) % semset 131072 Current value: 0x10; flags on: SEM_DLVR_MAIL(1x) % semset 131072 Current value: 0x10; flags on: SEM_DLVR_MAIL(1x) % semset 131072 Current value: 0x10; flags on: SEM_DLVR_MAIL(1x) - Multi-threading caveats and cautions
- Multi-threading can potentially create a much larger drain on your
system's resources. If the system-wide process table is not large
enough, you may run out of available processes. Furthermore, each
thread requires its own memory and makes separate demands on
processor and disk resources.
Perhaps most significantly, each thread opens its own sendmail processes. (The number of sendmail processes for each list depends on the number specified by the "multiple_recipients" directive in the config file. For example, if "multiple_recipients" is set to M, a list with S subscribers will spawn S/M sendmail processes.) Increasing the number of lists processing at one time will cause a proportionally larger number of sendmail processes to run.
An additional caveat with multi-threaded processing is that serverd must wait for all outstanding threads to quit before it can re-initialize. (This is essentially also what happens with 7.x - it is just that with only one thread running, the wait time will probably not be very long.) Once the server receives a notification that it must re-initialize, no new threads are spawned while serverd waits for the existing threads to complete operation. Once all threads have finished, serverd can rebuild its internal data structures.
- The LISTS command is slow because it must check each hidden list to see if the requester is either an owner or a subscriber. The new routine is 50% faster than the previous version.
- Checking whether a user is an owner is now done in memory; this speeds up the LISTS request as well as all list-specific requests.
- The speed of subscriber lookups has been increased between 30 and 100 times, depending on the request. Some of the changes that allow this increase required the removal of comment support for the subscribers file.
- Additional speed up of QUERY requests by eliminating system calls.
- The SET request has been sped up by a factor of 25. The user's entry is overwritten with blanks and a new entry is placed at the end of the file. Blank entries are cleaned up once a day, when the list is sorted.
- Sped up the error analysis routine by overwriting subscriber entries to be removed with blanks.
- Sped up look for alternate addresses when a user is not subscribed. The search time was cut in half.
- Small performance enhancement in list.c by keeping track of internal unprocessed information (messages, and file offsets).
- Sped up the following requests while in interactive, casual user mode: LISTS, GET, INDEX, SEARCH, REVIEW, STATS. Since casual users do not provide an email address, they are never authorized to recieve information about private lists. Thus, ListProc no longer checks the subscribers file of hidden lists to see if casual users are subscribed.
- Sped up SET ... PREF requests by avoiding subscriber look up. Since this command is only valid for list owners, the subscriber file look up is unnecessary.
The accounting program is passed the following information:
- the list alias
- the value of LPDIR
- the total number of regular mail recipients (including peers)
- the total number of digest recipients
- the total number of newsgroup postings
- the filename that contains the headers of the message distributed
- the filename that contains the body of the message distributed
- Loop detection has been enhanced to deal with an additional type of loop. Consider the following scenario: An error message comes in for a user, and a notification that it was ignored is sent to the list owner. If the message to the list owner bounces, ListProc will receive a bounced mail message. This will create another error message, which is again sent to the owner. This problem was not caught by the present loop detection system, since the returned error messages kept growing. To solve this problem, ListProc now looks for Message-ID: and X-Listprocessor-Version: header lines in the body *if* the message is sent to the errors folder. When a message id is found that has already been processed, the message is flushed. Since the problem could well be the owner's address, the owner is NOT notified, even if CCIGNORE is set. If X-Listprocessor-Version: is found in the body, the owner WILL be notified if CCIGNORE is set, unless a message id is found that has already been processed.
- ListProc 8.0 no longer performs loop-detection on the Reply-To: header of messages. These checks were unnecessary to catch mail loops, and they inhibited legitmate use of the "Reply-To:" to force replies to individual messages to be sent to a specified list or lists.
header * {
Headerline:
!Headerline:
...
}
By specifying a * all lists and the server are affected. Header
lines to be propagated are listed as before, and header lines not
to be propagated should start with an exclamation point ("!").
Headers propagated (or not propagated) to individual lists can be
controlled by using a list name instead of the * as follows:
header foo-list {
Headerline:
!Headerline:
...
}
Notice that "header lines" are actually regular expressions as
before, so, for example, the following is valid:
header * {
!X-.+-expire:
}
The following header lines will be preserved regardless of the
header directive:
From: Reply-To:
Resent-From: Date:
Sender: Control:
Resent-Sender: Approved:
Message-Id: Archive-Name:
Resent-Message-Id: all MIME headers
Example: here is how to avoid propagating header
lines that cause loops and error messages with auto-responders:
header * {
!Registered-Mail-Reply-Requested-By:
!X-Confirm-Reading-To:
}
If list-specific "header" directives are also defined, these are
appended to the list inherited by "header *".
- The system manager is now required to use the system password to
issue the following requests:
EDIT, PUT, HOLD, FREE, LOCK, UNLOCK, CONFIGURATION
Moreover, ARCHIVE operations CREATE, MODIFY, REMOVE will work with the manager password only.Without these restrictions, it is possible for a list owner to fake email from the system manager, and use the list password and get around restrictions posed on owners for the above requests.
- Review now shows REFLECTOR setting.
- CONFIGURE is a new alias for CONFIGURATION.
- ListProc-generated Subject: lines on outgoing server messages have
been limited to 132 characters in length.
- The ILP time-out has been changed from an absolute time-out to an
idle time-out. The idle time-out is specified with the -i flag to
serverd. For example "serverd -i 180" enables interactive sessions
with an idle time-out of 3 minutes. The implementation is as
follows:
- A fixed 60 second login time-out is enforced.
- Upon login, the specified idle time-out is enforced.
- Every request (including just hitting return) except TIMELEFT resets the clock.
- When a connection is older than 24 hours it will be preempted if all ports are busy and a new connection request comes in. (Note that this time period is presently hard coded.)
- Catmail exits with code 75 when files cannot be opened or
locked. This forces the MTA (e.g. sendmail) process on the sender's
side to queue the message, and try again later. The site manager is
sent a notification message when this occurs.
- Subscription managers (as well as list owners) can now EDIT and PUT
subscribers and aliases files.
- Digests are now distributed in MIME format. MIME headers for
individual messages are also preserved. In previous versions MIME
encoded messages could not be read in digest form. These changes
solve that problem.
- The "queued" utility has been modified to make it easy to kill.
The "start" program has also been changed so that it kills "queued"
when appropriate.
- Approval requests to moderators are now sent in MIME format. With
previous versions, moderators were forced to pipe MIME messages to
munpack or similar utilities in order to read them, with mixed
success. The new behavior is as follows:
- If the list is set to MODERATED-NO-EDIT, a MIME encoded approval request is sent even if the message to be considered is not MIME encoded. This is because catmail does not scan the contents of the message.
- If the list is set to MODERATED-EDIT, a MIME approval-request is sent only when the message to be considered is MIME encoded. If the message is not MIME encoded, regular email is sent.
In all cases, approved MIME messages are preserved correctly with the correct MIME headers and boundaries. The only exception is with MODERATED-EDIT when the moderator indents the text with "> " or some other string. Since this breaks the MIME encoding, the moderator should NEVER indent segments of MIME encoded messages.
- Added CCSEARCH as a valid manager preference.
- Fixed various typos in the help files.
- Each moderated-no-edit list now has its own tag counter. In
pre-8.0 versions of ListProc, this counter was system-wide. Also,
the Subject: header line now shows the tag id when sending request
for approval of a message.
- Solaris exhibited a problem with ilp. Because BSD signal handling
is used and Solaris is SVR4, signals were sent too fast for the
process to catch them again, which resulted in ilp being killed
(default behavior for SIGIO, SIGURG, and SIGPOLL). Therefore,
signal handling in ilp.c, serverd.c and silp.c has been modified so
that on SVR3 and SVR4 systems true System V signal handling is
used.
- Alias lookups have been modified. In previous versions, aliases
were interpreted just as pairs of strings, separated by whitespace.
This caused strange problems if one line had one too many or one
too few fields, as all the records after this would be shifted over
by one.
We now read pairs of expressions per line and report extra garbage. The "dbglpfiles" utility has been modified to do the same as well.
- Apostrophes are now allowed in user addresses. The multi-threading
code required getting rid of the use of the system() command. As a
side effect, ListProc is no longer reliant on the shell's
interpretation of command line parameters. Hence, the security
issues involved in allowing apostrophes in user names have
disappeared.
- A number of changes have been made to the internal handling of the
subscribers, aliases, ignored, and owners files are now handled
slightly differently, in order to improve efficiency.
- The subscribers file is now sorted at most once a day, at least 24 hours after the last sort.
- ListProc now handles unsubscriptions differently. The entry to be removed from the subscribers file is simply overwritten with blanks. (It will be removed at the next sorting.) The code for SUBSCRIBE, UNSUBSCRIBE, REVIEW, STATS, SET, and list delivery, and error analysis were changed to make this work.
- The SET request now uses a similar mechanism. The old user entry is overwritten with blanks and a new entry is placed at the end of the file.
- SUBSCRIBE and SET requests will reuse blank entries if the new entry fits in one of them.
- Removal of relevant aliases when a user is unsubscribed is now done using the same "write with blanks" technique which speeds up things a little more. The drawback is that aliases files will now contain blanks in them. Unlike subscribers files, they are not resorted so the blanks will remain until someone removes them by hand.
- SET ... PREF for owners has also been sped up by overwriting entries with blanks and reusing blank entries.
- Removals and additions of owners have been sped up as well by overwriting old entries with blanks, and by using blank entries when there is space for the new entry. Note that the owners file is NOT resorted, so this procedure will create an accumulation of blank lines over time.
- The following changes have been made regarding aliases:
- UNSUBSCRIBE, SET ... ADDRESS and auto-deletion no longer remove aliases from $LPDIR/.aliases
- SET ... ADDRESS no longer writes the alias to $LPDIR/.aliases. It continues to append aliases to the list's .aliases file.
Hence aliases work as follows:
- A SET ... ADDRESS request will write to the list's .aliases file only (new).
- An ALIAS request will write to the list's aliases file if issued by the owner, and to $LPDIR/.aliases if issued by the manager (unchanged).
- UNSUBSCRIBE, SET ... ADDRESS and auto-deletion will remove relevant aliases from the list's .aliases only (new).
These changes ensure that the manager has absolute control of $LPDIR/.aliases.
- The OS fault tolerance has been improved in the routines that deal
with SUBSCRIBE, UNSUBSCRIBE, and SET requests. If an error occurs,
mail is sent to the manager (regardless of his/her error
preferences) and the user who issued the command is notified. The
error messages and their causes are listed below:
subscribe: ERROR: subscribe(): stat() failed for subscribers file CAUSE: the subscribers file disappeared before sorting RESULT: ListProc exits ERROR: subscribe(): fatal exit code from system() -- see .warning file CAUSE: The shell failed to execute the sorting pipe; system ran out of swap space, temp space, processes or other resources RESULT: user was subscribed, but the subscribers file will remain unsorted ERROR: subscribe(): stat() failed for temp subscribers file; list XXX; user YYY; .subscribers file unsorted CAUSE: the shell failed to execute the sorting pipe; system ran out of swap space, temp space, processes or other resources RESULT: user was subscribed, but the subscribers file will remain unsorted ERROR: subscribe(): Internal error: subscriber file sizes differ; list XXX; user YYY; .subscribers file unsorted. CAUSE: sort truncated its output; system ran out of swap space, temp space, processes or other resources RESULT: user was subscribed, but the subscribers file will remain unsorted ERROR: mv() failed CAUSE: unknown; see .warning; ListProc exits RESULT: ListProc exits unsubscribe: ERROR: unsubscribe(): stat() failed for subscribers file CAUSE: the subscribers file disappeared before update RESULT: ListProc exits ERROR: unsubscribe(): stat() failed for temp subscribers file; subscribers file not updated; request not completed; list XXX; user YYY CAUSE: system ran out of swap space, temp space, processes or other resources RESULT: user remains subscribed and is notified ERROR: unsubscribe(): Internal error: temp subscribers file truncated; original intact; request not completed; list XXX; user YYY CAUSE: system ran out of swap space, temp space, processes or other resources RESULT: user remains subscribed and is notified ERROR: mv() failed CAUSE: unknown; see .warning; ListProc exits RESULT: ListProc exits set: ERROR: set(): stat() failed for subscribers file CAUSE: the subscribers file disappeared before update RESULT: ListProc exits ERROR: set(): stat() failed for temp subscribers file; subscribers file not updated; request not completed; list XXX; user YYY CAUSE: system ran out of swap space, temp space, processes or other resources RESULT: the new setting did not take effect and the user is notified ERROR: set(): Internal error: temp subscribers file truncated; original intact; request not completed; list XXX; user YYY CAUSE: system ran out of swap space, temp space, processes or other resources RESULT: the new setting did not take effect and the user is notified ERROR: mv() failed CAUSE: unknown; see .warning; ListProc exits RESULT: ListProc exits
- Fixed core dump with SET list-name PREF invalid-pref FOR owner.
- Fixed bug with WHICH, LISTS, INDEX and PURGE. If a user was
subscribed to a list via an alias, the aliased address was used for
subsequent lookups, thus overriding the original sender address.
- Fixed bug with REPORTS. It did not send the current list report.
- Fixed bug with error analysis which did not work correctly with
quoted addresses with spaces while forcing POSTPONE on them.
- SET ... ADDRESS ... FOR address and SET ... PASSWORD
... FOR address failed to shadow the password in the
Subject: of the reply to the sender. This is now fixed.
- SET ... PREFERENCE for owners did not work correctly with quoted
addresses with spaces in them; this is now fixed.
- SET ... FOR and DELETE requests did not notify owner of alternate
addresses. This is now fixed.
- A bug existed in the SET code since the 6.0c days: a partial digest
was sent to a user with mail mode digest when (s)he changed his/her
password, conceal mode, mail mode or address, when it should only
be sent when she/he changes the *mail* mode. This is now fixed: a
partial digest is sent only when changing the mail mode from digest
to ack or no ack.
- GET, INDEX and SEARCH requests were not working correctly as far as
copying the manager were concerned. GET and INDEX were copying
random owners, and SEARCH was copying random owners if the manager
had set CCALL or CCINDEX. This has been corrected.
- When the mailmethod chosen was not "system" or "telnet" the Cc:
list was not displayed correctly. This has been fixed.
- Fixed a bug with RFC 822 address parsing. Addresses such as:
From: "<< C. Larkins >>"were not parsed correctly. - Fixed bug with ARCHIVE. Due to an uninitialized variable, the
manager could not ADD/DELETE or UPDATE a file.
Last updated: Monday, December 10, 2007, 09:38AM PST