University of Southern California

ITS Information Technology Services

A division of the Office of the Chief Information Officer

ListProc 8.0 Release Notes

Contents


NEW FEATURES

FULLY MULTI-THREADED

ListProc 8.0 is fully multi-threaded, which allows it to process mail for several lists simultaneously.

The "threads" config file directive
A config file directive "threads" controls the number of threads ListProc will create. Usage of the "threads" directive is as follows: 
	
     threads number
where number is the number of threads to use. Valid numbers of threads are between 1 and 256, inclusive. A value of 1 results in the previous, unthreaded behavior.

If no value of "threads" is specified, ListProc defaults to 1 thread.

Synchronized access to data files
Accesses to aliases and ignored files have been synchronized for multiple threads. Synchronization is done via the system's semaphore. In previous versions, this was only necessary when ILP was enabled. The use of multiple threads in 8.0 requires the use of semaphores at all times.

Error mail processing
As long as the config file defines more than one thread, ListProc will handle error mail for all lists asynchronously with a single thread, rather than dealing with it when processing the list mail. This has the following implications:
  • There is at most one thread processing error mail at any one time, regardless of the number of threads defined, and the number of threads still free.
  • It is possible for ListProc to concurrently process list mail and error mail for a single list. In this case, the error mail is given lower priority.
  • This will significantly speed up the processing of each list, since each list's thread will only need to handle regular list mail and digests.

ListProc's flow of control
Simultaneous processing is performed on as many lists as there are threads defined in the config file. Hence, if your config file contained "threads 10", ListProc would be able to concurrently process mail from 10 different lists.

The spawning of threads is still subject to delays specified by the "frequency" directive in the config file. For example, suppose your config file contained the line "frequency 0 30". In ListProc 7.X, the flow of list processing would look like this:

      process list 1 (regular mail, digests, error mail)
      process server mail
      process list 2 (regular mail, digests, error mail)
      process server mail
      ...
      process final list (regular mail, digests, error mail)
      process server mail
      wait 30 seconds
      repeat
ListProc 8.0 would instead do the following:


      spawn list 1 thread (regular mail, digests)
      spawn error thread if error mail exists and 
	    no error thread is already running.  
      spawn server mail thread unless one already exists

      spawn list 2 thread (regular mail, digests)
      spawn error thread if error mail exists and 
	    no error thread is already running.  
      spawn server mail thread unless one already exists

      ...
      ...

      spawn final list thread (regular mail, digests)
      spawn error thread if error mail exists and 
	    no error thread is already running.  
      spawn server mail thread unless one already exists

      wait 30 seconds
      repeat
Notice that in 8.0 ListProc does not need to wait for the processing of one list to finish before it goes spawns a thread to process the next list.

Within each list, processing is still serialized: regular mail and digests are never handled concurrently for a given list. As noted above, the error processing has been taken out of the list processing loop. Error mail is processed by the first available thread after error mail arrives.

Replacement of system() calls
The multi-threaded code required the use of fork() and exec() rather than system(). This also allows us to circumvent the problems with the shell that's spawned by system(). Consequently, this allows us to lift the restriction of no apostrophes in a user address. (see the GENERAL CHANGES section below) All binaries have been converted to make use of the new sysexec() routine that performs the fork(), exec() and wait() functions. All of these required significant rewriting of signal handling routines in serverd.c.

Updated semset utility
The semset utility included in the distribution has been updated to show how many lists are delivering mail at the time of invocation. Invoked with no arguments, semset returns basic information about ListProc's interprocess communication (the output has been wrapped to fit an 80 column display):

      %semset

      IPC status from  as of Thu Feb  8 12:22:48 1996
      Message Queue facility not in system.
      Shared Memory facility not in system.
      T     ID     KEY        MODE       OWNER    GROUP  CREATOR   
      Semaphores:
      s 131072 0x0x00000000 --ra-------   server  staff   root     

      CGROUP NSEMS   OTIME    CTIME 
      staff     1 12:20:26 13:49:31
When semset is called with the semaphore ID as an argument, it returns specific information about that semaphore's current state. The following flag information can be displayed:

      SEM_REQ_ID        Requesting a tag ID, request #, etc.
      SEM_SYSFILES      Accessing/Processing global server files
      SEM_LISTFILES     Accessing/Processing list specific files
      SEM_ARCHIVES      Accessing/Processing a list archive
      SEM_DLVR_MAIL(N)  Delivering mail (N = number of lists
                        currently delivering)
The sample output below shows ListProc's processing on our server over the course of about 30 seconds:


      % semset 131072
      Current value: 0x10; flags on: SEM_DLVR_MAIL(1x)
      % semset 131072
      Current value: 0x10; flags on: SEM_DLVR_MAIL(1x)
      % semset 131072
      Current value: 0x10; flags on: SEM_DLVR_MAIL(1x)
      % semset 131072
      Current value: 0x14; flags on: SEM_LISTFILES SEM_DLVR_MAIL(1x)
      % semset 131072
      Current value: 0x20; flags on: SEM_DLVR_MAIL(2x)
      % semset 131072
      Current value: 0x20; flags on: SEM_DLVR_MAIL(2x)
      % semset 131072
      Current value: 0x20; flags on: SEM_DLVR_MAIL(2x)
      % semset 131072
      Current value: 0x24; flags on: SEM_LISTFILES SEM_DLVR_MAIL(2x)
      % semset 131072
      Current value: 0x10; flags on: SEM_DLVR_MAIL(1x)
      % semset 131072
      Current value: 0x10; flags on: SEM_DLVR_MAIL(1x)
      % semset 131072
      Current value: 0x10; flags on: SEM_DLVR_MAIL(1x)
Multi-threading caveats and cautions
Multi-threading can potentially create a much larger drain on your system's resources. If the system-wide process table is not large enough, you may run out of available processes. Furthermore, each thread requires its own memory and makes separate demands on processor and disk resources.

Perhaps most significantly, each thread opens its own sendmail processes. (The number of sendmail processes for each list depends on the number specified by the "multiple_recipients" directive in the config file. For example, if "multiple_recipients" is set to M, a list with S subscribers will spawn S/M sendmail processes.) Increasing the number of lists processing at one time will cause a proportionally larger number of sendmail processes to run.

An additional caveat with multi-threaded processing is that serverd must wait for all outstanding threads to quit before it can re-initialize. (This is essentially also what happens with 7.x - it is just that with only one thread running, the wait time will probably not be very long.) Once the server receives a notification that it must re-initialize, no new threads are spawned while serverd waits for the existing threads to complete operation. Once all threads have finished, serverd can rebuild its internal data structures.

SPEEDUPS

USAGE AND ACCOUNTING DATA

ListProc now allows the option of calling an external accounting program each time list digests and regular list mail are disributed. The accounting program is NOT invoked when error messages are sent out. The environment variable ULISTPROC_LIST_ACCOUNTING_PROG specifies the full path to the accounting program to use. (If this variable is not set, no accounting program is called.)

The accounting program is passed the following information:

The accounting program is responsible for calculating file sizes, keeping track of the date and time, running totals, etc.

LOOP DETECTION

HEADER CONFIG FILE DIRECTIVE

The "header" config directive has been expanded to affect all lists and the server, and to allow for header lines to not be propagated, as follows: 
      header * {
	  Headerline:
	  !Headerline:
	  ...
      }
By specifying a * all lists and the server are affected. Header lines to be propagated are listed as before, and header lines not to be propagated should start with an exclamation point ("!"). Headers propagated (or not propagated) to individual lists can be controlled by using a list name instead of the * as follows: 
      header foo-list {
	  Headerline:
	  !Headerline:
	  ...
      }
Notice that "header lines" are actually regular expressions as before, so, for example, the following is valid: 
      header * {
	  !X-.+-expire:
      }
The following header lines will be preserved regardless of the header directive: 
	 
      From:		           Reply-To:
      Resent-From:                 Date: 
      Sender:                      Control: 
      Resent-Sender:               Approved: 
      Message-Id:                  Archive-Name: 
      Resent-Message-Id:           all MIME headers
Example: here is how to avoid propagating header lines that cause loops and error messages with auto-responders:

      header * {
	  !Registered-Mail-Reply-Requested-By:
	  !X-Confirm-Reading-To:
      }
If list-specific "header" directives are also defined, these are appended to the list inherited by "header *".



GENERAL CHANGES



BUG FIXES

USC Listproc Main Page

Last updated:
June 01, 2007

print this page email this page
The use of all USC computing resources is governed by the USC Computing Policies.