Filtering FAQ

From: FAQ Editor 
Subject: Filtering Mail FAQ
Followup-To: comp.mail.misc
Date: 4 Oct 1996 23:37:54 -0400

Archive-name: mail/filtering-faq
Posting-frequency: approximately monthly
Last-modified: 4 October 1996
Copyright: See section 7.0
FAQ-launcher-URLs: http://www.jazzie.com/ii/faqs/archive/mail/filtering-faq/
                   http://www.best.com/~ii/faqs/archive/mail/filtering-faq/

 All current hypertext versions of this FAQ are accessible through either
 of the above two "FAQ Launcher" URLs.  If you want to link to this FAQ,
 please link to these launchers rather than to only one version of the FAQ
 (e.g., rather than linking to the version you are viewing right now!). 
 This will give your users a choice of which version they access, e.g., a
 single-file version or an index-plus-multiple-files version.  Also, please
 give links to both mirrors of the launcher since sometimes one or the
 other is down. 


           _____________________________________________________
  ________|                                                     |________
  \       |                  FILTERING MAIL FAQ                 |       /
   \      |    Copyright (c) 1994-96 Nancy McGough and others   |      /
   /      |_____________________________________________________|      \
  /___________)                                              (__________\


  TABLE OF CONTENTS

  0.0 Preliminaries
      0.1 Getting the Latest Version of this FAQ
          0.1.1 Hypertext
          0.1.2 Plain Text
      0.2 Terminology
      0.3 Notation
  1.0 Mail Folder Strategies
      1.1 Naming Incoming Mail Folders
      1.2 Reading Incoming Mail Folders
  2.0 Procmail
      2.1 Setting Up Procmail
          2.1.1 Is Procmail on Your System?
          2.1.2 Setting Up Procmail for Testing
          2.1.3 Testing
          2.1.4 Setting Up Procmail to Filter Mailing List Messages
      2.2 Troubleshooting Procmail
          2.2.1 General Strategies
          2.2.2 Alternate .forward files
      2.3 Explanation of Test Recipe
      2.4 Tracking Your Incoming Mail
      2.5 Procmail References
  3.0 Mailagent
      3.1 Setting Up Mailagent
      3.2 Tracking Your Incoming Mail
      3.3 How Safe Mailagent Processing Is?
      3.4 Locking Under Mailagent
      3.5 Folder Types Supported
      3.6 Mailagent References
  4.0 Filter
      4.1 Setting up Filter
      4.2 Tracking Your Incoming Mail
      4.3 Filter References
  5.0 Mailer and Newsreader References
  6.0 Contributors
      6.1 Acknowledgements
      6.2 Contributing to this FAQ
  7.0 Copyright Notice

----------------------------------------------------------------------

From: FAQ Editor 
Subject: 0.0 Preliminaries

Q: How can I have my incoming mailing-list messages automatically put
   into appropriate folders?

This is one of the most frequently asked questions about email.  This
FAQ provides basic instructions for Unix users to set up either
procmail, mailagent, or elm's filter to filter incoming mailing-list
messages.  If you have a choice, I recommend procmail which is
powerful, robust, and is actively developed and supported.

All these mail processors can also be set up to filter other types of
incoming messages, send automated replies, etc.  For instructions see
the relevant references listed below.

------------------------------

From: FAQ Editor 
Subject:     0.1 Getting the Latest Version of this FAQ

If this FAQ is over a couple months old, there may be an updated
version.  Please get the latest hypertext or plain text version from
one of the places listed below.

------------------------------

From: FAQ Editor 
Subject:         0.1.1 Hypertext

The best way to read this FAQ (and most other FAQs too) is to view
the hypertext version using a Web browser such as Cello, Lynx,
Mosaic, Netscape Navigator, Microsoft Internet Explorer, OmniWeb, 
SpiderWoman or WinWeb.  This will allow you to easily jump:
   * between subjects in the FAQ
   * to any Uniform Resource Locator (URL) in the FAQ
   * to an Internet Request For Comments document (RFC)
   * to some manual pages

This, and all FAQs that are crossposted to news.answers, are available at
the following hypertext FAQ archives:
 http://www.cs.ruu.nl/cgi-bin/faqwais/
 http://www.lib.ox.ac.uk/search/search_faqs.html
 http://www.smartpages.com/bngfaqs/top.html

All hypertext versions of this particular FAQ are accessible through
either of these "FAQ Launcher" URLs:
  http://www.jazzie.com/ii/faqs/archive/mail/filtering-faq/
  http://www.best.com/~ii/faqs/archive/mail/filtering-faq/

------------------------------

From: FAQ Editor 
Subject:         0.1.2 Plain Text

The plain text version is regularly posted to comp.mail.misc,
comp.mail.elm, comp.mail.mh, comp.mail.mush, comp.mail.pine,
comp.mail.zmail, comp.answers, and news.answers.  It's
in digest format which means that you may be able to use your
newsreader to easily move between digest items (e.g., nn uses G% to
burst a digest and trn uses ^G to jump to the next digest item).

The plain text FAQ is also available through:

Anonymous FTP:
 ftp://rtfm.mit.edu/pub/usenet/news.answers/mail/filtering-faq

Email:
 Send mail to:
    mail-server@rtfm.mit.edu 

 Containing the following in the body of the message:
    send usenet/news.answers/mail/filtering-faq

UUCP:
 uunet!/archive/usenet/news.answers/mail/filtering-faq

Hard Copy:
 A printed version of this FAQ is in Chapter 25 of the book 
 "Internet Secrets" by John R. Levine and Carol Baroudi; published 
 1995 by IDG Books; ISBN 1-56884-452-2.

------------------------------

From: FAQ Editor 
Subject:     0.2 Terminology

  Term               Meaning
  ====               =======
  mailer or MUA      Mail user agent such as pine or elm
  pico               PIne COmposer - a user friendly editor
  reader             Mailer, newsreader, or Web browser that can read 
                     mail folders
  regular expression Text that can include "wild cards" (such as . to
                     match any single character); used for searching
  ^x                 Press the Ctrl key and then, while holding down the 
                     Ctrl key, press the x key
  ~ or $HOME         Your home directory.  You can always get to your
                     home directory by typing `cd'.

------------------------------

From: FAQ Editor 
Subject:     0.3 Notation

Notation  Meaning
========  =======
TextName  placeholder; replace with appropriate text
    placeholder; replace with appropriate text without the angle brackets
`text'    text you type but without the quotes
"text"    text you type including the double quotes
'text'    text you type including the single quotes
[Key]     press the key, e.g., [Space], [Enter], or [Tab]

------------------------------

From: FAQ Editor 
Subject: 1.0 Mail Folder Strategies

It is a good idea to come up with a system for naming and reading
your mail folders.  With a good system, you will have an easier time
managing the hundreds (or thousands!) of messages you will receive.

------------------------------

From: FAQ Editor 
Subject:     1.1 Naming Incoming Mail Folders

For my incoming mail folders I use names that start with `IN'.  For
example, I put mail sent to the procmail mailing list into a folder
named IN.procmail.  This way when all my folders are listed
alphabetically the incoming folders are together and near the top.
They are near the top because Unix is case sensitive and upper case
letters come before lower case letters in an ascii sort.

Of course, you can use any names you like for your mail folders!

------------------------------

From: FAQ Editor 
Subject:     1.2 Reading Incoming Mail Folders

Here are some instructions for opening a mail folder that's in the
default folder directory for some mailers and newsreaders.

         Default Folder  
 Reader   Directory      Command line command  From within reader
 ======  ==============  ====================  ==================
 elm     ~/Mail          elm -f =folder        c =folder
 pine    ~/mail          pine -if folder       G folder
 mail    ~/mail          mail -f /path/folder
 nn      ~/News          nn +folder            G +folder
 
(Please send me information for other mailers and newsreaders.)

Pine has an incoming-folder variable which you can use to list your
incoming folders in a separate section of your folder list (to view
your folder list in Pine, type `L').  Setting this variable also
allows you to use the Tab key to step through all new messages in all
your incoming folders.  For more information see the question "How
can I filter messages into different incoming folders?" in the Pine
FAQ:

  http://www.cac.washington.edu/pine/QandA/FAQs.html#xtocid178164


IMPORTANT NOTE 
============== 
If you are going to be editing your incoming mail folder, e.g.,
deleting messages, then your filtering program and your folder reader
should use locking schemes that work together.  Otherwise, if mail is
delivered at the same time that you are deleting a message your
folder may be corrupted.  To learn more about locking see the
question "What is folder locking and how does it work?" in the Pine
FAQ.  This is useful to read even if you don't use Pine as your
mailer.

 http://www.cac.washington.edu/pine/QandA/sysadmins.html#xtocid7145

Most mailers, such as pine and elm, use lock a file named
`folder.lock' to lock the folder named `folder'.  Note that nn does
not lock folders so you probably don't want to use nn to delete
messages in a folder that is receiving incoming messages.  (Please
send me info about locking in other mailers and newsreaders.)

For more information about specific mailers and newsreaders, see the
relevant documentation, such as man pages and Web pages.  Some
pointers to documentation are listed in section 5.0.

------------------------------

From: FAQ Editor 
Subject: 2.0 Procmail

Procmail is a powerful mail processor that can be used to process your
mail messages either as they arrive or after they are in a mail folder.  
To find out how to process an existing mail folder see the NOTES section
of the procmail(1) man page.

------------------------------

From: FAQ Editor 
Subject:     2.1 Setting Up Procmail

To set up and test procmail, follow the ten steps given in sections
2.1.1 through 2.1.4 below.

------------------------------

From: FAQ Editor 
Subject:         2.1.1 Is Procmail on Your System?

1]  To find out if procmail is on your system and what the full path to 
    it is, type one of the following commands:

      Shell      Command
      =====      =======
      csh        which procmail
      sh or ksh  type procmail
      various    whereis procmail
      various    where procmail

    Make a note of the full path to procmail because this is needed in
    step 5b below.  You can find out the version by typing:

      procmail -v

   The latest released version, as of 15 April 1995, is 3.10.
   Version 3.11 should be out soon.  If your system doesn't have
   procmail or doesn't have the latest version, you may want to ask
   your system administrator to install it.  The procmail package of
   tools is at:

 ftp://ftp.informatik.rwth-aachen.de/pub/packages/procmail/procmail.tar.gz

   If your sys admin isn't able to do this, use a different mail
   processor like mailagent (described in section 3 of this FAQ) or
   filter (described in section 4).

------------------------------

From: FAQ Editor 
Subject:         2.1.2 Setting Up Procmail for Testing

2a] Create ~/.procmailrc.  

     cd
     pico .procmailrc

    NOTE: Throughout this article I use pico for editing files.
          Replace `pico' with your editor.

2b] Enter a modified version of the following in your ~/.procmailrc.
    Note that lines that begin with # are comments and are ignored by
    procmail.

     #Set on when debugging
     VERBOSE=off
 
     #Replace `mail' with your mail directory (Pine uses mail, Elm uses Mail)
     MAILDIR=$HOME/mail
     
     #Directory for storing procmail log and rc files
     PMDIR=$HOME/.procmail
     
     LOGFILE=$PMDIR/log
     INCLUDERC=$PMDIR/rc.testing
     INCLUDERC=$PMDIR/rc.maillists

3] Create the directory where you will store your procmail log and rc files 
   (this is $PMDIR that you set above).

     cd
     mkdir .procmail


4a] Create an rc (run commands) file for testing:

     cd .procmail
     pico rc.testing


4b] Enter the following in ~/.procmail/rc.testing:

      :0:
      * ^Subject:.*test
      IN.testing

    Note that the first line contains a zero (0), not the letter "oh".
    For now, don't worry about the meaning of this recipe.  It is
    explained in 2.3 "Explanation of Test Recipe."


5a] Create a ~/.forward file by typing the following. (Pico's -w flag tells 
    pico not to auto wrap lines.)

     cd
     pico -w .forward



5b] Enter a modified version of the following in your ~/.forward:


     "|IFS=' ' && exec /usr/local/bin/procmail -f- || exit 75 #nancym"


    == IMPORTANT NOTES ==
    * Make sure you include all the quotes, both double (") and single (').
    * The vertical bar (|) is a pipe.
    * Replace /usr/local/bin with the correct path for procmail (see step 1).
    * Replace `nancym' with your userid.  You need to put your userid in 
      your .forward so that it will be different than any other .forward file 
      on your system.
   
    * Do NOT use ~ or environment variables, like $HOME, in your .forward
      file.  If procmail resides below your home directory write out the 
      *full* path.


5c] On many systems you need to make your .forward world 
    readable and your home directory world searchable in order for the
    mail transport agent to "see" it.  To do this type:
   
      cd
      chmod 644 .forward
      chmod a+x .

------------------------------

From: FAQ Editor 
Subject:         2.1.3 Testing

6] Send yourself two test messages: one with `test' in the subject
   and one without `test' in the subject.


7] Start your mailer (pine, elm, etc.) and check that the messages
   were delivered correctly.  The one with `test' in the subject
   should be in the folder $MAILDIR/IN.testing and the one without 
   `test' in the subject should be in your inbox.  

   If these were not delivered correctly, see section 2.2 on
   "Troubleshooting Procmail" section below.

------------------------------

From: FAQ Editor 
Subject:         2.1.4 Setting Up Procmail to Filter Mailing List Messages

8a] Once you have successfully tested procmail in steps 6 and 7, create 
     rc.maillists for filtering incoming mailing-list messages into mail 
     folders.
  
      cd
      cd .procmail
      pico rc.maillists


8b] In rc.maillists, create a recipe, like the two example recipes
    below, for each of your mailing lists.

     :0:
     * ^TOwww-talk
     IN.www-talk

     :0:
     * ^TOprocmail
     IN.procmail
     
     The first recipe filters the www-talk mailing list and the second 
     recipe filters the procmail mailing lists.  The meaning of the first 
     recipe is as follows:

     Notation      Meaning
     ========      =======
     :0            Begin a recipe
       :           Use a lock file
     *             Begin a condition 
       ^TO         Match ``To:'' ``Cc:'' or other synonyms for To at the
                   beginning of a line, followed by any or no characters,
                   followed by....
          www-talk  ``www-talk''
     IN.www-talk   If successful match, put in folder $MAILDIR/IN.www-talk

  
     IMPORTANT NOTES
     ===============
     * ^TO is not a normal regular expression; it is a special
       procmail expression that is designed to catch any destination
       specification.  For details, see the MISCELLANEOUS section of
       the procmailrc(5) man pages.

     * Do not put a space between the caret (^) and the word `TO' in
       `^TO'.

     * Do not put a space between the `^TO' and the text that you are
       matching on; it must be `^TOtext'.

     * Both letters in `TO' must be capitalized.
       

9]  Repeat steps 6 and 7 to make sure that things are still working.


10]  Comment out the rc.testing line in you .procmailrc file so that it looks 
     like this:
          
       VERBOSE=off
       MAILDIR=$HOME/mail
       PMDIR=$HOME/.procmail
       LOGFILE=$PMDIR/log
       # INCLUDERC=$PMDIR/rc.testing
       INCLUDERC=$PMDIR/rc.maillists

     NOTE: Leaving the rc.testing line in your .procmailrc file is
           useful for future testing.

------------------------------

From: FAQ Editor 
Subject:     2.2 Troubleshooting Procmail

If messages are not delivered correctly, here are steps you can use to
try to solve the problem.

------------------------------

From: FAQ Editor 
Subject:         2.2.1 General Strategies

1]    Look at your $LOGFILE (~/.procmail/log) to see if you can
      determine what the problem is.

2]    Check these three files for typos:
        ~/.forward
        ~/.procmailrc
        ~/.procmail/rc.testing

3]    Check the file and directory permissions of your .forward (set in 
      5c in "Setting up Procmail for Testing" above).

      Type...          In order to...
      -------          --------------
      cd               Go to your home directory.
      
      ls -l .forward   Check the permission: it should say -rw-r--r--
      
      ls -ld .         Check permission of home dir: it should say drwx?-x?-x
                       The ?'s may be r's or hyphens or one of each (i.e.,
                       drwx--x--x, drwxr-xr-x, drwxr-x--x, drwx--xr-x are
                       each acceptable.)
      

4]    If the above three steps do not locate the problem edit your 
      ~/.procmailrc so that it contains:

        VERBOSE=on

      Test procmail by following steps 6 and 7 again.  Look at your
      $LOGFILE (which will contain verbose messages) to see if you
      can now determine what the problem is.

      If you are still having problems see the next section on
      "Alternate .forward Files."  After you get procmail to work, 
      you probably will want to set VERBOSE back to off.

------------------------------

From: "Stephen R. van den Berg" 
Subject:         2.2.2 Alternate .forward files

If the .forward template in 5b above doesn't work the following alternatives 
might be helpful:

In a perfect world:
	"|exec /usr/local/bin/procmail #nancym"
In an almost perfect world:
	"|exec /usr/local/bin/procmail USER=nancym"
In another world:
	"|IFS=' ';exec /usr/local/bin/procmail #nancym"
In a different world:
	"|IFS=' ';exec /usr/local/bin/procmail USER=nancym"
In a smrsh world:
	"|/usr/local/bin/procmail #nancym"

These formats can be tried in different combinations, the leading "| can
be tried as |" instead, or vice versa.

NOTE
====
If your system uses procmail as its local delivery agent, you do not
need a .forward file; simply having a .procmailrc files suffices.  To
find out if procmail is the local delivery agent, ask your system
administrator.

------------------------------

From: FAQ Editor 
Subject:     2.3 Explanation of Test Recipe

The recipe you used for testing is:

:0:
* ^Subject:.*test
IN.testing


The meaning of this recipe is:

Notation            Meaning
========            =======
:0                  Begin a recipe
  :                 Use a lock file
*                   Begin a condition 
  ^                 Match the beginning of a line followed by....
   Subject:         ``Subject:'' followed by....
           .        any character (.) followed by....
            *       0 or more of preceding character (any character in this 
                       case) followed by....
              test  ``test''
IN.testing          If successful match, put in folder $MAILDIR/IN.testing

------------------------------

From: FAQ Editor 
Subject:     2.4 Tracking Your Incoming Mail

You can use mailstat, a useful script that is part of the procmail
package, to check your procmail log file. Check to see if it is 
on your system by typing either `which mailstat' or `type mailstat'.  
If it's on your system type:

  mailstat $HOME/.procmail/log

This displays a concise version of your log file and moves your log 
file to log.old.  You may want to put the above line in your .login so that 
each time you log in you will see a listing of how many messages you've
received since the last time you ran mailstat, and what folders these
messages were delivered to.

You can get a mailstat listing of log.old by using the -o flag:

  mailstat -o $HOME/.procmail/log


If mailstat is not on your system ask your system administrator to
install it.  The script is located with all the other procmail tools
(see 2.1.1 above for the ftp location).

------------------------------

From: FAQ Editor 
Subject:     2.5 Procmail References

Web Page: http://www.jazzie.com/ii/internet/procmail/
          http://www.best.com/~ii/internet/procmail/ (mirror of above URL)

Man Pages:    procmail(1)   - autonomous mail processor
              procmailrc(5) - procmail rc file
              procmailex(5) - procmail rc file examples
              procmailsc(5) - procmail weighted scoring techique
              egrep(1)      - search file for regular expression (procmail 
                              uses egrep-style regular expressions along with 
                              some of its own expressions like ^TO)
              formail(1)    - mail reformatter
              sendmail(8)   - send mail over the internet

Newsgroup:    comp.mail.misc

Mailing List: procmail@informatik.rwth-aachen.de
           Subscribe to the procmail mailing list by sending mail:
             To: procmail-request@informatik.rwth-aachen.de 
             Subject: subscribe

Procmail Archives:
           Get a list of files available at the procmail mail server by 
           sending mail:
             To: procmail-request@informatik.rwth-aachen.de 
             Subject: archive ls

           Get Best of the Procmail mailing list by sending mail (you'll 
           need gzip and a MIME-decoder to unpack it):
             To: procmail-request@informatik.rwth-aachen.de 
             Subject: archive get best_of_procmail_list*

------------------------------

From: Raphael Manfredi 
Subject: 3.0 Mailagent

Mailagent is a powerful mail processing package that can be used to
process your mail messages, either at arrival time via a .forward hook
or later on while they are already saved in a mail folder.

Mailagent is written in Perl and hence has all the advantages of being
interpreted, i.e. it is easy to enhance dynamically and to customize.

This version of the FAQ describes the basics of setting up mailagent to
process incoming mail messages from within a .forward. To find out about
the more advanced features of mailagent, please refer to the mailagent(1)
manual page.

------------------------------

From: Raphael Manfredi 
Subject:     3.1 Setting Up Mailagent

1] First, make sure mailagent is available on your system. The easiest
   way to do this is to run:

    mailagent -V

   which will print the mailagent version if it is available, or the shell
   will issue an error message "mailagent: not found" or something like it.

2a] Now you need to understand the MTA (Mail Transport Agent, the program
    that delivers the mail; usually, sendmail) will NOT deliver to mailagent
    directly, rather to an intermediate (small) filter program. Two versions
    are available: shell or C -- refer to the mailagent(1) manpage to choose,
    but I recommend you use the C version first, and move to the shell
    version if you can't run a binary from your .forward file.

2b] Locate the filter program (it will be filter or filter.sh depending on
    whether you choose the C or the shell version respectively) under some
    directory like /usr/local/lib/mailagent. From now on, we'll assume we
    use the C filter and that it is located under /usr/local/lib/mailagent.

3] Copy the file /usr/local/lib/mailagent/mailagent.cf as ~/.mailagent and
   edit it to configure your system correctly. You will see two distinct
   sections in that file and you need to set-up properly the first one, the
   "Configuration section".

   If you have a version of mailagent that is recent enough (at least 3.0 PL32)
   then you can create an initial configuration very easily and quickly by
   running the following command:

    mailagent -I

   which will set-up an almost ready-to-use ~/.mailagent file. All you
   need to do at this point is go through its configuration section
   to make sure mailagent made the right choices...

   The minimal set of variables that MUST be correctly set (i.e. for which
   you cannot rely on the default set in the file) are (# introduces comment
   in a shell-like manner, which run up to the end of the line):

    home:  # Your HOME directory, as reported by "echo $HOME", usually.
    path:  # Path to be used to locate mailagent and perl, at least.
    p_xxx: # Path to be appended to "path" above when running on machine "xxx".
    user:  # Your login name.
    name:  # Your name.
    level: # Logging level. I recommend you raise it to 20 for testing.

   The mail folder directory is ~/Mail by default, but it can be changed
   only from within your rule file by putting

    maildir = ~/mail;

   at its top, for instance, to make it ~/mail. The rule file is defined as
   the "rules" parameter, and is set to ~/.rules by default.

4] Ensure the directories configured in your ~/.mailagent under "logdir" and
   "spool" and "queue" do exist. If you use the standard setting, this
   requires the following commands:

    cd
    mkdir var
    cd var
    mkdir log mailagent
    cd mailagent
    mkdir queue

5a] Create a rule file (named ~/.rules by default) for testing:

    cd
    vi .rules

5b] Enter the following in ~/.rules

    Subject: /test/     { SAVE testing };

    The meaning of that rule should be pretty obvious: If we receive a mail
    whose subject line contains the word "test", then we save that mail in
    a folder named "testing", under the default folder directory (~/Mail).

5c] Create a ~/.forward file as follows:

    "|exec /usr/local/lib/mailagent/filter >> /export/home/ram/.bak 2>&1"

    The meaning of that line is the following: every mail should be piped
    (hence the leading "|" character) onto the filter program, and any
    output from that program (i.e. errors) are appended to some file in
    your home directory, with stderr following stdout (2>&1) in traditional
    sh syntax.

    == IMPORTANT NOTES ==
    * Your .forward is always processed by sh, regardless of your login shell.
    * Replace /export/home/ram with your proper login directory full path.
      That's a part that makes your .forward unique (for zealous optimizing
      sendmail that are dead wrong about optimizing!) and that can save
      you a lot of trouble if anything goes wrong! Just look at your ~/.bak!
    * Replace /usr/local/lib/mailagent/filter with the proper filter path
      on your machine.

5d] Note that on many systems, you need to ensure your .forward will be
    readable by everybody, and that your home directory has the "x" bit set
    for everybody (i.e. can be part of a lookup path) so that sendmail can see
    and parse your .forward file. To ensure this, type:

        cd
        chmod a+r .forward
        chmod a+x .

6] Send yourself two test messages: one with "test" in the subject, and one
   without "test" in the subject.

7a] Checkout your ~/.bak file: it should be empty!

7b] Checkout your ~/var/log/agentlog file to see what really happened to your
    messages. Watch out for any ERROR or WARNING logs. This assumes you have
    left at least the default logging level (9) in the ~/.mailagent file (the
    "level" variable). But for testing, that level should be raised to 20 to
    help you diagnose what's going on.

7c] Look out in ~/Mail/testing. You should find there the message whose Subject
    line contained the word "test". Then make sure the other message has been
    delivered to your regular mailbox. (Since no match occurred in your rule
    file, the mail is left in your mailbox by default).

7d] TROUBLESHOOTING
    * If your mail was not properly delivered, please make sure your rule file
      and configuration file are correct by issuing the following command:

        mailagent -d

    * If the previous command does not output the single rule you should have
      put in ~/.rules, then please make sure step 3 and 4 above were correctly
      performed (those being the crucial steps ensuring a proper configuration).
    
    * Check the ~/.bak file for error messages.

    * Check for typos in any of these files:
        ~/.forward
        ~/.mailagent
        ~/.rules
    
    * Check the file and directory permissions of your .forward (set in step
      5c above).

      Type...          In order to...
      -------          --------------
      cd               Go to your home directory.
      
      ls -l .forward   Check the permission: it should say -rw-r--r--
      
      ls -ld .         Check permission of home dir: it should say drwx?-x?-x
                       The ?'s may be r's or hyphens or one of each (i.e.,
                       drwx--x--x, drwxr-xr-x, drwxr-x--x, drwx--xr-x).
      
    * If none of the previous hints helped you identify problem, and you can't
      figure it out from the output in ~/.bak or in the ~/var/log/agentlog file
      (or whatever file you have configured for logging within your ~/.mailagent
      file, variables "log" and "logdir"), then make sure your mail is not
      waiting in the MTA's queue: this might be the case if the agentlog file
      is empty. If you are using sendmail as MTA, you can run:

        /usr/lib/sendmail -bp
    
      to print out the queue.
    
    * As a last resort, please look at the mailagent(1) manual page under
      the section "Testing Your Installation" for more tips and things
      to look at.

8] Once you have successfully tested mailagent in steps 6 and 7 above,
   you're on! Mailagent is ready to process your mail. All you have to
   do is extend the ~/.rules file to add more rules.

   For instance:

   To Cc: www-talk      { SAVE www-talk };
   To Cc: agent-users   { SAVE agent-users };

   Those two rules filter the two mailing lists www-talk and agent-users into
   their respective folders, whether the mailing list address appear in the To
   OR Cc header.

   Since rules are not qualified as a pattern match (contrary to our test
   above), they match on logins in the address, i.e. they will match things
   like www-talk@chip.com or chip!www-talk, or a plain simple www-talk if
   this is a local alias. (This implicit matching on logins works only for
   some selectors like To, Cc or From which are know to contain addresses).

   If you wish to sort on patterns appearing in the Subject of messages for
   instance, then you must use a pattern matching syntax, as in:

   Subject: /star trek/ { SAVE star-trek; };

   to save in a folder "star-trek" all the messages whose subject contains
   the words "star trek". Case matters, but keep on reading...

9] As an advanced topic, since mailagent is written in Perl, you have all the
   power of Perl's regular expressions at your disposal. Which means you can
   write things like this:

   To Cc: agent-users                    { REJECT AGENT };
    Subject: /^\w*subscribe/i     { DELETE };
    *                             { UNIQUE -a; SAVE agent };

   The second lines makes use of that perl extended regular expression syntax:
   \w matches an alphanumeric character plus "_", while the trailing "i" option
   requests a case-insensitive match.

   You also note you have a real automaton at your disposal. You can enter
   a special state (AGENT in our example) and continue parsing by only
   scanning for rules tagged by this mode. The first match stops the automaton,
   unless you REJECT to continue processing. When not restricted by a mode list,
   a rule is always taken into account. For example, assuming the automaton
   is in the state "NEWS", it will not consider rules tagged , as in
   the above example. The automaton begins in mode "INITIAL".

   The "UNIQUE -a" action followed by a SAVE ensures only one copy per
   Message-ID will ever end-up in your agent folder -- no duplicates!
   Also note you can have more than one action per rule, and that the last
   one uses '*' to match anything, i.e. its action part between {} will
   always be executed in AGENT mode, when reached by the automaton.

   Also, since in Perl regular expression syntax, \b matches a word-boundary
   and \s any space or tab character, we can write our Star Trek message
   sorting into a much more robust form:

   Subject: /\bstar\s+trek\b/i  { SAVE star-trek; };

   (\s+ matches one or more white spaces, while \s* would match zero or more,
   see the Perl manual page for a complete description of regular expressions.)

   which will match on various subject strings like "Last Star Trek season"
   or "I am addicted to Star trek", but not on "Tristar treks" -- whatever
   that means :-)

   All in all, the filtering automaton syntax is pretty much intuitive and
   easy to read. You have to learn which actions are possible and what they
   mean, naturally.

------------------------------

From: Raphael Manfredi 
Subject:     3.2 Tracking Your Incoming Mail

If you are curious about what mailagent does to your mail, you have two
options:

    * Look at your "agentlog" file, with a log level set to 9.
    * Get a summary of all the actions performed by running:

        mailagent -summary

      This last option is only possible if you have initialized the statistics
      gathering process by creating a ~/var/mailagent/mailagent.st file (under
      the default setting from ~/.mailagent).

      You will get a clear picture of your processing, by seeing which rule
      match, how often, in which state, etc... You will also know how many
      times you SAVE or DELETE messages for instance.

      Actually, mailagent statistics are triggered by a simple

        mailagent -s
    
      command, the letters given after in the {u,m,a,r,y} set being options
      that alter the output given by the command.

    * Look at the ~/.bak occasionally to make sure no error is pending...

------------------------------

From: Raphael Manfredi 
Subject:     3.3 How Safe Mailagent Processing Is?

As soon as the filter program has taken a hold on your message, you can rest
assured the mail will get filtered one way or the other. If filter can't
queue your mail, it will exit with an exit status of 75, that status
being recognized by "sendmail" as a "deliver later on" hint, in which case
the mail message will safely wait in sendmail's queue.

So if filter gets your message, it immediately forks and exits with a 0
status for sendmail, letting it know its work is finished and releasing it
to save resources. It then calls mailagent on the queued message (in mailagent's
private queue) to actually process the message.

Only after successful processing will mailagent delete the queued message.
Hence, under an heavily loaded system, the worst that could happen would be
a duplicate processing of a message, or a bounce back when sendmail cannot
fork and exec the filter program from your .forward.

Under catastrophic conditions, filter or mailagent will simply dump the message
on stdout, for ~/.bak to catch, preceded by the reason why processing was
aborted.

------------------------------

From: Raphael Manfredi 
Subject:     3.4 Locking Under Mailagent

By default, mailagent proceeds with a fixed locking scheme (.lock extension)
plus flock() if asked to do so at Configure time. However, mailagent provides
support for NFS-secure locks and also can use non-standard locking procedures,
configurable from within ~/.mailagent (variables "nfslock" and "mboxlock").

However, it cannot support locking on a rule basis (yet!). The author is
willing to raise the priority of that item if one comes up with a legitimate
need for that feature that could not be worked-around by a PERL escape. ;-)

------------------------------

From: Raphael Manfredi 
Subject:     3.5 Folder Types Supported

Mailagent can deliver mail to plain UNIX folders (also known as "mbox format"),
to MMDF folders, to MH folders (with unseen sequence update built-in and using
locks, not like rcvstore which does not!) or to directories (ala MH, but
without unseen sequence support and with alternate naming possible).

It also supports delivery to folders with the "x" bit set, in which case
mailagent interprets those as being hooks. It either pipes the message
to the "program" or further interprets the hook to do more processing.
See the manual page mailagent(1) under the section "MAIL HOOKS" for more
details.

------------------------------

From: Raphael Manfredi 
Subject:     3.6 Mailagent References

Man Pages   mailagent(1) - reference manual page (about 47 pages troff'ed).
            perl(1)      - reference manual for Perl
            sendmail(8)  - send mail over the Internet

Examples:   agent/examples/rules - a commented rule file sample, from the
                                   distribution source tree.

Newsgroup: comp.mail.misc

Mailing List: agent-users@foretune.co.jp

    Subscribe to the list by sending mail:

      To: majordomo@foretune.co.jp
      subscribe agent-users
    
Mailagent archives:

    ftp://ftp.foretune.co.jp/pub/network/mail/mailagent

    EMAIL: Send mail to the author's mailagent:
    
      To: ram@hptnos02.grenoble.hp.com
      Subject: Command
      @SH maildist - mailagent -

------------------------------

From: FAQ Editor 
Subject: 4.0 Filter

Filter is part of the Elm package of tools.  Note that you can use filter
to filter your incoming mail even if you are not using Elm to read your mail.

IMPORTANT NOTE
==============
If your system has both procmail and filter installed then you should use 
procmail which is *much* more robust and powerful than filter.  This 
recommendation is almost universal; even the developers of Elm and Filter
recommend procmail over filter.  IT IS POSSIBLE TO LOSE MAIL MESSAGES
WHEN USING FILTER; this is rare but it has happened.

------------------------------

From: FAQ Editor 
Subject:     4.1 Setting up Filter
Followup-To: comp.mail.elm

1] Find out if filter is on your system and what the full path to it is.  
   If you are using the C shell (csh) type:

     which filter

   Or, if you are using the Korn shell (ksh) or the Bourne shell (sh) type:

     type filter

   If neither ``which'' nor ``type'' are on your system try ``where'' and
   ``whereis''.  If your system doesn't have filter ask your system 
   administrator to install it; or even better ask her to install procmail.


2] Note the full path of your home directory by typing:

     cd
     pwd


3a] Create ~/.elm/filter-rules.  (Note that throughout this article I use
    pico for editing files.  Replace ``pico'' with your editor.)

     cd
     mkdir .elm
     cd .elm
     pico filter-rules

  
3b] Enter a modified version of the following in your ~/.elm/filter-rules

     if (subject contains "test") then save "/j/nancym/Mail/IN.testing"
   
   Replace /j/nancym with your home directory path (see step 2).
   Replace /Mail with the name of the directory where your mail folders are
   stored.  Pine and Berkeley Mail use /mail (lower case m) and Elm uses 
   /Mail (upper case M).
   

4] To see what the filter rule will do type the following at your Unix 
   prompt:  

     filter -r


5a] Create a ~/.forward file by typing the following. (Pico's -w flag tells 
    pico not to auto wrap lines.)

      cd
      pico -w .forward


5b] Enter a modified version of the following in your ~/.forward:


     "|/usr/local/bin/filter -o /j/nancym/.elm/filter-errors"


    == IMPORTANT NOTES ==
    * Make sure you include the quotes (").
    * The vertical bar (|) is a pipe.
    * Replace /usr/local/bin with the correct path for filter (see step 1).
    * Replace /j/nancym with your home directory (see step 2).
   
    * Do NOT expect environment variables, like $HOME, to work in your 
      .forward file.
    * Do NOT expect ~ to mean your home directory in the .forward file.


5c] Note that on many systems you need to make your .forward is world 
    readable and your home directory world searchable in order for the
    mail transport agent to "see" it.  To do this type:
   
      cd
      chmod 644 .forward
      chmod a+x .


6] Send yourself two test messages: one with ``test'' in the subject
   and one without ``test'' in the subject.


7a] Start your mailer (pine, elm, etc.) and check that the messages
    were delivered correctly.  The one with ``test'' in the subject
    should be in the folder IN.testing and the one without ``test'' 
    in the subject should be in your inbox.  If these were delivered 
    correctly go on to step 8.


7b] TROUBLESHOOTING
    * If the two messages were not delivered correctly look at your
      ~/.elm/filter-errors to see if you can determine what the problem 
      is.  

    * Check these two files for typos:
        ~/.forward
        ~/.elm/filter-rules

    * Check the file and directory permissions of your .forward (set in 
      5c above).

      Type...          In order to...
      -------          --------------
      cd               Go to your home directory.
      
      ls -l .forward   Check the permission: it should say -rw-r--r--
      
      ls -ld .         Check permission of home dir: it should say drwx?-x?-x
                       The ?'s may be r's or hyphens or one of each (i.e.,
                       drwx--x--x, drwxr-xr-x, drwxr-x--x, drwx--xr-x).
      

    * If none of these turn up the problem edit your ~/.forward so
      that filter will be verbose with it's output (use the -vo flag).

        "|/usr/local/bin/filter -vo /j/nancym/.elm/filter-errors"

      And repeat steps 6 and 7.  After you get filter to work you 
      will probably want to change the ``-vo'' flag back to ``-o''.

   
8]  After you have successfully tested filter in steps 6 and 7, edit
    ~/.elm/filter-rules so that it contains a modified version of the
    following:

     # if (subject contains "test") then save "/j/nancym/Mail/IN.testing"
     if (to contains "www-talk") then save "/j/nancym/Mail/IN.www-talk"
     if (to contains "hopos-l") then save "/j/nancym/Mail/IN.hopos"

     Replace /j/nancym with your home directory path and
     /Mail with the name of your mail directory.  Replace the mailing 
     list string (e.g., "www-talk") and the name of the mail folder
     (e.g., IN.www-talk) with text for your mailing lists.

     Note that ``to contains...'' means either the To or Cc header contains...

     Rather than deleting the test line, it's useful to just turn it
     into a comment (by preceding it with #) so that you can easily
     use it for future testing.

------------------------------

From: FAQ Editor 
Subject:     4.2 Tracking Your Incoming Mail
Followup-To: comp.mail.elm

You can get a short summary of filter's activity by typing:

    filter -s

For a longer summary type:

    filter -S

Or you can look at the log file itself, ~/.elm/filterlog.

You should regularly look at ~/.elm/filter-errors to make sure things 
are working.  You can automatically check filter-errors each time you
log in by putting the following in your .login:

    tail ~/.elm/filter-errors

Also you probably want to regularly delete filterlog and filter-errors so 
they don't fill up your disk space.  To get a summary of the filter log and 
clear it type:

    filter -cs

------------------------------

From: FAQ Editor 
Subject:     4.3 Filter References
Followup-To: comp.mail.elm

Web Pages: http://www.myxa.com/elm.html
           http://www.math.fu-berlin.de/~guckes/elm/elm.filter.html
Man Pages: filter(1)
Newsgroup: comp.mail.elm

------------------------------

From: FAQ Editor 
Subject: 5.0 Mailer and Newsreader References

PINE
====
Web Pages: http://www.cac.washington.edu/pine/
           http://www.jazzie.com/ii/internet/pine/
           http://www.best.com/~ii/internet/pine/ (mirror of above URL)
Man Pages: pine(1), pico(1), pilot(1)
Newsgroup: comp.mail.pine (linked to Pine mailing list)
Mailing List: pine-info@cac.washington.edu (linked to Pine newsgroup)
           Subscribe to the pine-info mailing list by sending mail to:
                     majordomo@cac.washington.edu
           With...
                     subscribe pine-info
           in the body of the message.

ELM
===
Web Pages: http://www.myxa.com/elm.html
           http://www.math.fu-berlin.de/~guckes/elm/
Man Pages: elm(1)
Newsgroup: comp.mail.elm

Emacs Mail Mode
===============
Newsgroups: gnu.emacs.help and comp.emacs

MH
==
Man Pages: mh(1)
Newsgroup: comp.mail.mh

MAIL
====
Man Pages: mail(1)
Newsgroup: comp.mail.misc

NN
==
Web Pages: http://www.jazzie.com/ii/internet/nn/
           http://www.best.com/~ii/internet/nn/ (mirror of above URL)
Man Pages: nn(1)
Newsgroup: news.software.nn


Please send me pointers to other mailer and newsreader references
and let me know what newsreaders can read mail folders.

------------------------------

From: FAQ Editor 
Subject: 6.0 Contributors

This FAQ, like many others, is a collaborative effort.  I learned a
lot of the information in newsgroups and mailing lists, especially:
  comp.mail.*
  procmail mailing list

Also, lots of people have mailed me information and I've tried to
acknowledge them below.

------------------------------

From: FAQ Editor 
Subject:     6.1 Acknowledgements

Thanks to these people who sent suggestions:
   David L. Miller 
   Cookie Monster 
   Jim Showalter 
   David W. Tamkin 
   Rick Troxel 
   Stephen R. van den Berg 
   Syd Weinstein 

Special thanks to:
   Thomas A. Fine  for setting up and
   maintaining the hypertext archive of FAQs.  Congratulations
   to him for winning O'Reilly and Associates' "The Best of the
   Net" award!

Please let me know if I've left you, or anyone else, out of this list.

------------------------------

From: FAQ Editor 
Subject:     6.2 Contributing to this FAQ

If you have any corrections, suggestions, or new digest items to
contribute to this FAQ please send them to faq-editor@ii.com.  If your
reader understands the following URL, you can use it to send me mail:

  mailto:faq-editor@ii.com.  

Things I'm especially interested in are:
 * a comparison of mail filtering tools
 * what newsreaders can read mail folders
 * instructions for opening a specific folder in various readers
 * lock scheme and files used by various readers
 * other Unix filtering tools, like deliver
 * filtering tools for other operating systems like the Mac and DOS/Windows

------------------------------

From: FAQ Editor 
Subject: 7.0 Copyright Notice

This FAQ is Copyright (c) 1994-96 Nancy McGough, except sections 2.2.2 and
3.0 through 3.6 which are Copyright (c) 1994-96 by the authors named in
those sections. 

No portion of this work may be sold or put to commercial use without
express written consent of the authors.  This restriction covers
publication in any form, or distribution by any method, which permits this
work to be visually perceived, either directly or with the aid of any
machine or device.  Permission is granted to republish or redistribute
this article in its entirety for noncommercial use if this copyright
notice is not removed or altered.  For more information, see Infinite
Ink's Legal Notice, which is available at: 

  http://www.jazzie.com/ii/legalese/
  http://www.best.com/~ii/legalese/  (mirror of above URL)
 

End of Filtering Mail FAQ Digest
********************************

-- 
<*<*<*<*<*<*<*<*<*<*<*<*<*<*<*<*<*<*<*<*<*<*<*<*<*<*<*<*<*<*<*<*<*<*<
            @..@            Nancy McGough           /\_/\
           (----)           Infinite Ink           ( o.o )
          ( >__< )    http://www.jazzie.com/ii/     > ~ <