E-Mail, Advanced Topics

This tutorial discusses advanced topics concerning e-mail on SDF: spam filtering, automatic processing, forwarding. It is probably only useful for ARPA members.

Contents

1. Introduction, and what this is not
2. Forwarding E-Mail
3. Fetching E-Mail
4. Fighting Spam
5. Personal White List Against Spam
6. Using Procmail With a DNSBL

Introduction, and what you will not find here

Here we discuss advanced topics for e-mail processing on SDF, mainly automatic processing of incoming messages with procmail, and forwarding issues. As procmail is only available to ARPA members, the contents of this tutorial will be mostly useless to user members.
For basic info about e-mail (addresses, reader programs, file size, etc), please read the corresponding FAQ entry - and if you still have problems with basic sending and receiving of messages via SDF, this tutorial is most probably not for you!

Forwarding E-Mail

However, if you want to do more than forwarding to one address, use of procmail is highly recommended! In fact, it is best to set up nospam -e, which will generate a .forward file with a line calling procmail, and then to modify the resulting .procmailrc.

Fetching E-Mail

Use fetchmail to collect messages from accounts you might have elsewhere. The messages are given to procmail (see Forwarding E-Mail above) for further processing.

Sample .fetchmailrc

poll pop.provider.net
     proto pop3
     user "john.doe"
     pass "secret"
     is "jdoe" here
     mda "/path/to/procmail -f- ~/.procmailrc" 
     ssl;

Fighting Spam

1. As a general remark, think before you publish or tell somebody your e-mail address! And try to make clear to people that you don't want your address on a visible list of 500 recipients from somebody's address book; one should use Bcc: for sending to a lot of people in parallel, not To:!
2. SDF also supports plus-addressing; you can use this to join mailing lists and do filters. This can be used by adding +something before the @ in your email address.
3. The FAQ has more information about nospam, available on SDF.
4. The automatic mail processor procmail (see man procmail for more information) allows to filter incoming mail following user defined rules; in particular, you can generate a white list of sender addresses you know as good, and put messages from other addresses into a special folder which you browse through now and then, to check for good messages and adding the corresponding addresses to your white list.

nospam

nospam is a script which can set up .forward and .procmailrc files so that certain e-mails will be stored in "bulk folders".
nospam -i gives some additional information, and nospam -e will enable this filtering. We will suppose that this has been done also for the following discussion of white lists.

Personal White List Against Spam

# procmail spamfilter using whitelist (2006 Yargo Bonetti)
# :: use at your own risk and any way you want! ::
# whitelist: file containing one valid e-mail address per line
# (only generic xxx@yyy.zz form, without "Name.." <*> parts)
WHITELIST=$MAILDIR/.whitelist
# spamfilter
FROM=`formail -c -x 'From:'|sed -e 's/.*<\(.*@[^>]*\)>.*/\1/'`
:0 hb:
* ! ? grep -i -F -e "$FROM" $WHITELIST >/dev/null
$MAILDIR/quarantine

This will compare the address in the From: field of incoming messages to the ones saved in the file $MAILDIR/.whitelist (of course, name and location can be changed), and when an address is not found in this file, the message will be saved in the folder $MAILDIR/quarantine (which can be modified as well) and not show up in the normal Inbox (and not be processed further). Now and then, one can look through the quarantine folder for "good messages", and add the corresponding addresses to the .whitelist file.

If you want to use the mail directories proposed by nospam, you may prepend the above noted fragment by

MAILDIR=$HOME/mail

which will result in storing the "possibly bad" messages in ~/mail/quarantine and expect the whitelist in ~/mail/.whitelist.

A sample .whitelist may look like this:

my.friend@his.domain.com
president@whitehouse.gov
myself@gmail.com

All messages not coming from these three addresses would end up in the "quarantine" folder defined in the .procmailrc fragment.

Using Procmail With a DNSBL

DNSBL stands for "DNS blacklist", and it is essentially a collection of IP addresses known to send out spam and other bad stuff. Two popular DNSBLs are SpamCop and Spamhaus. Typically the mail server will check the source of incoming mail with the DNSBL before even accepting the message. SDF does not do this, but using Procmail to check a DNSBL at the user level is easy. In this tutorial we will use Spamhaus.

(Thank you Benya for the original instructions.)

First make sure that ~/.forward is set-up to forward incoming mail to Procmail. Then we'll edit ~/.procmailrc. The beginning of this file should declare basic variables such as SHELL=/bin/sh, LOGFILE=$HOME/.procmaillog, ORGMAIL, DEFAULT, and MAILDIR. ORGMAIL and DEFAULT should point to your inbox, and MAILDIR should point to the directory that contains your saved mail.

Next Procmail should extract the IP address from which the message was sent. This is done using formail to get the headers, grep to find the correct line, and sed to find the actual IP address, which is then saved as SENDERIP.

SENDERIP = `formail -c -XReceived | grep "by sdf.lonestar.org" | \
grep -v "from sdf.lonestar.org" | \
sed "s/^Received: from .*\[\([0-9]*\.[0-9]*\.[0-9]*\.[0-9]*\)\].*by sdf.lonestar.org.*$/\1/"`

Then we'll begin the actual Procmail recipe, which will only be executed after checking to make sure that the SENDERIP variable exists and is in the correct format. Anything written to the LOG variable will be inserted into the log file. (New lines must be explicitly stated.)

:0
* SENDERIP ?? ^^[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*^^
{
	LOG = "The sender's IP address is "
	LOG = $SENDERIP
	LOG = "
"

The next step is to reverse the IP address because Spamhaus wants it that way. Procmail will then use host to determine the IP address that [reversed IP address].zen.spamhaus.org resolves to.

	SENDER_REVERSED = `expr "$SENDERIP" | \
	sed "s/\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\)/\4.\3.\2.\1/"`

	KNOWNOFFENDER = `host "$SENDER_REVERSED".zen.spamhaus.org | \
	sed "s/^.*\(127\.0\.0\.[0-9]*\)$/\1/"`

If Spamhaus returns an IP address between 127.0.0.0 and 127.0.0.9, then we know that this particular email is of dubious origin. In that case we can put it in the folder $MAILDIR/Spam. Otherwise, the message will get written to your inbox by default.

	:0
	* KNOWNOFFENDER ?? ^^127.0.0.[0-9]*^^
	{
		LOG = "This sender is a known source of spam.
"

		:0:
		Spam
	}

	LOG = "This sender is not a known source of spam.
"
}

This completes the Procmail recipe.