This tutorial discusses advanced topics concerning e-mail on SDF: spam filtering, automatic processing, forwarding. It is probably only useful for ARPA members.
Here we discuss advanced topics for e-mail processing on SDF, mainly
automatic processing of incoming messages with procmail
, and
forwarding issues. As procmail
is only available to ARPA
members, the contents of this tutorial will be mostly useless to user
members.
For basic info about e-mail (addresses, reader programs, file size, etc),
please read the corresponding
FAQ entry - and if
you still have problems with basic sending and receiving of messages
via SDF, this tutorial is most probably not for you!
The sendmail
system allows for automatic forwarding of
incoming mail to other addresses or even programs (filters).
The file $HOME/.forward
may contain the following types of
lines:
filterprogram
/bin/sh
with the
argument filterprogram
However, if you want to do more than forwarding to one address,
use of procmail
is highly recommended! In fact, it is
best to set up nospam -e, which will generate
a .forward file with a line calling procmail, and then to modify
the resulting .procmailrc
.
Use fetchmail
to collect messages from accounts
you might have elsewhere. The messages are given to procmail
(see Forwarding E-Mail above) for further
processing.
.fetchmailrc
poll pop.provider.net proto pop3 user "john.doe" pass "secret" is "jdoe" here mda "/path/to/procmail -f- ~/.procmailrc" ssl;
To fight unwanted bulk e-mail (spam), there are several possibilities available on SDF:
procmail
(see man
procmail
for more information)
allows to filter incoming mail following user defined rules; in particular,
you can generate a white list of sender addresses
you know as good, and put messages from other addresses into a special
folder which you browse through now and then, to check for good messages
and adding the corresponding addresses to your white list.nospam
is a script which can set up .forward and .procmailrc
files so that certain e-mails will be stored in "bulk folders".
nospam -i
gives some additional information, and nospam
-e
will enable this filtering. We will suppose that this has been done
also for the following discussion of white
lists.
White lists contain e-mail addresses which are always
to be accepted as good. It is easy to implement this with
procmail
: Supposing that nospam -e
has been
executed, so that basic procmail processing is set up, one can add the
following fragment to .procmailrc
.
Note: For this to be working, the following must have been set up:
.forward
file which pipes incoming e-mail through
procmail
.procmailrc
file, where this fragment is attached
or insertednospam -e
as mentioned above, then adding
the fragment to the generated .procmailrc
.
# procmail spamfilter using whitelist (2006 Yargo Bonetti) # :: use at your own risk and any way you want! :: # whitelist: file containing one valid e-mail address per line # (only generic xxx@yyy.zz form, without "Name.." <*> parts) WHITELIST=$MAILDIR/.whitelist # spamfilter FROM=`formail -c -x 'From:'|sed -e 's/.*<\(.*@[^>]*\)>.*/\1/'` :0 hb: * ! ? grep -i -F -e "$FROM" $WHITELIST >/dev/null $MAILDIR/quarantine
This will compare the address in the From: field of incoming messages to the ones saved in the file $MAILDIR/.whitelist (of course, name and location can be changed), and when an address is not found in this file, the message will be saved in the folder $MAILDIR/quarantine (which can be modified as well) and not show up in the normal Inbox (and not be processed further). Now and then, one can look through the quarantine folder for "good messages", and add the corresponding addresses to the .whitelist file.
If you want to use the mail directories proposed by nospam, you may prepend the above noted fragment by
MAILDIR=$HOME/mail
which will result in storing the "possibly bad" messages in
~/mail/quarantine
and expect the whitelist in
~/mail/.whitelist
.
A sample .whitelist may look like this:
my.friend@his.domain.com president@whitehouse.gov myself@gmail.com
All messages not coming from these three addresses would end up in the "quarantine" folder defined in the .procmailrc fragment.
DNSBL stands for "DNS blacklist", and it is essentially a collection of IP addresses known to send out spam and other bad stuff. Two popular DNSBLs are SpamCop and Spamhaus. Typically the mail server will check the source of incoming mail with the DNSBL before even accepting the message. SDF does not do this, but using Procmail to check a DNSBL at the user level is easy. In this tutorial we will use Spamhaus.
(Thank you Benya for the original instructions.)
First make sure that ~/.forward
is set-up to forward incoming
mail to Procmail. Then we'll edit ~/.procmailrc
. The beginning
of this file should declare basic variables such as SHELL=/bin/sh
,
LOGFILE=$HOME/.procmaillog
, ORGMAIL
,
DEFAULT
, and MAILDIR
. ORGMAIL
and
DEFAULT
should point to your inbox, and MAILDIR
should point to the directory that contains your saved mail.
Next Procmail should extract the IP address from which the message was
sent. This is done using formail
to get the headers,
grep
to find the correct line, and sed
to find the
actual IP address, which is then saved as SENDERIP
.
SENDERIP = `formail -c -XReceived | grep "by sdf.lonestar.org" | \ grep -v "from sdf.lonestar.org" | \ sed "s/^Received: from .*\[\([0-9]*\.[0-9]*\.[0-9]*\.[0-9]*\)\].*by sdf.lonestar.org.*$/\1/"`
Then we'll begin the actual Procmail recipe, which will only be executed
after checking to make sure that the SENDERIP
variable exists
and is in the correct format. Anything written to the LOG
variable
will be inserted into the log file. (New lines must be explicitly stated.)
:0 * SENDERIP ?? ^^[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*^^ { LOG = "The sender's IP address is " LOG = $SENDERIP LOG = " "
The next step is to reverse the IP address because Spamhaus wants it that
way. Procmail will then use host
to determine the IP address that
[reversed IP address].zen.spamhaus.org resolves to.
SENDER_REVERSED = `expr "$SENDERIP" | \ sed "s/\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\)/\4.\3.\2.\1/"` KNOWNOFFENDER = `host "$SENDER_REVERSED".zen.spamhaus.org | \ sed "s/^.*\(127\.0\.0\.[0-9]*\)$/\1/"`
If Spamhaus returns an IP address between 127.0.0.0 and 127.0.0.9, then
we know that this particular email is of dubious origin. In that case we can
put it in the folder $MAILDIR/Spam
. Otherwise, the message will
get written to your inbox by default.
:0 * KNOWNOFFENDER ?? ^^127.0.0.[0-9]*^^ { LOG = "This sender is a known source of spam. " :0: Spam } LOG = "This sender is not a known source of spam. " }
This completes the Procmail recipe.
Using SSH's port-forwarding capabilities, MetaARPA members can create an encrypted tunnel which makes the SDF SMTP mail server appear local, avoiding need for authentication or additional encryption.
Port-forwarding can occur on ports 53, 110, 143, 443, and 993 on most SDF hosts.
ex:
Use port 443 on SDF host "otaku", tunnel local port 2525 to port 25 on SDF host "mx" (SMTP server); restrict
ssh to IPv4, no remote commands, run in background after connecting:% ssh -4 -fN -p 443 -L 2525:mx:25 sdf_user@otaku.sdf.org
Note if public key authentication isn't setup a password prompt occurs.
telnet(1) can be used; type "quit" to exit session:
% telnet localhost 2525 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. 220 sdf.lonestar.org ESMTP Sendmail 8.14.5/8.14.3; Tue 13 Dec 2011 07:45:59 GMT quit 221 2.0.0 sdf.lonestar.org closing connection Connection closed by foreign host.
Configure MUA to use "127.0.0.1" (localhost), port 2525 for out-going messages. Also ensure the From: and ReplyTo: addresses are routeable. Handling of rejected email is MUA-dependent - some do queuing, others write to $HOME/dead.letter or /dev/null:
ex:
Configure and test the Heirloom Mailx MUA for user "frog" on localhost "mud.bog"
to use "localhost:2525" for SMTP (off-site email):# $HOME/.nailrcset smtp="localhost:2525" set from="You <someone@sdf.org>" set replyto="You <someone@sdf.org>"
% hmail -v someone@sdf.org Subject: tunneled SMTP test test 123 . Resolving host localhost . . . done. Connecting to 127.0.0.1:2525 . . . connected. 220 sdf.lonestar.org ESMTP Sendmail 8.14.5/8.14.3; Tue, 13 Dec 2011 08:21:22 GMT >>> HELO mud.bog 250 sdf.lonestar.org Hello IDENT:root@otaku.sdf.org [192.94.73.6], pleased to meet you >>> MAIL FROM:250 2.1.0 ... Sender ok >>> RCPT TO: 250 2.1.5 ... Recipient ok >>> DATA 354 Enter mail, end with "." on a line by itself >>> . 250 2.0.0 pBD8LM7d000515 Message accepted for delivery >>> QUIT 221 2.0.0 sdf.lonestar.org closing connection
The SSH tunnel can be torn down by identifying the appropriate PID (process ID) and sending it a kill signal, via the kill(1) command or similar tool. The following illustrates the teardown process using sockstat(1) to determine the PID on the NetBSD operating system:
% sockstat -np 2525 USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS frog ssh 24362 7 tcp 127.0.0.1.2525 *.* % kill -HUP 24362 Killed by signal 1.
Network connections can fail for various reasons; a keep-alive script can help:
(script contributed by SDF user nullogic - thanks!)
#!/bin/bash # sdftun.sh: Keep alive script for local ssh tunnel from home to SDF SMTP SSH_USER= SSH_HOST='faeroes.freeshell.org' SSH_PORT=443 SSH_ARGS='-N -f -L' TAG_HOST=mx TAG_PORT=25 LIS_PORT=8080 LOG_FILE=~/l0g/sdftun.log SLEEP="30m" while [ 1 ]; do SSH_PID=`ps aux | sed -n -e /sed/d -e /ssh\ \-N/p | wc -m` if [ $SSH_PID -eq 0 ]; then ssh $SSH_ARGS $LIS_PORT:$TAG_HOST:$TAG_PORT $SSH_USER@$SSH_HOST \ -p$SSH_PORT &>/dev/null DATE_TIME=`date '+%y.%m.%d %T'` echo "$DATE_TIME SSH Tunnel restarted.">>$LOG_FILE fi sleep $SLEEP done
Writing the PID to a file at the time of tunnel creation can simplify the ID process.