The basic idea behind SPF is that a domain should specify (in DNS) precisely which servers are authorized to send its email. Thus, an email claiming to be from a particular domain (such as ebay.com) but not originating from one of that domain's approved servers is considered a forgery. This works well in most circumstances, but its drawback is it breaks a widely used feature of email called forwarding, and not just for SPF users, but for anyone contacting an SPF-using domain.

People often have email addresses that are merely forwarding addresses such that all mail sent to that address is resent to another address. On virtually every email server prior to the invention of SPF, forwarded email was delivered unmodified. The problem with this approach has been illustrated with the help of the following figure:

Imagine that the address [email protected] is a forwarding address that forwards to [email protected]. If a message is sent from [email protected] to [email protected], the example2.net server then sends the message to the example3.org server. From the example3.org server's perspective, the example2.net server is sending email claiming to be from example.com. If example.com happens to publish an SPF record asserting that its email only originates from its own servers (which it did), and if example3.org blocks messages based on SPF information, it will refuse the [email protected] message forwarded by the example2.net server. This thwarts a viable (and popular) method of managing email accounts. The SPF workaround is to re-encode email return addresses so that each server only sends from its own domain. For example, example2.net could re-encode the return address to look like [email protected], so the message does not appear to be claiming to be from example.com. If the message bounced back from example3.org for some reason, example2.net would receive it and could then decode the return address and bounce it back to the original sender, [email protected]. The problem with this solution is that a spammer could convince the example2.net server to relay spam by sending spam to [email protected] or a similar address. To prevent such abuse, each server will need to remember every email it forwarded for every user, so that it can refuse to relay bounces (or spam that looks like a bounce) that are for messages it didn't send. To be more to the point, each server will have to remember all of these messages until it can be certain that these emails will not bounce, which is an unpredictable amount of time. For example, the example2.net server cannot know whether the example3.org server is the message's final stop or whether it will be forwarded again and then bounce—keep in mind that a message can sit in a server's queue for a long time (typically up to seven days) before it is bounced.

The DomainKeys concept is to cryptographically sign all messages with a public key encryption system, where the signature asserts that a server approved by the domain has sent the email. The information necessary to verify the validity of the signature is published in DNS, so that any server may validate the signature and prove that the message was signed by a server approved by that domain to send email. Thus, for a domain that asserts (in a DomainKeys policy record stored in DNS) that it signs all of its email, a message missing a signature or containing an incorrect signature is considered a forgery. Unlike SPF, this allows emails to be forwarded without being modified. The caveat is that DomainKeys requires that messages not be modified (between the DomainKeys signature and the end of the message). Thus, while Received headers and any other headers prepended to the message are acceptable, other modifications—such as spam-filter headers or signatures appended to either the end of the message or the end of the headers—can invalidate the DomainKeys signature. The DomainKeys specification includes the ability to name, in the signature (in an h= tag), what headers were originally included in the signature. This makes the signature much more tolerant of header modification, as additional headers (such as spam-filter headers) inserted in the wrong place in the message can be excluded from the signature. However, the body of the message and the headers that were included in the signature cannot change without invalidating the signature.

Russ Nelson has contributed significantly to qmail's ability to verify and sign messages with DomainKeys. Most significantly, he is the original author of the libdomainkeys library (http://domainkeys.sourceforge.net), which provides many ways of creating and validating DomainKeys signatures.

Russ Nelson also wrote a patch to the qmail source that creates a new component, qmail-dk (http://www.qmail.org/qmail-1.03-dk-0.53.patch), that can be used as a wrapper around qmail-queue for signing and verifying DomainKeys signatures. This program is convenient, but not always necessary. The libdomainkeys package includes a utility called dktest that can, with the appropriate shell-script glue, provide some of the basic features that qmail-dk provides. For example, to verify DomainKeys signatures in incoming email and add a header proclaiming the result, a script like the following will work:

#!/bin/sh
[ "$DKQUEUE" ] || DKQUEUE=/var/qmail/bin/qmail-queue
if printenv | grep -q '^DKVERIFY=' ; then
tmp=`mktemp -t dk.verify.XXXXXXXXXXX`
cat - >"$tmp"
(dktest -v <"$tmp" 2>/dev/null | awk 'NR>1' ; 
cat "$tmp" ) | "$DKQUEUE"
retval=$?
rm "$tmp"
exit $retval
else
exec "$DKQUEUE"
fi

This script can, in conjunction with the QMAILQUEUE patch (or as a qmail-queue replacement, provided that the real qmail-queue can still be used with a different name) tag each incoming email with a "DomainKey-Status" header, recording the results of checking the DomainKeys signature. The DKQUEUE environment variable is consulted for the location of the qmail-queue binary, and the DKVERIFY environment variable is used to enable verification.

The qmail-dk program serves two purposes: it verifies signatures (like the above script), and it creates signatures. Because qmail-dk is a qmail-queue wrapper, it must decide whether to sign a message when the message is queued. In order to sign all messages, every part of the system that queues messages must use it—from qmail-smtpd to qmail-inject to qmail-local (which queues messages in response to delivery instructions). Unfortunately, qmail-dk does not support the inclusion of an h= tag in the signature.

An alternative to wrapping qmail-queue with something like qmail-dk is to wrap qmail-remote. This signs all remotely delivered messages, no matter how they were enqueued. While qmail-dk does not serve easily as a qmail-remote wrapper, the following script (using dktest) is a suitable option.

This script, which supports the h= tag, presumes that its name is qmail-remote and that the original qmail-remote was renamed qmail-remote.orig:

#!/bin/bash
[ "$DKSIGN" ] || DKSIGN="/etc/domainkeys/%/default"
[ "$DKREMOTE" ] || DKREMOTE=/var/qmail/bin/qmail-remote.orig
if [[ $DKSIGN == *%* ]] ; then
DOMAIN="${DOMAIN:-${2##*@}}"
DKSIGN="${DKSIGN%%%*}${DOMAIN}${DKSIGN#*%}"
fi
if [ -f "$DKSIGN" ] ; then
tmp=$( mktemp -t dk.sign.XXXXXXXXXXX )
cat - >"$tmp"
(dktest -s "$DKSIGN" -c nofws -h <"$tmp" 2>/dev/null | 
sed 's/; d=.*;/; d='"$DOMAIN"';/'; 
cat "$tmp" ) | 
"$DKREMOTE" "$@"
retval=$?
rm "$tmp"
exit $retval
else
exec "$DKREMOTE" "$@"
fi

With this script, the environment variable DKSIGN specifies the location of the private key for the signing process, DKREMOTE specifies the location of the original qmail-remote program, and finally, DOMAIN specifies the sending domain (if it is unspecified, the script will assume the domain is the sender's domain). More conveniently, just like qmail-dk, DKSIGN can contain a percent sign (%) that is replaced by the signing domain name, thus making signing for multiple domains with separate keys more convenient.

Lightweight methods for identifying spam are considered lightweight because they do not rely on complex analysis of email in order to make decisions, and the decisions are almost always binary: each email is accepted or rejected, with no gray area in between. How this decision is reached generally depends on how much information is available when the decision is made. In some cases, email is considered spam before the sender has sent even a single character of the email. If it is blocked at that point, bandwidth is not wasted in receiving the message. The methods described here are not an exhaustive list, but merely a set of popular examples.

While lightweight spam identification techniques are often effective, their simplicity leaves them vulnerable to avoidance or compensation techniques implemented by spammers. Heavyweight spam identification, on the other hand, is designed to be more robust and to adapt as spammers change their tactics.

Perhaps the most involved filters are the kind that hold email messages hostage while a human decides whether they are spam (and should be deleted) or are ham (and should be delivered). These types of filters fall into two categories: quarantines and challenges. With a quarantine the recipient decides whether a message is spam or not, while with a challenge, the sender must certify a message is not spam. The more common of the two is quarantine, usually combined with heavyweight scoring. Because heavyweight identification typically places email somewhere on a continuum between spam and ham, messages not considered ham are often delivered to a special folder for holding spam. This folder is a quarantine where messages are examined by the recipient, and mistaken categorizations can be identified.

Challenge-based spam identification is less popular, but also effective. One of the most popular forms of challenge-based spam identification is called Tagged Message Delivery Agent (TMDA). When a message is received, it is stored in a holding area while a challenge message is sent back to the sender. The challenge message contains instructions to be followed by the sender so as to identify himself or herself as a non-spammer, such as visiting a message-specific URL or replying to a message-specific email address. When the sender's validating action is performed, the original message is delivered, and in some cases the sender is then added to a list of known good senders so that no challenges are made in the future. This technique is effective against spam, because it requires a valid return address in addition to some of the spammer's precious time. On a usual spammer scale, where millions of messages are sent in an average hour, responding to the TMDA message is not worth the effort. On the other hand, TMDA challenges are frequently effective against ordinary busy people as well. Some people consider challenges to prove that they are not spammers rude or, at the very least, a waste of time. Consider, for example, a person asking an expert a question. The expert might wish to respond, but balk at spending time verifying his non-spammer status. Of course, such situations are easily avoided through judicious use of white lists, but TMDA and other challenge-based methods make such problems easy to overlook.

One of the most critical things to consider when evaluating any anti-spam technique is the problem of mistakes. No technique is perfect, and the questions to answer when evaluating a technique include:

In the lightweight categories, detecting false positives is problematic because messages identified as spam are often prevented from being delivered. For example, if a domain is mistakenly listed in a DNSBL, all email communication with that domain is cut off, making it difficult for users of that domain to complain that their messages are not being delivered. Often, the only way for domains using the DNSBL to discover the error is for someone from the blocked domain to inform their intended recipient of the problem via a non-email method. This is, of course, only possible if all affected parties have both the desire and ability to use a non-email method of contact. Rather than blocking email that is considered spam, the alternative is to tag it with an identifying header. Email so tagged is delivered to a special spam folder, quarantine area, or something similar. This allows spam-like email to be reviewed for mistakes, but does not take advantage of the primary purpose of lightweight filters: to make quick, final decisions and avoid spending time and resources analyzing each email in depth. Thus tagging email only makes sense when using a heavyweight filter.

Whether mistakes can be fixed is another issue. With learning filters, such as Bayesian classifiers, correcting mistakes and thus improving accuracy is fundamental to the filter's design. On the other hand, informing the filter of its mistakes is frequently inconvenient, particularly with a large group of users who use different mail clients. Some filters, like DSPAM, provide a full web interface for submitting incorrectly tagged email messages, while others rely on command-line access to the mail server or some concoction by the system administrator. A good way to handle such tools in an IMAP-based environment is by creating magic folders that will—either by some server-based hook, cron, or some other method—cause the spam identification system to re-interpret messages placed in them.

On the other hand, correcting errors in a DNSBL or a spam database ranges anywhere from impossible to difficult, depending on the philosophy of those who maintain it. It is often necessary to use a local hand-written white list to avoid or countermand known misidentifications in such lightweight spam prevention techniques.

A partial solution to the problem of bounce-back spam is to modify qmail so that it will only accept messages addressed to valid recipients. For example, a modified (or wrapped) qmail can check whether a given user exists in the recipient domain rather than simply checking whether the recipient domain is listed in the control/rcpthosts file. Checking whether a user exists can be a complicated task. For example, mailing lists often use temporary addresses for administrative tasks (like handling user subscription requests) and virtual domains can have external user databases that qmail cannot access easily (such as vpopmail or GNU Mailman domains). As such, there are many different patches to qmail that use different methods of determining whether a user exists or not. Some of the most popular are:

This task does not absolutely require patching qmail; a qmail-queue wrapper can also perform it. Using a wrapper rather than a patch in this case has the drawback that the entire message must be received before the qmail-queue wrapper is triggered. A patch can check the recipients' validity as the sender lists them. If the recipients are invalid, rejecting them earlier saves bandwidth.

None of these methods is a full solution to the problem of bounce-back spam, because none of them can guarantee that a given message is deliverable.

User validation is sufficient to prevent bounces in many cases, but some common examples where bounce-back spam cannot be prevented include:

Though bounce-back spam can never be fully eliminated, validating recipients still dramatically reduces the problem.

One of the downsides of validating recipients, however, is it allows spammers to quickly discover (via the guess-and-check method) what users are valid on such a system, and in future the spammer can direct spam to only those recipients. While this is technically possible, in practice, it is uncommon that spammers spend the time necessary to track the success of each address, because obtaining valid addresses to spam can be achieved with much less time-consuming methods.