The foundation of knowledge presented so far in this book is enough to provide some intuition about how to get started with implementing most desired email server features. To provide some examples for building upon this foundation, this chapter explores SSL encryption (also known as TLS) and efficient mailing list implementations. There is, of course, a nearly unlimited set of further topics and features that could be discussed, but these two are a good start.
The Internet is extremely powerful and flexible because of the way it works, although people are frequently surprised by how it works. When information (like a bit of text, a picture, or an email) is sent across the network, the sending computer puts the information into a packet, or series of packets, and hands them to a computer closer to the destination computer. In the end, this process resembles taking a postcard and handing it to someone else to be delivered. The person receiving the postcard looks at the address and hands it to someone else who is a little bit closer to the addressee. Computer networks are somewhat more formal and have a better sense (usually) of who the next-closest computer is, but the process is essentially the same. It is common for seventeen or so computers to handle a packet before it reaches its destination. The text of the postcard, or content of the packet, is available for anyone to read, if they so choose. Most computers do not examine the contents of the packets they relay, because they have other things to do, but there is nothing preventing them from doing so.
The entire scope of email content transmitted over the Internet this way—everything from stock tips to love letters to bank passwords—is exposed to unauthorized interception as it crosses networks, from machine to machine. In other words, things not everyone should read are transmitted in a way that anyone in the middle is able to intercept. In the case of regular mail, the solution is to forgo the use of postcards for anything private and instead write a letter secured in an envelope. In the world of computers, the solution is to use encryption.
There are many forms of encryption available for different purposes, but the basic concept is common to all of them: encryption is designed to guarantee only those who should read something can read it. For email, there are many ways to use encryption. When discussing email servers, encryption usually involves encrypting communication between SMTP servers using Secure Sockets Layer (SSL) encryption or Transport Layer Security (TLS) encryption. The SSL acronym is a reference to network connections in UNIX, which are called sockets. The difference between SSL and TLS is somewhat blurry because they both use the same encryption technology and are occasionally used interchangeably. To be more precise, however, the two terms are used to distinguish between encryption that starts at the same time as the network connection (SSL) and encryption that begins after the connection is established (TLS). Because the encryption must begin as the network connection is established, SSL encryption generally requires a distinct network port but does not change the underlying communication protocol. For example, SMTPS (the SSL version of SMTP) uses port 465 rather than the SMTP-standard port 25. TLS encryption, on the other hand, is an extension to the protocol implemented in a backwards-compatible way. In the case of SMTP, TLS encryption adds a single command, STARTTLS, to the protocol that directs the server to begin encrypting the network conversation. As such, it can be added to an SMTP server without requiring the server's clients to be aware of the change.
Qmail does not support SSL or TLS by default but this feature can be added to qmail in several ways, depending on when and where encryption is used. There are two primary opportunities for using encryption: when receiving email and when sending email. As is usually the case with the qmail architecture, there are two ways of implementing encryption: with a patch and with a wrapper.
The difference between using a patch and using a wrapper is based on their fundamental design difference: one modifies qmail (a patch), one uses qmail (a wrapper). Because a wrapper is a separate program, it creates an additional need for communication between different programs on the server: the wrapper and the qmail component it wraps. This leads to an increased overhead and latency. On the other hand, using a wrapper does not require modification to qmail itself, so it is less likely to cause security problems and can be easily removed if there is a problem with it.
The primary SSL/TLS patch was written by Frederick Vermeulen (http://inoa.net/qmail-tls/). There are several options for providing SSL service by means of a wrapper, but the most popular include André Oppermann's patch to tcpserver (http://www.nrg4u.com/), and stunnel (http://www.stunnel.org/). Another valid option, of course, is to replace qmail-smtpd with a program that understands SSL and TLS itself, such as Bruce Guenter's mailfront (http://untroubled.org/mailfront/).
The reasons to use encryption when receiving email are numerous. The standard reason is for privacy—i.e. keeping the content of email away from prying eyes. However, another common reason for needing the security and cryptographic privacy that encryption affords is SMTP-AUTH. When submitting email via SMTP and using the SMTP-AUTH extension to authorize the submission, usually both a username and password are sent over the network to the mail server. This information makes protecting this communication from interception especially important.
The most basic operational difference between handling the encryption within qmail-smtpd (i.e. with Vermeulen's patch or a replacement like mailfront) and handling the encryption in a wrapper around qmail-smtpd (i.e. Oppermann's patch or stunnel) is that changing the SMTP server makes it easier to support TLS (as it is an extension to the SMTP protocol). Oppermann's patch, for example, only supports SMTPS, (SSL encryption) not the STARTTLS extension to the SMTP protocol. stunnel can support the STARTTLS extension, but currently this requires patching the stunnel source code.
When evaluating the options, it is important to consider the security ramifications of the choice. Bruce Guenter's mailfront is a powerful tool (see Chapters 2 and 6); it is essentially a replacement for qmail-smtpd that adds many useful features, including support for SSL, TLS, and SMTP-AUTH. The fact that these additional features are bundled together adds a level of convenience to this option, if the other features of mailfront are used. Because mailfront, Vermeulen's patch, and the stunnel program all operate within the same security region as qmail-smtpd—i.e. as a user with restricted permissions—they are subject to essentially the same caveats: a security breach in one has the same impact as a breach within qmail-smtpd. But by operating with the same security restrictions as qmail-smtpd, they all benefit from qmail's privilege-separation architecture in the same way that qmail-smtpd does: if exploited, they run as a user with insufficient permissions to do very much. Capitalizing on such an exploit is much more difficult. This is not to say that any of them has a history of security flaws, but merely that their history of security is not as long (Vermeulen's patch and mailfront both have flawless security track records).
Using a patch to add SSL support to tcpserver is an interesting idea because it has the potential to undermine the security design of the qmail architecture on most UNIX systems. Since listening to the network requires root permissions, tcpserver normally runs briefly as the root user, and thus has sufficient permissions to do virtually anything. (Some UNIX variants, such as SELinux, allow permissions to be highly customized, and so tcpserver can technically be run as a user with permission to listen to the network but without sufficient permissions to do anything else.) This is usually acceptable for two reasons: tcpserver drops the root power quickly, and its task is extremely simple. Because tcpserver does not attempt to understand or interpret any of the network traffic, it is virtually impossible to exploit.
However, implementing SSL within tcpserver means that tcpserver performs a more complex job and interprets untrusted user (or attacker) input. This is risky, which is precisely why qmail-smtpd runs with restricted permissions. Oppermann's patch is very careful to only use OpenSSL after tcpserver has dropped its root privileges. If Oppermann was not as careful, the patch could have become a severe liability. Other patches that perform a similar task may not be as careful.
It is important to note that mailfront, Oppermann's patch to tcpserver, and stunnel enable qmail to receive encrypted connections but not to make encrypted connections to other SMTP servers. These options only enable clients to send mail to the qmail server in an encrypted form. If the goal, however, is to make qmail use encryption for its outbound connections, then none of these three solutions is up to the task. Frederick Vermeulen's patch, on the other hand, is the only mainstream addition to qmail that provides this ability. With Vermeulen's patch, qmail-remote uses and understands the extended SMTP EHLO semantics: it tests recipient servers to determine whether they support TLS and uses TLS encryption if they do. With this patch, qmail-remote also automatically uses encryption when connecting to port 465 (the SMTPS port) on any server; this behavior is specified by the smtproutes file.
The primary drawback to Frederick Vermeulen's patch is its size—it patches both qmail-smtpd and qmail-remote and alters and adds a significant amount of code to both. Although it has been tested in many places for a long time, it might still contain bugs. In addition, these extensive code additions have more potential to conflict with other qmail patches. For example, one of the most common situations for encryption is when using SMTP-AUTH, to protect the passwords transmitted during authentication. Most of the SMTP-AUTH patches, however, conflict with Vermeulen's patch. Because this is so common, a small segment at the beginning of the patch can be removed to allow it to cooperate with the popular SMTP-AUTH patches. Despite this concession to compatibility, the patch still conflicts with, or confuses, many other unrelated patches to qmail-smtpd, and these conflicts usually require manual resolution.
One of the most important and most common tasks that mail servers do is the distribution of email via mailing lists. Many organizations rely on mailing lists not only to reach their customers or constituents, but also to provide a means of communication within the organization or a group within that organization. The complexity and difficulty of this task varies depending on the number of recipients on a given list, and the level of automatic maintenance required. Choosing a mailing list management strategy or software package requires careful consideration of the needs and goals of the mailing list or lists.
The most basic form of sending an email to a mailing list is simply sending an email with multiple addresses in the To header. This technique is easy, simple, and for a small number of recipients, it makes perfect sense. The intermediate level is the alias-based mailing list. This is akin to having a group in an address book that the email server accesses. With this, one sends an email to all of the addresses listed in the alias by simply sending an email to a special email address representing that group. For example, in qmail, creating the file ~alias/.qmail-mylist creates the address [email protected] (assuming the qmail server's name is yourdomain.com). This file is filled with addresses (one per line), each of which receives a copy of every message sent to [email protected]. This is a lightweight mailing list, because it is so simple: it is defined in a single file and maintained by manually editing that file.
The problem with such a list is someone (who has permissions to edit ~alias/.qmail files) must maintain the list's membership. For small mailing lists or lists whose membership does not change often, this is not usually a problem. However, once the membership of a list gets large (over a hundred people, for example), maintenance of this file becomes a significant chore. It is often desirable to establish a mailing list where people can join or leave without involving the mailing-list administrator. This self-serve option requires software and that capability distinguishes a heavyweight list from a lightweight list.
Heavyweight list management software, of course, frequently includes other convenient features, including the abilities to easily archive all list messages, to prevent non-subscribers from posting, to allow message moderation, to provide some users with daily digests of all list messages, and many others. Examples of popular heavyweight list management software packages that work with qmail include ezmlm (http://cr.yp.to/ezmlm.html) by qmail's author, Dr. Bernstein; ezmlm-idx (http://www.ezmlm.org/) by Bruce Guenter (based on ezmlm); and GNU Mailman (http://www.gnu.org/software/mailman/).
When comparing mailing-list manager software, important details to evaluate include:
How many messages can the mailing-list manager handle per day or at a time?
How does the mailing-list manager perform under load?
How does the manager store its recipient list, and how quickly can it find a particular recipient?
The most fundamental question regarding mailing lists is: what are the speed limitations of posting to the list? In many cases, the real limitation is the outbound network bandwidth from the mail server: only a limited number of messages can be transmitted over the same wires at one time. This is not always the case. Poor software design can cause mismanagement or inefficient use of bandwidth. For example, the email server may use too much bandwidth retrying undeliverable messages or the mailing list management software may spend too much time managing itself and not spend enough time queuing messages for delivery.
One of the tasks mailing list management software often performs is automatic handling of email bounces to detect and remove undeliverable email addresses from its membership list. In a list with tens of thousands of members, the likelihood of some being undeliverable is rather high. When a bounce message comes back informing the list of a delivery failure, it must locate that recipient in its internal records and record the bounce. In a list of tens of thousands of names, simply searching through the list linearly takes too much time. And, of course, every undeliverable message causes a recipient lookup. If the mailing list is restricted such that only subscribers can post, every single message sent also requires a search of the membership rolls to verify that the sender of the message is indeed subscribed. At the same time, every posting requires that all of the addresses be collected and fed to the email server. For very large, very busy mailing lists or mailing-list servers, allowing a subscription request, a bounce, or a membership check to take more than a second of CPU time is too much.
The most popular list management software packages used with qmail store their subscriber lists differently. ezmlm and ezmlm-idx store user lists in hashed directories of files, while GNU Mailman stores its user lists in binary database files. The database files are faster than hashed files if they're kept in memory, while the hashed files are often (but not always) faster than database files if the database files cannot be kept in memory. Hashed files, however, are editable by hand if necessary, while GNU Mailman's binary database format cannot be reconstructed or hand-edited easily.
The number of processes that handle each posting to each list is a common way of comparing mailing-list efficiency. The idea is the more processes required, the greater the load placed on the server for each message. For email servers with minimal amounts of mailing-list traffic, this is essentially irrelevant, but in high-load situations, this is much more important. ezmlm and ezmlm-idx invoke multiple programs (typically, three or four) every time a message is sent to the list. Different
numbers of programs are used depending on the enabled options for that particular list. Handing the posted messages to the email server for delivery requires another two processes (qmail-inject and qmail-queue). GNU Mailman, on the other hand, invokes only two programs (preline and mailman) to receive every message, and another three (sendmail, qmail-inject, and qmail-queue) to queue messages for delivery. Thus, if large volumes of email are sent, ezmlm and ezmlm-idx spawn more programs than GNU Mailman and impose more program-loading overhead on the server. On the other hand, the programs spawned by ezmlm and ezmlm-idx are all small and simple. GNU Mailman's mailman program is a wrapper around a Python program. Because Python is an interpreted language and its interpreter is large and must parse the entire input program before doing any useful work, starting up a Python program takes longer than a simpler, compiled program. To mitigate the overhead of starting multiple large Python programs, GNU Mailman has a daemon (qrunner) to handle most of the operations of the mailing lists, allowing the mailman program to be simpler and merely submit job requests to qrunner. In any case, the efficiency differences between the ezmlm/ezmlm-idx and GNU Mailman cannot be treated as simply a case of one starting more programs than the other. Not only is the overhead per process different depending on the process, but on many servers, the cost of invoking processes is greatly overshadowed by other costs, such as the cost of bounce message handling.
Not all recipients can be contacted every time mailing list messages are distributed. It is ordinarily desirable to track which members of a list have not received messages, and for what reason. For example, if a given address no longer exists, the mailing-list manager could warn the list administrator and temporarily remove that address from the distribution list. This procedure is complicated by the fact that mailing-list software cannot know whether mail was delivered immediately. If the software is to automatically monitor delivery success, it must receive and interpret bounce messages.
While receiving bounce messages is relatively easy by simply setting the envelope sender address of each outbound list message to a special list-specific email address, reliably interpreting those bounce messages is difficult. Because the format and language of bounce messages varies, reliably extracting the reason for the bounce and even the recipient responsible is challenging.
To address this problem, a technique exists to encode the recipient of a list message in the return address. There are many ways of doing it, but the formalized technique is known as Variable Envelope Return Path (VERP). Messages are normally fed to a mail server (such as qmail) with a single sender and one or more recipients. In order to give every recipient a unique sender, a new copy of the message body normally must be given to the mail server for each recipient. For small lists this is not a problem, but for large lists, this presents a storage and efficiency dilemma, as the queue must track and store each message separately.
The solution, in this case, is VERP and transferring the responsibility of changing the sender of each message from the mailing-list software to the email-server software. By standardizing the method of encoding a recipient into the sending address, the encoding can occur at any location, including within the email server. In this case, qmail-remote is either patched or wrapped so messages from mailing lists (with a special return address to indicate that VERP encoding should be employed) are rewritten as they exit the queue, rather than as they enter the queue. This drastically reduces the overhead of VERP while maintaining its functionality. Both ezmlm and GNU Mailman support VERP. ezmlm uses it by default; and GNU Mailman can be configured to do so. Neither one relies on an implementation in the email server (qmail), though Frederik Lindberg has written a patch (http://www.ezmlm.org/archive/patches/qmail-verh-0.06.tar.gz) to both qmail and ezmlm that allows them to cooperate this way. GNU Mailman usually restricts its use of VERP to probe messages. This avoids using extra resources for each normal mailing list message without relying on the email server to implement particular support for VERP. However, this can lead to a failure to detect some failed deliveries.
Not all mailing-list software works optimally with the qmail architecture because mailing-list software frequently is tightly bound to specific email-server software to take best advantage of its features. For example, the popular mailing list management software Majordomo was originally designed specifically for the Sendmail email server, and relies on the details of the Sendmail aliasing system. Since qmail can be made to interpret Sendmail alias files, and since Majordomo can be altered to create qmail aliases, they can be made to work together, though not generally as efficiently and/or conveniently as software designed for use with qmail. (It is expected that Majordomo 2.0 will support qmail directly, but work on Majordomo 2.0 has been in progress since even before qmail was written and as of this writing does not have a set release date.) Of the three software packages recommended here—ezmlm, ezmlm-idx, and GNU Mailman—the first two are designed explicitly for qmail (though they can work with servers like Postfix). Because of this, ezmlm and ezmlm-idx are easy to install and get running with qmail. GNU Mailman, on the other hand, is designed to work with a broad range of email servers and therefore requires glue between itself and the email server.
For example, GNU Mailman establishes several standard email addresses for each of its mailing lists (similar to ezmlm and ezmlm-idx) to handle administrative requests, bounces, and other specific tasks. Enabling delivery of mailing-list addresses with these software packages requires either several .qmail files (one per address) or a way to deliver all of the mailing list messages to a single address prefix and a script that delivers messages to their intended destinations. ezmlm and ezmlm-idx handle .qmail file creation themselves. A common approach with GNU Mailman is to create a virtual domain for all GNU Mailman mailing lists (e.g. lists.example.com) and put a .qmail-default file in that domain's home directory that directs qmail-local to feed messages to a script (a Python version is provided with GNU Mailman). This script determines the original destination address and then calls mailman with the correct arguments. While this script (glue) makes management of GNU Mailman mailing lists less involved, it also increases the overhead required to process every message posted to GNU Mailman lists.
One of the most popular and visible features of mailing lists is their ability to keep an archive of messages distributed on the list, and make this archive available from a web browser. GNU Mailman makes this relatively easy because its entire management system is based around a web-browser interface. Making list archives available via the Web requires only enabling the archives and properly configuring the web server. Providing a web interface for ezmlm archives requires more work because it does not come with such an interface. The easiest option is the ezmlm-www software (http://ezmlm-www.sourceforge.net/), a Perl-based CGI script. ezmlm-idx adds more extensive web support to ezmlm. ezmlm-idx comes with a program called ezmlm-cgi that—when properly configured—allows web-based browsing of list archives. However, its interface is not particularly user friendly. A more attractive alternative is ezmlm-browse (http://untroubled.org/ezmlm-browse/) written by Bruce Guenter. Because Guenter wrote both ezmlm-browse and ezmlm-idx, they work well together.
It is often convenient to administer, not just browse, mailing lists from a web browser, executing administrative tasks such as moderation and changing settings. ezmlm and ezmlm-idx make it possible to do advanced administrative tasks entirely via email, but a web browser is typically more convenient. The GNU Mailman software is controlled entirely from its built-in web interface—many configuration details can be set only from its web interface. ezmlm and ezmlm-idx, however, are both oriented primarily towards command-line-based configuration, though they can be configured via email. The ezmlm-web software package (https://systemausfall.org/toolforge/ezmlm-web/), however, provides a web-based interface for configuring both ezmlm and ezmlm-idx.
This chapter has covered two common needs of qmail administrators: encryption and mailing-list management. This discussion has built upon the knowledge of qmail architecture presented in the preceding chapters, as examples of how to approach important features with qmail's architecture in mind. The next chapter will cover how to optimize a qmail server for particular environments, and how to maintain a qmail server over