The virtualdomains file is one of the most powerful, useful, and stunningly simple mechanisms for configuring qmail. Virtual domains and even virtual users can be created, independently of one another, and assigned to controlling users. Virtual domains are fully independent, and as they are assigned to users (or, more accurately, prefixes) they can be in different UNIX protection domains if desired. This file can also define virtual users and similarly assign them to controlling users. This file thus makes it both possible and easy to intercept specific addresses and do something special with them.

Perhaps surprisingly for the power this file wields, the control/virtualdomains file is only slightly more complicated than similar control files, such as control/locals. Rather than a list of domains, the virtualdomains file is a list of patterns and their associated prefixes. The entries are of the form:

matching-pattern: prefix

Only one prefix is associated with each pattern. The matching pattern can be one of the following three things: a domain (that follows the same wildcarding semantics used in the control/rcpthosts file), an email address, or an empty string. The empty string is considered to match anything. An email address matches any message addressed to that email address, and a domain matches any message addressed to a user in that domain. If there are multiple possible matches, the longest match is used.

Emails that match patterns listed in the control/virtualdomains file are considered to be local emails, similar to those addressed to domains in the control/locals file. Messages that match patterns in the virtualdomains file must first undergo a simple modification before delivery. While messages are prepared for delivery, if they have an envelope destination address that matches a pattern in the virtualdomains file, that destination address is prepended with the matching pattern's associated prefix. This turns the original local part of the address into an extension of the prepended user. In this way a domain can be mapped to a user, giving that user full control over the mailboxes (or users) within that domain. The virtualdomains file can also be used to define exceptions to the matching rules, if the prefix is blank.

It is important to note that the control/virtualdomains file is considered after the control/locals file, which means that if an email is addressed to a domain listed in locals, the virtualdomains file will not apply. Also, virtualdomains rewriting occurs before the mail is given to qmail-lspawn for delivery.

This sounds complex, but is easy to understand with an example.

An example virtualdomains file that demonstrates most of the file's features is as follows:

example.com:foo
.example.com:foo-bar
two.example.com:baz
[email protected]:qux
domain.com:
:garply

Presuming that the users mentioned i.e. foo, baz, qux, and garply are the only local users on this system; these lines cause the following results:

In this example, the foo user is essentially in charge of the entirety of the example.com domains. Users that are specific to example.com are defined by creating appropriately named .qmail files in foo's home directory.

For example, to establish the standard [email protected] address, foo would create a file named .qmail-postmaster in its home directory, containing the instructions for delivering postmaster's email. The foo user could also establish a [email protected] address by creating a file named .qmail-foo in his or her home directory. In this way, once the example.com mapping has been established, the foo user can set up and maintain the users in the example.com domain without ever requiring further permission from or contact with the system administrator.

It is worth pointing out that precisely who controls what can get more complex if real usernames have extension separator characters (a hyphen, by default) in them. For example, if the address [email protected] is rewritten to be [email protected], it is typically delivered according to the instructions in ~foo/.qmail-postmaster or, if that file does not exist, ~foo/.qmail-default.

However, if there is a user named foo-postmaster, that user will receive email addressed to [email protected].

If a .qmail file (such as ~foo/.qmail-postmaster) cannot be located for a given extension address (such as [email protected]), the alias user's directory is checked for .qmail files. Specifically, .qmail-foo-postmaster, .qmail-foo-default, and .qmail-default. If none of these files exist, the message is considered undeliverable and is bounced.

Note that during delivery of such an email, the standard environment variables are defined per the rewritten destination rather than per the original address. For example, if a message addressed to [email protected] is delivered using the example configuration, the affected environment variables will be defined as follows (assuming foo's home directory is /home/foo):

To better understand how qmail's virtual domain mechanism works, let's attempt to understand it visually. The following chart roughly illustrates the decision-making process that qmail performs when delivering a message (the process starts at the top left):

To new qmail administrators, the users/assign file is probably the most mysterious file in qmail's configuration. Its purpose is to allow the administrator to define mailboxes (or users) that qmail will treat as potential delivery targets. With this purpose comes the ability to define the name of the .qmail file that will be used to configure the defined user and the way in which that user's extensions will be separated (qmail extensions are typically separated by a hyphen unless changed at compile time in conf-break).

Being able to define mailboxes independent of the underlying operating-system users can be very useful for many purposes. One reason this file is often used is for speed; it is compiled into a CDB-formatted database file, which is faster to read than text, and thus can make the process of looking up users fast. This is particularly valuable in cases when looking up users the usual way (i.e. via getpwent()) is slow—such as when there are very large numbers of users. Being able to create mailboxes independent of the underling operating system can also be used for many creative purposes and organizational schemes. The most popular use of this file, though, is to assist in organizing virtual domains. Each domain can be given its own directory, which may or may not have a unique system user assigned to it.

While this may sound complicated, and the file format is more detailed than any other qmail configuration file, the underlying concept is very simple. For any mailbox, two things are obviously required: a name (i.e. the part of the associated email address to the left of the at @ symbol) and a directory associated with the name. When dealing with users defined by the operating system, this is the username and the user's home directory. Because all files must be owned by a system user of some kind, qmail must know a UID, GID, and username to be used when delivering to this mailbox. For a user (mailbox) defined by the operating system, these are the UID, GID, and username of the user in question, and this same information is defined in the users/assign file. Finally, qmail also allows both the definition of alternative extension separators and an extension to specify the default .qmail file in the defined user's home directory. These definitions are compiled into a CDB file—users/cdb—for qmail's use by running the qmail-newu program, which must be run whenever the users/assign file is changed.

The /etc/passwd file, which defines UNIX users, generally has a format as follows:

username:password:UID:GID:groupname:homedirectory:shell

For example:

john:x:100:101:doe:/home/john:/bin/sh

The users/assign file has most of this same information, but comes in two forms: simple assignments and wildcard assignments. Simple assignments are the easiest, and begin with an equals (=) sign:

=mailboxname:username:uid:gid:homedir:dash:ext:

The dash and ext parts are concatenated to define a .qmail file for delivery. A simple translation of the above example /etc/passwd user would be something like the following:

=john:john:100:101:/home/john:::

Note that this provides an alternative method of defining aliases. For example:

=zeke:john:100:101:/home/john:::

This line will deliver mail sent to zeke as if it had been sent to john. If zeke mail must be sent to the john user but must be delivered according to a different .qmail file, a line like this could be used:

=zeke:john:100:101:/home/john:-:zeke:

This would tell qmail to deliver mail addressed to zeke according to the instructions in

/home/john/.qmail-zeke.

The above example lines are all simple assignments, and do not support extensions. In other words, it will direct email addressed to [email protected], but not [email protected]. That's what wildcard assignments, which begin with a plus (+) sign, are for. Wildcard assignments have the following form:

+mailboxprefix:username:uid:gid:homedir:dash:pre:

This is essentially equivalent to having an entry for every possible string that could replace EXT:

=mailboxnameEXT:username:uid:gid:homedir:dash:extEXT:

For example, the following entry will handle [email protected] and all similar [email protected] extensions:

+john-:john:100:101:/home/john:-::

One of the places where this ability to create arbitrary mailboxes can be very useful is in organizing virtual domains. For example, imagine that you are configuring example.com to be a virtual domain. The control/virtualdomains file would have an entry like the following:

example.com:example 

Now that all example.com email will be directed to the example mailbox, the example mailbox must be defined. One way to do this is to have a UNIX user named example. Unfortunately, this means that the system will also accept mail addressed to [email protected] (because if example is a full UNIX user, it is also a valid recipient). A more restrictive method is to create a wildcard entry in the users/assign file, as follows:

+example-:john:100:101:/home/example.com:-::

This will make sure that all example.com users are handled by .qmail files in the folder /home/example.com, and that all such messages will be owned by the john UNIX user. This also prevents [email protected] from being a valid recipient address.

As managing virtual domains and virtual users is a common frustration, several programs that provide assistance are available. Two of the most popular qmail-compatible ways of managing virtual domains are vpopmail by Inter7 (http://www.inter7.com/index.php?page=vpopmail), and VMailMgr by Bruce Guenter (http://www.vmailmgr.org/). The two are very similar in what they provide.

The way vpopmail works is straightforward. At the basic level, it maintains a database of users (it can use Oracle, Sybase, MySQL, or LDAP databases, or use its own domain-specific CDB files), provides a checkpassword-compliant interface for authenticating against that database, and automatically modifies qmail's configuration files as necessary when domains are added. Each domain has an entry in the control/rcpthosts and control/virtualdomains files, as well as an entry in the users/assign file. The users/assign file is used to give each domain its own home directory without creating a UNIX user for each domain. It can, of course, be told to use different UNIX users for each domain. There is a .qmail-default file in each domain's home directory that feeds email messages to a vpopmail-specific delivery program that verifies the recipient and delivers the email. Additionally, .qmail files in each domain's home directory define aliases or forwards. Each virtual user has a unique home directory within its domain's home directory that can contain user-specific configuration information, such as delivery instructions and spam detection preferences. vpopmail emulates .qmail file handling for these virtual users, so a .qmail file within the virtual user's home directory controls the email delivery for that user. By default, mail is delivered to a Maildir-formatted mailbox named Maildir inside each virtual user's home directory.

For clarity's sake, imagine an email addressed to a vpopmail-managed domain: [email protected]. Imagine that vpopmail has been configured to keep domains in /var/lib/vpopmail/domains/. Here's how delivery would work, from network to disk:

As you can see, it's a complex process.

VMailMgr works very similarly to vpopmail. It also maintains a database of users and passwords in CDB files, and provides both a checkpassword-compliant authentication program and a CourierIMAP module for accessing these files (CourierIMAP supports vpopmail's interface natively). When new domains are added, qmail's configuration files—rcpthosts and virtualdomains—must be modified manually. Each domain is required to have its own UNIX user and UID, which negates the need to modify the users/assign file (though it can be used if desired, to speed looking up user information). The virtual users are organized similarly to vpopmail within each virtual domain's home directory, though there is no emulation of a per-user .qmail file. The difference in delivery operations from vpopmail is that Step 8 becomes a lookup of the example.com user in /etc/passwd, Step 10 uses the example.com user, Steps 12 through 15 involve a program called vdeliver instead of vdelivermail, and Step 14 doesn't happen.

Virtual domains are almost always a part of a larger system of storing and retrieving email. Because each virtual user does not have a corresponding UNIX user account, other mail operations must use an abstracted interface such as IMAP or POP3 to securely access a virtual user's mail storage. Additionally, a virtual email domain system also needs to cooperate frequently with other virtual domain systems, such as a virtual web domain system or database system.

There are two facets to cooperating with other virtual domain systems and other related and dependent services. The first is file organization and layout, and the second is user authentication. File organization and layout is the primary consideration during the set up of a new domain: frequently all support files for new domains, regardless of service (email or web or whichever) are collected into a single location or file hierarchy. Though this is usually unnecessary, it has a certain

aesthetic appeal, and lends itself to some administrative tasks such as implementing cross-service per-domain disk quotas. On the other hand, it is also sometimes more convenient to separate files by service. For example, mail files tend to be small and may be better served by a different filesystem than web or database files. The home directory qmail will use for each domain can be easily configured using the users/assign file or, for management systems like VMailMgr, using /etc/passwd. This setup is sufficiently flexible to deal with most rational organization schemes.

Authentication is frequently the most difficult facet of virtual domains to configure because so many different software packages must use it. For example, if virtual domain and virtual user information is stored in a vpopmail- or VMailMgr-specific set of CDB files, getting an IMAP server to authenticate users from these files may be very difficult. The IMAP server must use the checkpassword-style authentication program provided by both packages and be able to understand the user databases without help, or use an intermediary such as the CourierIMAP authentication service that does understand them. If these users are then going to be used by the web server for authentication purposes, the web server must support one of these authentication methods as well. This problem of having multiple services, possibly on different physical computers that need to authenticate from the same set of user data is one of the attractions of service-based authentication mechanisms like LDAP.

The crucial detail to setting up and maintaining multiple qmail installations is to alter the conf-qmail file in the qmail source. This file is what defines both where qmail will install its binaries and where those binaries will expect to find each other and their queue. The location of the queue is compiled into the binaries directly, so maintaining a separate queue requires a separate set of binaries.

Assuming that one instance of qmail has already been successfully installed in /var/qmail (the default), a second qmail instance can be installed in (for example) /var/qmail2 by performing the following commands from within the qmail source directory:

echo /var/qmail2 > ./conf-qmail
make setup check

Once this is done, a new installation of qmail will reside in /var/qmail2. To complete the setup, the appropriate configuration files from /var/qmail/control, such as me, should be copied to /var/qmail2/control. Also, the new qmail-send process should be started, (if the new queue will be used) usually with a script nearly identical to that of the original installation.

For example, if the original qmail-send process was started with a daemontools run script, another supervise directory and run script for the new qmail-send process should be prepared. The second qmail-send process is not necessary, if the second qmail installation will share a queue with the first. In that case, all that is necessary is to symlink the qmail-queue binary from the first installation into the second (and delete the second's qmail-queue binary).

If desired, another supervise directory and run script may be created to run the new qmail-smtpd as well. The trick to get both qmail-smtpd instances (the original and this new one) to run together is to specify different IP addresses for them to listen to. The original example script in Chapter 1 specified the arguments to tcpserver as follows:

tcpserver -R -l "$LOCAL" -H
 -x /etc/tcp.smtp.cdb 
-u "$QUID" -g "$QGID" 
0 smtp  /var/qmail/bin/qmail-smtpd 2>&1

This—or more specifically, the 0 in the fourth line—specifies that tcpserver will listen for SMTP connections on all network interfaces. This prevents other programs, including other tcpserver instances, from listening for SMTP connections on any network interface. If, for example, the original qmail installation only needs to listen to the single publicly accessible IP address (for example, 192.168.1.1), then the original setup should be changed to specify that address rather than simply 0. Thus, it would look as follows:

tcpserver -R -l "$LOCAL" -H 
-x /etc/tcp.smtp.cdb 
-u "$QUID" -g "$QGID" 
192.168.1.1 smtp  /var/qmail/bin/qmail-smtpd 2>&1

With the original configuration more specific, the second qmail installation is free to listen to a different address. For example, it could listen to the loopback interface via IP address 127.0.0.1, as follows:

tcpserver -R -l "$LOCAL" -H 
-x /etc/tcp.smtp.cdb 
-u "$QUID" -g "$QGID" 
127.0.0.1 smtp 
/var/qmail2/bin/qmail-smtpd 2>&1

To what address the second qmail-send instance should listen depends very much on what its purpose is. If the purpose is to provide different connection policies (e.g. different DNS blacklists) for different domains, the server computer will need to manage multiple publicly accessible IP addresses, each with its own tcpserver and qmail-smtpd. The reason different connection policies cannot be implemented for different domains on a single shared IP address is that a connection policy, by definition, takes effect before data is sent. Thus, which domain's policy should be used is unknown until qmail-smtpd is run, which is too late.

When a given computer has multiple qmail installations, it is sometimes desirable to hide the details of the different qmail queues from the users. For example, if a single computer is hosting several domains, each with different queue lifetimes, each mail that is sent must be fed to the correct queue, depending on which domain sent it. Most webmail applications will either use a single binary (usually /usr/sbin/sendmail) or a single address (usually 127.0.0.1, port 25) to send mail, regardless of which user is sending the mail. Unless this situation is handled specially, all mail sent from that webmail application will go through a single queue, rather than a different queue for each domain, and so they will all use the same queue settings (which invalidates the whole point of setting up separate queues in the first place).

There are multiple ways of handling this problem. A labor-intensive way is to install several versions of the webmail application, one for each domain. While conceptually simple, this quickly becomes an administrative nightmare, particularly when upgrading or reconfiguring the webmail software. A more transparent and maintainable technique is to create a wrapper script around the qmail-queue binary to deliver the message to the correct queue depending on its return address. As a simple example, imagine a computer responsible for both the 192.168.1.1 and 192.168.1.2 IP addresses. The 192.168.1.1 address is used for the example.com domain, and the 192.168.1.2 address is used for the somewhere.net domain. Each domain has its own queue—example.com's queue is in /var/qmail/queue and somewhere.net's queue is in /var/qmail2/queue—but the /usr/sbin/sendmail file (used by their shared webmail installation) can only be a symbolic link to one of them. In this case, it is a link to the example.com qmail installation's sendmail binary. To ensure that mail is delivered by the correct domain's qmail installation, a wrapper script around example.com's qmail-queue binary can be used.

Here is a simple example:

#!/bin/bash
# Read envelope data (from file descriptor 1)
read -u 1 -d $'' sender
i=0
while read -u 1 -d $'' recipient ; do
[ -z "$recipient" ] && break
recipients[$i]="$recipient"
i=$(($i+1))
done
# Build the envelope data back up
TMPFILE="$(mktemp -t qmailqueue.XXXXXXXXXXXXXXXXXX)"
printf '%s' "$sender" > "$TMPFILE"
for ((i=0;$i<${#recipients[*]};i=$i+1)) ; do
printf '%s' "${recipients[$i]}" >> "$TMPFILE"
done
printf '' >> $tmpfile
# Extract the domain from the sender's address FROMDOMAIN=${sender##*@}
# turn on case-insensitive matching
shopt -s -q nocasematch
# Invoke the domain-specific qmail-queue binary
case "$FROMDOMAIN" in
example.com)
/var/qmail/bin/qmail-queue-real 1<"$TMPFILE"
exitvalue=$? ;;
*)
/var/qmail2/bin/qmail-queue 1<"$TMPFILE"
exitvalue=$? ;;
esac
# cleanup
rm $TMPFILE
exit $exitvalue

To use this script most transparently, one must first rename the real qmail-queue binary for the example.com domain from /var/qmail/bin/qmail-queue to /var/qmail/bin/qmail-queue-real. Then, save this script as /var/qmail/bin/ qmail-queue and make sure that it is both executable and readable. Thereafter, whenever mail is queued with example.com's sendmail binary, (or qmail-smtpd, qmail-queue, qmail-inject, qmail-qmtpd, or any other component that uses qmail-queue) this script will be invoked instead of the real qmail-queue binary. The script will make sure that the mail gets added to the queue corresponding to the sender's domain. The upside is that this works for any standard method of sending mail. Of course, the sender address this script relies upon can be forged (or mistyped) and may not reflect the domain of the actual sender.

In most cases, this is not a severe problem, but can expose the fact that there are multiple queues. The problem of message headers and envelope information potentially being forged is a difficult one. It is possible to be more thorough, and attempt making decisions about which queue to use based on other, more reliable information (such as the user name provided during SMTP-AUTH) but this is more involved and does not typically address every possible way of queueing mail. However, getting the mail into the right queue is only a problem when attempting to hide either the virtual nature of the server or the separation of its queues. By exposing the separation, such as by providing different IP addresses or port numbers to use for sending mail from each domain, the system's behavior becomes far more predictable. Not because the headers are restricted, but because which domain's queue is used becomes an explicit choice the user makes by submitting messages to a specific IP address or port number.

Virtualization is usually an attempt to share limited resources, such as servers, IP addresses, or system administrator's time. At some level, there is almost always a way to break the veil and discover the virtualization because some detail of the sharing is visible. In general, few people mind it, as long as mail gets where it needs to go.