Objectives
How email clients and servers function to transmit email from one user to another
How to install and configure SendMail to act as a mail transfer agent (MTA)
To configure the firewall for email
To configure name services to accommodate email with an MX record
How to test email using a command line email client
To use email headers to trace the origin and route of an email
To configure a host to use the email server as a smart host
To configure the aliases file to forward system level email intended for root to another email address like the student user
Introduction
Email is a ubiquitous messaging service and is available on devices ranging from work and home desktop computers to various mobile devices such as smart phones and tablets. There are two sides to email. The IMAP and POP protocols are used to receive email on your device and the SMTP protocol is used to send email from your device to and between email servers.
Email is an asynchronous messaging protocol at the macro level. That is, if I send you an email message, you do not have to be at the receiving computer at that moment in order to receive the message. The computer does not even need to be turned on. The message is stored at the server until the computer is turned on and you retrieve it.
A typical synchronous messaging system is a face-to-face conversation or a telephone call which requires both parties to the conversation to be on the line or in the same place at the same time. Voicemail is just a corrupted version of a telephone call in which we leave messages for each other – another form of asynchronous messaging.
Email services were originally limited to users on a local Unix computer. All users of an email system had to be connected via a hardware terminal to the Unix computer. Because computers were not normally connected in any way, email systems were very localized. As slow dial-up connections became available, remote computers could be connected but only for specified and relatively short periods of time. Email messages could be stored on the sending server until the connection was made and all messages intended for the remote email server would be sent using that temporary connection. Email bound for other remote servers were held on the local server until a connection to the destination remote server was made. It is these ancient requirements that helped to define many of today’s email protocols such as the ability to store messages for a period of time until the remote server is available.
Today we can send email to almost anyone on the planet, but spam is a major issue. With so many people connected to the same Internet that allows us to communicate with email, there are also those who use email for scamming the rest of us. We will discuss dealing with spam in Chapter 9 of this volume.
Definitions
Protocol – A set of formal rules describing how to transmit data, especially across a network.
SMTP – Simple Mail Transfer Protocol: A protocol used to transfer electronic mail between computers.
POP – Post Office Protocol: A simple protocol designed to allow single user computers to retrieve electronic mail from a POP server. Once retrieved by the client, the email is deleted from the server.
IMAP – Internet Message Access Protocol: A protocol allowing a client to access and manipulate electronic mail messages on a server. Emails are retained on the server until explicitly deleted by the user.
MTA – Mail Transfer Agent: An agent such as Sendmail that transfers email from one host to another. These transfers may not only be between email servers but also from a sending email client to an email server.
SendMail – A very common MTA that has been around for many years.
Email data flow
- 1.
The client adds an initial set of headers to the email to be sent. This includes the subject line, a date stamp, and the From: and To: lines.
- 2.
The sending client uses SMTP to send the email to the local email (SMTP) server, SMTP server1, as defined in the client configuration. So, for clients in the domain example.com, their email would typically go to an email server for example.com. One common method is to identify that server as mail.example.com in the internal name services (DNS) database.
- 3.
SMTP Server1 receives the email and adds a Received: line to the headers that lists where the email came from with IP address and host name if possible, along with a timestamp. The header entry also indicates the addressee.
- 4.
SMTP Server1 parses the address(es) to which the email is destined.
- 5.
SMTP Server1 uses DNS to specifically request the MX (Mail eXchanger) record for the target domain.
- 6.
SMTP Server1 then sends the email to the receiving server, SMTP Server2, through the Internet.
- 7.
SMTP Server2 adds another Received: line to the headers.
- 8.
SMTP Server2 holds the email in the user’s inbox until the client connects to the server to retrieve the email. Using the relatively more common and newer IMAP protocol on the receiving client, the user can view the email on SMTP Server2.
The flow of data for email messages
Structure of an email
The primary structure of an email message has two parts as defined in RFC 822, the headers and the message body. The headers are separated from the message body by a single blank line.
The message body can contain ASCII plain text or MIME1 components consisting of HTML messages, images, or other types. The text body content of an email message is limited to 7-bit ASCII which is why MIME is used to attach data types based on 8-bit data.
Email headers
The email headers provide a record of the email's travels and can help us identify their true source. Each MTA adds one or more lines to the headers to record the email's passage. The email headers are normally hidden from users by the email clients, but a SysAdmin can access them to use in the task of problem determination for email delivery issues. I refer to email headers frequently for various types of problems including spam source identification, to determine where an email may have been delayed in its transit across the Internet from sender to receiver, and to use as the basis for blocking spam.
Typical email headers
This line is an indicator that the body of the email is ASCII plain text, or that there is a non-text attachment. It can also mean that the message body has multiple parts, that is, more than one type such as text and image.
Every message has an ID, and this is the ID for the message from which our headers were extracted. This message ID was generated after the sending software, Fail2Ban, sent it to the local email MTA on the local host.
Each message has a different ID one every server through which it travels. This is to prevent the possibility of having a message sent from one server having the same ID as a message sent from another server. These message IDs are stored in the headers as a permanent record which enables us to locate log entries pertaining to the message in each server.
This line identifies the sending host of the email. This, like many of the other headers, can be spoofed so that it looks like it came from another email account entirely. We will explore that later in this chapter. But for this example, we can be sure that none of the headers have been tampered with.
This header tells us that the email MTA on host1 received the email from the localhost. It might sound confusing, but so far we are still working on host1, which originated the email message. Each MTA that the email passes through always adds its own received header.
This, the third received header, shows that the email was received by the email server for the example.net domain.
Note that there is no time difference within the 1 second granularity of these headers between the original date stamp placed on the email and this header. All of these time stamps so far place the date and time at Wed, 26 Jun 2019 03:54:35 -0400. So far the email has been processed on two computers.
The email has now been received by my email server, yorktown.both.org. We now notice a 3-second time difference since the previous header. This is due to two factors: the time required to process the email through the spam detection software on the mailserver.example.net system and the time needed to connect with my server and to perform the handshaking and data transfer. Almost all of this time is due to the spam filtering.
Different email servers use somewhat different formats for some of the headers and they also may insert headers in different places in the stream. As we proceed through this chapter and these next chapters that also relate closely to email, I suggest that you take time to view the headers of the emails we send as part of the experiments.
SendMail on the server
Sendmail is a common email transfer agent. It has been around since 1983 and is still widely used on many email servers. There are a number of other good MTAs available, many of which are open source. Understanding SendMail – at least what we will be able to do here in this course – will provide a good basis for understanding email and mail transfer agents in general.
Although I use SendMail for my domain email server, it is also useful on a host that is not being used as the primary email server for a domain. Sendmail can be used on any host to provide an MTA to deal with emails sent to root by various system level applications and servers. If not sent to an email server, the local emails will be sent to the root user on the local host. Thus, they may never be read and acted upon. SendMail is required for each host that is intended to send its system management emails to a central mail server for further relay and distribution. When a network of hosts is configured to send all emails to the email server for the domain, that email server is called a smart host.
In the experiments below, we will install and configure SendMail as the primary mail server for our domain on StudentVM2 and we will install it on StudentVM1 to act as a transfer agent which can forward emails to the domain email server on StudentVM2. From there, these emails can be sent to any email client account.
Sendmail installation
Let’s start by installing SendMail on both of our virtual machines.
Experiment 7-1
Perform this experiment as the root user on both VMs. We will install the sendmail and sendmail-cf packages on both of our virtual machines in this experiment.
StudentVM1 may already have SendMail installed. Even so, the sendmail-cf RPM must also be installed. The sendmail-cf package provides the makefiles2 and configuration files that allow configuration and recompilation of sendmail.mc and other SendMail configuration files and databases.
We also install mailx, an email client that can be used as a text mode email client and as a command in a pipeline to send a data stream from its STDIN to the local mail transfer agent. This is a good tool for use to send emails in scripts. We can also use it from the command line to easily send test email.
This does not require a reboot.
SendMail configuration
Sendmail is already well configured by Red Hat in its distributions including Fedora. SendMail does still need a bit of additional configuration and it needs to be configured just a bit differently for the domain email server than for a system that will only send emails to the smart host.
It is only necessary to make some minor changes to the SendMail configuration itself.
Experiment 7-2
Perform this experiment as root on StudentVM2. We will make StudentVM1 into the domain mail server by configuring SendMail. For the moment, we will concentrate on sending email from our server although this first change is for inbound email.
Use a text editor to make the changes to the configuration files.
In the M4 language used in sendmail.mc, dnl means “delete through newline,” which further translates into something meaningful as “ignore the rest of this line.” It is an instruction for the specialized compiler used by SendMail.
Verify that the timestamps for the *.db files that correspond to the altered files have been changed.
This error occurs because we set the virtual machines’s host name without using the fully qualified domain name (FQDN).
What?! You did not think I would divulge all of my secrets at once, did you? I learned a lot about configuring SendMail from the many mistakes I made while doing so the first several times I did it. It took me hours to work my way through some of these problems even using search engines. So my intent here is to give you a feel for SendMail, not just what we need to do but also the why of it.
Also check the /etc/hostname file which is where the hostname is stored. We could have changed the host name in that file but activating it would require a reboot. Using the hostnamectl command does all of that for us.
You should notice immediately that the command only took a very short time. You should also see an informational message on the screen following the maillog but no errors.
You should also see four log entries added to the maillog file. The last one should have a status of Sent.
Do not delete this email. That worked and you could use q<Enter> to quit from mailx, but let’s just leave it open because we will be using it a bit more.
Notice that there are very few headers because the message was delivered by the email server on the local host, StudentVM2. It is time to send an email message to the outside world. For this, you will need an external email account that you can access from wherever you are taking this course. A computer, mobile phone, or tablet that you have configured to access your real-world email account would work. If you do not have one of these, you will be able to determine the success of sending these emails from the maillog. This is what SysAdmins and especially email administrators need to do in real life anyway.
So, did you figure it out? It took me a long time at first, so let me explain it. The first set of five log entries from Jun 27 13:02:17 through Jun 27 13:02:19 are from the outbound interaction with the remote email server. In my case, this was the email server for the both.org domain. The last log entry for this series shows that the message was accepted by the remote server.
However, starting almost immediately, the next set of log entries shows that the mail server for both.org has sent us a notification via the connection that we initiated to send the mail in the first place, that the user is unknown. The final line is the indication that our email server sent an email to the sender indicating that this was the case.
Note the 553 message that says, “Domain of sender address [email protected] does not exist).” Do you see the problem now? The domain for this email is not really a domain; it is a host name studentvm2.example.com. It includes the domain name. I ran into this problem, too, the first couple times I set up an email server. It is an unintended side effect of specifying the hostname of our server with the FQDN.
The reason for this failure is that the Internet name servers, not the ones in our own network, do not have any domains named <hostname>.example.com. They do have example.com. The remote mail server, in my case, mail.both.org, checks to see that DNS has an IP address for the domain name in the From: header of the email message. If studentvm2.example.com does not exist, the mail server rejects the email.
These lines now masquerade hostnames like studentvm1.example.com and studentvm2.example.com to a true, two-part domain name, example.com.
Now send the email to your real-world email account. The log should show a successful delivery with no entries to indicate a return error message. Check your external email account to verify the successful receipt of the test email.
Our email server is now capable of sending emails to external, real-world mail servers so long as the email originates from a local account.
Firewall and DNS configuration
Now that our email server can send emails, we need it to also receive emails. Let us start with using it as a “smart host” so that it accepts email from other hosts on our network and can pass them on to the external world. We also want to set up a CNAME record for mail.example.com and a Mail eXchanger (MX) record that explicitly defines the mail server for a domain no matter its given hostname.
Experiment 7-3
Perform this Experiment as root on StudentVM2. We will configure both DNS and the firewall.
The email server is now ready to accept emails from hosts inside our virtual network.
SendMail on the client
Now we can configure SendMail on the StudentVM1 client host. We already installed it in Experiment 7-1.
Experiment 7-4
Now let’s test our configuration. On StudentVM1, open a terminal window as root and use it to tail -f /var/log/maillog. Do the same thing on StudentVM2. On StudentVM1, enter the following command and watch the log files. You may see some log entries indicating deliveries of LogWatch notifications to [email protected]. This is normal and I had about 30 days’ worth.
You should first see some log messages on StudentVM1 indicating that its own instance of SendMail has received the message and various steps in processing it. A moment or so later, you should also see some messages in the log for StudentVM2 indicating it has received the email and is processing it.
We now know that our email server is working and that it is being used as the smart host by StudentVM1. Let’s send our message a bit further afield. This time we send the email to an external email account.
Trace the route of the email through the various hosts using the headers and the mail logs on the virtual hosts.
We now have a working email system with a server and a simple client. Note that with the setup we currently have, the student user needs to login to the StudentVM2 host to retrieve email using mailx. We will discuss email clients and the server requirements to support them in more detail in Chapter 8 of this volume.
SMTP – The protocol
SMTP – Simple Mail Transfer Protocol – is an ASCII plain text conversation used to transfer email between servers and from a sending email client to a server and is defined in RFC821. SMTP uses TCP port 25. The SMTP protocol is well defined in the Internet RFCs so it is an open standard. SMTP servers are known as mail transfer agents (MTAs) because their function is to transfer email messages between one another.
Let’s watch this conversation.
Experiment 7-5
As the student user on StudentVM1, send an email using the -v option of the mailx command. The SMTP protocol commands are shown with preceding >>> characters. These lines are highlighted to enhance their visibility. The responses from the mail server are not highlighted and begin with message ID numbers.
The SMTP return code classifications
Sometimes these messages, especially when they are error codes, are embedded in a returned email rejection. Other times the only way to see them is to use a tool like mailx and to observe the conversation for yourself.
Email-only accounts
Although the student user is a valid user in our experimental environment, it is a login account. Email servers need to be secure so that the owners of email accounts are unable to actually log in to the server. This is accomplished by creating nologin accounts for email-only users.
Experiment 7-6
Create a password for this account. The password is used when an email client attempts to retrieve email from this account while the nologin shell prevents a login as a Linux user. Test to verify that you cannot login with this new account by attempting to do so as the user email1 using a virtual console.
Who gets email for root?
Many system level services can send email to root@localhost to notify the root user of the completion of an at job, for example, the daily LogWatch report, and more, depending upon the specific tools and their configuration. These emails can get missed and ignored on many hosts that the SysAdmin does not login to frequently. Even with only a few hosts to login to each day, I found it a chore to do so just to check root’s emails.
I found an easy way to fix that now that our internal StudentVM1 host can use StudentVM2 as a smart host. I use the /etc/aliases file to send the email to my personal email address.
The /etc/aliases file contains aliases for the system users that defines who gets email sent to them. Many system services have user accounts associated and some of those services send email notifications to root or to another user. Web sites also have nonspecific email accounts such as [email protected] or [email protected]. So if someone sends an email to [email protected], the aliases file tells SendMail to route that email to root. This email ends up in root's local mailbox on the local host. If that is not what you want, and it usually is not, we need to change the /etc/aliases file.
I like to get email that is addressed to root sent to me at one of my regular email accounts so that I will be sure to get it. This allows me to keep track of notifications that might indicate a problem of some sort.
Experiment 7-7
Start this experiment as the root user on StudentVM1. We will change a few things to send notifications intended for root to the student user instead.
First, copy the current version of the /etc/aliases file to /tmp for a short-term backup. Then open /etc/aliases with a text editor. Study the entries and notice that some like the ftp entry send emails to root, while further down, four other entries, ftpadm, ftpadmin, ftp-adm, and ftp-admin, all redirect email to ftp.
Now make the same change to the aliases file on StudentVM2 and test it with emails sent to root or one of the other aliases like ftp.
By making this one change in the aliases file, I do not need to change the default email address in many different services.
Things to remember
There are some things to remember about email.
It is not instant
One of the most common misconceptions that end users have about email is that it is instant. It is not. Email may get held up at one of the MTAs for various reasons. Heavy traffic can delay emails and anything marked as bulk in one of the headers will be placed at the bottom of the queue and only sent when all emails with higher precedence have been sent. Any email without a precedence header is considered to be normal. Bulk email is sent from listservs and may have many addressees at any one domain.
I had one situation when working in a government organization where a PHB tried to ream me out and threatened me with some sort of disciplinary action because an email he sent did not get to the people in his building immediately upon being sent. The email he sent was to warn of an imminent tornado which was, in fact, bearing down on that part of the city at the time. But the email was sent to a list, and between being bulk mail as well as having hundreds of recipients in an email system that received more than 20,000,000 (yes 20 million) emails per day, it took some time to process the delivery of all those emails.
And, as we have mentioned before, one must be sitting at their computer with the email client up and running and watching for new emails to come in for this asynchronous communication system to be effective. Email is just not an appropriate communication method for that type of imminent danger.
There is no delivery guarantee
Another problem is the misconception that email will always get delivered. It won’t. Many email systems drop emails that don’t conform to their anti-spam or bulk mail policies. They may reject emails for many reasons and there is nothing that we on the sending end can do about it. Sure, we can call or email the designated contact for the domain, but in most cases, they ignore this type of complaint.
Emails also get dropped when routers become overburdened and start dropping packets. In this case, the sending server may try to send the email again, but there is still no guarantee of its ultimate delivery.
Chapter summary
In this chapter, we have learned to use SendMail as an SMTP mail transfer agent. We have configured SendMail to be our mail server as well as to forward internal emails from our own network hosts to the mail server as a smart host. We have used the mailx email client on both hosts to retrieve and send emails. We have also added an MX record and a supporting record to our DNS server and added a rule to the firewall on the server to allow incoming SMTP packets on port 25.
Although we cannot receive email from the outside world on our email server, be assured that it would work. Once it can receive email from internal network hosts, and with configuration of SendMail to listen on the appropriate external network interface, we would also be able to receive emails from outside domains.
There is more to be done to make this into a fully functioning email system, but we are well on the way.
Exercises
- 1.
Where are emails located in your inbox stored? Be specific with the host and the complete directory path.
- 2.
Why do we need to masquerade the email sending addresses?
- 3.
What TCP port does SMTP use?
- 4.
Why does email on our virtual network get sent to our own instance of example.com and not to the outside world instance?
- 5.
When sending email to an alias like www on StudentVM1, what is the To: address when the email arrives in the student email account on StudentVM2?
- 6.
How does an email only account differ from a regular Linux user account?
- 7.
What other things can you think of that we need to do to make our email server more functional and more secure?