SMTP, MSAs, and MTAs

The lingua franca for sending email is the Simple Mail Transfer Protocol (SMTP). While it is possible to use SMTP to send an email directly to the recipient’s mail server, this is generally a bad idea: unless you are a “trusted sender” like Google or Yahoo!, chances are your email will be tossed directly into the spam bin. It’s better to use a mail submission agent (MSA), which will deliver the email through trusted channels, reducing the chance that your email will be marked as spam. In addition to ensuring that your email arrives, MSAs handle nuisances like temporary outages and bounced emails. The final piece of the equation is the mail transfer agent (MTA), which is the service that actually sends the email to its final destination. For the purposes of this book, MSA, MTA, and SMTP server are essentially equivalent.

So you’ll need access to an MSA. While it is possible to get started using a free consumer email service such as Gmail, Outlook, or Yahoo!, these services are no longer as friendly to automated emails as they once were (in an effort to cut down on abuse). Fortunately, there are a couple of excellent email services to choose from that have a free option for low-volume use: Sendgrid and Mailgun. I’ve used both services, and I like them both. The examples in this book will be using SendGrid.

If you’re working for an organization, the organization itself may have an MSA; you can contact your IT department and ask them if there’s an SMTP relay available for sending automated emails.

If you’re using SendGrid or Mailgun, go ahead and set up your account now. For SendGrid, you’ll need to create an API key (which will be your SMTP password).

Receiving Email

Most websites only need the ability to send email, like password reset instructions and promotional emails. However, some applications need to receive email as well. A good example is an issue-tracking system that sends out an email when someone updates an issue, and if you reply to that email, the issue is automatically updated with your response.

Unfortunately, receiving email is much more involved and will not be covered in this book. If this is functionality you need, you should allow your mail provider to maintain the mailbox and have a periodic process to access it with an IMAP agent such as imap-simple.

Email Headers

An email message consists of two parts: the header and the body (very much like an HTTP request). The header contains information about the email: who it’s from, who it’s addressed to, the date it was received, the subject, and more. Those are the headers that are normally displayed to the user in an email application, but there are many more headers. Most email clients allow you to look at the headers; if you’ve never done so, I recommend you take a look. The headers give you all the information about how the email got to you; every server and MTA that the email passed through will be listed in the header.

It often comes as a surprise to people that some headers, like the “from” address, can be set arbitrarily by the sender. When you specify a “from” address other than the account from which you’re sending, it’s often referred to as spoofing. There is nothing preventing you from sending an email with the “from” address Bill Gates <[email protected]>. I’m not recommending that you try this, just driving home the point that you can set certain headers to be whatever you want. Sometimes there are legitimate reasons to do this, but you should never abuse it.

An email you send must have a “from” address, however. This can sometimes cause problems when sending automated email, which is why you often see email with a return addresses like DO NOT REPLY <[email protected]>. Whether you want to take this approach or have automated emails come from an address like Meadowlark Travel <[email protected]> is up to you; if you take the latter approach, though, you should be prepared to respond to emails that come to [email protected].

Email Formats

When the internet was new, all email was simply ASCII text. The world has changed a lot since then, and people want to send email in different languages and do more sophisticated things like include formatted text, images, and attachments. This is where things start to get ugly: email formats and encoding are a horrible jumble of techniques and standards.

Fortunately, we won’t really have to address these complexities. Nodemailer will handle that for us. What’s important for you to know is that your email can be either plain text (Unicode) or HTML.

Almost all modern email applications support HTML email, so it’s generally pretty safe to format your emails in HTML. Still, there are “text purists” out there who eschew HTML email, so I recommend always including both text and HTML email. If you don’t want to have to write text and HTML email, Nodemailer supports a shortcut that will automatically generate the plain text version from the HTML.

HTML Email

HTML email is a topic that could fill an entire book. Unfortunately, it’s not as simple as just writing HTML as you would for your site: most mail clients support only a small subset of HTML. Mostly, you have to write HTML as if it were still 1996; it’s not much fun. In particular, you have to go back to using tables for layout (cue sad music).

If you have experience with browser compatibility issues with HTML, you know what a headache it can be. Email compatibility issues are much worse. Fortunately, there are some things that can help.

First, I encourage you to read MailChimp’s excellent article about writing HTML email. It does a good job covering the basics and explaining the things you need to keep in mind when writing HTML email.

The next is a real time-saver: HTML Email Boilerplate. It’s essentially a very well-written, rigorously tested template for HTML email.

Finally, there’s testing. You’ve read up on how to write HTML email, and you’re using HTML Email Boilerplate, but testing is the only way to know for sure your email is not going to explode on Lotus Notes 7 (yes, people still use it). Feel like installing 30 different mail clients to test one email? I didn’t think so. Fortunately, there’s a great service that does it for you: Litmus. It’s not an inexpensive service; plans start at about $100 a month. But if you send a lot of promotional emails, it’s hard to beat.

On the other hand, if your formatting is modest, there’s no need for an expensive testing service like Litmus. If you’re sticking to things like headers, bold/italic text, horizontal rules, and some image links, you’re pretty safe.

Nodemailer

First, we need to install the Nodemailer package:

npm install nodemailer

Then, require the nodemailer package and create a Nodemailer instance (a transport in Nodemailer parlance):

const nodemailer = require('nodemailer')

const mailTransport = nodemailer.createTransport({

  auth: {
    user: credentials.sendgrid.user,
    pass: credentials.sendgrid.password,
  }
})

Notice we’re using the credentials module we set up in Chapter 9. You’ll need to update your .credentials.development.json file accordingly:

{
  "cookieSecret": "your cookie secret goes here",
  "sendgrid": {
    "user": "your sendgrid username",
    "password": "your sendgrid password"
  }
}

Common configuration options for SMTP are the port, authentication type, and TLS options. However, most of the major mail services use the default options. To find out what settings to use, consult your mail service documentation (try searching for sending SMTP email or SMTP configuration or SMTP relay). If you’re having trouble sending SMTP email, you may need to check the options; see the Nodemailer documentation for a complete list of supported options.

try {
  const result = await mailTransport.sendMail({
    from: '"Meadowlark Travel" <[email protected]>',
    to: '[email protected]',
    subject: 'Your Meadowlark Travel Tour',
    text: 'Thank you for booking your trip with Meadowlark Travel.  ' +
      'We look forward to your visit!',
  })
  console.log('mail sent successfully: ', result)
} catch(err) {
  console.log('could not send mail: ' + err.message)
}

try {
  const result = await mailTransport.sendMail({
    from: '"Meadowlark Travel" <[email protected]>',
    to: '[email protected], "Jane Customer" <[email protected]>, ' +
      '[email protected]',
    subject: 'Your Meadowlark Travel Tour',
    text: 'Thank you for booking your trip with Meadowlark Travel.  ' +
      'We look forward to your visit!',
  })
  console.log('mail sent successfully: ', result)
} catch(err) {
  console.log('could not send mail: ' + err.message)
}

// largeRecipientList is an array of email addresses
const recipientLimit = 100
const batches = largeRecipientList.reduce((batches, r) => {
  const lastBatch = batches[batches.length - 1]
  if(lastBatch.length < recipientLimit)
    lastBatch.push(r)
  else
    batches.push([r])
  return batches
}, [[]])
try {
  const results = await Promise.all(batches.map(batch =>
    mailTransport.sendMail({
      from: '"Meadowlark Travel", <[email protected]>',
      to: batch.join(', '),
      subject: 'Special price on Hood River travel package!',
      text: 'Book your trip to scenic Hood River now!',
    })
  ))
  console.log(results)
} catch(err) {
  console.log('at least one email batch failed: ' + err.message)
}

Better Options for Bulk Email

While you can certainly send bulk email with Nodemailer and an appropriate MSA, you should think carefully before going this route. A responsible email campaign must provide a way for people to unsubscribe from your promotional emails, and that is not a trivial task. Multiply that by every subscription list you maintain (perhaps you have a weekly newsletter and a special announcements campaign, for example). This is an area in which it’s best not to reinvent the wheel. Services like Emma, Mailchimp, and Campaign Monitor offer everything you need, including great tools for monitoring the success of your email campaigns. They’re very affordable, and I highly recommend using them for promotional emails, newsletters, etc.

Sending HTML Email

So far, we’ve just been sending plain-text email, but most people these days expect something a little prettier. Nodemailer allows you to send both HTML and plaintext versions in the same email, allowing the email client to choose which version is displayed (usually HTML) (ch11/03-html-email.js in the companion repo):

const result = await mailTransport.sendMail({
  from: '"Meadowlark Travel" <[email protected]>',
  to: '[email protected], "Jane Customer" <[email protected]>, ' +
    '[email protected]',
  subject: 'Your Meadowlark Travel Tour',
  html: '<h1>Meadowlark Travel</h1>
<p>Thanks for book your trip with ' +
    'Meadowlark Travel.  <b>We look forward to your visit!</b>',
  text: 'Thank you for booking your trip with Meadowlark Travel.  ' +
    'We look forward to your visit!',
})

Providing both HTML and text versions is a lot of work, especially if very few of your users prefer text-only email. If you want to save some time, you can write your emails in HTML and use a package like html-to-formatted-text to automatically generate text from your HTML. (Just keep in mind that it won’t be as high quality as hand-crafted text; HTML won’t always translate cleanly.)

<img src="//meadowlarktravel.com/email/logo.png"
  alt="Meadowlark Travel Logo">

<p>Thank you for booking your trip with Meadowlark Travel,
  {{cart.billing.name}}!</p>
<p>Your reservation number is {{cart.number}}, and an email has been
sent to {{cart.billing.email}} for your records.</p>

<table cellpadding="0" cellspacing="0" border="0" id="backgroundTable">
  <tr>
    <td valign="top">
      <table cellpadding="0" cellspacing="0" border="0" align="center">
        <tr>
          <td width="200" valign="top"><img class="image_fix"
            src="//placehold.it/100x100"
            alt="Meadowlark Travel" title="Meadowlark Travel"
            width="180" height="220" /></td>
        </tr>
        <tr>
          <td width="200" valign="top"><p>
            Thank you for booking your trip with Meadowlark Travel,
            {{cart.billing.name}}.</p><p>Your reservation number
            is {{cart.number}}.</p></td>
        </tr>
        <tr>
          <td width="200" valign="top">Problems with your reservation?
          Contact Meadowlark Travel at
          <span class="mobile_link">555-555-0123</span>.</td>
        </tr>
      </table>
    </td>
  </tr>
</table>

app.post('/cart/checkout', (req, res, next) => {
  const cart = req.session.cart
  if(!cart) next(new Error('Cart does not exist.'))
  const name = req.body.name || '', email = req.body.email || ''
  // input validation
  if(!email.match(VALID_EMAIL_REGEX))
    return res.next(new Error('Invalid email address.'))
  // assign a random cart ID; normally we would use a database ID here
  cart.number = Math.random().toString().replace(/^0.0*/, '')
  cart.billing = {
    name: name,
    email: email,
  }
  res.render('email/cart-thank-you', { layout: null, cart: cart },
    (err,html) => {
        console.log('rendered email: ', html)
        if(err) console.log('error in email template')
        mailTransport.sendMail({
          from: '"Meadowlark Travel": [email protected]',
          to: cart.billing.email,
          subject: 'Thank You for Book your Trip with Meadowlark Travel',
          html: html,
          text: htmlToFormattedText(html),
        })
          .then(info => {
            console.log('sent! ', info)
            res.render('cart-thank-you', { cart: cart })
          })
          .catch(err => {
            console.error('Unable to send confirmation: ' + err.message)
          })
    }
  )
})

const nodemailer = require('nodemailer')
const htmlToFormattedText = require('html-to-formatted-text')

module.exports = credentials => {

  const mailTransport = nodemailer.createTransport({
    host: 'smtp.sendgrid.net',
    auth: {
      user: credentials.sendgrid.user,
      pass: credentials.sendgrid.password,
    },
  })

  const from = '"Meadowlark Travel" <[email protected]>'
  const errorRecipient = '[email protected]'

  return {
    send: (to, subject, html) =>
      mailTransport.sendMail({
        from,
        to,
        subject,
        html,
        text: htmlToFormattedText(html),
      }),
  }

}

const emailService = require('./lib/email')(credentials)

emailService.send(email, "Hood River tours on sale today!",
  "Get 'em while they're hot!")

Conclusion

In this chapter, you learned the basics of how email is delivered on the internet. If you were following along, you set up a free email service (most likely SendGrid or Mailgun) and used the service to send text and HTML email. You learned how we can use the same template rendering mechanism we use for rendering HTML in our Express applications to render HTML for email.

Email remains an important way your application can communicate to your users. Be mindful not to abuse this power! If you’re like me, you have an inbox overflowing with automated email that you mostly ignore. When it comes to automated emails, less is more. There are legitimate and useful reasons your application could send an email to your users, but you should always ask yourself, “Do my users really want this email? Is there another way to communicate this information?”

Now that we’ve covered some basic infrastructure that we’ll need to create applications, we’ll spend some time talking about the eventual production launch of our application and the kinds of things we’ll want to consider to make that launch successful.