Chapter 10. Email-based integration

 

 

If JMS is the first thing that comes to your mind when associating Java and messaging, then chances are that associating internet and messaging will make you think of email. We all know it well. For decades, email has been the primary method of exchanging digital messages, and has made its way from being the killer app of the nascent internet to becoming an indispensable personal and business communication tool.

But this isn’t a book about email etiquette, nor do we want to settle the age-old question of top-posting versus bottom-posting; we’re interested only in the implications of using email as an enterprise application integration tool. Besides its primary role as a means of communication between people, email is also a useful interaction medium for applications, either with users—by sending notifications or receiving requests—or with other applications. Email is a complex messaging mechanism, supporting broadcast, data delivery as attachments, and store-and-forward transmission with message relaying, and allowing a choice of message receiving protocols, either event-driven (IMAP push) or polling (POP, IMAP pull). Also, email is continuously evolving in response to changes in technology and challenges such as spam and fraud, incorporating more sophisticated mechanisms for secure data transmission, authentication, and authorization, to name just a few areas of interest.

In this chapter, we provide an overview of the most significant use cases that require email support, and you’ll learn how you can use Spring Integration to add it to your application. We show you how to send and receive email, the various options for doing so, and how to choose among them. Let’s start with sending emails, the more typically encountered use case.

10.1. Sending email

Event-driven interaction isn’t something that happens only between applications and systems; humans have been exchanging messages since the dawn of history, and electronic mail is a way of using modern technology for implementing an ancient communication pattern. But incorporating it in working applications opens the way for interaction workflows that improve and transform the user experience. For example, you may be waiting for a shipment (maybe your print copy of Spring Integration in Action), but you don’t know the exact date and time when you have to be home to receive it. You may try to find out when you’re supposed to be home by repeatedly checking the shipper’s site, but this can become a tedious exercise, and often a futile one, because the information may change frequently. “Hold on,” you say, “I’d rather get a notice when you know something,” and you subscribe to receive a notification when the status changes. Now you may carry on with your own business, assured that you’ll be informed without having to repeatedly check the shipper’s site for a status report.

You can find many other examples where a notification system is a valuable add-on to an application, and for a notification system to be efficient, it must use a mechanism that allows easy access on the recipient’s side. This assumption holds true for email; it’s widely available, either as a public service or as part of a corporate communication infrastructure, and because mobile devices have become so popular, email is accessible even when you’re on the go.

Although email is a widely used notification method, there are other ways of sending short notifications to users, such as Short Message Service (SMS) and social networking media, such as tweets and chat. Email differentiates itself from these other personal messaging systems through features such as the ability of carrying attachments, broadcast, store-and-forward delivery, delivery status availability, and failure notification, which make it interesting for use as an intersystem message exchange transport. Email isn’t a typical first choice for integrating applications hosted on the same machine or deployed in the same local network, where specifically designed messaging middleware, accessible via Java Message Service (JMS) or Advanced Message Queuing Protocol (AMQP), is a far better fit when considering any relevant criterion (performance, throughput, transaction capabilities, delivery guarantees). But emails can be used as an integration mechanism in other situations, for example for systems communicating over the Web. They also don’t require as much special setup for remote access, like exposing ports and addresses, as a message broker may normally require.

The process of sending email from within an application can be divided in two logical steps: (1) preparing an email message by populating its subject, body, attachments, and recipients list and (2) the operation of sending it. A complete solution involves a chain of Spring Integration endpoints with specific roles, typically similar to what you can see in figure 10.1.

Figure 10.1. Sending email requires a channel adapter and includes optional transformers for preparing outbound messages.

Because the key component of this chain (and the only one whose presence is mandatory) is the channel adapter, we look at that first.

10.1.1. The outbound channel adapter

The operation of sending email messages from your application is fairly straightforward and intuitive, considering that it’s based on a pattern you’re already familiar with: the outbound channel adapter. As a first step, you must configure such an adapter, which can be as simple as in the following code snippet, where the username and password attributes describe the login credentials for using the Simple Mail Transfer Protocol (SMTP) services provided by the host:

<mail:outbound-channel-adapter channel="outboundMail"
    host="${host}" username="${user}" password="${password}"/>

What you send in a message can be, in the simplest case, equivalent to the following snippet; the outbound channel adapter composes a message with the payload as the body and the recipients lists (to, cc, bcc) and subject as header values:

Message<String> message = MessageBuilder("Hello World!")
                .setHeader(MailHeaders.TO, "[email protected]")
                .setHeader(MailHeaders.SUBJECT, "Greeting")
                .build();
outboundMailChannel.send(message);

We said equivalent because sending mail in a programmatic fashion isn’t the only way to send it and perhaps isn’t even the most common: email messages are usually produced by upstream endpoints or by a publisher interceptor. The end result, regardless of what led to it, is that a Spring Integration message with a mail-specific payload and mail-specific headers is sent to a channel to which an outbound email channel adapter listens, and then composes an email message based on payload and header content and sends it.

One aspect to consider when composing messages for the email outbound channel adapter is that it supports only a limited number of payload choices. A string is the most straightforward alternative and can be either a predefined message or content generated from a template. Another type of payload accepted by the channel adapter is a byte array, in which case the resulting email message adds the binary content as an attachment. You can use additional headers for describing the attachment (filename, content type, and so on). A complete list of the message headers and how they’re used for constructing the outgoing message is found in table 10.1. The header name refers to string constants defined in the MailHeaders helper class.

Table 10.1. Spring Integration mail message headers

Header name	Meaning
MailHeaders.FROM	The sender of the message
MailHeaders.TO	List of recipients
MailHeaders.CC	List of carbon-copy (CC) recipients
MailHeaders.BCC	List of blind carbon-copy (BCC) recipients
MailHeaders.SUBJECT	Email subject
MailHeaders.REPLY_TO	Reply-to address
MailHeaders.MULTIPART_MODE	Indicates whether the message supports alternative texts, attachments, and inline content (used only for byte array messages)
MailHeaders.CONTENT_TYPE	Can be used to set an alternative content type for message body (for example, to send HTML messages instead of plain text)
MailHeaders.ATTACHMENT_FILENAME	Filename of the attachment (used only for byte array messages)

To understand more advanced ways of composing email messages and configuring outbound email channel adapters, we need to look closer at the innards of the mail-sending process.

10.1.2. Advanced configuration options

Under the hood, the Spring Integration email outbound channel adapter is based on the JavaMail API and on additional utilities provided by the Spring Framework.

The underlying usage of JavaMail means that you can configure the channel adapter by inserting additional properties through the java-mail-properties attribute of the <mail:outbound-channel-adapter/> element. This mechanism is used for finely tuning the configuration parameters, adding connectivity features such as Secure Sockets Layer (SSL) support for sending messages, as shown in the following snippet. For a complete list of the configuration options and their meanings, please consult the JavaMail documentation:

<mail:outbound-channel-adapter channel="outboundMail"
       host="${host}" username="${user}" password="${password}"
       java-mail-properties="javaMailProperties"/>

<util:properties id="javaMailProperties">
  <prop key="mail.imap.socketFactory.class"> javax.net.ssl.SSLSocketFactory</prop>
  <prop key="mail.imap.socketFactory.fallback">false</prop>
  <prop key="mail.store.protocol">imaps</prop>
  <prop key="mail.transport.protocol">smtps</prop>
  <prop key="mail.smtps.auth">true</prop>
  <prop key="mail.smtp.starttls.enable">true</prop>
</util:properties>

For applications that need to send email (especially the ones that don’t use Spring Integration), using the JavaMail API directly can become a cumbersome exercise, which involves operating directly with the low-level components of the API and direct manipulation of resources. To simplify the general process of sending and receiving email, the Spring Framework provides a higher-level API for sending emails, the JavaMailSender.

In a nutshell, sending or receiving email requires interacting with instances of the type javax.mail.internet.MimeMessage, which is the JavaMail representation of an email message. When receiving messages, dealing directly with such objects is fairly simple because they provide access to all the properties of the email (as explained in the following section). But when sending messages, creating and manipulating Mime-Message instances requires access to a low-level resource, the javax.mail.Session that represents the connection to the mail server. Out of the box it’s impossible to separate concerns in a good manner and create a service layer that’s responsible for composing an infrastructure-independent email, which is a problem when considering, for example, the need for unit testing such code. Spring therefore provides the JavaMailSender abstraction, an intermediate layer that uses a Spring-specific object, the MailMessage, as an input and interacts with the infrastructure for sending messages, as shown in figure 10.2. For more details, we encourage you to read the email section of Spring in Action by Craig Walls (Manning, 2011).

Figure 10.2. Spring Integration uses the mail sending layer provided by the Spring Framework proper. The components from each layer, as well as the mail message abstractions, are shown on the right side.

Using the JavaMailSender has two consequences. First, users can inject a JavaMailSender object directly in the outbound channel adapter, which can also act as a general configuration for sending emails (rather than, for example, individually configuring each channel adapter). Second, the outbound channel adapter accepts two other types of object as payload: the raw JavaMail MimeMessage and the Spring-specific MailMessage, which allows developers to compose messages using the Spring Framework utilities and send them through the channel adapter.

Sending email is usually part of a larger chain of events, and to illustrate that, let’s consider a business example. You saw in chapter 6 that as soon as the application is notified of a flight schedule change, it must inform all the customers who are booked on that flight. The upstream service activator takes care of searching for accounts and producing one or more Notification objects.

10.1.3. Transforming outbound messages

What can you do with messages whose payloads are not strings, byte arrays, or Mail-Message/MimeMessage instances? If you can’t deal elegantly with domain objects, the adapter will serve only a limited set of corner cases. One of the strengths of Spring Integration is decoupling between components, so it may not seem too far a stretch that a service activator won’t want to produce dedicated MailMessage instances, but domain-specific notifications, leaving it to the downstream components to translate them into email-ready instances. You may also need this functionality when supporting more than one kind of notification type by the application; besides email, you may have SMS messages or even direct phone calls. In such cases, an upstream component will create a generic Notification instance, which is sent to a publish-subscribe channel, with the email-sending component being just one of the subscribers.

The pipes-and-filters architecture of Spring Integration has a simple solution for this: all you need to do is include a message transformer that converts the message with a plain old Java object (POJO) payload into one of the four compatible payloads, as shown in figure 10.1.

Generally speaking, it’s not just about the payload. If the transformer logic is only dealing with the payload (for example, creating a string message based on the content of the Notification object), you can use a dedicated header enricher component for populating the message headers with appropriate values. A header enricher can use static values or Spring Expression Language (SpEL) expressions evaluated against the message to be enriched. In general, the best practice is to insert the header enricher first because the transformer may remove information that’s required for populating the headers; for example, the recipient of the message may be one of the properties of the Notification object but may not be contained in the message body payload. A complete example, including the transformer and the optional header enricher, is shown in the following snippet. To simplify the configuration, the endpoints are chained together:

<chain input-channel="notificationRequest"
       output-channel="outboundMail">
  <service-activator ref="notificationService"/>
  <mail:header-enricher>
    <mail:to expression="payload.email"/>
    <mail:from value="${settings.email.from}"/>
    <mail:subject value="Notification"/>
  </mail:header-enricher>
  <transformer ref="templateMessageGenerator"/>
</chain>

This concludes the section dedicated to sending email. It’s time now to discuss how to handle inbound email messages.

10.2. Receiving email

Sending notification emails is the most common use case, but a significant number of applications also support an email-driven application flow. Sending documents as email attachments can be an alternative to uploading them on a web page, especially when you don’t want to put up with creating a dedicated website. Also, emails carry the identity of the sender, which works pretty well in request-reply scenarios, because the identity of the sender and the reply-to information is communicated as part of the email. And, as we mentioned in the previous section, email can serve as transport for carrying data over the web when using a message broker would create too many logistical and infrastructural issues.

For example, your application may offer users the opportunity to perform flight status checks through email. The company provides an email address to which customers send an email with the keyword STATUS and the flight number in the subject, and the application replies with the flight information for today.

Our application acts as an email client, reading the contents of the mailbox and processing every incoming message as a request. The general structure of an email-receiving application is shown in figure 10.3. The inbound channel adapter creates messages with a JavaMail MimeMessage payload, and the messages are sent on the adapted channel for further processing downstream. For separating concerns in the application, the MimeMessages are converted to domain objects downstream, typically using a transformer.

Figure 10.3. Receiving emails requires a channel adapter and includes optional transformers for extracting message content.

The first step for creating such an application is deciding how to receive emails—what kind of a channel adapter you’d like to include. You can either poll the mailbox or be notified when new messages arrive, so let’s examine these two options before we discuss how to process inbound messages.

10.2.1. Polling for emails

Polling is the most basic email-receiving strategy, and it’s supported by all mail servers. A client connects to the mailbox and downloads any unread messages that have arrived since the last polling cycle.

A polling channel adapter can be configured as in the following example:

<mail:inbound-channel-adapter id="mailAdapter"
     store-uri="imaps://[username]:[password]@imap.gmail.com:993/INBOX"
     java-mail-properties="javaMailProperties"
     channel="emails"
     should-delete-messages="true"
     should-mark-messages-as-read="true">
     <poller max-messages-per-poll="1" fixed-rate="5000"/>
</mail:inbound-channel-adapter>

This example describes an Internet Message Access Protocol (IMAP)-based channel adapter, but Spring Integration supports both of the two major email access protocols, Post Office Protocol 3 (POP3) and IMAP, through the same channel adapter element. Whether the adapter uses one or the other protocol is indicated in the store-uri attribute, which in this example indicates that secure IMAP (imaps) is in use, and additional JavaMail properties are set on the adapter through the java-mail-properties attribute. The protocol that will be used in a particular case depends on the setup of the mail server where the incoming mailbox is located, and it’s not typically left to you as an option.

Though considering the differences between protocols may not help you choose one over the other, it may help you understand how the protocols work and what consequences configuration options such as should-delete-messages have on the application.

POP3 is a protocol designed for downloading messages locally. Mailboxes are set up as a temporary storage, and the assumed permanent destination of the messages is the local store provided by the email client. This means the server is somewhat aware that a message has been read, but only for the duration of a session. In Spring Integration, an email client session is started when the application starts and is closed on shutdown, so any messages that aren’t deleted from the server before shutdown may be received multiple times. To solve the issue of duplicate messages, an application can choose to delete messages from the server as soon as they are read by setting should-delete-messages to true, but destructive reading carries its own risks as well. Because there’s no such concept as a transaction, a failure that occurs between the moment the email has been read and the moment it has been processed may result in the message being lost. Moreover, using the mailbox in a multiple-client, publish-subscribe scenario would be problematic if one of the subscribers started deleting emails. Because there’s no single best choice here, Spring Integration forces you to state explicitly which one of the two values you want to set, essentially deciding between providing a contingency on loss or duplication.

IMAP is a protocol designed for storing messages remotely. Mailboxes are the permanent storage for emails, and email clients, by default, provide a view of the remote mailbox state rather than retrieving content locally. Because the remote mailbox maintains state, deleting messages isn’t necessary to avoid duplicate reception. In the case of IMAP, you can also decide whether you want to mark processed messages as read. Setting this option on a POP3 adapter is possible, but the value will be ignored because POP3 doesn’t provide an updatable persistent mailbox state.

IMAP provides a more robust alternative to POP3 in polling scenarios, but its true advantage is that, under certain circumstances, you can replace polling with event-driven reception.

10.2.2. Event-driven email reception

Although polling is available for any email server, for particular server configurations it’s also possible to set up an event-driven inbound channel adapter for receiving emails. This functionality is available when connecting to IMAP servers that support IMAP-Idle, a feature that allows clients to request real-time notifications on mailbox changes. As a result, clients don’t have to poll the server for new messages; they’re notified when new messages are added to the mailbox.

This approach is lighter on resource consumption and traffic because the application doesn’t have to actively check the server for new messages. It also improves responsiveness because applications are notified as soon as a message arrives, without the lag introduced by a polling cycle. But it’s important to note that the IMAP-Idle command has a timeout of, at most, 30 minutes, which means that clients have to reissue it periodically.

For using the IMAP-Idle message retrieval strategy, all you need to do is use a different type of channel adapter:

<mail:imap-idle-channel-adapter id="mailAdapter"
     store-uri="imaps://[username]:[password]@imap.gmail.com:993/INBOX"
     channel="emails"
     auto-startup="true"
     should-delete-messages="false"
     should-mark-messages-as-read="true"
     java-mail-properties="javaMailProperties"/>

Whether it’s polling or event-driven, the inbound email channel adapter will create a message with a payload of the JavaMail’s Message type. To handle these messages in the application, you must transform them to a more neutral format, such as a string or a POJO.

10.2.3. Transforming inbound messages

Once you configure your inbound channel adapter, the next step is to process the received messages downstream. For handling the lower-level JavaMail Message objects in the domain-aware services, it’s typical to include a transformer before the objects enter the application flow.

The simplest solution is to use a SpEL-based transformer that’s evaluated against the JavaMail Message payload of the inbound message. For example, the transformer in the following snippet extracts the message body and creates a new Spring Integration message, using it as a payload. You can write more complex SpEL expressions to include other properties of the message, such as the subject, sender, or date:

<chain input-channel="emails"
       output-channel="handled-emails">
  <transformer expression="payload.content"/>
  <service-activator ref="emailHandlingService"/>
</chain>

Spring Integration provides an out-of-the box transformer you can use for extracting the email body or the binary content into a string payload. One of the benefits of using it is that it also extracts the other email properties as message headers, mirroring what happens when you send a message. Review table 10.1 for a reminder of the mail property/message headers mapping. Here’s an example where it replaces the previous SpEL-based transformer:

<chain input-channel="emails"
       output-channel="handled-emails">
  <mail:mail-to-string-transformer />
  <service-activator ref="emailHandlingService"/>
</chain>

If you want to customize the transformation rather than extracting the payload as a string, you can replace the <mail-to-string-transformer> here with a regular transformer that accepts an inbound javax.mail.Message payload. Extend the AbtsractMailMessageTransformer class provided by the framework and implement the logic for extracting the message content, as in the following snippet:

public class MyMailMessageTransformer extends AbstractMailMessageTransformer<InputData> {
    @Override
    protected MessageBuilder<InputData> doTransform(javax.mail.Message mailMessage) throws Exception {
        InputData inputData = new InputData();
        // populate inputData from message
        return MessageBuilder.withPayload(inputData);
    }
}

Then you can add your implementation to the chain using a regular custom transformer setup:

<chain input-channel="emails"
       output-channel="handled-emails">
  <transformer>
    <beans:bean class="example.MyMailMessageTransformer"/>
  </transformer>
  <service-activator ref="emailHandlingService"/>
</chain>

The resulting transformer will also extract the mail properties as header values, as in the case of the string transformer. One of the advantages of the latter two approaches over the SpEL transformer is that they automatically provide access to entries, such as the originator of the message and the reply address, which may be useful when handling the message and are important if you want to implement an ad hoc gateway that sends back an email response to an inbound email request. For implementing such a round-trip solution, all you need to do is create a sequence of endpoints that starts with an inbound channel adapter and ends with an outbound channel adapter.

10.3. Summary

Email is a ubiquitous modern communication technology that can be used for human interaction, for allowing applications to send notifications or receive requests via email from users, and as a transport mechanism, especially between remote applications. On criteria such as performance and reliability, email can’t compete with dedicated messaging middleware, but it has some qualities that make it attractive in internet communication: it’s widely available and accessible to users, it can transfer information over the internet without requiring custom firewall access, and it supports small to mid-sized attachments.

Spring Integration provides support for sending and receiving emails through dedicated channel adapters, which cover all the major connectivity options and email transfer protocols (POP3, IMAP, SMTP). It provides out-of-the box support for handling the most typical use cases, such as text messages (including HTML) and file attachments; and special situations, such as complex email structures and conversion to POJOs, can be easily addressed by including customized transformers in the processing chain.

This wraps up our introduction to email support in Spring Integration, and it’s time to look at our next topic: file support.