9

Behavioral Design Patterns

In the last two chapters, we have learned patterns that aid us in the creation of objects and with building complex object structures. Now it's time to move onto another aspect of software design, which concerns the behavior of components. In this chapter, we will learn how to combine objects and how to define the way they communicate so that the behavior of the resulting structure becomes extensible, modular, reusable, and adaptable. Problems such as "How do I change parts of an algorithm at runtime?", "How can I change the behavior of an object based on its state?", and "How can I iterate over a collection without knowing its implementation?" are the typical kinds of problems solved by the patterns presented in this chapter.

You've already met a notable member of this category of patterns, and that is the Observer pattern, which we presented in Chapter 3, Callbacks and Events. The Observer pattern is one of the foundational patterns of the Node.js platform as it provides us with a simple interface for dealing with events and subscriptions, which are the life force of Node's event-driven architecture.

If you are already familiar with the Gang of Four (GoF) design patterns, in this chapter, you will witness once again how the implementation of some of those patterns can be radically different in JavaScript compared to a purer object-oriented approach. A great example of this thesis can be found in the Iterator pattern, which you will meet later in the chapter. To implement the Iterator pattern, in fact, we won't need to extend any class or build any complex hierarchy. Instead, we will just need to add a special method to a class. Moreover, one particular pattern in this chapter, Middleware, tightly resembles another popular GoF pattern, which is the Chain of Responsibility pattern, but its implementation in Node.js has become such a standard that it can be considered a pattern of its own.

Now, it's time to roll up your sleeves and get your hands dirty with some behavioral design patterns. In this chapter, you will learn about the following:

• The Strategy pattern, which helps us change parts of a component to adapt it to specific needs
• The State pattern, which allows us to change the behavior of a component based on its state
• The Template pattern, which allows us to reuse the structure of a component to define new ones
• The Iterator pattern, which provides us with a common interface to iterate over a collection
• The Middleware pattern, which allows us to define a modular chain of processing steps
• The Command pattern, which materializes the information required to execute a routine, allowing such information to be easily transferred, stored, and processed

Strategy

The Strategy pattern enables an object, called the context, to support variations in its logic by extracting the variable parts into separate, interchangeable objects called strategies. The context implements the common logic of a family of algorithms, while a strategy implements the mutable parts, allowing the context to adapt its behavior depending on different factors, such as an input value, a system configuration, or user preferences.

Strategies are usually part of a family of solutions and all of them implement the same interface expected by the context. The following figure shows the situation we just described:

Figure 9.1: General structure of the Strategy pattern

Figure 9.1 shows you how the context object can plug different strategies into its structure as if they were replaceable parts of a piece of machinery. Imagine a car; its tires can be considered its strategy for adapting to different road conditions. We can fit winter tires to go on snowy roads thanks to their studs, while we can decide to fit high-performance tires for traveling mainly on motorways for a long trip. On the one hand, we don't want to change the entire car for this to be possible, and on the other, we don't want a car with eight wheels so that it can go on every possible road.

We quickly understand how powerful this pattern is. Not only does it help with separating the concerns within a given problem, but it also enables our solution to have better flexibility and adapt to different variations of the same problem.

The Strategy pattern is particularly useful in all those situations where supporting variations in the behavior of a component requires complex conditional logic (lots of if...else or switch statements) or mixing different components of the same family. Imagine an object called Order that represents an online order on an e-commerce website. The object has a method called pay() that, as it says, finalizes the order and transfers the funds from the user to the online store.

To support different payment systems, we have a couple of options:

• Use an if...else statement in the pay() method to complete the operation based on the chosen payment option
• Delegate the logic of the payment to a strategy object that implements the logic for the specific payment gateway selected by the user

In the first solution, our Order object cannot support other payment methods unless its code is modified. Also, this can become quite complex when the number of payment options grows. Instead, using the Strategy pattern enables the Order object to support a virtually unlimited number of payment methods and keeps its scope limited to only managing the details of the user, the purchased items, and the relative price while delegating the job of completing the payment to another object.

Let's now demonstrate this pattern with a simple, realistic example.

Multi-format configuration objects

Let's consider an object called Config that holds a set of configuration parameters used by an application, such as the database URL, the listening port of the server, and so on. The Config object should be able to provide a simple interface to access these parameters, but also a way to import and export the configuration using persistent storage, such as a file. We want to be able to support different formats to store the configuration, for example, JSON, INI, or YAML.

By applying what we learned about the Strategy pattern, we can immediately identify the variable part of the Config object, which is the functionality that allows us to serialize and deserialize the configuration. This is going to be implemented by our strategies.

Let's create a new module called config.js, and let's define the generic part of our configuration manager:

import { promises as fs } from 'fs'
import objectPath from 'object-path'
export class Config {
  constructor (formatStrategy) {                           // (1)
    this.data = {}
    this.formatStrategy = formatStrategy
  }
  get (configPath) {                                       // (2)
    return objectPath.get(this.data, configPath)
  }
  set (configPath, value) {                                // (2)
    return objectPath.set(this.data, configPath, value)
  }
  async load (filePath) {                                  // (3)
    console.log(`Deserializing from ${filePath}`)
    this.data = this.formatStrategy.deserialize(
      await fs.readFile(filePath, 'utf-8')
    )
  }
  async save (filePath) {                                  // (3)
    console.log(`Serializing to ${filePath}`)
    await fs.writeFile(filePath,
      this.formatStrategy.serialize(this.data))
  }
}

This is what's happening in the preceding code:

1. In the constructor, we create an instance variable called data to hold the configuration data. Then we also store formatStrategy, which represents the component that we will use to parse and serialize the data.
2. We provide two methods, set() and get(), to access the configuration properties using a dotted path notation (for example, property.subProperty) by leveraging a library called object-path (nodejsdp.link/object-path).
3. The load() and save() methods are where we delegate, respectively, the deserialization and serialization of the data to our strategy. This is where the logic of the Config class is altered based on the formatStrategy passed as an input in the constructor.

As we can see, this very simple and neat design allows the Config object to seamlessly support different file formats when loading and saving its data. The best part is that the logic to support those various formats is not hardcoded anywhere, so the Config class can adapt without any modification to virtually any file format, given the right strategy.

To demonstrate this characteristic, let's now create a couple of format strategies in a file called strategies.js. Let's start with a strategy for parsing and serializing data using the INI file format, which is a widely used configuration format (more info about it here: nodejsdp.link/ini-format).

For the task, we will use an npm package called ini (nodejsdp.link/ini):

import ini from 'ini'
export const iniStrategy = {
  deserialize: data => ini.parse(data),
  serialize: data => ini.stringify(data)
}

Nothing really complicated! Our strategy simply implements the agreed interface, so that it can be used by the Config object.

Similarly, the next strategy that we are going to create allows us to support the JSON file format, widely used in JavaScript and in the web development ecosystem in general:

export const jsonStrategy = {
  deserialize: data => JSON.parse(data),
  serialize: data => JSON.stringify(data, null, '  ')
}

Now, to show you how everything comes together, let's create a file named index.js, and let's try to load and save a sample configuration using different formats:

import { Config } from './config.js'
import { jsonStrategy, iniStrategy } from './strategies.js'
async function main () {
  const iniConfig = new Config(iniStrategy)
  await iniConfig.load('samples/conf.ini')
  iniConfig.set('book.nodejs', 'design patterns')
  await iniConfig.save('samples/conf_mod.ini')
  const jsonConfig = new Config(jsonStrategy)
  await jsonConfig.load('samples/conf.json')
  jsonConfig.set('book.nodejs', 'design patterns')
  await jsonConfig.save('samples/conf_mod.json')
}
main()

Our test module reveals the core properties of the Strategy pattern. We defined only one Config class, which implements the common parts of our configuration manager, then, by using different strategies for serializing and deserializing data, we created different Config class instances supporting different file formats.

The example we've just seen showed us only one of the possible alternatives that we had for selecting a strategy. Other valid approaches might have been the following:

• Creating two different strategy families: One for the deserialization and the other for the serialization. This would have allowed reading from a format and saving to another.
• Dynamically selecting the strategy: Depending on the extension of the file provided, the Config object could have maintained a map extension → strategy and used it to select the right algorithm for the given extension.

As we can see, we have several options for selecting the strategy to use, and the right one only depends on your requirements and the tradeoff in terms of features and the simplicity you want to obtain.

Furthermore, the implementation of the pattern itself can vary a lot as well. For example, in its simplest form, the context and the strategy can both be simple functions:

function context(strategy) {...}

Even though this may seem insignificant, it should not be underestimated in a programming language such as JavaScript, where functions are first-class citizens and used as much as fully-fledged objects.

Between all these variations, though, what does not change is the idea behind the pattern; as always, the implementation can slightly change but the core concepts that drive the pattern are always the same.

In the wild

Passport (nodejsdp.link/passportjs) is an authentication framework for Node.js, which allows a web server to support different authentication schemes. With Passport, we can provide a login with Facebook or login with Twitter functionality to our web application with minimal effort. Passport uses the Strategy pattern to separate the common logic used during an authentication process from the parts that can change, namely the actual authentication step. For example, we might want to use OAuth in order to obtain an access token to access a Facebook or Twitter profile, or simply use a local database to verify a username/password pair. For Passport, these are all different strategies for completing the authentication process and, as we can imagine, this allows the library to support a virtually unlimited number of authentication services. Take a look at the number of different authentication providers supported at nodejsdp.link/passport-strategies to get an idea of what the Strategy pattern can do.

State

The State pattern is a specialization of the Strategy pattern where the strategy changes depending on the state of the context.

We have seen in the previous section how a strategy can be selected based on different variables such as a configuration property or an input parameter, and once this selection is done, the strategy remains unchanged for the rest of the lifespan of the context object. In the State pattern, instead, the strategy (also called the state in this circumstance) is dynamic and can change during the lifetime of the context, thus allowing its behavior to adapt depending on its internal state.

The following figure shows us a representation of the pattern:

Figure 9.2: The State pattern

Figure 9.2 shows how a context object transitions through three states (A, B, and C). With the State pattern, at each different context state, we select a different strategy. This means that the context object will adopt a different behavior based on the state it's in.

To make this easier to understand, let's consider an example: imagine we have a hotel booking system and an object called Reservation that models a room reservation. This is a typical situation where we have to adapt the behavior of an object based on its state.

Consider the following series of events:

• When the reservation is initially created, the user can confirm (using a method called confirm()) the reservation. Of course, they cannot cancel it (using cancel()), because it's still not confirmed (the caller would receive an exception, for example). They can, however, delete it (using delete()) if they change their mind before buying.
• Once the reservation is confirmed, using the confirm() method again does not make any sense; however, now it should be possible to cancel the reservation but no longer delete it, because it has to be kept for the records.
• On the day before the reservation date, it should not be possible to cancel the reservation anymore; it's too late for that.

Now, imagine that we have to implement the reservation system that we just described in one monolithic object. We can already picture all the if...else or switch statements that we would have to write to enable/disable each action depending on the state of the reservation.

Figure 9.3: An example application of the State pattern

As illustrated in Figure 9.3, the State pattern is, instead, perfect in this situation: there would be three strategies, all implementing the three methods described (confirm(), cancel(), and delete()) and each one implementing only one behavior—the one corresponding to the modeled state. By using this pattern, it should be very easy for the Reservation object to switch from one behavior to another; this would simply require the activation of a different strategy (state object) on each state change.

Let's now work on a more concrete example so that we can apply what we learned about the State pattern.

Implementing a basic failsafe socket

Let's build a TCP client socket that does not fail when the connection with the server is lost; instead, we want to queue all the data sent during the time in which the server is offline and then try to send it again as soon as the connection is reestablished. We want to leverage this socket in the context of a simple monitoring system, where a set of machines sends some statistics about their resource utilization at regular intervals. If the server that collects these resources goes down, our socket will continue to queue the data locally until the server comes back online.

Let's start by creating a new module called failsafeSocket.js that defines our context object:

import { OfflineState } from './offlineState.js'
import { OnlineState } from './onlineState.js'
export class FailsafeSocket {
  constructor (options) {                                  // (1)
    this.options = options
    this.queue = []
    this.currentState = null
    this.socket = null
    this.states = {
      offline: new OfflineState(this),
      online: new OnlineState(this)
    }
    this.changeState('offline')
  }
  changeState (state) {                                    // (2)
    console.log(`Activating state: ${state}`)
    this.currentState = this.states[state]
    this.currentState.activate()
  }
  send (data) {                                            // (3)
    this.currentState.send(data)
  }
}

The FailsafeSocket class is made of three main elements:

1. The constructor initializes various data structures, including the queue that will contain any data sent while the socket is offline. Also, it creates a set of two states: one for implementing the behavior of the socket while it's offline, and another one when the socket is online.
2. The changeState() method is responsible for transitioning from one state to another. It simply updates the currentState instance variable and calls activate() on the target state.
3. The send() method contains the main functionality of the FailsafeSocket class. This is where we want to have a different behavior based on the offline/online state. As we can see, this is done by delegating the operation to the currently active state.

Let's now see what the two states look like, starting from the offlineState.js module:

import jsonOverTcp from 'json-over-tcp-2'                  // (1)
export class OfflineState {
  constructor (failsafeSocket) {
    this.failsafeSocket = failsafeSocket
  }
  send (data) {                                            // (2)
    this.failsafeSocket.queue.push(data)
  }
  activate () {                                            // (3)
    const retry = () => {
      setTimeout(() => this.activate(), 1000)
    }
    console.log('Trying to connect...')
    this.failsafeSocket.socket = jsonOverTcp.connect(
      this.failsafeSocket.options,
      () => {
        console.log('Connection established')
        this.failsafeSocket.socket.removeListener('error', retry)
        this.failsafeSocket.changeState('online')
      }
    )
    this.failsafeSocket.socket.once('error', retry)
  }
}

The module that we just created is responsible for managing the behavior of the socket while it's offline. This is how it works:

1. Instead of using a raw TCP socket, we will use a little library called json-over-tcp-2 (nodejsdp.link/json-over-tcp-2). This will greatly simplify our work since the library will take care of all the parsing and formatting of the data going through the socket into JSON objects.
2. The send() method is only responsible for queuing any data it receives. We are assuming that we are offline, so we'll save those data objects for later. That's all we need to do here.
3. The activate() method tries to establish a connection with the server using the json-over-tcp-2 socket. If the operation fails, it tries again after one second. It continues trying until a valid connection is established, in which case the state of failsafeSocket is transitioned to online.

Next, let's create the onlineState.js module, which is where we will implement the OnlineState class:

export class OnlineState {
  constructor (failsafeSocket) {
    this.failsafeSocket = failsafeSocket
    this.hasDisconnected = false
  }
  send (data) {                                            // (1)
    this.failsafeSocket.queue.push(data)
    this._safeWrite(data)
  }
  _safeWrite (data) {                                      // (2)
    this.failsafeSocket.socket.write(data, (err) => {
      if (!this.hasDisconnected && !err) {
        this.failsafeSocket.queue.shift()
      }
    })
  }
  activate () {                                            // (3)
    this.hasDisconnected = false
    for (const data of this.failsafeSocket.queue) {
      this._safeWrite(data)
    }
    this.failsafeSocket.socket.once('error', () => {
      this.hasDisconnected = true
      this.failsafeSocket.changeState('offline')
    })
  }
}

The OnlineState class models the behavior of the FailsafeSocket when there is an active connection with the server. This is how it works:

1. The send() method queues the data and then immediately tries to write it directly into the socket, as we assume that we are online. It'll use the internal _safeWrite() method to do that.
2. The _safeWrite() method tries to write the data into the socket writable stream (see the official docs at nodejsdp.link/writable-write) and it waits for the data to be written into the underlying resource. If no errors are returned and if the socket didn't disconnect in the meantime, it means that the data was sent successfully and therefore we remove it from the queue.
3. The activate() method flushes any data that was queued while the socket was offline and it also starts listening for any error event; we will take this as a symptom that the socket went offline (for simplicity). When this happens, we transition to the offline state.

That's it for our FailsafeSocket. Now we are ready to build a sample client and a server to try it out. Let's put the server code in a module named server.js:

import jsonOverTcp from 'json-over-tcp-2'
const server = jsonOverTcp.createServer({ port: 5000 })
server.on('connection', socket => {
  socket.on('data', data => {
    console.log('Client data', data)
  })
})
server.listen(5000, () => console.log('Server started'))

Then, the client-side code, which is what we are really interested in, goes into client.js:

import { FailsafeSocket } from './failsafeSocket.js'
const failsafeSocket = new FailsafeSocket({ port: 5000 })
setInterval(() => {
  // send current memory usage
  failsafeSocket.send(process.memoryUsage())
}, 1000)

Our server simply prints to the console any JSON message it receives, while our clients are sending a measurement of their memory utilization every second, leveraging a FailsafeSocket object.

To try the small system that we built, we should run both the client and the server, then we can test the features of failsafeSocket by stopping and then restarting the server. We should see that the state of the client changes between online and offline and that any memory measurement collected while the server is offline is queued and then resent as soon as the server goes back online.

This sample should be a clear demonstration of how the State pattern can help increase the modularity and readability of a component that has to adapt its behavior depending on its state.

Template

The next pattern that we are going to analyze is called Template and it has a lot in common with the Strategy pattern. The Template pattern defines an abstract class that implements the skeleton (representing the common parts) of a component, where some of its steps are left undefined. Subclasses can then fill the gaps in the component by implementing the missing parts, called template methods. The intent of this pattern is to make it possible to define a family of classes that are all variations of a family of components. The following UML diagram shows the structure that we just described:

Figure 9.4: UML diagram of the Template pattern

The three concrete classes shown in Figure 9.4, extend the template class and provide an implementation for templateMethod(), which is abstract or pure virtual, to use C++ terminology. In JavaScript, we don't have a formal way to define abstract classes, so all we can do is leave the method undefined or assign it to a function that always throws an exception, indicating that the method has to be implemented. The Template pattern can be considered a more traditionally object-oriented pattern than the other patterns we have seen so far, because inheritance is a core part of its implementation.

The purpose of Template and Strategy is very similar, but the main difference between the two lies in their structure and implementation. Both allow us to change the variable parts of a component while reusing the common parts. However, while Strategy allows us to do it dynamically at runtime, with Template, the complete component is determined the moment the concrete class is defined. Under these assumptions, the Template pattern might be more suitable in those circumstances where we want to create prepackaged variations of a component. As always, the choice between one pattern and the other is up to the developer, who has to consider the various pros and cons for each use case.

Let's now work on an example.

A configuration manager template

To have a better idea of the differences between Strategy and Template, let's now reimplement the Config object that we defined in the Strategy pattern section, but this time using Template. As in the previous version of the Config object, we want to have the ability to load and save a set of configuration properties using different file formats.

Let's start by defining the template class. We will call it ConfigTemplate:

import { promises as fsPromises } from 'fs'
import objectPath from 'object-path'
export class ConfigTemplate {
  async load (file) {
    console.log(`Deserializing from ${file}`)
    this.data = this._deserialize(
      await fsPromises.readFile(file, 'utf-8'))
  }
  async save (file) {
    console.log(`Serializing to ${file}`)
    await fsPromises.writeFile(file, this._serialize(this.data))
  }
  get (path) {
    return objectPath.get(this.data, path)
  }
  set (path, value) {
    return objectPath.set(this.data, path, value)
  }
  _serialize () {
    throw new Error('_serialize() must be implemented')
  }
  _deserialize () {
    throw new Error('_deserialize() must be implemented')
  }
}

The ConfigTemplate class implements the common parts of the configuration management logic, namely setting and getting properties, plus loading and saving it to the disk. However, it leaves the implementation of _serialize() and _deserialize() open; those are in fact our template methods, which will allow the creation of concrete Config classes supporting specific configuration formats. The underscore at the beginning of the template methods' names indicates that they are for internal use only, an easy way to flag protected methods. Since in JavaScript we cannot declare a method as abstract, we simply define them as stubs, throwing an error if they are invoked (in other words, if they are not overridden by a concrete subclass).

Let's now create a concrete class using our template, for example, one that allows us to load and save the configuration using the JSON format:

import { ConfigTemplate } from './configTemplate.js'
export class JsonConfig extends ConfigTemplate {
  _deserialize (data) {
    return JSON.parse(data)
  }
  _serialize (data) {
    return JSON.stringify(data, null, '  ')
  }
}

The JsonConfig class extends our template class, ConfigTemplate, and provides a concrete implementation for the _deserialize() and _serialize() methods.

Similarly, we can implement an IniConfig class supporting the .ini file format using the same template class:

import { ConfigTemplate } from './configTemplate.js'
import ini from 'ini'
export class IniConfig extends ConfigTemplate {
  _deserialize (data) {
    return ini.parse(data)
  }
  _serialize (data) {
    return ini.stringify(data)
  }
}

Now we can use our concrete configuration manager classes to load and save some configuration data:

import { JsonConfig } from './jsonConfig.js'
import { IniConfig } from './iniConfig.js'
async function main () {
  const jsonConfig = new JsonConfig()
  await jsonConfig.load('samples/conf.json')
  jsonConfig.set('nodejs', 'design patterns')
  await jsonConfig.save('samples/conf_mod.json')
  const iniConfig = new IniConfig()
  await iniConfig.load('samples/conf.ini')
  iniConfig.set('nodejs', 'design patterns')
  await iniConfig.save('samples/conf_mod.ini')
}
main()

Note the difference with the Strategy pattern: the logic for formatting and parsing the configuration data is baked into the class itself, rather than being chosen at runtime.

With minimal effort, the Template pattern allowed us to obtain a new, fully working configuration manager by reusing the logic and the interface inherited from the parent template class and providing only the implementation of a few abstract methods.

In the wild

This pattern should not look entirely new to us. We already encountered it in Chapter 6, Coding with Streams, when we were extending the different stream classes to implement our custom streams. In that context, the template methods were the _write(), _read(), _transform(), or _flush() methods, depending on the stream class that we wanted to implement. To create a new custom stream, we needed to inherit from a specific abstract stream class, providing an implementation for the template methods.

Next, we are going to learn about a very important and ubiquitous pattern that is also built into the JavaScript language itself, which is the Iterator pattern.

Iterator

The Iterator pattern is a fundamental pattern and it's so important and commonly used that it's usually built into the programming language itself. All major programming languages implement the pattern in one way or another, including, of course, JavaScript (starting from the ECMAScript2015 specification).

The Iterator pattern defines a common interface or protocol for iterating the elements of a container, such as an array or a tree data structure. Usually, the algorithm for iterating over the elements of a container is different depending on the actual structure of the data. Think about iterating over an array versus traversing a tree: in the first situation, we need just a simple loop; in the second, a more complex tree traversal algorithm is required (nodejsdp.link/tree-traversal). With the Iterator pattern, we hide the details about the algorithm being used or the structure of the data and provide a common interface for iterating over any type of container. In essence, the Iterator pattern allows us to decouple the implementation of the traversal algorithm from the way we consume the results (the elements) of the traversal operation.

In JavaScript, however, iterators work great even with other types of constructs, which are not necessarily containers, such as event emitters and streams. Therefore, we can say in more general terms that the Iterator pattern defines an interface to iterate over elements produced or retrieved in sequence.

The iterator protocol

In JavaScript, the Iterator pattern is implemented through protocols rather than through formal constructs, such as inheritance. This essentially means that the interaction between the implementer and the consumer of the Iterator pattern will communicate using interfaces and objects whose shape is agreed in advance.

The starting point for implementing the Iterator pattern in JavaScript is the iterator protocol, which defines an interface for producing a sequence of values. So, we'll call iterator an object implementing a next() method having the following behavior: each time the method is called, the function returns the next element in the iteration through an object, called the iterator result, having two properties—done and value:

• done is set to true when the iteration is complete, or in other words, when there are no more elements to return. Otherwise, done will be undefined or false.
• value contains the current element of the iteration and it can be left undefined if done is true. If value is set even when done is true, then it is said that value contains the return value of the iteration, a value which is not part of the elements being iterated, but it's related to the iteration itself as a whole (for example, the time spent iterating all the elements or the average of all the elements iterated if the elements are numbers).

Let's use a quick example to demonstrate how to implement the iterator protocol. Let's implement a factory function called createAlphabetIterator(), which creates an iterator that allows us to traverse all the letters of the English alphabet. Such a function would look like this:

const A_CHAR_CODE = 65
const Z_CHAR_CODE = 90
function createAlphabetIterator () {
  let currCode = A_CHAR_CODE
  return {
    next () {
      const currChar = String.fromCodePoint(currCode)
      if (currCode > Z_CHAR_CODE) {
        return { done: true }
      }
      currCode++
      return { value: currChar, done: false }
    }
  }
}

The logic of the iteration is actually very straightforward; at each invocation of the next() method, we simply increment a number representing the letter's character code, convert it to a character, and then return it using the object shape defined by the iterator protocol.

The important aspect to note is that an iterator is very often a stateful object since we have to keep track in some way of the current position of the iteration. In the previous example, we managed to keep the state in a closure (the currCode variable) but this is just one of the ways we can do so. We could have, for example, kept the state in an instance variable. This is usually better in terms of debuggability since we can read the status of the iteration from the iterator itself at any time, but on the other side, it does not prevent external code from modifying the instance variable and hence tampering with the status of the iteration. It's up to you to decide the pros and cons of each option.

Iterators can indeed be fully stateless as well. Examples are iterators returning random elements and either completing randomly or never completing, and iterators stopping at the first iteration.

Now, let's see how we can use the iterator we just built. Consider the following code fragment:

const iterator = createAlphabetIterator()
let iterationResult = iterator.next()
while (!iterationResult.done) {
  console.log(iterationResult.value)
  iterationResult = iterator.next()
}

As we can see from the previous code, the code that consumes an iterator can be considered a pattern itself. However, as we will see later in this section, it's not the only way we have to consume an iterator. JavaScript has, in fact, much more convenient and elegant ways to use iterators.

The iterable protocol

The iterable protocol defines a standardized means for an object to return an iterator. Such objects are called iterables. Usually, an iterable is a container of elements, such as a data structure, but it can also be an object virtually representing a set of elements, such as a Directory object, which would allow us to iterate over the files in a directory.

In JavaScript, we can define an iterable by making sure it implements the @@iterator method, or in other words, a method accessible through the built-in symbol Symbol.iterator.

Such an @@iterator method should return an iterator object, which can be used to iterate over the elements represented by the iterable. For example, if our iterable is a class, we would have something like the following:

class MyIterable {
  // other methods...
  [Symbol.iterator] () {
    // return an iterator
  }
}

To show how this works in practice, let's build a class to manage information organized in a bidimensional matrix structure. We want this class to be implementing the iterable protocol, so that we can scan all the elements in the matrix using an iterator. Let's create a file called matrix.js containing the following content:

export class Matrix {
  constructor (inMatrix) {
    this.data = inMatrix
  }
  get (row, column) {
    if (row >= this.data.length ||
      column >= this.data[row].length) {
      throw new RangeError('Out of bounds')
    }
    return this.data[row][column]
  }
  set (row, column, value) {
    if (row >= this.data.length ||
      column >= this.data[row].length) {
      throw new RangeError('Out of bounds')
    }
    this.data[row][column] = value
  }
  [Symbol.iterator] () {
    let nextRow = 0
    let nextCol = 0
    return {
      next: () => {
        if (nextRow === this.data.length) {
          return { done: true }
        }
        const currVal = this.data[nextRow][nextCol]
        if (nextCol === this.data[nextRow].length - 1) {
          nextRow++
          nextCol = 0
        } else {
          nextCol++
        }
        return { value: currVal }
      }
    }
  }
}

As we can see, the class contains the basic methods for getting and setting values in the matrix, as well as the @@iterator method, implementing our iterable protocol. The @@iterator method will return an iterator, as specified by the iterable protocol and such an iterator adheres to the iterator protocol. The logic of the iterator is very straightforward: we are simply traversing the matrix's cells from the top left to the bottom right, by scanning each column of each row; we are doing that by leveraging two indexes, nextRow and nextCol.

Now, it's time to try out our iterable Matrix class. We can do that in a file called index.js:

import { Matrix } from './matrix.js'
const matrix2x2 = new Matrix([
  ['11', '12'],
  ['21', '22']
])
const iterator = matrix2x2[Symbol.iterator]()
let iterationResult = iterator.next()
while (!iterationResult.done) {
  console.log(iterationResult.value)
  iterationResult = iterator.next()
}

All we are doing in the previous code is creating a sample Matrix instance and then obtaining an iterator using the @@iterator method. What comes next, as we already know, is just boilerplate code that iterates over the elements returned by the iterator. The output of the iteration should be '11', '12', '21', '22'.

Iterators and iterables as a native JavaScript interface

At this point, you may ask: "what's the point of having all these protocols for defining iterators and iterables?" Well, having a standardized interface allows third party code as well as the language itself to be modeled around the two protocols we've just seen. This way, we can have APIs (even native) as well as syntax constructs accepting iterables as an input.

For example, the most obvious syntax construct accepting an iterable is the for...of loop. We've just seen in the last code sample that iterating over a JavaScript iterator is a pretty standard operation, and its code is mostly boilerplate. In fact, we'll always have an invocation to next() to retrieve the next element and a check to verify if the done property of the iteration result is set to true, which indicates the end of the iteration. But, worry not, simply pass an iterable to the for...of instruction to seamlessly loop over the elements returned by its iterator. This allows us to process the iteration with an intuitive and compact syntax:

for (const element of matrix2x2) {
  console.log(element)
}

Another construct compatible with iterables is the spread operator:

const flattenedMatrix = [...matrix2x2]
console.log(flattenedMatrix)

Similarly, we can use an iterable with the destructuring assignment operation:

const [oneOne, oneTwo, twoOne, twoTwo] = matrix2x2
console.log(oneOne, oneTwo, twoOne, twoTwo)

The following are some JavaScript built-in APIs accepting iterables:

• Map([iterable]): nodejsdp.link/map-constructor
• WeakMap([iterable]): nodejsdp.link/weakmap-constructor
• Set([iterable]): nodejsdp.link/set-constructor
• WeakSet([iterable]): nodejsdp.link/weakset-constructor
• Promise.all(iterable): nodejsdp.link/promise-all
• Promise.race(iterable): nodejsdp.link/promise-race
• Array.from(iterable): nodejsdp.link/array-from

On the Node.js side, one notable API accepting an iterable is stream.Readable.from(iterable, [options]) (nodejsdp.link/readable-from), which creates a readable stream out of an iterable object.

for (const letter of createAlphabetIterator()) {
  //...
}

JavaScript itself defines many iterables that can be used with the APIs and constructs we've just seen. The most notable iterable is Array, but also other data structures, such as Map and Set, and even String all implement the @@iterable method. In Node.js land, Buffer is probably the most notable iterable.

Generators

The ES2015 specification introduced a syntax construct that is closely related to iterators. We are talking about generators, also known as semicoroutines. They are a generalization of standard functions, in which there can be different entry points. In a standard function, we can have only one entry point, which corresponds to the invocation of the function itself, but a generator can be suspended (using the yield statement) and then resumed at a later time. Among other things, generators are very well suited to implement iterators, in fact, as we will see in a bit, the generator object returned by a generator function is indeed both an iterator and an iterable.

Generators in theory

To define a generator function, we need to use the function* declaration (the function keyword followed by an asterisk):

function * myGenerator () {
  // generator body
}

Invoking a generator function will not execute its body immediately. Rather, it will return a generator object, which, as we already mentioned, is both an iterator and an iterable. But it doesn't end here; invoking next() on the generator object will start or resume the execution of the generator until the yield instruction is invoked or the generator returns (either implicitly or explicitly with a return instruction). Within the generator, using the keyword yield followed by a value x is equivalent to returning {done: false, value: x} from the iterator, while returning a value x is equivalent to returning {done: true, value: x}.

A simple generator function

To demonstrate what we just learned, let's see a simple generator called fruitGenerator(), which will yield two names of fruits and return their ripening season:

function * fruitGenerator () {
  yield 'peach'
  yield 'watermelon'
  return 'summer'
}
const fruitGeneratorObj = fruitGenerator()
console.log(fruitGeneratorObj.next())                      // (1)
console.log(fruitGeneratorObj.next())                      // (2)
console.log(fruitGeneratorObj.next())                      // (3)

The preceding code will print the following text:

    { value: 'peach', done: false }
    { value: 'watermelon', done: false }
    { value: 'summer', done: true }

This is a short explanation of what happened:

1. The first time fruitGeneratorObj.next() was invoked, the generator started its execution until it reached the first yield command, which put the generator on pause and returned the value 'peach' to the caller.
2. At the second invocation of fruitGeneratorObj.next(), the generator resumed, starting from the second yield command, which in turn put the execution on pause again, while returning the value 'watermelon' to the caller.
3. The last invocation of fruitGeneratorObj.next() caused the execution of the generator to resume from its last instruction, a return statement, which terminates the generator, returns the value 'summer', and sets the done property to true in the result object.

Since a generator object is also an iterable, we can use it in a for...of loop. For example:

for (const fruit of fruitGenerator()) {
  console.log(fruit)
}

The preceding loop will print:

peach
watermelon

Controlling a generator iterator

Generator objects are more than standard iterators, in fact, their next() method optionally accepts an argument (whereas, as specified by the iterator protocol, it does not need to accept one). Such an argument is passed as the return value of the yield instruction. To show this, let's create a new simple generator:

function * twoWayGenerator () {
  const what = yield null
  yield 'Hello ' + what
}
const twoWay = twoWayGenerator()
twoWay.next()
console.log(twoWay.next('world'))

When executed, the preceding code prints Hello world. This means that the following has happened:

1. The first time the next() method is invoked, the generator reaches the first yield statement and is then put on pause.
2. When next('world') is invoked, the generator resumes from the point where it was put on pause, which is on the yield instruction, but this time we have a value that is passed back to the generator. This value will then be set to the what variable. The generator then appends the what variable to the string 'Hello ' and yields the result.

Two other extra features provided by generator objects are the optional throw() and return() iterator methods. The first behaves like next() but it will also throw an exception within the generator as if it was thrown at the point of the last yield, and returns the canonical iterator object with the done and value properties. The second, the return() method, forces the generator to terminate and returns an object such as the following: {done: true, value: returnArgument} where returnArgument is the argument passed to the return() method.

The following code shows a demonstration of these two methods:

function * twoWayGenerator () {
  try {
    const what = yield null
    yield 'Hello ' + what
  } catch (err) {
    yield 'Hello error: ' + err.message
  }
}
console.log('Using throw():')
const twoWayException = twoWayGenerator()
twoWayException.next()
console.log(twoWayException.throw(new Error('Boom!')))
console.log('Using return():')
const twoWayReturn = twoWayGenerator()
console.log(twoWayReturn.return('myReturnValue'))

Running the previous code will print the following to the console:

Using throw():
{ value: 'Hello error: Boom!', done: false }
Using return():
{ value: 'myReturnValue', done: true }

As we can see, the twoWayGenerator() function will receive an exception as soon as the first yield instruction returns. This works exactly as if an exception was thrown from inside the generator, and this means that it can be caught and handled like any other exception using a try...catch block. The return() method, instead, will simply stop the execution of the generator causing the given value to be provided as a return value by the generator.

How to use generators in place of iterators

Generator objects are also iterators. This means that generator functions can be used to implement the @@iterator method of iterable objects. To demonstrate this, let's convert our previous Matrix iteration example to generators. Let's update our matrix.js file as follows:

export class Matrix {
  // ...rest of the methods (stay unchanged)
  * [Symbol.iterator] () {                                 // (1)
    let nextRow = 0                                        // (2)
    let nextCol = 0
    while (nextRow !== this.data.length) {                 // (3)
      yield this.data[nextRow][nextCol]
      if (nextCol === this.data[nextRow].length - 1) {
        nextRow++
        nextCol = 0
      } else {
        nextCol++
      }
    }
  }
}

There are a few interesting aspects in the code fragment we've just seen. Let's analyze them in more detail:

1. The first thing to notice is that the @@iterator method is now a generator (note the asterisk * before the method name).
2. The variables we use to maintain the state of the iteration are now just local variables for the generator, while in the previous version of the Matrix class, those two variables were part of a closure. This is possible because when a generator is invoked, its local state is preserved between reentries.
3. We are using a standard loop to iterate over the elements of the matrix. This is certainly more intuitive than trying to imagine a loop that invokes the next() method of an iterator.

As we can see, generators are an excellent alternative to writing iterators from scratch. They will improve the readability of our iteration routine and will offer the same level of functionality (or even better).

Async iterators

The iterators we've seen so far return a value synchronously from their next() method. However, in JavaScript and especially in Node.js, it's very common to have iterations over items that require an asynchronous operation to be produced.

Imagine, for example, to iterate over the requests received by an HTTP server, or over the results of an SQL query, or over the elements of a paginated REST API. In all those situations, it would be handy to be able to return a promise from the next() method of an iterator, or even better, use the async/await construct.

Well, that's exactly what async iterators are; they are iterators returning a promise, and since that's the only extra requirement, it means that we can also use an async function to define the next() method of the iterator. Similarly, async iterables are objects that implement an @@asyncIterator method, or in other words, a method accessible through the Symbol.asyncIterator key, which returns (synchronously) an async iterator.

Async iterables can be looped over using the for await...of syntax, which can only be used inside an async function. With the for await...of syntax, we are essentially implementing a sequential asynchronous execution flow on top of the Iterator pattern. Essentially, it's just syntactic sugar for the following loop:

const asyncIterator = iterable[Symbol.asyncIterator]()
let iterationResult = await asyncIterator.next()
while (!iterationResult.done) {
  console.log(iterationResult.value)
  iterationResult = await asyncIterator.next()
}

This means that the for await...of syntax can also be used to iterate over a simple iterable (not just async iterables) as, for example, over an array of promises. It will work even if not all the elements (or none) of the iterator are promises.

To quickly demonstrate this, let's build a class that takes a list of URLs as input and allows us to iterate over their availability status (up/down). Let's call the class CheckUrls:

import superagent from 'superagent'
export class CheckUrls {
  constructor (urls) {                                     // (1)
    this.urls = urls
  }
  [Symbol.asyncIterator] () {
    const urlsIterator = this.urls[Symbol.iterator]()      // (2)
    return {
      async next () {                                      // (3)
        const iteratorResult = urlsIterator.next()         // (4)
        if (iteratorResult.done) {
          return { done: true }
        }
        const url = iteratorResult.value
        try {
          const checkResult = await superagent             // (5)
            .head(url)
            .redirects(2)
          return {
            done: false,
            value: `${url} is up, status: ${checkResult.status}`
          }
        } catch (err) {
          return {
            done: false,
            value: `${url} is down, error: ${err.message}`
          }
        }
      }
    }
  }
}

Let's analyze the previous code's most important parts:

1. The CheckUrls class constructor takes as input a list of URLs. Since we now know how to use iterators and iterables, we can say that this list of URLs can be just any iterable.
2. In our @@asyncIterator method, we obtain an iterator from the this.urls object, which, as we just said, should be an iterable. We can do that by simply invoking its @@iterable method.
3. Note how the next() method is now an async function. This means that it will always return a promise, as requested by the async iterable protocol.
4. In the next() method, we use the urlsIterator to get the next URL in the list, unless there are no more, in which case we simply return {done: true}.
5. Note how we can now use the await instruction to asynchronously get the result of the HEAD request sent to the current URL.

Now, let's use the for await...of syntax we mentioned earlier to iterate over a CheckUrls object:

import { CheckUrls } from './checkUrls.js'
async function main () {
  const checkUrls = new CheckUrls([
    'https://nodejsdesignpatterns.com',
    'https://example.com',
    'https://mustbedownforsurehopefully.com'
  ])
  for await (const status of checkUrls) {
    console.log(status)
  }
}
main()

As we can see, the for await...of syntax is a very intuitive way to iterate over an async iterable and, as we will see in a while, it can be used in conjunction with some interesting built-in iterables to obtain alternative new ways to access asynchronous information.

Async generators

As well as async iterators, we can also have async generators. To define an async generator function, simply prepend the keyword async to the function definition:

async function * generatorFunction() {
  // ...generator body
}

As you can well imagine, async generators allow the use of the await instruction within their body and the return value of their next() method is a promise that resolves to an object having the canonical done and value properties. This way, async generator objects are also valid async iterators. They are also valid async iterables, so they can be used in for await...of loops.

To demonstrate how async generators can simplify the implementation of async iterators, let's convert the CheckUrls class we saw in the previous example to use an async generator:

export class CheckUrls {
  constructor (urls) {
    this.urls = urls
  }
  async * [Symbol.asyncIterator] () {
    for (const url of this.urls) {
      try {
        const checkResult = await superagent
          .head(url)
          .redirects(2)
        yield `${url} is up, status: ${checkResult.status}`
      } catch (err) {
        yield `${url} is down, error: ${err.message}`
      }
    }
  }
}

Interestingly, using an async generator in place of a bare async iterator allowed us to save a few lines of code and the resulting logic is also more readable and explicit.

Async iterators and Node.js streams

If we stop for a second and think about the relationship between async iterators and Node.js readable streams, we would be surprised by how similar they are in both purpose and behavior. In fact, we can say that async iterators are indeed a stream construct, as they can be used to process the data of an asynchronous resource piece by piece, exactly as it happens for readable streams.

It's not a coincidence that stream.Readable implements the @@asyncIterator method, making it an async iterable. This provides us with an additional, and probably even more intuitive, mechanism to read data from a readable stream, thanks to the for await...of construct.

To quickly demonstrate this, consider the following example where we take the stdin stream of the current process and we pipe it into the split() transform stream, which will emit a new chunk when it finds a newline character. Then, we iterate over each line using the for await...of loop:

import split from 'split2'
async function main () {
  const stream = process.stdin.pipe(split())
  for await (const line of stream) {
    console.log(`You wrote: ${line}`)
  }
}
main()

This sample code will print back whatever we have written to the standard input only after we have pressed the Return key. To quit the program, you can just press Ctrl + C.

As we can see, this alternative way of consuming a readable stream is indeed very intuitive and compact. The previous example also shows us how similar the two paradigms—iterators and streams—are. They are so similar that they can interoperate almost seamlessly. To prove this point even further, just consider that the function stream.Readable.from(iterable, [options]) takes an iterable as an argument, which can be both synchronous or asynchronous. The function will return a readable stream that wraps the provided iterable, "adapting" its interface to that of a readable stream (this is also a good example of the Adapter pattern, which we have already met in Chapter 8, Structural Design Patterns.

So, if streams and async iterators as so closely related, which one should you actually use? This, as always, depends on the use case and many other factors; however, to help you with the decision, this is a list of aspects that set the two constructs apart:

• Streams are push, meaning that data is pushed into the internal buffers by the stream and then consumed from the buffers. Async iterators are pull by default (unless another logic is explicitly implemented by the iterator), meaning that data is only retrieved/produced on demand by the consumer.
• Streams are better suited to process binary data since they natively provide internal buffering and backpressure.
• Streams can be composed using a well-known and streamlined API, pipe(), while async iterators do not offer any standardized composition method.

In the wild

Iterators and, in particular, async iterators are quickly gaining popularity in the Node.js ecosystem. In fact, in many circumstances, they are becoming a preferred alternative to streams and are replacing custom-built iteration mechanisms.

For example, the packages @databases/pg, @databases/mysql and @databases/sqlite are popular libraries for accessing Postgres, MySQL, and SQLite databases respectively (more at nodejsdp.link/atdatabases).

They all expose a function called queryStream(), which returns an async iterable, which can be used to easily iterate over the results of a query. For example:

for await (const record of db.queryStream(sql`SELECT * FROM my_table`)) {
  // do something with record
}

Internally, the iterator will automatically handle the cursor for a query result, so all we have to do is simply loop with the for await...of construct.

Another example of a library heavily relying on iterators for its API is the zeromq package (nodejsdp.link/npm-zeromq). We'll see a detailed example of it in the next section, about the Middleware pattern, as we move on to other behavioral patterns.

Middleware

One of the most distinctive patterns in Node.js is definitely Middleware. Unfortunately, it's also one of the most confusing for the inexperienced, especially for developers coming from the enterprise programming world. The reason for the disorientation is probably connected to the traditional meaning of the term middleware, which in enterprise architecture jargon represents the various software suites that help to abstract lower-level mechanisms such as OS APIs, network communications, memory management, and so on, allowing the developer to focus only on the business case of the application. In this context, the term middleware recalls topics such as CORBA, enterprise service bus, Spring, JBoss, and WebSphere, but in its more generic meaning, it can also define any kind of software layer that acts as glue between lower-level services and the application (literally, the software in the middle).

Middleware in Express

Express (nodejsdp.link/express) popularized the term middleware in the Node.js world, binding it to a very specific design pattern. In Express, in fact, middleware represents a set of services, typically functions, that are organized in a pipeline and are responsible for processing incoming HTTP requests and relative responses.

Express is famous for being a very non-opinionated and minimalist web framework and the Middleware pattern is the main reason for that. Express middleware is, in fact, an effective strategy for allowing developers to easily create and distribute new features that can be easily added to an application, without the need to grow the minimalistic core of the framework.

An Express middleware has the following signature:

function (req, res, next) { ... }

Here, req is the incoming HTTP request, res is the response, and next is the callback to be invoked when the current middleware has completed its tasks, and that in turn triggers the next middleware in the pipeline.

Examples of the tasks carried out by Express middleware include the following:

• Parsing the body of the request
• Compressing/decompressing requests and responses
• Producing access logs
• Managing sessions
• Managing encrypted cookies
• Providing Cross-Site Request Forgery (CSRF) protection

If we think about it, these are all tasks that are not strictly related to the main business logic of an application, nor are they essential parts of the minimal core of a web server. They are accessories, components providing support to the rest of the application and allowing the actual request handlers to focus only on their main business logic. Essentially, those tasks are "software in the middle."

Middleware as a pattern

The technique used to implement middleware in Express is not new, in fact, it can be considered the Node.js incarnation of the Intercepting Filter pattern and the Chain of Responsibility pattern. In more generic terms, it also represents a processing pipeline, which reminds us of streams. Today, in Node.js, the word middleware is used well beyond the boundaries of the Express framework, and indicates a particular pattern whereby a set of processing units, filters, and handlers, under the form of functions, are connected to form an asynchronous sequence in order to perform the preprocessing and postprocessing of any kind of data. The main advantage of this pattern is flexibility. In fact, the Middleware pattern allows us to obtain a plugin infrastructure with incredibly little effort, providing an unobtrusive way to extend a system with new filters and handlers.

The following diagram shows the components of the Middleware pattern:

Figure 9.5: The structure of the Middleware pattern

The essential component of the pattern is the Middleware Manager, which is responsible for organizing and executing the middleware functions. The most important implementation details of the pattern are as follows:

• New middleware can be registered by invoking the use() function (the name of this function is a common convention in many implementations of the Middleware pattern, but we can choose any name). Usually, new middleware can only be appended at the end of the pipeline, but this is not a strict rule.
• When new data is received for processing, the registered middleware is invoked in an asynchronous sequential execution flow. Each unit in the pipeline receives the result of the execution of the previous unit as input.
• Each piece of middleware can decide to stop further processing of the data. This can be done by invoking a special function, by not invoking the callback (in case the middleware uses callbacks), or by propagating an error. An error situation usually triggers the execution of another sequence of middleware that is specifically dedicated to handling errors.

There is no strict rule on how the data is processed and propagated in the pipeline. The strategies for propagating the data modifications in the pipeline include:

• Augmenting the data received as input with additional properties or functions
• Maintaining the immutability of the data and always return fresh copies as the result of the processing

The right approach depends on the way the Middleware Manager is implemented and on the type of processing carried out by the middleware itself.

Creating a middleware framework for ZeroMQ

Let's now demonstrate the pattern by building a middleware framework around the ZeroMQ (nodejsdp.link/zeromq) messaging library. ZeroMQ (also known as ZMQ, or ØMQ) provides a simple interface for exchanging atomic messages across the network using a variety of protocols. It shines for its performance, and its basic set of abstractions are specifically built to facilitate the implementation of custom messaging architectures. For this reason, ZeroMQ is often chosen to build complex distributed systems.

The interface of ZeroMQ is pretty low-level as it only allows us to use strings and binary buffers for messages. So, any encoding or custom formatting of data has to be implemented by the users of the library.

In the next example, we are going to build a middleware infrastructure to abstract the preprocessing and postprocessing of the data passing through a ZeroMQ socket, so that we can transparently work with JSON objects, but also seamlessly compress messages traveling over the wire.

The Middleware Manager

The first step toward building a middleware infrastructure around ZeroMQ is to create a component that is responsible for executing the middleware pipeline when a new message is received or sent. For this purpose, let's create a new module called zmqMiddlewareManager.js and let's define it:

export class ZmqMiddlewareManager {
  constructor (socket) {                                     // (1)
    this.socket = socket
    this.inboundMiddleware = []
    this.outboundMiddleware = []
    this.handleIncomingMessages()
      .catch(err => console.error(err))
  }
  async handleIncomingMessages () {                          // (2)
    for await (const [message] of this.socket) {
      await this
        .executeMiddleware(this.inboundMiddleware, message)
        .catch(err => {
          console.error('Error while processing the message', err)
        })
    }
  }
  async send (message) {                                     // (3)
    const finalMessage = await this
      .executeMiddleware(this.outboundMiddleware, message)
    return this.socket.send(finalMessage)
  }
  use (middleware) {                                         // (4)
    if (middleware.inbound) {
      this.inboundMiddleware.push(middleware.inbound)
    }
    if (middleware.outbound) {
      this.outboundMiddleware.unshift(middleware.outbound)
    }
  }
  async executeMiddleware (middlewares, initialMessage) {    // (5)
    let message = initialMessage
    for await (const middlewareFunc of middlewares) {
      message = await middlewareFunc.call(this, message)
    }
    return message
  }
}

Let's discuss in detail how we implemented our ZmqMiddlewareManager:

1. In the first part of the class, we define the constructor that accepts a ZeroMQ socket as an argument. In the constructor, we create two empty lists that will contain our middleware functions, one for inbound messages and another one for outbound messages. Next, we immediately start processing the messages coming from the socket. We do that in the handleIncomingMessages() method.
2. In the handleIncomingMessages() method, we use the ZeroMQ socket as an async iterable and with a for await...of loop, we process any incoming message and we pass it down the inboundMiddleware list of middlewares.
3. Similarly to handleIncomingMessages(), the send() method will pass the message received as an argument down the outboundMiddleware pipeline. The result of the processing is stored in the finalMessage variable and then sent through the socket.
4. The use() method is used for appending new middleware functions to our internal pipelines. In our implementation, each middleware comes in pairs; it's an object that contains two properties, inbound and outbound. Each property can be used to define the middleware function to be added to the respective list. It's important to observe here that the inbound middleware is pushed to the end of the inboundMiddleware list, while the outbound middleware is inserted (using unshift()) at the beginning of the outboundMiddleware list. This is because complementary inbound/outbound middleware functions usually need to be executed in inverted order. For example, if we want to decompress and then deserialize an inbound message using JSON, it means that for the outbound, we should instead first serialize and then compress. This convention for organizing the middleware in pairs is not strictly part of the general pattern, but only an implementation detail of our specific example.
5. The last method, executeMiddleware(), represents the core of our component as it's the part responsible for executing the middleware functions. Each function in the middleware array received as input is executed one after the other, and the result of the execution of a middleware function is passed to the next. Note that we are using the await instruction on each result returned by each middleware function; this allows the middleware function to return a value synchronously as well as asynchronously using a promise. Finally, the result of the last middleware function is returned back to the caller.

Implementing the middleware to process messages

Now that we have implemented our Middleware Manager, we can create our first pair of middleware functions to demonstrate how to process inbound and outbound messages. As we said, one of the goals of our middleware infrastructure is to have a filter that serializes and deserializes JSON messages. So, let's create a new middleware to take care of this. In a new module called jsonMiddleware.js, let's include the following code:

export const jsonMiddleware = function () {
  return {
    inbound (message) {
      return JSON.parse(message.toString())
    },
    outbound (message) {
      return Buffer.from(JSON.stringify(message))
    }
  }
}

The inbound part of our middleware deserializes the message received as input, while the outbound part serializes the data into a string, which is then converted into a buffer.

In a similar way, we can implement a pair of middleware functions in a file called zlibMiddleware.js, to inflate/deflate the message using the zlib core module (nodejsdp.link/zlib):

import { inflateRaw, deflateRaw } from 'zlib'
import { promisify } from 'util'
const inflateRawAsync = promisify(inflateRaw)
const deflateRawAsync = promisify(deflateRaw)
export const zlibMiddleware = function () {
  return {
    inbound (message) {
      return inflateRawAsync(Buffer.from(message))
    },
    outbound (message) {
      return deflateRawAsync(message)
    }
  }
}

Compared to the JSON middleware, our zlib middleware functions are asynchronous and return a promise as a result. As we already know, this is perfectly supported by our Middleware Manager.

You can note how the middleware used by our framework is quite different from the one used in Express. This is totally normal and a perfect demonstration of how we can adapt this pattern to fit our specific needs.

Using the ZeroMQ middleware framework

We are now ready to use the middleware infrastructure that we just created. To do that, we are going to build a very simple application, with a client sending a ping to a server at regular intervals and the server echoing back the message received.

From an implementation perspective, we are going to rely on a Request/Reply messaging pattern using the req/rep socket pair provided by ZeroMQ (nodejsdp.link/zmq-req-rep). We will then wrap the sockets with our ZmqMiddlewareManager to get all the advantages from the middleware infrastructure that we built, including the middleware for serializing/deserializing JSON messages.

The server

Let's start by creating the server-side of our application in a file called server.js:

import zeromq from 'zeromq'                                  // (1)
import { ZmqMiddlewareManager } from './zmqMiddlewareManager.js'
import { jsonMiddleware } from './jsonMiddleware.js'
import { zlibMiddleware } from './zlibMiddleware.js'
async function main () {
  const socket = new zeromq.Reply()                          // (2)
  await socket.bind('tcp://127.0.0.1:5000')
  const zmqm = new ZmqMiddlewareManager(socket)              // (3)
  zmqm.use(zlibMiddleware())
  zmqm.use(jsonMiddleware())
  zmqm.use({                                                 // (4)
    async inbound (message) {
      console.log('Received', message)
      if (message.action === 'ping') {
        await this.send({ action: 'pong', echo: message.echo })
      }
      return message
    }
  })
  console.log('Server started')
}
main()

The server-side of our application works as follows:

1. We first load the necessary dependencies. The zeromq package is essentially a JavaScript interface over the native ZeroMQ library. See nodejsdp.link/npm-zeromq.
2. Next, in the main() function, we create a new ZeroMQ Reply socket and bind it to port 5000 on localhost.
3. Then comes the part where we wrap ZeroMQ with our middleware manager and then add the zlib and JSON middlewares.
4. Finally, we are ready to handle a request coming from the client. We will do this by simply adding another middleware, this time using it as a request handler.

Since our request handler comes after the zlib and JSON middlewares, we will receive a decompressed and deserialized version of the received message. On the other hand, any data passed to send() will be processed by the outbound middleware, which in our case will serialize and then compress the data.

The client

On the client-side of our little application, in a file called client.js, we will have the following code:

import zeromq from 'zeromq'
import { ZmqMiddlewareManager } from './zmqMiddlewareManager.js'
import { jsonMiddleware } from './jsonMiddleware.js'
import { zlibMiddleware } from './zlibMiddleware.js'
async function main () {
  const socket = new zeromq.Request()                      // (1)
  await socket.connect('tcp://127.0.0.1:5000')
  const zmqm = new ZmqMiddlewareManager(socket)
  zmqm.use(zlibMiddleware())
  zmqm.use(jsonMiddleware())
  zmqm.use({
    inbound (message) {
      console.log('Echoed back', message)
      return message
    }
  })
  setInterval(() => {                                      // (2)
    zmqm.send({ action: 'ping', echo: Date.now() })
      .catch(err => console.error(err))
  }, 1000)
  console.log('Client connected')
}
main()

Most of the code of the client application is very similar to that of the server. The notable differences are:

1. We create a Request socket, rather than a Reply socket, and we connect it to a remote (or local) host rather than binding it on a local port. The rest of the middleware setup is exactly the same as in the server, except for the fact that our request handler now just prints any message it receives. Those messages should be the pong reply to our ping requests.
2. The core logic of the client application is a timer that sends a ping message every second.

Now, we're ready to try our client/server pair and see the application in action. First, start the server:

node server.js

We can then start the client in another terminal with the following command:

node client.js

At this point, we should see the client sending messages and the server echoing them back.

Our middleware framework did its job. It allowed us to decompress/compress and deserialize/serialize our messages transparently, leaving the handlers free to focus on their business logic.

In the wild

We opened this section by saying that the library that popularized the Middleware pattern in Node.js is Express (nodejsdp.link/express). So, we can easily say that Express is also the most notable example of the Middleware pattern out there.

Two other interesting examples are:

• Koa (nodejsdp.link/koa), which is known as the successor of Express. It was created by the same team behind Express and it shares with it its philosophy and main design principles. Koa's middleware is slightly different than that of Express since it uses modern programming techniques such as async/await instead of callbacks.
• Middy (nodejsdp.link/middy) is a classic example of the Middleware pattern applied to something different than a web framework. Middy is, in fact, a middleware engine for AWS Lambda functions.

Next, we are going to explore the Command pattern, which, as we will see shortly, is a very flexible and multiform pattern.

Command

Another design pattern with huge importance in Node.js is Command. In its most generic definition, we can consider a command any object that encapsulates all the information necessary to perform an action at a later time. So, instead of invoking a method or a function directly, we create an object representing the intention to perform such an invocation. It will then be the responsibility of another component to materialize the intent, transforming it into an actual action. Traditionally, this pattern is built around four major components, as shown in Figure 9.6:

Figure 9.6: The components of the Command pattern

The typical configuration of the Command pattern can be described as follows:

• Command is the object encapsulating the information necessary to invoke a method or function.
• Client is the component that creates the command and provides it to the invoker.
• Invoker is the component responsible for executing the command on the target.
• Target (or receiver) is the subject of the invocation. It can be a lone function or a method of an object.

As we will see, these four components can vary a lot depending on the way we want to implement the pattern. This should not sound new at this point.

Using the Command pattern instead of directly executing an operation has several applications:

• A command can be scheduled for execution at a later time.
• A command can be easily serialized and sent over the network. This simple property allows us to distribute jobs across remote machines, transmit commands from the browser to the server, create remote procedure call (RPC) systems, and so on.
• Commands make it easy to keep a history of all the operations executed on a system.
• Commands are an important part of some algorithms for data synchronization and conflict resolution.
• A command scheduled for execution can be canceled if it's not yet executed. It can also be reverted (undone), bringing the state of the application to the point before the command was executed.
• Several commands can be grouped together. This can be used to create atomic transactions or to implement a mechanism whereby all the operations in the group are executed at once.
• Different kinds of transformation can be performed on a set of commands, such as duplicate removal, joining and splitting, or applying more complex algorithms such as operational transformation (OT), which is the base for most of today's real-time collaborative software, such as collaborative text editing.

The preceding list clearly shows us how important this pattern is, especially on a platform such as Node.js where networking and asynchronous execution are essential players.

Now, we are going to explore in more detail a couple of different implementations of the Command pattern, just to give you an idea of its scope.

The Task pattern

We can start off with the most basic and trivial implementation of the Command pattern: the Task pattern. The easiest way in JavaScript to create an object representing an invocation is, of course, by creating a closure around a function definition or a bound function:

function createTask(target, ...args) {
  return () => {
    target(...args)
  }
}

This is (mostly) equivalent to doing:

const task = target.bind(null, ...args)

This should not look new at all. In fact, we have used this pattern already so many times throughout the book, and in particular in Chapter 4, Asynchronous Control Flow Patterns with Callbacks. This technique allowed us to use a separate component to control and schedule the execution of our tasks, which is essentially equivalent to the invoker of the Command pattern.

A more complex command

Let's now work on a more articulated example leveraging the Command pattern. This time, we want to support undo and serialization. Let's start with the target of our commands, a little object that is responsible for sending status updates to a Twitter-like service. We will use a mockup of such a service for simplicity (the statusUpdateService.js file):

const statusUpdates = new Map()
// The Target
export const statusUpdateService = {
  postUpdate (status) {
    const id = Math.floor(Math.random() * 1000000)
    statusUpdates.set(id, status)
    console.log(`Status posted: ${status}`)
    return id
  },
  destroyUpdate (id) => {
    statusUpdates.delete(id)
    console.log(`Status removed: ${id}`)
  }
}

The statusUpdateService we just created represents the target of our Command pattern. Now, let's implement a factory function that creates a command to represent the posting of a new status update. We'll do that in a file called createPostStatusCmd.js:

export function createPostStatusCmd (service, status) {
  let postId = null
  // The Command
  return {
    run () {
      postId = service.postUpdate(status)
    },
    undo () {
      if (postId) {
        service.destroyUpdate(postId)
        postId = null
      }
    },
    serialize () {
      return { type: 'status', action: 'post', status: status }
    }
  }
}

The preceding function is a factory that produces commands to model "post status" intentions. Each command implements the following three functionalities:

• A run() method that, when invoked, will trigger the action. In other words, it implements the Task pattern that we have seen before. The command, when executed, will post a new status update using the methods of the target service.
• An undo() method that reverts the effects of the post operation. In our case, we are simply invoking the destroyUpdate() method on the target service.
• A serialize() method that builds a JSON object that contains all the necessary information to reconstruct the same command object.

After this, we can build an invoker. We can start by implementing its constructor and its run() method (the invoker.js file):

import superagent from 'superagent'
// The Invoker
export class Invoker {
  constructor () {
    this.history = []
  }
  run (cmd) {
    this.history.push(cmd)
    cmd.run()
    console.log('Command executed', cmd.serialize())
  }
  // ...rest of the class

The run() method is the basic functionality of our Invoker. It is responsible for saving the command into the history instance variable and then triggering the execution of the command itself.

Next, we can add to the Invoker a new method that delays the execution of a command:

delay (cmd, delay) {
  setTimeout(() => {
    console.log('Executing delayed command', cmd.serialize())
    this.run(cmd)
  }, delay)
}

Then, we can implement an undo() method that reverts the last command:

undo () {
  const cmd = this.history.pop()
  cmd.undo()
  console.log('Command undone', cmd.serialize())
}

Finally, we also want to be able to run a command on a remote server, by serializing and then transferring it over the network using a web service:

async runRemotely (cmd) {
  await superagent
    .post('http://localhost:3000/cmd')
    .send({ json: cmd.serialize() })
  console.log('Command executed remotely', cmd.serialize())
}

Now that we have the command, the invoker, and the target, the only component missing is the client, which we will implement in a file called client.js. Let's start by importing all the necessary dependencies and by instantiating Invoker:

import { createPostStatusCmd } from './createPostStatusCmd.js'
import { statusUpdateService } from './statusUpdateService.js'
import { Invoker } from './invoker.js'
const invoker = new Invoker()

Then, we can create a command using the following line of code:

const command = createPostStatusCmd(statusUpdateService, 'HI!')

We now have a command representing the posting of a status message. We can then decide to dispatch it immediately:

invoker.run(command)

Oops, we made a mistake, let's revert our timeline to the state it was before posting the last message:

invoker.undo()

We can also decide to schedule the message to be sent in 3 seconds from now:

invoker.delay(command, 1000 * 3)

Alternatively, we can distribute the load of the application by migrating the task to another machine:

invoker.runRemotely(command)

The little example that we have just implemented shows how wrapping an operation in a command can open a world of possibilities, and that's just the tip of the iceberg.

As the last remarks, it is worth noting that a fully-fledged Command pattern should be used only when strictly necessary. We saw, in fact, how much additional code we had to write to simply invoke a method of the statusUpdateService. If all that we need is only an invocation, then a complex command would be overkill. If, however, we need to schedule the execution of a task or run an asynchronous operation, then the simpler Task pattern offers the best compromise. If instead, we need more advanced features such as undo support, transformations, conflict resolution, or one of the other fancy use cases that we described previously, using a more complex representation for the command is almost necessary.

Summary

We opened this chapter with three closely related patterns, which are Strategy, State, and Template.

Strategy allows us to extract the common parts of a family of closely related components into a component called the context and allows us to define strategy objects that the context can use to implement specific behaviors. The State pattern is a variation of the Strategy pattern where the strategies are used to model the behavior of a component when under different states. The Template pattern, instead, can be considered the "static" version of the Strategy pattern, where the different specific behaviors are implemented as subclasses of the template class, which models the common parts of the component.

Next, we learned about what has now become a core pattern in Node.js, which is Iterator. We learned how JavaScript offers native support for the pattern (with the iterator and iterable protocols), and how async iterators can be used as an alternative to complex async iteration patterns and even to Node.js streams.

Then, we examined Middleware, which is a very distinctive pattern born from within the Node.js ecosystem. We learned how it can be used to preprocess and postprocess data and requests.

Finally, we had a taste of the possibilities offered by the Command pattern, which can be used to implement a myriad of functionality, from simple undo/redo and serialization, to more complex operational transformation algorithms.

We have now arrived at the end of the last chapter dedicated to "traditional" design patterns. By now, you should have added to your toolbelt a series of patterns that will be enormously useful in your everyday programming endeavors.

In the next chapter, we'll shift our attention to a topic that goes beyond the boundaries of server-side development. Thanks to Node.js, in fact, we can create "Universal" JavaScript applications, or in other words, applications that can run as seamlessly on the server as they run on the browser. Stay tuned, then, to learn about the most useful Universal JavaScript patterns.

Exercises

• Exercise 9.1 Logging with Strategy: Implement a logging component having at least the following methods: debug(), info(), warn(), and error(). The logging component should also accept a strategy that defines where the log messages are sent. For example, we might have a ConsoleStrategy to send the messages to the console, or a FileStrategy to save the log messages to a file.
• Exercise 9.2 Logging with Template: Implement the same logging component we defined in the previous exercise, but this time using the Template pattern. We would then obtain a ConsoleLogger class to log to the console or FileLogger class to log to a file. Appreciate the differences between the Template and the Strategy approaches.
• Exercise 9.3 Warehouse item: Imagine we are working on a warehouse management program. Our next task is to create a class to model a warehouse item and help track it. Such a WarehouseItem class has a constructor, which accepts an id and the initial state of the item (which can be one of arriving, stored, or delivered). It has three public methods:
  • store(locationId) moves the item into the stored state and records the locationId where it's stored.
  • deliver(address) changes the state of the item to delivered, sets the delivery address, and clears the locationId.
  • describe() returns a string representation of the current state of the item (for example, "Item 5821 is on its way to the warehouse," or "Item 3647 is stored in location 1ZH3," or "Item 3452 was delivered to John Smith, 1st Avenue, New York."

  The arriving state can be set only when the object is created as it cannot be transitioned to from the other states. An item can't move back to the arriving state once it's stored or delivered, it cannot be moved back to stored once it's delivered, and it cannot be delivered if it's not stored first. Use the State pattern to implement the WarehouseItem class.

• Exercise 9.4 Logging with Middleware: Rewrite the logging component you implemented for exercises 9.1 and 9.2, but this time use the Middleware pattern to postprocess each log message allowing different middlewares to customize how to handle the messages and how to output them. We could, for example, add a serialize() middleware to convert the log messages to a string representation ready to be sent over the wire or saved somewhere. Then, we could add a saveToFile() middleware that saves each message to a file. This exercise should highlight the flexibility and universality of the Middleware pattern.
• Exercise 9.5 Queues with iterators: Implement an AsyncQueue class similar to one of the TaskQueue classes we defined in Chapter 5, Asynchronous Control Flow Patterns with Promises and Async/Await, but with a slightly different behavior and interface. Such an AsyncQueue class will have a method called enqueue() to append new items to the queue and then expose an @@asyncIterable method, which should provide the ability to process the elements of the queue asynchronously, one at a time (so, with a concurrency of 1). The async iterator returned from AsyncQueue should terminate only after the done() method of AsyncQueue is invoked and only after all items in the queue are consumed. Consider that the @@asyncIterable method could be invoked in more than one place, thus returning an additional async iterator, which would allow you to increase the concurrency with which the queue is consumed.