The New Crop of Hypermedia Formats

Starting in the early 2010s, a new crop of text-based formats emerged that offered more than just structure for describing data; they included instructions on how to manipulate the data as well. These are formats I refer to as hypermedia formats. These formats represent another trend in APIs and, as we will see later in the book, can be a valuable tool in creating API-based services that support a wide range of service changes without breaking existing client applications. In some cases, they even allow client applications to “auto-magically” acquire new features and behaviors without the need for rewriting and redeploying client-side code.

The Fallacy of The Right One

So, despite all the new hypermedia formats out there (and the continued use of XML in enterprises), with the current trend pointing toward JSON as the common output format, it would seem an easy decision, right? Any time you start implementing an API, just use JSON and you’re done. Unfortunately, that’s almost never the way it goes.

First, some industry verticals still rely on XML and SOAP-based formats. If you want to interact with them, you’ll need to support SOAP or some other custom XML-based formats. Examples might be partner APIs that you work with on a regular basis, government or other standards-led efforts that continue to focus on XML as their preferred format, and even third-party APIs that you use to solve important business goals.

Second, many companies invested heavily in XML-based APIs over the last decade and are unwilling to rewrite these APIs just to change the output format. Unless there is a clear advantage to changing the messages format (e.g., increased speed, new functionality, or some other business metric), these XML-based services are not likely to change any time soon.

Finally, some data storage systems are XML-native or default to outputting data as XML documents (e.g., dbXML, MarkLogic, etc). While some of these services may offer an option to output the data in a JSON-based format, many continue to focus on XML and the only clear way of converting this data to JSON is to move it to JSON-native data storage systems like MongoDB, CouchDB, and others.

So, deciding on a single format for your team’s service may not be feasible. And, as your point of view widens from a single team to multiple teams within your company, multiple products within an enterprise, on up to the entire WWW itself, getting everyone to agree to both produce and consume a single output format is not a reasonable goal.

As frustrating as this may be for team leaders and enterprise-level software architects, there is no single “Right One” when it comes to message formats. It may be possible to control a single team’s decision (either through consensus or fiat) but one’s ability to exert this control wanes as the scope of the community grows.

And that means the way forward is to rethink the problem, not work harder at implementing the same solution.

Reframing the Problem

One way to face a challenge that seems insurmountable is to apply the technique of reframing—to put the problem in a different light or from a new point of view. Instead of working harder to come up with a solution to the perceived problem, reframing encourages us to step outside the frame and change our perspective. Sometimes this allows us to recognize the scenario as a completely different problem—one that may have an easier or simpler solution.

In our case (the challenge of selecting a single format for your APIs), it can help to ask “Why do we need to decide on a single format?” or, to put it another way, “Why not support many formats for a single API?” Asking these questions gives us a chance to lay out some of the reasons for and against supporting multiple formats. In this way, we’ve side-stepped the challenge of picking one format. Now we’re focused on a new aspect of the same problem. Why not support multiple formats?

Separating Format from Functionality

All too often, I see service implementations that are bound too tightly to a single format. This is a common problem for SOAP implementations—usually because the developer tooling leads programmers into relying on a tight binding between the internal object model and the external output format. It is important to treat all formats (including SOAP XML output) as independent of the internal object model. This allows some changes in the internal model to happen without requiring changes to the external output format.

To manage this separation, we’ll need to employ some modularity to keep the work of converting the domain model into a message external from the work of manipulating the domain model itself. This is using modularity to split up the assignment of work. Typically modularity is used to collect related functionality in a single place (e.g., all the functionality related to users or customers or shoppingCarts). The notion of using modularity as primarily a work assignment tactic comes from David Parnas’s 1972 paper “On the Criteria to be Used in Decomposing Systems into Modules.” As Parnas states:

[M]odule is considered to be a responsibility assignment rather than a subprogram. The modularizations include the design decisions which must be made before the work on independent modules can begin. [Emphasis in the original]

Viewing the work of converting internal domain data into external output formats as a responsibility assignment leads us to isolate the conversion process into its own module. Now we can manage that module separately from the one(s) that manipulate domain data. A simple example of this clear separation might look like this:

var convert = new ConversionModule(HALConverter);
var output = convert.toMessage(domainData);

In that imaginary pseudo-code, the conversion process is accessed via an instance of the conversionModule() that accepts a message-specific converter (in this case, HALConverter) and uses the toMessage function that accepts a domainData instance to produce the desired output. This is all quite vague right now, but at least we have a clear target for implementing safe, cheap, easy support for multiple output formats.

Once the functionality of the internal domain model is cleanly separated from the external format, we need some guidance on how to consistently convert the domain model into the desired output format. But before that, we’ll need a pattern for selecting which format is appropriate.

The Selection Algorithm

An important implementation detail when supporting multiple output formats for an API is that of the output selection process. There needs to be some consistent algorithmic way to select the correct output format at runtime. The good news is that HTTP—still the most common application-level protocol for web APIs—has this algorithm already defined: content negotiation.

Section 3.4 of the HTTP specification (RFC7231) describes two patterns of content negotiation for “representing information”:

Proactive

The server selects the representation based on the client’s preferences.

Reactive

The server provides a list of possible representations to the client and the client selects the preferred format.

The most common pattern in use on the Web today is the proactive one and that’s what we’ll implement in our representor. Specifically, clients will send an Accept HTTP header that contains a list of one or more format preferences, and the server will use that list to determine which format will be used in the response (including the selected format identifier in the server’s Content-Type HTTP header).

A typical client request might be:

GET /users HTTP/1.1
Accept: application/vnd.hal+json, application/vnd.uber+json
...

And, for a service that supports HAL but does not support UBER, the response would be:

HTTP/1.1 200 OK
Content-Type: application/vnd.hal+json
...

GET /users/ HTTP/1.1
application/vnd.uber+json;q=0.3,
application/vnd.hal+json;q=1

So, that’s what it looks like on the “outside”—the actual HTTP conversation. But what pattern is used internally to make this work on the server side? Thankfully, a solution for this kind of selection process was worked out in the 1990s.

Adapting and Translating

Many of the challenges of writing solid internal code can be summed up in a common pattern. And one of the most important books on code patterns is the 1994 book Design Patterns by Gamma, Helm, Johnson, and Vlissides (Addison-Wesley Professional). Those are rather tough names to remember and, over time, this group of authors has come to be known as the Gang of Four (GoF). You’ll sometimes even hear people refer to the Gang of Four book when discussing this important text.

There are about twenty patterns in the GoF book, categorized into three types:

• Creational patterns

• Structural patterns

• Behavioral patterns

The pattern that will help us in our quest to implement support for multiple output formats for our API that is safe, cheap, and easy is the Adapter structural pattern.

The Adapter pattern

As established on OODesign.com, the intent of the Adapter pattern is to:

Convert the interface of a class into another interface clients expect. Adapter lets classes work together that couldn’t otherwise because of incompatible interfaces.

And that’s essentially what we need to do—convert an internal class (or model) into an external class (or message).

There are four participants to the Adapter pattern (Figure 3-1):

Target

Defines the domain-specific interface that client uses. This will be the message model or media type we want to use for output.

Client

Collaborates with objects conforming to the target interface. In our case, this is our API service—the app that is using the Adapter pattern.

Adaptee

Defines an existing interface that needs adapting. This is the internal model that needs to be converted into the target message model.

Adapter

Adapts the interface of adaptee to the target interface. This will be the specific media-type plug-in that we write to handle the work of converting the internal model to the message model.

So, we need to write an adapter plug-in for each target media type (HTML, HAL, Siren, Collection+JSON, etc.). That’s not too tough. But the challenge is that each internal object model (the adaptee) is going to be different. For example, writing a plug-in to handle Users then writing one to handle Tasks, and on and on. That can mean lots of code is needed to write the adapters and—even more disappointing—these adapters may not be very reusable.

To try to reduce the need for tightly coupled adapter code, I’m going to introduce another pattern—one based on the Adapter pattern: the Message Translator.

Figure 3-1. The Adapter pattern

That means we need to spend a few minutes on what the Message Translator pattern looks like and how it can be used to standardize the process of converting internal object models into external message models.

The Message Translator pattern

To cut down on lots of custom adapters, I’m going to introduce another pattern—derived from the Adapter pattern—called the Message Translator. This comes from Gregor Hohpe and his book Enterprise Integration Patterns (Addison-Wesley Professional).

Hohpe describes the Message Translator as:

Use a special filter, a Message Translator, between other filters or applications to translate one data format into another.

A message translator is a special form of the adapter class in the GoF set of patterns.

To make this all work, I’ll introduce a general message format—the Web Service Transition Language (WeSTL)—in the next section of this chapter. That will act as a standardized adaptee and make it possible to generalize the way adapter plug-ins can be coded. Now, the process of translating can be turned into an algorithm that doesn’t need to rely on any domain-specific information. As illustrated in Figure 3-2, we can write WeSTL-to-HAL or WeSTL-to-Cj or WeSTL-to-Siren translators and then the only work is to convert the internal model into a WeSTL message. This moves the complexity to a new location, but does so in a way that reduces the amount of custom code needed to support multiple formats.

So, armed with this background, we can now look at a set of concrete implementation details to make it all happen.

Figure 3-2. The Message Translator pattern

Handling the HTTP Accept Header

The first two items on that list are rather trivial to implement in any WWW-aware codebase. For example, identifying acceptable output formats for a request means reading the Accept HTTP header. Here is a snippet of NodeJS code that does that:

// rudimentary accept-header handling
var contentType = '';
var htmlType = 'text/html';
var contentAccept = req.headers['accept'];
if(!contentAccept || contentAccept==='*/*') {
  contentType = htmlType;
}
else {
  contentType = contentAccept.split(',')[0];
}

Note that the preceding code example makes two key assumptions:

1. If no Accept header is passed or the Accept header is set to “anything”, the Accept header will be set to text/html.

2. If the Accept header lists more than one acceptable format, this service will just grab the first one listed.

This implementation is very limited. It does not support the use of q values to help the server better understand client preferences, and this service defaults to the text/html type for API responses. Both of these assumptions can be altered and/or improved through additional coding, but I’ve skipped over that for this book.

Implementing the Message Translator Pattern

Now that we have the requested format value—the output context for this request—we can move on to the next step: implementing the Message Translator pattern in NodeJS. For this book, I’ve created a simple module that uses a switch … case element that matches the request context string (the accepted format) with the appropriate translator implementation.

The code looks like this:

// load representors 
var html = require('./representors/html.js');
var haljson = require('./representors/haljson.js');
var collectionJson = require('./representors/cj.js');
var siren = require('./representors/siren.js');

function processData(domainData, mimeType) {
  var doc;

  // clueless? assume HTML 
  if (!mimeType) {
    mimeType = "text/html";
  }

  // dispatch to requested representor 
  switch (mimeType.toLowerCase()) {
    case "application/vnd.hal+json":
      doc = haljson(object);
      break;
    case "application/vnd.collection+json":
      doc = collectionJson(object);
      break;
    case "application/vnd.siren+json":
      doc = siren(object);
      break;
    case "text/html":
    default: 
      doc = html(object);
      break;
  }
  return doc;
}

In the preceding code snippet, you can see that a set of representors are loaded at the top (see ). The code in these modules will be covered in “Runtime WeSTL”). Next (), if the mimeType value is not passed (or is invalid) it is automatically set to text/html. This is a bit of defensive coding. And then (at ) the switch … case block checks the incoming mimeType string with known (and supported) mime type strings in order to select the appropriate format processing module. Finally, in case an unknown/unsupported format is passed in, the default statement () makes sure that the service runs the html() module to produce valid output.

We now have the basics of the representor outlined. The next step is to actually implement each format-specific translator (HTML, HAL, etc.). To solve this challenge, we need to take a side road on our journey that establishes a general format understood by all translators—the WeSTL format.

General Representor Modules

In the Message Translator pattern, each format module (html(), haljson(), etc.) is an instance of a translator. While implementing these modules as domain-specific converters (e.g., userObjectToHTML, userObjectToHAL, etc.) would meet the needs of our implementation, that approach will not scale over time. Instead, what we need is a general-purpose approach to a translator that will not have any domain-specific knowledge. For example, the translator module used to handle user domain data will be the same translator module used to handle customer and accounting or any other domain-specific domain data.

To do that, we’ll need to create a common interface for passing domain data into format modules that is independent of any single domain model.

The WeSTL Format

For this book, I’ve worked up a common interface in the form of a standardized object model, one that service developers can quickly load with domain data and pass to format modules. I also took the opportunity to reframe the challenge of defining interfaces for web APIs. Instead of focusing on defining resources, I chose to focus on defining state transitions. For this reason, I’ve named this interface design the Web Service Transition Language, or WeSTL (pronounced wehs’-tul).

Basically, WeSTL allows API service developers to use a standardized message model for converting internal object models into external message formats. This reduces the cost (in time and effort) to craft new Translators and pushes the complexity of converting internal models into messages to a single instance—converting from the internal model to WeSTL.

Figure 3-3 provides a visual representation of the request/response life cycle of services that use the WeSTL format.

Figure 3-3. The WeSTL transformation cycle

When designing and implementing web APIs with WeSTL, the service developer collects all the possible state transitions and describes them in the WeSTL model. By state transitions, I mean all the links and forms that could appear within any service response. For example, every response might have link to the home page. Some responses will have HTML-style input forms allowing API clients to create new service data or edit existing data. There may even be service responses that list all the possible links and forms (state transitions) for the service!

A simple example of how WeSTL can be used to describe transitions is shown here:

{
  "wstl" : {
    "transitions" : [
      {
        "name" : "home", 
        "type" : "safe",
        "action" : "read",
        "prompt" : "Home Page",
      },
      {
        "name" : "user-list", 
        "type" : "safe",
        "target" : "user list"
        "prompt" : "List of Users"
      }
      {
        "name" : "change-password", 
        "type" : "unsafe",
        "action" : "append"
        "inputs" : [ 
          {
            "name" : "userName",
            "prompt" : "User",
            "readOnly" : true
          },
          {
            "name" : "old-password",
            "prompt" : "Current Password",
            "required" : true,
          },
          {
            "name" : "old-password",
            "prompt" : "New Password (5-10 chars)",
            "required" : true,
            "pattern" : "^[a-zA-Z0-9]{5,10}$"
          }
        ]
      }
    ]
  }
}

As you can see from this WeSTL document, it contains three transition descriptions named home (), user-list (), and change-password (). The first two transitions are marked safe. That means they don’t write any data, only execute reads (e.g., HTTP GET). The third one, however (change-password) is marked unsafe since it writes data to the service (à la HTTP POST). You can also see several input elements described for the change-password transition (). These details will be used when creating an API resource for the User Manager service.

There are a number of details left out in this simple example, but you can see how WeSTL works; it describes the transitions that can be used within the service. What’s important to note is that this document does not define web resources or constrain where (or even when) these transitions will appear. That work is handled by service developers elsewhere in the code.

So, this is what a WeSTL model looks like at design time, before the service is up and running. Typically, a service designer uses WeSTL in this mode. There is also another mode for WeSTL documents: runtime. That mode is typically used when implementing the service.

var transitions = require('./wstl-designtime.js');
var domainData = require('./domain.js');

function userResource(root) {
  var doc, coll, data;

  data = [];
  coll = [];

  // pull data for this resource 
  data = domain.getData('user',root.getID());

  // add transitions for this resource 
  tran = transitions("home");
  tran.href = root +"/home/";
  tran.rel = ["http:"+root+"/rels/home"];
  coll.splice(coll.length, 0, tran);

  tran = transitions("user-list");
  tran.href = root +"/user/";
  tran.rel = ["http:"+root+"/rels/collection"];
  coll.splice(coll.length, 0, tran);

  tran = transitions("change-password");
  tran.href = root +"/user/changepw/{id}";
  tran.rel = ["http:"+root+"/rels/changepw"];
  coll.splice(coll.length, 0, tran);

  // compose wstl model 
  doc = {};
  doc.wstl = {};
  doc.wstl.title = "User Management";
  doc.wstl.transitions = coll;
  doc.wstl.data =  data;

  return doc;
}

var transitions = require('./wstl-designtime.js');
var domainData = require('./domain.js');

function userResource(root) {
  var doc, coll, data;

  data = [];
  coll = [];

  // pull data for this resource 
  data = domain.getData('user',root.getID());

  // add transitions for this resource 
  tran = transitions("home");
  tran.href = root +"/home/";
  tran.rel = ["http:"+root+"/rels/home"];
  coll.splice(coll.length, 0, tran);

  tran = transitions("user-list");
  tran.href = root +"/user/";
  tran.rel = ["http:"+root+"/rels/collection"];
  coll.splice(coll.length, 0, tran);

  tran = transitions("change-password");
  tran.href = root +"/user/changepw/{id}";
  tran.rel = ["http:"+root+"/rels/changepw"];
  coll.splice(coll.length, 0, tran);

  // compose wstl model 
  doc = {};
  doc.wstl = {};
  doc.wstl.title = "User Management";
  doc.wstl.transitions = coll;
  doc.wstl.data =  data;

  return doc;
}

A Sample Representor

Now that resources are represented using a generic interface using WeSTL, we can build a general format module that converts the standardized WeSTL model into an output format. Basically, the code accepts a runtime WeSTL document and then translates domain data, element by element, into the target message format for an API response.

To see how this might look, here is a high-level look at a simplified implementation of a general HAL representor.

function haljson(wstl, root, rels) { 
  var hal;

  hal = {};
  hal._links = {};

  for(var segment in wstl) {
    hal._links = getLinks(wstl[segment], root, segment, rels);
    if(wstl[segment].data && wstl[segment].data.length===1) {
      hal = getProperties(hal, wstl[segment]);
    }
  }
  return JSON.stringify(hal, null, 2); 
}

// emit _links object 
function getLinks(wstl, root, segment, relRoot) {
  var coll, items, links, i, x;

  links = {};

  // list-level actions
  if(wstl.actions) {
    coll = wstl.transitions;
    for(i=0,x=coll.length;i<x;i++) {
      links = getLink(links, coll[i], relRoot);
    }

    // list-level objects
    if(wstl.data) {
      coll = wstl.data;
      items = [];
      for(i=0,x=coll.length;i<x;i++) {
        item = {};
        item.href = coll[i].meta.href;
        item.title = coll[i].title;
        items.push(item);
      }
      links[checkRel(segment, relRoot)] = items;
    }
  }
  return links;
}

// emit root properties 
function getProperties(hal, wstl) {
  var props;

  if(wstl.data && wstl.data[0]) {
    props = wstl.data[0];
    for(var p in props) {
      if(p!=='meta') {
        hal[p] = props[p];
      }
    }
  }
  return hal;
}

/* additional support functions appear here */

While this code example is just a high-level view, you should be able to figure out the important details. The first argument of the top level function (haljson()) accepts a WeSTL model along with some runtime request-level data (). That function “walks” the WeSTL runtime instance and (a) processes any links (transitions) in the model () and then (b) deals with any name–value pairs in the WesTL instance (). Once all the processing is done, the resulting JSON object (now a valid HAL document) is returned to the caller (). An example of what this code might produce follows:

{
  "_links" : { 
    "self" : {
      "href": "http://localhost:8282/user/mamund"
    },
    "http://localhost:8282/rels/home": {
      "href": "http://localhost:8282/",
      "title": "Home",
      "templated": false
    },
    "http://localhost:8282/rels/collection": {
      "href": "http://localhost:8282/user/",
      "title": "All Users",
      "templated": false
    },
    "http://localhost:8282/rels/changepw": {
      "href": "http://localhost:8282/user/changepw/mamund",
      "title": "Change Password"
    }
  },
  "userName": "mamund", 
  "familyName": "Amundsen",
  "givenName": "Mike",
  "password": "p@ss",
  "webUrl": "http://amundsen.com/blog/",
  "dateCreated": "2015-01-06T01:38:55.147Z",
  "dateUpdated": "2015-01-25T02:28:12.402Z"
}

Now you can see how the WeSTL document has led from design-time mode to runtime instance and finally (via the HAL translator module) to the actual HAL document. The WeSTL transitions appear in the HAL _links section () and the related data for this user appears as name–value pairs called properties in HAL documents (starting at ).

Of course, HAL is just one possible translator implementation. Throughout this book, you’ll find general message translators for a handful of formats. Hopefully, this short overview will give enough guidance to anyone who wishes to implement their own (possibly better) general representors for HAL and many other registered formats.

Image credits

• Diogo Lucas, Figures 3-1 and 3-2