Structural Organization: Find Patterns by Scope

Most of our patterns focus on API building blocks at different levels of abstraction and detail; some concern the API as a whole and its documentation, both technical and commercial. The resulting architectural scopes are API as a whole, endpoint, operation, and message. We introduced these basic concepts in the API domain model in Chapter 1. Figure 4.2 calls out patterns for these five scopes.

Patterns such as API DESCRIPTION and SERVICE LEVEL AGREEMENT concern the API as a whole. Others, such as PROCESSING RESOURCE and DATA TRANSFER RESOURCE, operate on single endpoints. Many patterns deal with operation or message design; some of these primarily target request messages (API KEY, WISH LIST), and others are more focused on response messages (PAGINATION, ERROR REPORT). Element stereotypes may appear both in requests and responses (IDENTIFIER ELEMENT, METADATA ELEMENT).

Call to action: When being confronted with an API design task, ask yourself which of these scopes you are about to deal with and refer to Figure 4.2 to find patterns of interest for this task.

Theme Categorization: Search for Topics

We grouped the patterns into five categories. Each category answers several related topical questions:

• Foundation patterns: Which types of systems and components are integrated? From where should an API be accessible? How should it be documented?

• Responsibility patterns: What is the architectural role played by each API end-point? What are the operation responsibilities? How do these roles and responsibilities impact service decomposition and API granularity?

• Structure patterns: What is an adequate number of representation elements for request and response messages? How should these elements be structured? How can they be grouped and annotated?

• Quality patterns: How can an API provider achieve a certain level of design time and runtime qualities while using its resources in a cost-effective way? How can the API quality trade-offs be communicated and accounted for?

• Evolution patterns: How can life-cycle management concerns such as support periods and versioning be dealt with? How can backward compatibility be promoted and unavoidable breaking changes be communicated?

These theme categories organize the decision models in Chapter 3, “API Decision Narratives,” and the Web site supporting this book.² Figure 4.3 groups the patterns by book chapter. The theme categories and Chapters 4 to 8 correspond with each other but for only two exceptions: API DESCRIPTION from the foundation category and three patterns related to quality management (RATE LIMIT, PRICING PLAN, SERVICE LEVEL AGREEMENT) are factored out into a separate Chapter 9, “Document and Communicate API Contracts.” The patterns API KEY, ERROR REPORT, and CONTEXT REPRESENTATION are related to quality but appear in Chapter 6, “Design Request and Response Message Representations,” due to their role of special-purpose representations. The cheat sheet in Appendix A follows the same structure.

2. https://api-patterns.org

Call to action: Think about an API design issue you were confronted with recently. Does it fit in any of the preceding categories? Do any of the questions and pattern names suggest that the pattern might be able to resolve the issue? If so, you may want to go to the respective chapter and pattern right now (and return here later). If you need more information, you can consult the cheat sheet in Appendix A.

Time Dimension: Follow Design Refinement Phases

Roughly following the “Unified Process” [Kruchten 2000], an API design evolves from project/product inception to design elaboration, implementation construction iterations, and project/product transition. Table 4.1 categorizes patterns by process phases; note that some patterns can be applied in multiple phases.

APIs endpoints are identified and characterized by their roles in the overall system/architecture at the early stages (inception). Next, operations are drafted with their request and response message structure conceptualized and designed initially (elaboration). Quality improvements follow (construction). An approach to versioning and a support/lifetime strategy is specified for APIs when they go live (transition); updates are possible later on.

While Table 4.1 has an order from top to bottom (as all tables do), it can be walked through multiple times, even within a single two-week sprint. We do not propose a waterfall model here; it is perfectly fine to go back and forth, for instance when applying agile project organization practices. In other words, each sprint may contain inception, elaboration, construction, and transition tasks (and apply related patterns).

You might be wondering how the Align-Define-Design-Refine (ADDR) phases (see introduction to Part 2) relate to the phases in the Unified Process and Table 4.1. Our take is: Align corresponds to inception, the Define activities happen during elaboration. Design work stretches from elaboration to construction iterations; construction and transition (and later evolution and maintenance phases) provide opportunities to Refine the design continuously.

Call to action: Which phase is your current API design effort in? Do the listed patterns suggest being eligible for consideration in your design? You may want to revisit Table 4.1 each time your design reaches a certain milestone or each time you pick an API-related story from the product backlog at the start of a sprint.

How to Navigate: The Map to MAP

You can use the three navigation aids structure from this section—structure/scope, theme category/chapter, and time/phase—to explore the language for your immediate needs if you are not yet ready to read Part 2 page by page. Having picked one or more entry points, you can then follow the “Related Patterns” pointers provided in each pattern to move on; you can also return to one of the three organizers (scope, topic, phase). Having studied a few patterns, you may want to check out the Lakeside Mutual case study or the real-world pattern stories in Chapter 10, “Real-World Pattern Stories,” to return to the big picture and learn how individual patterns can be combined.

The following section introduces basic API foundation and message structure patterns eligible during the Align phase of ADDR, with Chapters 5 to 9 covering the remaining phases Define, Design, and Refine as well as additional topics.

Pattern: FRONTEND INTEGRATION

In Chapter 1, we motivated the emergence of mobile apps and cloud-native applications as one reason why APIs matter so much. They provide mobile apps and Web clients of cloud applications with data and access to provider-side processing capabilities.

The application frontends serving end users may be internal ones or be part of external systems. FRONTEND INTEGRATION APIs are consumed by API clients in these types of application frontends. Figure 4.5 positions the FRONTEND INTEGRATION pattern in its context.

The Business Logic Layer [Fowler 2002] in the backend is a natural entry point. Sometimes the user interface is also split up between client and server. In such cases, the API might reside on the user interface level as well.

Pattern: BACKEND INTEGRATION

In Chapter 1, we discussed that cloud-native applications and microservices-based systems require APIs to both connect and separate their parts. APIs also play a key role in software ecosystems. More generally speaking, any backend system may benefit from and rely on remote APIs when requiring information from or desiring activity in other systems.

Such BACKEND INTEGRATION APIs are never directly used by frontend clients of the distributed application but are consumed by other backends exclusively.

Figure 4.6 positions the pattern in the first of its two application contexts, business-to-business (or system-to-system) integration.

Figure 4.7 illustrates the second usage context of the pattern, application-internal decomposition of business logic into service components exposing a SOLUTION-INTERNAL API.

The entry to the business logic layer is a suitable location for a BACKEND INTEGRATION API. Access control, authorization enforcement, system transaction management, and business rule evaluation typically are already located here. In some data-centric scenarios not requiring much logic, it may make sense to integrate on the data persistence layer instead (this is not shown in Figure 4.7).

Pattern: PUBLIC API

APIs exposed to the World Wide Web do not aim to limit their target audience and accessibility but often control access to them with API KEYS.

Figure 4.8 sketches the PUBLIC API pattern in an exemplary scenario.

Pattern: COMMUNITY API

Some APIs are shared by clients in different organizations and might be deployed in and accessible via networks available only to community members.

Figure 4.9 sketches the COMMUNITY API pattern in its architectural context.

Pattern: SOLUTION-INTERNAL API

Some APIs structure applications into components, for instance, services/microservices or program-internal modules. In such cases, API clients and their providers often run in the same data center or even on the same physical or virtual compute node.

Figure 4.10 sketches two instances of the SOLUTION-INTERNAL API pattern, supporting an application frontend and another backend component, with sample API clients and backend implementation.

Foundation Patterns Summary

This concludes our coverage of five foundation patterns in this chapter. Chapter 3 features these patterns as decisions required and problem-solution pairs.

Note that FRONTEND INTEGRATION is sometimes called vertical integration, while BACKEND INTEGRATION is called horizontal integration. The notion originates from a rather common visualization of distributed systems (and their layers and tiers) that places frontends at the top of figures/model diagrams and backends at the bottom; if several systems are displayed, this is done along the x-axis of the figure. Note that a left-to-right organization of such figures also is seen frequently.

You might be wondering why we call out the integration type and API visibility in pattern form; aren’t all these APIs just APIs with endpoints, operations, and messages? They are. However, practical experience suggests that the business contexts and requirements for the two integration types are different; therefore, APIs serving frontend and backend fulfill other purposes and are designed differently. For instance, the protocol choice might differ in the two cases: HTTP often is a natural (or the only) choice in FRONTEND INTEGRATION, while message queuing is attractive in BACKEND INTEGRATION. The request and response message structures may vary too in terms of their breadth and depth. An API that does both either makes design compromises or has to offer optional features, which tends to complicate its usage. Similar concerns apply to API visibility; for instance, a PUBLIC API often has more advanced security requirements and stability needs than an internal one; the error reporting has to consider that API clients and providers might not even know each other (which is less likely for SOLUTION-INTERNAL APIS).

Next, we look at the building blocks of request and response messages, abstracting from the data definition concepts in exchange formats such as JSON.

Pattern: ATOMIC PARAMETER

Known from programming languages, basic types are the simplest unit of transmission in message exchanges between API clients and API providers (of all visibilities and integration types introduced earlier in this section).

Decide whether the atom is single- or set-valued. Describe the meaning of the transported values, at least informally, including, for instance, a unit of measure. Consider specifying a value range to constrain the type of the ATOMIC PARAMETER. Make this value-range information explicit—statically in the schema definition language for the chosen message exchange format (for example, JSON Schema, Protocol Buffers, GraphQL Schema Language, or XML Schema) and/or dynamically in runtime metadata.

Figure 4.12 visualizes a single-valued string parameter as a single instance of the pattern appearing in a request message.

In the Lakeside Mutual sample case, ATOMIC PARAMETERS can be found in all API operations, for instance, those dealing with services concerning customer information. The first example is single-valued:

"city":Data<string>

The notation of this example is Microservice Domain-Specific Language (MDSL; see Appendix C for an introduction). In the API of the Customer Core application at Lakeside Mutual, such parameters can be used to retrieve the city of a customer:

Click here to view code image

curl -X GET --header 'Authorization: Bearer b318ad736c6c844b' 
http://localhost:8110/customers/gktlipwhjr?fields=city
{
  "customers": [{
    "city": "St. Gallen",
    "_links": {
      "self": {
        "href": "/customers/gktlipwhjr?fields=city"
      },
      "address.change": {
        "href": "/customers/gktlipwhjr/address"
      }
    }
  }],
  ...
}

Note that city is not the only ATOMIC PARAMETER in the example. The customer identifier gktlipwhjr in the path of the URI also qualifies as such.

Atomic parameters may come as collections of basic types, which is expressed by making the atom set-valued *, as shown in the following MDSL example:

"streetAddress":D<string>*

A JSON instance of the preceding definition is

Click here to view code image

{ "streetAddress": [ "sampleStreetName1", "sampleStreetName2"]}

ATOMIC PARAMETERS appear in all operation definitions and in their schema components. Appendix B presents an OpenAPI specification from the Lakeside Mutual case.

Pattern: ATOMIC PARAMETER LIST

Sometimes, a single ATOMIC PARAMETER is not expressive enough. Two or more such ATOMIC PARAMETERS might have strong semantic ties, or the content of a request or response message might need several parts that are worth distinguishing from an API client, API provider, or intermediary point of view.

The ATOMIC PARAMETER LIST as a whole, but also its elements, can be optional or set-valued. These properties should be expressed as cardinalities in the API DESCRIPTION.

Figure 4.13 sketches an application of the pattern in a request message. The data transfer representation in the figure has three ATOMIC PARAMETER entries.

In the Lakeside Mutual case, an ATOMIC PARAMETER LIST may represent customer addresses (MDSL notation):

Click here to view code image

data type AddressRecord (
  "streetAddress":D<string>*,
  "postalCode":D<int>?,
  "city":D<string>
)

The streetAddress is set-valued, indicated by the asterisk *. The postalCode is marked as optional in this example, indicated by the question mark ?.

A JSON representation of sample data adhering to this definition is

Click here to view code image

{
  "street": ["sampleStreetName"],
  "postalCode": "42",
  "city": "sampleCityName"
}

Revisiting the Customer Core example from the ATOMIC PARAMETER, it might be required to specify multiple fields in the request. In that case, a single fields=city,postalCode parameter, which is an ATOMIC PARAMETER LIST, allows the API client to indicate that it wants the provider to include certain (but not all) fields in the response:

Click here to view code image

curl -X GET --header 'Authorization: Bearer b318ad736c6c844b' 
http://localhost:8110/customers/gktlipwhjr?
fields=city,postalCode

The client does not identify the individual fields by a key but by position in the GET request. The provider iterates through the list to decide whether or not to include a field in the response. This, in fact, is the essence of an API quality pattern called WISH LIST (presented in Chapter 7).

Pattern: PARAMETER TREE

Listing basic representation elements in a flat ATOMIC PARAMETER LIST that by definition contains only plain ATOMIC PARAMETERS often is not sufficient, for instance, when publishing rich domain data such as an order that contains order items or products that are sold to many customers (that in turn buy many products).

Note that the pattern is defined recursively to yield the desired nested structures. In HTTP APIs, nested JSON objects provide the tree structure expressed by this pattern; set-valued tree nodes can be represented with JSON arrays containing JSON objects corresponding to the nodes.

Figure 4.14 illustrates the pattern conceptually.

In the Lakeside Mutual case, PARAMETER TREES can be found in several API operations that deal with customer and contract data. Picking up the example in Figure 4.1 at the beginning of this chapter, an example of a two-level nesting is as follows (note that the AddressRecord in the example has already been defined as an ATOMIC PARAMETER LIST above):

Click here to view code image

data type MoveHistory {
  "from":AddressRecord, "to":AddressRecord, "when":D<string>
}
data type CustomerWithAddressAndMoveHistory {
  "customerId":ID<int>,
  "addressRecords":AddressRecord+, // one or more
  "moveHistory":MoveHistory*       // type reference, collection
}

This MDSL data definition CustomerWithAddressAndMoveHistory might produce the following JSON object-array structure at runtime:

Click here to view code image

{
  "customerId": "111",
  "addressRecords": [{
    "street": "somewhere1",
    "postalCode": "42",
    "city": "somewhere2"
  }],

  "moveHistory": [{
    "from": {
      "street": "somewhere3",
      "postalCode": "44",
      "city": "somewhere4"
    },
    "to": {
      "street": "somewhere1",
      "postalCode": "42",
      "city": "somewhere2"
    },
    "when": "2022/01/01"
  }]
}

The MDSL Web site⁶ provides more examples.

6. https://microservice-api-patterns.github.io/MDSL-Specification/datacontract

Pattern: PARAMETER FOREST

Just as ATOMIC PARAMETERS may form ATOMIC PARAMETER LISTS, PARAMETER TREES can also be assembled into groups. This is useful only at the top level of a request or response message header or payload.

Figure 4.15 illustrates the pattern.

The PARAMETER TREES in the forest are accessed by position or name; in contrast to trees that may contain other trees, PARAMETER FORESTS may not contain other forests.

Click here to view code image

data type CustomerProductForest [
  "customers": { "customer":CustomerWithAddressAndMoveHistory }*,
  "products": { "product":ID<string> }
]

The JSON rendering of this specification looks very similar to that of a tree of the same structure:

Click here to view code image

{

  "customers": [{
    "customer": {
      "customerId": "42",
      "addressRecords": [{
        "street": "someText",
        "zipCode": "42",
        "city": "someText"
      }],
      "moveHistory": []
    }}],
  "products": [{ "product": "someText" }]
}

However, a Java interface of the service unveils the slight difference in the operation signature:

Click here to view code image

public interface CustomerInformationHolder {
    boolean uploadSingleParameter(
        CustomerProductForest newData);
    boolean uploadMultipleParameters(
        List<Customer> newCustomer, List<String> newProducts);
}

The uploadSingleParameter method uses a single class CustomerProduct-Forest as its input (containing customer and product trees), while uploadMul-tipleParameters works with two parameters, of type List<Customer> and List<String>. Note that the latter can easily be refactored into the former.

Basic Structure Patterns Summary

The data part of the API contract, established by the structures of request and response message payload, directly contributes to (or harms) the developer experience. Qualities such as interoperability and maintainability are at stake. Chapter 1 provides a deeper discussion of these and many more desired qualities (and related design challenges).

The use of the patterns results in platform-independent schema definitions that correspond to JSON schema (as used in OpenAPI), Protocol Buffer specifications, or GraphQL Schema Language (see Table 4.2).

Flat PARAMETER TREES and ATOMIC PARAMETER LISTS can be mapped to path parameters or the query string of a URI, for instance, via “deepObject” serialization [OpenAPI 2022]. This gets more difficult or might even be impossible for deeply nested trees; according to the OpenAPI Specification, the “behavior for nested objects and arrays is undefined.”

All four types of basic structure elements can be used and combined to create METADATA ELEMENTS, ID ELEMENTS, and LINK ELEMENTS as variations of general-purpose DATA ELEMENTS (patterns from Chapter 6). EMBEDDED ENTITIES often come as PARAMETER TREES, and LINKED INFORMATION HOLDERS use ATOMIC PARAMETER LISTS to define the link target (Chapter 7). A VERSION IDENTIFIER often is an ATOMIC PARAMETER (Chapter 8).

Optionally, data provenance information can be provided in the API DESCRIPTION. Such information might include the entities, people, and processes involved in producing the representation elements; the data origins; and where the data moves over time. Note that such information may increase coupling because the message receiver might start interpreting and depending on it, making the API harder to change. The element stereotypes in Chapter 6 describe how to add this and other semantic information to the representation elements: METADATA ELEMENT, ID ELEMENT, and LINK ELEMENT.

Chapter 3 covers the four basic structure patterns presented in this section with their problem-solution pairs.