2. Collaborative API Design

“Big design up front is dumb but doing no design up front is even dumber.”

Dave Thomas

An API design that looks good to the designer may not be the best design to solve the problems at hand. The initial assumptions about an API design may be incorrect as the API encounters the real world of customer, partner, and workforce needs.

API contract design is a separate and critical step of software delivery. Following an API design process encourages communication internally within the organization and externally between the organization and the developers ultimately tasked with integrating the API. It helps to identify incorrect assumptions and validate the assumptions that are correct. Finally, it encourages collaboration between API designers and the developers that will integrate the API.

This chapter presents a design process that is flexible to meet the needs of a single API product or mid-to-large scale enterprise API platform. Organizations from as small as 10 employees to those with a staff of over 10,000 developers have used this collaborative design process. Ultimately, it delivers business value with a customer-centric focus by applying outside-in design using the five principles outlined in Chapter 1.

Why an API Design Process?

Before presenting the design process overview, it is important to recognize that teams can design and deliver an API successfully without the need for a formal API design process. I have worked with many companies across the world that have managed to deliver an API into production without any kind of consistent approach to API design. However, the APIs they produced took longer to deliver as they required multiple iterations of breaking design changes. The APIs that weren’t properly designed lacked sufficient insights into how to use the API compared to those designed with an API design process.

An API design process encourages efficiency throughout the delivery process. By focusing on the API contract first, the design represents the needs of users and developers as a primary concern. Also, implementation details are less likely to leak into the API design, resulting in a fragile API design that must introduce breaking changes as implementation details change over time.

A backend API is the primary blocker for any front-end delivery schedule. If front-end developers are forced to wait until the backend developers have completed the API implementation, the end-to-end delivery process will take too long. Any errors in design won’t be identified until the front-end developers start to integrate the API. Customer feedback isn’t available until all of the integration work has been completed. Figure 2.1 visualizes this problem and the impact it has on the delivery schedule.

An API design process encourages an iterative, team-oriented design effort that allows for greater overall efficiency. The front-end and backend API teams work together to arrive at a design, then parallelize their specific tasks. Customer feedback may be incorporated earlier as well, avoiding last-minute rework efforts. As depicted in Figure 2.2, the process is repeated for each release, ensuring the design process becomes more rapid while incorporating feedback iteratively. Remember that the sooner that API design mistakes are caught, the cheaper they are to fix.

API Design Process Anti-Patterns

Failing to adopt an API design process or adopting a process that is less than effective can lead to the anti-patterns that result in negative impacts for the team and the entire organization. Review the common API design anti-patterns detailed here and see if any of them resonate.

The “Code the Design” Anti-Pattern

API designers without a formal API design process will start with code and work backwards into an API design. The API design will incorporate internal technology decisions, sometimes to the point of requiring familiarity with a particular database or cloud vendor.

For example, a public API product for a recommendation engine required the understanding of Apache Lucene to use the API. The API accepts configuration files via an HTTP POST using the Lucene configuration file format to manage the recommendation engine. The leaking of internal implementation details to API consumers resulted in the need to become Apache Lucene experts, rather than experts in using the recommendation engine API.

There is value in prototyping APIs or producing evolutionary API design through a mixture of code and design. However, this approach requires a focused effort to find the right balance of prototyping, followed by an outside-in design effort that incorporates the lessons learned. An effective API design process supports this iterative learning approach.

The “Next Release Design Fix” Anti-Pattern

Teams without an API design process may find themselves already planning the next API release, perhaps before the current version has been pushed to production. This is a result of API design decisions that become unchangeable with the current release. Design improvements that would result in breaking changes get moved as technical debt to the backlog.

This anti-pattern starts as an innocent design decision due to the complexity of the underlying code change. Perhaps the code change will take too long. Teams are forced to push an inadequate API design into production and support it for the foreseeable future. This can be something as small as a spelling mistake or small typo that has to remain part of the API to avoid introducing a breaking change to a large number of developers.

An API design process, combined with an effective API lifecycle strategy as discussed in Chapter 16, can mitigate this kind of problem. Techniques to address these issues with API lifecycle management will be addressed in a later chapter.

The “Heroic Design Effort” Anti-Pattern

Those more familiar with the business domain incorporate their understanding of customer and market needs to deliver an API that meets the needs of their target market. This may work well for small teams with a deep understanding of the customer and the market.

However, it doesn’t offer a predictable way to engage in emerging areas where subject matter expertise is in short supply, resulting in the need for heroic API design efforts. Long days of chaotic development that are filled with multiple design changes per day during the march to production are the signs of the heroic design effort anti-pattern. Last-minute calls with pilot customers that result in discovering significant design flaws are commonplace. Teams scramble to find a design solution that addresses the flaws before the release. Code is quickly patched to “make it work” with the limited time available.

While an API design process does not guarantee a perfect design the first time, it will help challenge assumptions quickly. It will also encourage early communication with subject matter experts and customers to address flawed design issues before they are too expensive to rectify.

The “Unused API” Anti-Pattern

Teams do not want their project to be considered a failure, or worse, languish with little-to-no use in production. Yet, this is often the case as API designs may miss the underlying goals and desires of the target audience. An API may be released to great fanfare, only to languish with only a few, if any, integrations. When an integration is finally started, bugs are encountered due to brittleness of the design and implementation. Rather than designing in isolation, an API design process should encourage validation early and often from stakeholders to avoid an API going unused.

The API Design-First Approach

An API design process is a predictable method of moving from business requirements to an API design. The goal of API design is to make it easier to discover, integrate, and deploy solutions in a way that is scalable for the organization and external parties.

An API design-first approach is important as APIs last forever. Once an API has at least one integration in production, it is nearly impossible to migrate consumers to the next version of an API.

Taking an API design-first approach starts by identifying the capabilities to deliver, then moves toward an API design to meet the desired outcomes - all before writing a line of production code.

Of course, reality doesn’t work exactly in this way. Code and data may already exist and must be leveraged from an existing system. API design-first doesn’t require strict adherence to a greenfield process that assumes no pre-existing code or data. It should, however, emphasize the API design effort as a separate and critical step of software delivery.

An API design-first approach has five rapidly executed, iterative phases, as shown in Figure 2.3:

1. Discover: Determine the digital capabilities the API needs to deliver, searching for APIs that may already exist to meet the requirements.

2. Design: Produce an initial API design or improve an existing API design to address the digital capabilities required but not available.

3. Prototype: Produce a prototype or mock API to gain feedback from stakeholders regarding the current design. Revisit previous steps based on the feedback.

4. Deliver: Deliver the API through a parallelized effort across developers, quality assurance, operations, and documentation teams. API capabilities are released iteratively rather than through a single release, driven by the agreed-upon API design.

5. Onboard: Customers, partners, and/or internal developers onboard with the API, integrating it with their solutions. Support is critical at this stage to help teams with complex integration needs.

Notice the iterative design process that occurs as stakeholder input is gathered. Feedback is incorporated early and often, making design changes along the way. This results in an API contract, which provides the specific details of how the design is realized. Prototypes or mock implementations help to see the API in action, prior to the full delivery process. Once delivery begins, the effort is parallelized across all teams, with the API contract as the primary communication artifact. After developers are onboarded with the API, additional feedback results in a new design effort.

Remaining Agile with API Design-First

An API design-first approach is focused on frequent feedback and opportunities to make adjustments throughout the design and delivery process. An API design-first approach does not specify that all design work must be complete before proceeding into code. To understand how, it is time to revisit the Agile Manifesto and see how it can be applied to an API design-first approach.

The Agile Manifesto Revisited

A quick review of the Agile Manifesto principles helps to better understand how API design-first fits with agile development. A few relevant to the concerns of API design first have been reprinted below:

■ “Our highest priority should be to satisfy the customer”

■ “We should welcome changing requirements, even late in development”

■ “We must strive to deliver working software frequently”

■ “Business people and developers must work together daily”

■ “Working software is the primary measure of progress.”

■ “Be attentive to technical excellence and good design, as this enhances agility.”

■ “Seek the simple by maximizing the amount of work not done.”

Keeping these principles in mind, teams have the opportunity to remain agile while communicating early and often with stakeholders on the API design. These stakeholders may include internal development teams, channel partners, and the developers tasked with integrating the API.

Delivering an API design progressively, rather than all at once, allows teams to meet the principle of welcoming changing requirements and delivering working software frequently. It also helps teams avoid last-minute scrambles that can negatively impact API design.

The “seek the simple” principle encourages teams to design in a simple way. Teams should design an API that avoids clever designs that require a higher cognitive load to understanding. Instead, designs should be intuitive based on the use cases it addresses and should use vocabulary that is appropriate for the solution domain. They should offer only the necessary information to support the use case.

The Agility of API Design-First

The goal of API design-first should be to gather sufficient details to limit the risk of a breaking change in the future. It doesn’t mandate that an entire API or a series of APIs design exist before development begins. Agile development and API design-first may co-exist.

Remember

Teams can always add to an API design, but it is impossible to take things away without breaking integrations that depend upon them. Take advantage of agile software development to incrementally design APIs with the needs of customers, partners, and the workforce in mind.

An API design-first process enables teams to move quickly, thoughtfully, and with the agility to make changes early in the process. It is the complete opposite to a waterfall-based approach to API design.

The Align-Define-Design-Refine Process

One of the biggest challenges most API design teams encounter is how to deliver an API design from business requirements in any variety of forms: use cases, spreadsheets, wireframes, etc. Those with a background in software business analysis may find this to be an easier task. However, there are still challenges when it comes to mapping a domain model and capabilities into a web-based API design. This includes the need to ensure alignment of scope and deliverables between all technical and non-technical team members.

As the name suggests, the Align-Define-Design-Refine (ADDR) process guides teams through an API design-first approach. The process groups the step-by-step process into four distinct phases:

■ Align: Ensures alignment of understanding and scope across business, product, and technology around a set of desired outcomes

■ Define: Maps business and customer requirements into digital capabilities that will form the basis of one or more APIs to deliver the desired outcomes

■ Design: Applies specific design steps for each API to meet the desired outcomes using one or more API styles

■ Refine: Refines the API design through feedback from developers, in addition to documentation, prototyping, and testing efforts

There are seven steps across the phases, which will be explored in-depth for the remainder of this book:

1. Identify digital capabilities: Identifies the customer needs and desired outcomes, including the corresponding digital capabilities that are required.

2. Capture activity steps: Expand the digital capabilities to include a unified understanding and clarity through collaborative API design sessions.

3. Identify API boundaries: Groups the digital capabilities into API boundaries and determine if the APIs already exist or if new APIs are required.

4. Model API profiles: Use a collaborative API modeling session to define the high-level API design including resources and operations into an API profile.

5. High-Level API designs: Select one or more API styles that each API profile will offer and document the high-level design elements.

6. Refine the design: Incorporate design feedback from API consumers using techniques that encourage improvement in the developer experience

7. Document the API: Complete the API documentation, including reference documentation and getting started guides, to accelerate integration.

Figure 2.4 summarizes the ADDR process that supports an API design first approach.

The process achieves the following goals:

■ Deliver an API design that emphasizes and solves the customer problems using a vocabulary they understand

■ Reduce the constant design churn common with informal design processes

■ Optimize the entire organization, not just developers, for API design and delivery

■ Avoid unnecessary steps whenever possible to expedite delivery

■ Create a repeatable process that delivers an API design with a mixture of technical and non-technical roles, some of whom don’t fully understand the nuances of API design but are able to contribute their insights

■ Produce artifacts that may be referenced within the team and shared across the organization, rather than some scribbles on a whiteboard that fail to communicate reason and intent about the resulting API design

These outcomes contribute to a healthy, sustainable, and successful API program. The remainder of this book examines the ADDR Process in detail and applies each step using a real-world design project:

■ Aligning and defining the APIs required to deliver desired outcomes based on the jobs to be done by developers and end users (Chapters 3-6)

■ Designing APIs that helps meet the desired outcomes of the target audience using the appropriate API styles along with common patterns and practices (Chapters 7-9)

■ Decomposing APIs into smaller services to shift complexity when needed (Chapter 10)

■ Improving the developer experience through a combination of robust documentation, helper libraries, command-line interfaces, and testing strategy ensure consumers get up-and-running quickly and with confidence (Chapters 11-13)

■ Evolving the API design is critical to sustainable, long-lived APIs (Chapter 14)

■ Protecting APIs to ensure that data is not leaked to unauthorized parties (Chapter 15)

■ Scaling the API design effort is important for larger initiatives (Chapter 16)

The Role of DDD in API Design

As mentioned, an API design process should emphasize and solve the customer problems using a vocabulary they understand. This requires a deep understanding of how the API will address market and customer needs, combined with business strategy. If the design and development of an API occurs without factoring in these concerns, it will often miss the mark for being a great API that is a joy to use.

Domain-driven design (DDD) is an approach to software development that encourages collaboration between business domain experts and software developers to address complex solutions. DDD’s core principles include discussion, listening, understanding, discovery, and delivering differentiating, strategic business value. Every member of the team across technical and non-technical roles contribute to the insightful depth of business innovations in the software solutions. Those new to DDD may wish to refer to the seminal book on DDD by Eric Evans and Vaughn Vernon’s book, Implementing Domain-Driven Design, which provides insights on implementing DDD in an organization.

The ADDR Process is built loosely upon concepts and practices found in DDD. However, organizations do not need to be practicing, or even familiar, with DDD to be effective at applying the process. Those familiar with DDD may recognize some of the concepts and techniques used. However, it is important to recognize that the ADDR Process may deviate with DDD practices when necessary to ensure that it remains approachable and repeatable in a variety of situations. As such, those familiar with DDD may wish to make adjustments to the process to best fit their needs and preferences.

API Design Involves Everyone

Most software development involves several people across a variety of roles. Business leaders and product owners analyze market needs. Software architects and technical leads map out the important design decisions for the solution. Developers design and write the code that makes it all work. Designers and user experience (UX) experts pull everything together as a user interface, with an eye toward usability.

Each person contributes their experience and can leverage their strengths and skills as part of the API design process. For smaller organizations, a single person may be required to fill multiple roles. Whenever possible, assign the more technical roles separately from the product and business roles. This will ensure a healthy balance of perspectives when designing APIs.

The roles typically involved in API design sessions may include, but are not limited to, the following:

■ API designers and architects that help facilitate the design process and bring in API design expertise.

■ Subject matter experts (SMEs) and domain experts that help to clarify requirements and shape the vocabulary used in API design.

■ Technical leads that will be responsible for guiding implementation efforts and may require additional clarifying questions for estimation purposes.

■ Product Managers that incorporate market opportunities and customer needs into the API design.

■ Technical writers who ask clarifying questions during scope and design sessions that will impact the capabilities delivered and drive the production of API documentation and getting started guides.

■ Scrum Masters and project managers provide input to assist in scheduling and identifying risks.

■ QA teams can provide input on designing testable APIs, how and when to test web APIs, and design test plans in parallel with development efforts.

■ Infrastructure and operations ensure network, server, container platforms, message brokers, streaming platforms, and other necessary resources are available for the teams that are building and consuming APIs.

■ Security teams review API designs for personally identifiable information (PII) and nonpublic information (NPI) concerns, identify risks, limit the surface area of attacks, and help to design APIs that will access sensitive data.

An API design process integrates perspectives from each of these roles to align the business with development teams, define the clear goals and outcomes of an API, and to design the API to meet the defined goals. Upcoming chapters explore this in further detail.

Applying the Process Effectively

The ADDR process may be integrated with any existing process. However, be prepared that some steps may seem uncertain or awkward at first. Over time, the processes will become more familiar, and the fruits of labor rewarded. Give the organization time to become familiar with the process. It may also be useful to spend time listing previous challenges and how the process seeks to address them.

Organizations may wish to incorporate this process incrementally. In this case, it is recommended to start with identifying the activities and steps needed for an API, as detailed in Chapter 4, then proceed with API modeling in Chapter 6. Additional steps may be introduced over time as they are needed.

Summary

The design of an API’s contract is a separate and critical step of software delivery. API design requires communication within the organization and with the developers using the API. It helps to course-correct wrong assumptions. It also encourages communication between business, product, and technology teams.

An API design-first approach takes an outside-in perspective on the design of an API by focusing on the customers and developers that are building the solution. Combined with design techniques that take a bottom-up approach, APIs will have a more balanced design that both reflect the domain and the needs of customers and developers. An API design process requires a variety of roles that help to align, define, and design the capabilities and outcomes that APIs will provide.

With the introduction to the art and fundamentals of API design complete, it is time to dive into the details of the ADDR Process with the first phase: Align.