Introducing the Seven Essential Evolutions of Design for Services - SEED(S) Method

As James Lewis and Martin Fowler point out in their seminal article about microservices, one of the main traits of microservice architecture is componentization of a system via services. Wherein by “services” they mean software components that are independently deployable and accessible over standard network protocols, such as: a web service request, or a remote procedure call. By exposing components as services, among other things, we commit to defining an explicit component interface. And this interface needs to be designed. More user-friendly and evolveable we make the interface of a microservice, more useful the service ends-up in the overall system, making the larger architecture robust, the developers of the larger system more efficient, and increasing the speed of development while maintaining the system safety, at scale.

SEED(S) process provides a repeatable, reliable, and battle-tested methodology for designing the interfaces of microservices that are user-friendly and robust.

It should also be noted that, as a generic approach, the SEED(S) methodology is actually useful beyond just microservices and can be effective in design of any number of service types: from front-end APIs to microservices. This universality of applicability should not be surprising. After all, from technical perspective, microservices are also a kind of APIs – whether of RESTful, gRPC, streaming or GraphQL nature – just sized with specific type of boundaries in mind - the one that minimizes coordination needs.

Without further ado, the seven steps of the SEED(S) process are:

1. Identifying Actors

2. Identifying Jobs-To-Be-Done (JTBD)

3. Designing queryable and actionable interaction patterns with Sequence Diagrams

4. Deriving high-level Actions and Queries based on JTBDs and the interaction patterns

5. Describing each query and action as a spec, with an open standard (such as Open API Specification or GraphQL schema)

6. Getting Feedback on the API spec

7. Implementing microservices

Let’s explore each of these steps in greater detail and see how we can master using them for service design.

Identifying Actors

In addition to being an evolutionary methodology SEED(S) takes a distinctly customer-centric approach, viewing services and APIs that it is used for the design of, as products. By now the “APIs are products” mantra is not particularly novel – we have been shouting it from all possible mountaintops for years. The good thing is: product-oriented perspective on APIs and services allows us to reuse the wealth of techniques from the business world, where it is nothing new; in fact the science and art of product management significantly predates that of APIs, or even the internet itself. Many people track product management as a field back to the 1930s Procter & Gamble and Neil H. McElroy’s attempts to improve the sales of P&G’s Camay brand of soap. In the ensuing decades product management has evolved significantly and there are a lot of lessons learned that we can reuse in the much more nascent API/services management space. If APIs are products, we should be able to use similar techniques to design APIs as what we use in product management.

When designing a product, and consequently an API or a service, we have to understand the “customer” - who is the service designed for? Typically, in the API and service management space we don’t call these personas “customers”, but rather we use the less commerce-oriented denomination of “an actor”, removing any accidental, unintended connotation of a financial transaction or interest being present between the service consumer and a publisher.

Usage of “Actors” in the first modeling step of SEED(S) methodology is inspired by the interaction design’s heritage of using “user personas” for a similar need. The notion of personas, as an interaction design tool, was introduced by Alan Cooper in his 1998 book The Inmates Are Running the Asylum and has gained significant adoption since. To be completely transparent: at this point “personas” have also received their share of criticism (which methodology has not?) and some product teams passionately advocate using real user data instead. Discussing the pros and cons of personas in product management is far, far beyond the scope of this book. Actors are inspired by, but are not identical to user personas and the purpose of actors is to aid in the modeling exercise at the stage in the design process, when actual user data is typically limited.

The main motivation for starting modeling with the definition of actors is to aid in scoping and prioritization. A typical plague of API and service design in our industry is over-abstraction and lack of clarity regarding user needs. Too many APIs are simply exposures of some database tables over HTTP or an attempt to provide direct networked access into application internals, via gRPC. Unsurprisingly, such lazy and naive approach to service design is seldom successful. If we don’t even bother to ask: “who will be using this API?” and “What are their needs?”, how can we possibly design solutions that solve for their needs? And yet - too many APIs and services are designed exactly this way: using service publisher’s goals, rather than that of the consumer. SEED(S) attempts to fix this upside-down problem from the very first step, by identifying the actors first.

There are several fundamental rules to identifying the right set of actors for your goals:

1. Much like with Cooper’s user personas, each actor must be specific, more so than precise. By this we mean that identifying the boundaries of what key traits differentiate various actors of our design is more important than identifying excruciating level of detail for who the actors are. We ought to always remember that we are in the process of modeling and any modeling exercise is by definition imprecise - it’s not that we cannot capture the details, rather we don’t care about every single detail and are trying to capture the prioritized view of the reality, relevant to us.

2. Overlapping actor definitions or actors defined too broadly are usually red flags. Actors also must be defined in the context. Having a company-wide “portfolio” of actors that are reused for each application design is more than an indication of trouble - it’s an “all alarms on, call 911” sure sign that the process has derailed and the process has been compromised.

3. As models, Actor definitions first and foremost represent needs/pain points and behaviors inherent to each actor archetype. Those needs and behaviors that distinguish one actor type from another, are relevant and there should be very limited overlap.

4. Less is more—you should use as few distinct actors as possible to describe your problem area, but no fewer than necessary. In most cases, if you have more than five actors for any API or a microservice, it may be an indication of prioritization gone awry.

Identifying Jobs to Be Done (JTBDs)

Once we identify a target class of customers (Actors, in our case) we need to spend significant amount of our time in understanding the jobs they have to get done, and only then create a solution which best addresses their needs. This is a critical point often misunderstood or ignored in the design of services and APIs, so let us try to explain the rationale behind its importance.

Any effective API/service design methodology, including SEED(S), is based on a fundamental premise we mentioned earlier: that APIs and microservices are types of products and in their design we can successfully employ the rich product management tool-set that has been developed over many decades. We already applied one such tool to our modeling process: identification of “Actors” in the manner of “user personas”, from interaction design. In this second step we are going to dive even deeper into product design so it may be worth re-iterating: why do we believe APIs are “products”? After all, a technical capability that is exposed over network, using standard protocols i.e. what we call an “API” doesn’t necessarily have an obvious resemblance with: hand soaps, winter jackets, smartphones and other physical products that we are more accustomed to.

Well, what is a general definition of a “product”, anyway? There is no one, true definition that we are aware of, so we might as well use the one Wikipedia references, because everything on Wikipedia is always absolute truth, right? (Kidding!).

We define a product as anything that can be offered to a market for attention, acquisition, use or consumption that might satisfy a want or need. Products include more than just tangible objects, such as cars, computers or mobile phones. Broadly defined, ‘products’ also include services, events, persons, places, organisations and ideas, or mixes of these.

Kotler et al., Principles of Maketing 7th Edition

Both APIs and microservices satisfy this definition – producers offer APIs and microservices to consumer(s), they satisfy a need of a consumer and this supply/demand can create a two-sided “market”.

So if APIs and microservices are products, how do we create better ones? The identification of actors is the first step, but what comes next? They must solve a customer’s problem. Alas, the unfortunate reality is that too many products are designed from the perspective of a solution provider, obsessing about what they have to offer, rather than concentrating on the problems customers have to solve. Probably the most succinct explanation of this problem comes from the famous words of Harvard Business School marketing professor Theodore Levitt: “People don’t want to buy a quarter-inch drill. They want a quarter-inch hole!” ¹. Indeed, if you are a product company producing drills, you need to realize that the real job customers are trying to get done may be hanging of a painting/photo on their walls, not – shopping for the most perfect general-purpose drill. And if you fail to realize this, continuing endless pursuit into perfecting a drill, you will eventually be outmaneuvered by some inventor who comes up with a simpler, alternative solution to getting quarter inch holes in the walls. It may be a chemical reaction of sorts or just pure telepathic magic – we wouldn’t know – but it will happen. If you look at the history of advancement of technology, it’s the problems that are timeless, solutions change and evolve all the time. Case in point - nobody uses floppy disks to save data, anymore, but the job of needing to save and transport data somehow has not gone anywhere, even if it is all “cloud"-based now.

Harvard Business School professor, Clayton Christensen explains that:

[Customers] often buy things because they find themselves with a problem they would like to solve.

Clayton Christensen, The Innovator’s Solution - Harvard Business Review Press

In his book, Innovator’s Solution, Professor Christensen names this observation the “theory of jobs to be done”. He further explains that product designs are successful and customers find them desirable when “the job, not the customer, is the fundamental unit of analysis” ².

Using Job Story Format to Capture JTBDs

For each of the actors we identify, we need to identify top jobs-to-be-dones for that actor. For the sake of uniformity, as well as to make sure key data points are well-documented, we capture JTBDs in a standard format. SEED(S) process uses “Job Stories” format as defined by Paul Adams ³: “when <a circumstance>, I want to <motivation>, so I can <goal>”.

Figure 6-1. Structure Of a Job Story Format

A job story centers around circumstances, the motivations for a job to be done, by the actor and the goal that they are trying to achieve.

If you are familiar with User Stories from Scrum or other agile methodologies, you may have noticed that the job story looks almost identical. However, as Alan Klement explains in his blog post “Replacing The User Story With The Job Story”, ⁴ there are crucial differences between the two. Specifically, user stories revolve around a user persona, they start with: “as a <persona>”, while job stories completely disregard the persona and instead emphasize the circumstance. This is important and aligned with Christensen’s “the job, not the customer, is the fundamental unit of analysis”. It is also spot on, because in the context of describing a specific job, persona does not matter, anymore. If I need to hang a painting on a wall, it doesn’t really matter whether I am a licensed contractor or a novice home-owner, I will need a quarter-inch hole in the wall (or several). It is the context, the circumstance in which we have a motivation to achieve a goal that matters, not - who we are. Long story short - we identify actors to scope the list of jobs, but at the point of describing each job, for that actor, we need to identify circumstances and not just repeat 10 times: “as a frequent flyer…”

Designing queryable and actionable interaction patterns with Sequence Diagrams

Technically, you could move from JTBDs directly into designing an API spec. However, Jobs are usually written by product managers from the business-value perspective and rarely correspond to microservice interactions one-to-one. We find it very helpful to explicitly visualize the corresponding interactions and understand technical dependencies, in addition to business ones, before we proceed with the next step. In the “Developing Microservices” chapter, further in this book, we will show a more advanced example that solidifies this approach. Also, sometimes the interactions in a complex system are sophisticated enough that coming up with linear list of queries and actions may not be sufficient progress on the design front. Instead, you may want to design an interaction diagram, explaining the sequence of events between various actors.

In the spirit of reusing existing, familiar standards, SEED(S) recommends using UML Sequence Diagrams for this task. You can use any other diagramming approach to express your model expressively, since the whole purpose here is to communicate the intent and the model. However, if you do choose to use UML sequence diagrams, then we highly recommend using one of the markdown-based diagramming formats, such as PlantUML - http://plantuml.com/.

We recommend this approach because modeling in a microservices team is a team activity. Using text-based format, instead of a graphic file allows team members to:

• Keep modeling separate from everybody’s personal choice of an editor. PlantUML and other similar formats can be edited in many different editors. As an example, PlantUML is supported in Atlassian’s Confluence, a knowledge-management software used by many software teams. There’re also free, online editors that can be used, such as https://liveuml.com/ and https://www.planttext.com/

• Easily and effectively version-control sources of the diagrams. Text files are easy to diff, merge and review in pull requests. None of which would be true for a binary graphic file.

• Conveniently integrate modeling into the release process. The diagrams become code and anything you can do with the code, you can now do with your diagrams, as well, if you also version-control them in a system like Git, for example.

Since, at this point, we are in the technical design phase, and JTBDs do not always map to technical interactions one-to-one, the interactions you model in your sequence diagrams do not necessarily end-up being between the actors described in the first and second steps, of the SEED(S) process. Rather, your interaction diagrams may go level deeper and show how the user-centric requirements translate into interactions between APIs and microservices, at a deeper, technical level.

For instance, a very simple diagram describing interactions related to the JTBDs we already identified earlier in this chapter, may read something like the following:

@startuml

actor Agent
participant "Agent Servicing" as AS
participant "Reservations API" as rAPI
participant "reservationCRUD" as rCRUD

AS -> rAPI: checkRes(reservationId)
rAPI -> rCRUD: reserve(data)
rAPI -> rCRUD: cancel(reservationId)

@enduml

which, in LiveUML, would render as Figure 6-2

Figure 6-2. A rendered PlantUML of the Sample UML Sequence

Once we have the sequence diagrams of the interactions, we can capture the technical requirements for a microservice, or an API, in the form of a set of actions and queries described using a standard syntax. Let’s explore, in the next section what those are and how they look.

Deriving actions and queries from JTBDs

While Job Stories provide great format for fluid conversations with subject matter experts and actual customers, translating JTBDs into more technically-oriented set of queries and actions can greatly aid in your design process.

Queries are lookups with defined inputs and outputs. It should be a well-understood contract between a client and a server: what input client sends and what response they expect. They are distinctly different from actions, in that queries do not modify system state (they “have no side effects”).

Actions, in contrast, are requests that cause some sort of state modification – they not only do have side effects, but their whole purpose is to cause side effects. Much like queries, actions also have well-defined contract – for inputs, expected outcome, and expected responses.

Similar to Job Stories, we recommend using a standard format for capturing queries and actions. The template for queries looks something like:

• An expressive description of a query

  • input: list of input variables

  • response: list of object of output data elements

Likewise, the standardized format for actions would look like the following:

• An expressive description of an action

  • input: list of input variables

  • expected outcome: description of the induced side-effect

  • response [optional]: list of data elements in the response (if any)

Please note that Job Stories do not always produce exactly one query or action. A Job story can be translated into multiple queries and actions, as well as a resulting query or action may combine multiple Job Stories as its source. SEED(S) is a process of modeling, design and discovery, not a robotic process that is ripe for removal of human judgement factor.

Describing Each Query And Action As A Spec, With An Open Standard

As a general rule, it is important to formally describe an interface contract of an API or a microservice, before we start implementing it in code. Such codified contracts serve as a mutually agreed-upon understanding between a service producer and consumers, API client developers. The contracts are also easily convertible into user-friendly documentation and interactive playgrounds. Contracts implemented using open standards such as: Open API Spec and GraphQL are widely supported by a rich set of tooling that allow easy rendering of documentation, streamlined creation of developer portals, etc.

In this section we are going to take a definition of an action that we described in the previous SEED(S) phase and we will design a RESTful endpoint for it, using Open API Specification.

The OpenAPI Specification is a specification for describing RESTful APIs, in a standard, tech stack-agnostic manner. OpenAPI Specification is governed by OpenAPI Initiative, a Linux Foundation Collaborative Project. At the time of writing of this book, the latest version of the specification is version 3.0.2 published at http://spec.openapis.org/oas/v3.0.2

Microservice interconnections do not have to be RESTful APIs. Other popular choices include: GraphQL, gRPC, and asynchronous communication using message queues. At the time of writing this book, using JSON-, ProtoBuf-, or Avro- encoded messages directly on Kafka streams, seems to be quite a popular choice. It does not matter which communication style you choose, any one of them will assume exchange of messages and the format of those messages should be well-documented and part of the exchange “contract”. For each one of those styles you can apply SEED(S) methodology in a way appropriate for a particular style. Since RESTful APIs are probably easiest and still most ubiquitous, we demonstrate the approach using a RESTful design, but the methodology works with others, as well.

You can use any tooling to edit and author your Open API specs. If you are looking for suggestions, however, an open-source setup that is available on most platforms, and seems to work well, includes: VS Code editor with Open API Designer plugin. Once you have the plugin installed, and a descriptor YAML file open inside the active tab, you can click “CTRL+ALT+P” on Windows, or “CMD+ALT+P” on Mac and choose the appropriate preview command to see the rendering of the spec as shown on Figure 6-3.

Figure 6-3. Selecting OpenAPI Spec Preview in VS Code

Example Open API Spec for an Action In Our Sample Project

A simple version of the Open API Spec, for the rebooking action we described earlier in this chapter may look something alomg the lines of the following:

openapi: 3.0.0
info:
  title: Airline Reservations Management API
  description: |
    API for Airline Management System
  version: 1.0.1
servers:
  - url: http://api.example.com/v1
    description: Production Server
paths:
  /reservations/{reservation_id}:
    put:
      # parameters syntax https://swagger.io/docs/specification/describing-parameters/
      summary: Book or re-book a reservation
      description: |
        Example request:
        ```
          PUT http://api.example.com/v1/reservations/d2783fc5-0fee
        ```
      parameters:
        - name: reservation_id
          in: path
          required: true
          description: Unique identifier of the reservation being created or
                       changed
          schema:
            type : string
          example: d2783fc5-0fee

      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              properties:
                outbound:
                  type: object
                  properties:
                    flight_num:
                      type: string
                      example: "AA 253"
                    flight_date:
                      type: string
                      example: "2019-12-31T08:01:00"
                    seats:
                      type: array
                      items:
                        type: string
                returning:
                  type: object
                  properties:
                    flight_num:
                      type: string
                      example: "AA 254"
                    flight_date:
                      type: string
                      example: "2020-01-07T14:16:00"
                    seats:
                      type: array
                      items:
                        type: string
            example: [
              {
                outbound: {
                  flight_num: "AA 253",
                  flight_date: "2019-12-31T08:01:00",
                  seats: [
                    "9C"
                  ]
                },
                returning: {
                  flight_num: "AA 254",
                  flight_date: "2020-01-07T14:16:00",
                  seats: [
                    "10A"
                  ]
                }
              }
            ]


      responses:
        '200':    # success response
          description: Successful Reservation
          content:
            application/json:
              schema:
                type: object
                properties:
                  reservation_id:
                    type: string
                    description: some additional description
        '403':
          description: seat(s) unavailable. Booking failed.
          content:
            application/json:
              schema:
                type: string
                description: detailed information

The rendered output with the VS Code plugin should look something like the one on Figure 6-4

Figure 6-4. Rendering Of A Sample Open API Spec Document

Producing a formal API contract is a huge milestone for the design of APIs and microservices. Some may even consider it job well done, at this point. However, good API designs cannot end at this stage. We wish things were that simple, but there is actually an additional, critical activity that still needs to be completed. Next step in SEED(S) process captures this activity.

Getting Feedback on the API spec

The initial version of the API and service design as captured by an OpenAPI Specification-based description, or some other standard, is an important milestone, but there is more modeling work that is necessary for a well-designed API.

We need to show the draft design of the endpoints to the client developers who will be asked to use these APIs and services, and collect their feedback. If previous steps were of active brainstorming and work, this is the stage of careful listening and reflection. It is an incredibly important step for designing APIs, if you care to design the kind of APIs and microservices that will stand the test of time and which your clients will love to use.

Generally, there’re two groups of customers that you need to keep in mind when designing services and APIs:

1. End-users of the system. You APIs enable the user experiences for them, and

2. Client developers who will code against your APIs, to build the user experiences.

At the beginning of the process, in step two of SEED(S) we interview the first group to collect and understand the jobs-to-be-done relevant to the end-users. However, later in the process, once the Open API Spec is produced, the second group, API client developers, must be interviewed to test the usability of the design, before we start coding APIs and microservices.

Both of the research activities are critical. The first study makes sure we build the right thing. The latter one makes sure we build it the right way!

Implementing Microservices

The last step in the SEED(S) methodology is actually implementing microservices. It is intentionally done at the very end of the process. Coding is one of the most expensive activities any software engineering team can engage into. Re-coding a functionality that was initially designed based on wrong assumptions a horrible, time-consuming, and expensive task. Which is why we engage in a carefully thought-out process, such as SEED(S) before we jump into coding microservices. Overall, it saves time and delivers better outcomes.

Before we wrap this chapter up, we need to clarify an important detail. Throughout this chapter, we have been saying “APIs and microservices” and we started by mentioning that the SEED(S) methodology can be equally successfully applied to both the design process of APIs, as well as that of microservices. This is in part true because APIs and microservices have a lot of similarities. But how are they different? Are microservices just small APIs? In the next section we will try to shed some light onto this important question.

Microservices vs. APIs

APIs and microservices do indeed have a lot in common. Microservices are capabilities exposed via standard network protocols, most commonly: HTTP. But capabilities exposed as HTTP endpoints had been known as web APIs, way before the coining of the term “microservices”. So - are the two essentially the same thing? Are microservices just a new flavor of APIs – smaller APIs? More importantly: do we even need conventional APIs once we start writing microservices, or do the smaller APIs (microservices) replace the bigger (conventional) APIs? We have often seen these questions cause a lot of confusion on teams trying to adopt Microservice Architecture.

With some frequency we have encountered developers referring to any small, focused APIs as “microservices”. In such approach microservices have the same role as APIs had before them, so they do indeed replace APIs of the old. In our experience, this is not an ideal approach for successful microservices thinking, and we offer an alternative, albeit opinionated, definition of what separates microservices from legacy APIs. Our approach builds on experience of some notable experts in the space, and is rooted in our own experiences with the successful microservices projects and teams.

We think that if microservices replace anything, the things they replace are the modular components you used to build your systems with. If before you would build a large system by linking (statically or dynamically) various submodules together, in Microservice Architecture the building blocks are networked services we call “microservices”. This approach can graphically be depicted on [Link to Come]:

Figure 6-5. Relationship Between Microservices and APIs

Please note that similar approaches of separating APIs into “internal ones - the ones you build with” and “the external ones - the ones that are optimized for consumption by front-ends” have been described by Phil Calçado as the Backends for Frontends pattern when he was at Soundcloud, and by Daniel Jacobson, during his time at Netflix. Daniel Jacobson explained how they separated APIs into Experience (front-end) and Ephemeral (back-end) APIs, at Netflix.

We have found that the ideal separation of duties happens when all of the business logic (capabilities) is implemented by microservices, while APIs act as a thin layer of orchestration in front of those microservices. Additionally, teams should try to avoid microservices directly “invoking” each other. Instead, for the sake of loose coupling, it’s best if any workflow is implemented in the API layer, without microservices knowing anything about each other. This is where the analogy between Unix Philosophy of building system as a collection of composable tools resonates well with the microservice architecture principles. One of the most powerful aspects of Unix Philosophy is that you can combine Unix tools (e.g. GNU tools) in a variety of ways using input/output piping on the command line or in shell scripts. However, in order to achieve it, it’s critical that various Unix tools act the same way for any input - they should not care who “calls” them or where their ouput goes. Components cannot explicitely know about each other, for them to become composable. Loose coupling is what makes the whole thing work, not just tools being small[-ish] and focused.

The same is true for microservices:

Summary

We set a critical foundation, in this chapter, for understanding the process of designing robust microservices. By establishing an effective and repeatable methodology - the SEED(S), we acquired a powerful understanding of many aspects of what traits make projects successful in their microservices journey and learned how to adapt these traits for our own circumstances.

In the following chapters we will leverage the insights from the understanding of SEED(S). In Chapter 10 we will continue implementing our sample project, reusing and expanding efforts started in this chapter.