Digital Infrastructure: Middleware and API
For the first 14 years of its existence, Lego was a wooden toy maker. Then in 1946, Lego bought the first injection molding in Denmark, and spent the next 12 years, it persisted, through many setbacks, to create a model of plastic bricks, which would provide the optimum level of adhesion for children to snap them together and then pull them apart, without the bricks falling apart of their own volition. But once the stud and tube coupling system (look under a Lego brick) was patented in 1958, it did more than hold two pieces of plastic together. It allowed the creation of the Lego brick architecture—which allowed any two pieces of Lego to be snapped together. It made Lego a platform of toys. A platform on which the next 60 years of success has been built. Most importantly, it turned the entire world of Lego into components with standard interfaces. It added unbounded imagination to the constructive powers of Lego without having to worry about compatibility, and how the pieces would work together. And a six-year-old can pick any two of them and snap them together, and pull them apart.
The Many Problems of Connecting Software
It should be intuitively obvious at this point why the world of software doesn’t work as elegantly as Lego, but here are four specific problems:
First, software is not an industrial, machine-made component. It’s made by humans who apply their judgment at every stage of the process. Consequently, almost no two pieces of software are exactly the same. Imagine if Lego bricks were made by hand by people from all over the world!
Second, software is made by thousands of different companies. Even if individuals can be given a set of rules, each firm has its own set of guidelines and approaches to writing software. Although, the reality is that within most firms, there is no consistency, and for the most part, no historical need for consistency at a granular level in the way the software is actually written. And the processes of standardization are themselves like tides in the ocean, which come and go, as each company in the industry strives for unique value and defendable market positions. This is like trying to fit Lego bricks with Meccano pieces.
Third, the way software is designed, conceptualized, built, and run have all changed over the past few decades, and continues to evolve as we speak. The state-of-the-art software written 20 years ago bears little resemblance to today’s code. We therefore often have the problem of backward compatibility—making sure new software can talk to old software. This is easier said than done, as the older systems may completely lack the ability to communicate efficiently, and may need expensive changes in order to be able to do so. This would be like fitting the plastic Lego pieces to the wooden ones, which were never designed to work that way.
Finally, there’s the constant change that software deals with. A single piece of software—one functionality with a system—say that deals with applying tax to a product price—can go through many changes driven by the environment—including regulatory changes, business changes, competitive decisions, process changes, compliance requirements, or the impact of other software being introduced. Imagine that the Lego pieces lost their snap over time and repeated use.
For all these reasons, we can’t simply assume that any two pieces of software will talk to each other, unless they are specifically designed to do so.
Why Does This Matter?
If you’re doing something meaningful in the digital environment—you are likely to require access to information in your company’s core systems (systems of record). You may need to change an existing core process, or update a record maintained in a key system. By this, I mean records that hold key information about customers, sales, prices, products, transactions, suppliers, or employees. Simple apps such as checking balance in a banking app or view loyalty points for an airline app or make a proximity-based offer for a retailer—all require this kind of backend system access. In the simplest case, you may just need to read information from those systems, but in more complex scenarios, you might be updating or adding to the data in the core systems.
Typically, this is the point at which your digital project needs the support of your IT team. And it’s one of the primary reasons why both IT and business have to be committed to making digital work, and the reason why your digital agency may struggle to deliver a scaled digital project by themselves. It is very likely that your company’s systems represent all the four challenges highlighted earlier. They are old and have been updated many times over, they represent a diverse set of suppliers and ages, and do not follow a standard interface model.
Yet, the way these applications interface and integrate to produce a usable service for your digital product or initiative is fundamentally critical to your success. This is why, middleware matters.
Middleware and Application Integration
As software systems have evolved sharply over the past couple of decades, there have been a number of different generations of application integration methods, tools, and concepts.
In lay terms, the starting point would be to simply couple any two applications that needed to talk to each other. Any application would typically have an application programming interface (API)—a way for another piece of software to ask it for information or to perform a task. The problem with this approach was that you couldn’t replace either application without affecting the interface, so any change would cause a decline in service levels. Second, as you added more and more applications into the mix, your effort and costs would grow exponentially. In a typical enterprise, you could have up to 200 systems that need to communicate with each other—you can do the math on the challenge of point to point connects. This is why over time, more elegant approaches evolved.
A driving principle in software has in fact been to make it more Lego-like, through modular architectures. This means that even for a complex system, the individual components are clearly identifiable and are reasonably self-contained, with the goal being that you can take one module out and replace it with another similar component without breaking the overall system.
Along with the modularity, another key step is the evolution of service-oriented architecture (SOA). In this world, you design a system in a way that the software (or a software module) is represented as a series of services that can be defined, discovered, and transacted with, using some common and shared ways (including metadata). In the SOA world, you don’t care if the backend system is a SAP or Oracle system. Or whether it sits on your premises or in the cloud. As long as another software (say a mobile application) can access the underlying services (e.g., tell me current inventory levels) through a well-defined interface, that’s all that matters. This has therefore led to the creation of APIs that follow the framework and principles defined by SOA.
SOAP and ReST
You may hear both these terms while discussing APIs and middleware. Without going into technical details, what you need to remember is that both of these are different ways of writing APIs. Simple object access protocol (SOAP) is more robust, and would be used for online banking and transactions—where it’s important for both parties to be able to confirm that a particular request went through and the transaction was recorded. Rest or representational state transfer (ReST) is a much more lightweight and more universal way of writing APIs and derives its advantage from the fact that the provider of the API and the consumer of the information know little about each other. Which works fine if you are a hotel website and you are consuming the Met Office’s weather API. It’s a loose relationship, the Met Office doesn’t need to know whether or not you actually used that API.
Microservices
The drive toward granularity has led to the creation of micro-services that, as the name suggests, are much more finely grained service definitions. These typically focus on just one task (or even what you might consider a subtask), and they are like smaller pieces of Lego you can combine differently. For example, a bit of code that just retotals the total in your shopping basket every time you make a change, adds tax and shipping, and calculates your net payment amount. The bits of software that add the tax and shipping are good potential examples of micro-services at work as well. Perhaps this figure is used in the shopping basket page but also as you continue to shop, this figure is displayed somewhere so the customer knows how much she is going to pay even as she continues to browse the virtual shelves.
API Management
As APIs proliferate, and boundaries of organizations become more and more porous, companies are finding themselves exchanging ever-higher amounts of information with external entities. If you work in a retail bank in Europe, for example, you are likely to be subject to the changes proposed by the PSD2 regulation,1 which requires banks to provide customer data to any third party who the customer has selected, to aggregate their financial data (another bank, a fin-tech startup or a retailer that offers financial services). Besides, in the omnichannel world, your customer data has to be available across the Web, mobile, kiosk, store, and customer call center already. All of this points to an explosion in the need to serve information to an ever-growing set of systems and channels. This is where API management comes in.
In a nutshell, you will have a large number of APIs, which need to be managed with minimum cost, and with greatest impact and control. They need to confirm to standards of design, security, and maintainability. API management platforms allow you to manage and govern this set of APIs and prevent them from becoming a jungle of weeds. Through API platforms, you can control who gets to see and use which APIs and manage exceptions. APIs have become so common today that we often forget how different the structure of digital services was earlier. When the Amazon home page loads, it consumes up to 150 internal APIs, which are typically set up as micro-services.
Most companies now think of the API management as an enterprisewide layer (nowadays sometimes referred to as an Integration Engine) to connect multiple apps and services to each other and to the backend systems. You should ideally plug into your organization’s API management system, rather than create your own. If your service publishes APIs for other to use, you probably want a dashboard to track who is using them, how often, and for what purposes.
Tip: Think about all the businesses that could to consume the service or information you offer, and imagine how they could integrate it into their customer offering.
The strategic role of API management can be seen in the way API management tools have been acquired by technology majors. Apigee was bought by Google, Mashery is now the Tibco Cloud API Management, and MuleSoft is now a part of Salesforce. In the ecosystem of businesses, APIs are the links between systems and information. Your taxi booking service works because it can consume a map API such as Google Maps, and tools like CityMapper work because they consume APIs from underlying transport systems in each city that they work in.
Consuming APIs
We have talked primarily about pushing out your own information as APIs so far, but you could be consuming external APIs for your product as well. In fact, you should actively scan the network for APIs, which you can be used to improve your service. If you are digitizing the customer experience for a golf course, you might start with location and weather APIs. Beyond this, you may find APIs from other golf sites, equipment makers, and magazines, which allow you to enrich your application, or social media APIs so that your customer can post her excellent score on her best friend’s wall! If your user experience is critically dependent on an external API, it needs to be underpinned by a service contract, which covers areas such as change of terms, or service model, or even discontinuation of a particular API.
As businesses become more networked, and the boundaries of organizations become more porous, we will collaborate and partner more voraciously in a networked environment, to create services that creatively combine functionality in real time. With the press of one button on your phone, you will trigger a series of events allowing you to check out of your hotel, summon an taxi, order a coffee at the coffee shop when you’re nearing the airport, and check for flight departure gate information as it becomes available. To do all of these, we’ll need APIs—the lubrication that allows the smooth ticking over of the digital universe.