The size ofclient-side libraries, and the features included in them, can vary widely. We will discuss the pros and cons of including various responsibilities, like exposing network errors, or making service calls, validating service responses, caching expensive API calls, handling errors, and alarming when things go wrong. Key topics in this chapter include
How much code should live in a client library?
Libraries should not have business logic, ever.
Handling errors is a must, and depends on the nature of the endpoint being called.
What Are Client Libraries?
In a microservices environment, we call client libraries, the application code used to interact with a service. It’s a set of code, that, when added to an application, will give developers access to the APIs of the service it was written for. We believe that client libraries should be a very thin service access layer that implements a convenient set of objects that mimic specific service APIs. For example, a client library for books-service should expose a Book class that has methods that correspond to the API response of the service, returning values of the correct type for each attribute, making it palatable and convenient to interact with the API.
In general, your client libraries will have a set of plumbing features that should be common to all client libraries , they provide functionality that is not associated with each underlying API, but is required in order to provide all the tools developers need to interact with services. We call the following plumbing features:
Network access. The main characteristic of distributed systems is that APIs require a network access layer in order to be accessed. That network access layer is implemented in client libraries; so, that developers don’t need to write network code in all their applications.
Object serialization/deserialization. APIs use a serialization format to retrieve and expose data, which needs to be handled at the client library level. It’s expected that client libraries will abstract the details of the serialization format to its users, making responses look like regular objects, as expected by users of the language in which the library is implemented.
Low level errors, like timeouts. Client libraries have to handle expected errors, such as access to invalid resources, or network timeouts. Depending on the programming language, client libraries will handle those type of errors, and will raise exceptions, or set specific response codes to service calls, allowing its users to write error handling code according to each failure coming from a service.
Expose application errors. Client libraries can choose to translate some service response into code that is common in programming languages. For example, when a service returns a “404” to a request for a nonexistent resource, the client library may decide to expose that as a ResourceNotFound exception, which may make programming more natural, instead of forcing API users to check response codes.
Caching. Client libraries may have code to cache service responses, thus increasing overall system performance. The caching layer in client libraries can have custom caching code, or it can just be HTTP-compliant to send and honor certain headers, performing tasks such as storing response based on ETags or expiry headers.
Service discovery. A client library needs to know how to connect to the services it exposes, and allow its users to override its default configuration.
This may seems like a lot to implement in what is supposed to be a “thin” layer; however, if you decided to use an IDL and/or a Hypermedia type, as explained in Chapters 4 and 5, you’ll probably be able to get a lot of this “for free” by using tools available for users of your IDL/Hypermedia type of choice. For example, in the sample project of this book, we decided to follow JSON API, and can use a wide array of libraries ( http://jsonapi.org/implementations/ ) to help us building client libraries.
We’ll go into more detail on some of the features we expect to see in most client libraries; however, we’ll first take a quick detour to talk about the type of code you should never have in a client library.
What Not to Do
Here are a few suggestions, based on our experience, about things to absolutely avoid when designing client libraries in a microservices environment.
Don’t wait eternally
In a microservices environment , a web page could make a large number of service requests for rendering each page, which will mean that slow response times from a single service could turn the user experience into something abysmal. It’s imperative that, when you consume your own APIs, you define timeouts that, if reached, will not make your site unusable.
The one thing that will always happen in a microservices environment is seeing service requests time out. In a distributed application, there are a multitude of the reason for seeing such timeouts, be they general networking snafus, network congestion or just general latency increases along the full trajectory of a service request.
Don’t make the clients smart
We recommend you never put any sort of business logic in a client library, be it some constant of default values, or any logic that may seem straightforward to code as you’re working on your gem. For example, in our book store example, we will need to know whether or not a specific book is available for purchase at a specific moment in time. Following the initial, very simple requirements for out hypothetical bookstore logic, a book would be considered available if we have more than 0 copies in stock, which seems fairly obvious, however also naive.
If we decide to implement an available? method in our Ruby library, it would look like:
def available?stock>= 1end
Someone, even yourself, could try to convince you that having that logic live in the client library is not harmful, and even more efficient than requiring the service to vend yet another attribute. Don’t be fooled by that influence because every piece of business logic starts simple and gets more complex with time. Keeping the theme of our hypothetical future evolution, your book store may start selling books that can be considered as “always available” because they can be printed on-demand; so, you decide to make tweaks to your “in-library” code.
def available?type == 'on_demand' || stock >= 1end
That wasn’t that bad; however, you now need to release a new version of your library, and get all consumers of that data to migrate to it, which may not be that simple, and will cause, even if for a small amount of time, to have different clients using different versions of the library that have a different implementation of that method in production.
In another unsurprising turn of events, you find out that the bookstore has a new supplier that promises same day delivery of books to your warehouse; so, the business wants to not lose any sales of books that belong to that supplier; so, you need to tweak your implementation, you decide to add a new attribute to Book resource called inventory_management, which, when set to allow means that a book can be sold even when it is not in stock. Off you go to update your client library, which now reads:
def available?inventory_management == 'allow' || type == 'on_demand' || stock >= 1end
The code is starting to look like spaghetti, we’ve seen how ugly it gets, don’t do that. Once again, we will have the issue of possibly having clients using different versions of the library at some point in time. Also, in an ideal world, you’d never have to expose the inventory_management to certain clients, it’s an implementation detail that should be kept inside the domain of the books-service; however, it had to be exposed to allow the logic in the library to be updated. We’ve been showing you the available? method implemented in Ruby, but, if your application happens to run in an environment where multiple languages are used, then, you would also have to implement that code in each one of those languages; so, trust us, business logic should never live in a client library. The general rule should be to keep your client libraries as dumb as possible, and move the business logic into service endpoints instead.
Service Orchestration
We can also offer an important positive recommendation: Try to understand how you can orchestrate the information retrieved, so you can optimize the workflow and cut down on load time.
In some cases, when working on user-facing applications, you may notice a specific pattern of service access that could be optimized. For example, suppose you know that on a book detail page, you need to access detailed book information from books-service, as well as a large amount of information from a set of other services, such as images, reviews, or categories. You may be tempted to add a class into your client library that could load data from those services in parallel, and return it in a faster fashion.
That type of connected service call is what is called service orchestration, and it is very helpful for increasing the efficiency from an end-user point of view. We’ll discuss in Chapter 7 the way backend-for-frontend (BFF) services have started to be used for exactly that reason, and we recommend you take that approach—add orchestration to a BFF, and have a thin client library for that service, which will give you all the advantages of optimizing your workflow to improve performance, and will keep logic away from your client libraries.
Caching in the Client
In order to increase speed, caches are often used in distributed systems, and a layer that sometimes becomes responsible for caching is the client library. It seems like a decent choice for caching because, by handling caching in the client library, it makes caching transparent to its clients, which can be seen as an advantage, reducing system complexity.
In quite a few cases, service clients can often tolerate data that is slightly stale. A good way to exploit this fact is to cache service responses within the client application (which can of course be either a consumer-facing front-end application or some other service that depends on data from your service). If, for example, the client can tolerate service data being 5 minutes old, then it will incur the interapplication communication costs only once every 5 minutes. The general rule here is that the fastest service calls are the ones your applications never need to make.
Build Caching into Your Client Library
Often the team that builds a service will also be developing the libraries (such as Ruby client gems) to access that service. Some of these client libraries are built to offer the option to have a cache store object injected into them via a configuration option. A cache store injected that way can be used to store all response objects returned by the service. A prudent approach when allowing cache store injection in your client library is to require as little as feasibly possible about the cache object that is injected. That strategy provides the client applications with more flexibility in choosing which cache store to use.
Caching should be entirely a client application’s responsibility. Some clients might not want to cache at all, so make sure to provide the ability to disable caching in your client. It is also the client application’s responsibility to set cache expiration policies, cache size, and cache store back-end. As an example, Rails client applications could simply use Rails.cache; some others might prefer a Dalli::Client instance (backed by Memcached), an ActiveSupport::Cache::MemoryStore (backed by in-process RAM), or even something that is entirely “hand-rolled” by your clients, as long as it conforms to the protocol your gem code expects of the cache store.
We’ll go into more detail about (mostly service-side) caching in Chapter 7, and just recommend that client libraries offset caching to standards—HTTP headers—or have a pluggable caching mechanism.
Error Handling
In a distributed environment, properly handling errors is extremely important because errors will happen due to the simple fact that there is a network between two systems making a call.
In systems that do not properly handle errors, one service could bring down an entire application when issues arise. We’ve seen many times a not-so-important service A start to return errors that are not properly handled by its clients, and thus generate a series of “cascading failures” that can potentially bring down other services or entire applications.
Timeouts
The main cause of failures in a microservices architecture will probably be timeouts, due to the network between client and server. In order to handle those types of errors, you should have timeouts—ideally with exponential back-offs—on connections and requests to all services you connect to. When services take too long to respond, a client can’t possibly know if the delay is caused by expensive computation, or because the connection has failed. In distributed systems, you should always assume that a slow synchronous response is a failure; and having timeout values set very low means that your client code will not wait too long for a service response, and can handle that condition faster.
We recommend you set aggressive timeouts that will keep your systems honest. If you have synchronous endpoints that are sometimes very slow to respond, you have to make sure you fix all performance bottlenecks in those services, instead of increasing the timeout on the client. We’ve seen clients that have very long timeouts; anything above 1 second should be unacceptable, which means that the timeout is not protecting the client from a slow service, because it’s still waiting too long for a response, and probably making the a user-facing application looks sluggish and incorrect—because of the service failure.
Circuit Breakers
A well-known pattern for handling service failures is the circuit breaker, also known as the feature flag. The circuit breaker design pattern is used to detect and isolate repeating failures, preventing system-wide failures caused by an issue with a specific endpoint/service. Circuit-breaker libraries usually allow their users to set a group of settings that characterize the circuit to be defined. Some of the properties that are usually defined in a circuit are
Timeout: As discussed in the previous section
Retries: The number of times a service call should be retried, in case of errors
Threshold: The number of errors that will cause the circuit to open
Window: The period of time to keep the circuit open before a new connection is attempted
In our sample bookstore application, we display a classic shopping cart icon in the top right of the user interface, and display the number of items currently in the cart, alongside the actual cost of the items in the cart. That interaction point is a good use case for a circuit breaker, and we have a circuit breaker in place. In that specific case, we should have a very slow timeout setting, we don’t really need retries in case of failures, and we have code in place to render an error scenario that doesn’t look like a failure. With the circuit breaker in place, our code properly handles failures, and it stops sending requests to the service in failure state, until it (hopefully) recovers and starts responding appropriately.
class BooksServiceinclude CircuitBreakerdef find_book(id)ApiClient::Books::Book.find(id).firstendcircuit_method :find_bookcircuit_handler do |handler|handler.logger = Logger.new(STDOUT)handler.failure_threshold = 5handler.failure_timeout = 2handler.invocation_timeout = 10endend
We recommend you get in the habit of always having a circuit breaker setup in place. Your application will be able to withstand failures in downstream dependencies, while at the same time, it will not overwhelm those services that are under some sort of failure scenario.
So far, we’ve only touched on handling the errors that happen in a downstream dependency; it’s also extremely important to have some information on how to actually fix those errors. The first place anyone will look when errors start to happen is in service logs; we discuss generic logging in detail in Chapter 13, but the message to convey is that your client libraries are a great location to add logging about service calls. You should log most details your client library uses when requesting data from a service.
Ideally, you should use an error-tracking system, whether built in-house or commercial, such as Airbrake ( https://airbrake.io/ ) or bugsnag ( https://bugsnag.com/ ). Using such tools has some advantages:
You can have a central location where your team can see what errors are happening throughout all your services. When a user-facing application starts to experience an increase in errors, the first challenge in a microservices environment is to find out the origin of the error; so having some visibility into error rates of all services is very useful, in stressful situations.
You can group similar errors. Some systems can start to generate a really large number of errors; in an ideal world, someone should be notified about that one error with some sort of indication that it is happening quite often. We’ve been in situations where services would generate tens of thousands of error emails that ended up being a single failure, but that would cause email servers to be overwhelmed, delaying the delivery of other, potentially new errors, that could be happening.
You can link alarming to escalation and severity policies. The main feature of an error-handling system is to notify the owning team that something is wrong. Depending on how teams are set up in your organization, the error-handling tool should notify one person, and have an escalation policy to alert other individuals if no action is taken. It’s also important to be able to assign a severity to errors, so that errors with a higher impact appear quickly and are given priority over other types of errors.
Performance
Performance is a critical characteristic in a microservices architecture, and we discuss it in detail in Chapter 7. From the point of view of a client library, the message we want to convey is that the performance of a service A can differ from the perceived performance from a client that calls client A. When a client calls a service, network latency, response deserialization, circuit checks, and potentially other tasks will translate into the perceived performance of each service call.
We have the following performance-related recommendations for client libraries:
Have metrics in place to measure service response time from a client perspective.
Only request what you need; avoid requesting 100 attributes when all you need is 2.
Check how much time is spent in each part of a call, as seen by the client.
Updating Client Libraries
Once your client library is used in the wild, it will start to evolve, and you should have an upgrade policy for those libraries, similar to how you support only a small number of versions of the service itself. Make sure that you have rules in place that make it easy for users of your client libraries to upgrade, and be as, up-to-date as possible most of the time.
You should follow a versioning policy for all your client libraries, to better communicate what type of changes your libraries are undergoing as you release new versions. We recommend you follow “Semantic Versioning” ( http://semver.org/ ), as it’s a well-defined policy that is both easy to follow and provides clear guidelines to how the version number should change. In Chapter 4, we talked about versioning the API endpoints in the service; we believe you should never tie the version of the service to the version of the API, as those can evolve in different ways, and having strict rules will not help you move forward quickly.
Briefly, given a version number major.minor.patch, increment them as follows:
The major version when you make incompatible API changes
The minor version when you add backward-compatible functionality
The patch version when you make backward-compatible bug fixes
Also make sure you have a test suite for your client library; it should most likely be slim. You just have to make sure that you can catch obvious errors before releasing a new version of the library. We cover testing in detail in Chapter 9.
Service Discovery
One of the things you will have to add to your client libraries is a way of specifying where the service endpoint for each service is to be found. This seems like a very simple configuration parameter, and can even be defined in the client library itself, as all services should follow a naming convention based on the service name and environment in which the service is running. An example of endpoints for the books service could be the following:
Development: books.development.bookstore.net
Staging: books.staging.bookstore.net
Production: books.prod.bookstore.com
Having endpoints defined in their libraries is a good practice, as it makes it easy for developers to start using the library. If you decide to distribute your service endpoints in your libraries, make sure you allow the configuration to accept an override, in case developers want or need to hit a version of the service in a nonstandard host or port.
One approach we’ve seen and recommend at the moment is to move the knowledge of where services are to be found to service discovery tools. We’ve successfully used Consul ( https://www.consul.io/ ), which is described as a tool that “makes it simple for services to register themselves and to discover other services via a DNS or HTTP interface. Register external services such as SaaS providers as well.” In our sample project, we make use of Consul, and have our client libraries ask it for service endpoints, which means that you can rely on Consul knowing the correct endpoints to connect to in each of the environments where your application runs.
module ServiceDiscoveryclass << selfdef url_for(service_name)client = Consul::Client.v1.httpservice_details = client.get("/catalog/service/#{service_name}").first# TODO raise if there is no service dataendpoint_parts = []if service_details["ServiceAddress"].empty?endpoint_parts << "http://localhost"elseservice_details["ServiceAddress"]endendpoint_parts << service_details["ServicePort"]URI.parse(endpoint_parts.join(":")).to_sendendend
We made the complete source of the Ruby service discovery library available at https://github.com/microservices-from-day-one/service_discovery .
Summary
This chapter focused on all aspects of consuming APIs. We defined client libraries and discussed their features and responsibilities. We talked about many concerns usually covered in client library implementations, such as…
In the next chapter will focus on what can be done to optimize the APIs on the service side. We expect to build APIs that respond quickly and efficiently, and will share some of the techniques we’ve seen that help you keep the promise of fast APIs.