Implementing microservice circuit breakers

The ability to invoke a remote microservice comes with an implicit risk--there is always a chance that the remote service is down.

Remember using @SpringCloudApplication? As a reminder, that annotation contains:

    @SpringBootApplication 
    @EnableDiscoveryClient 
    @EnableCircuitBreaker 
    public @interface SpringCloudApplication { 
    } 

The last annotation, @EnableCircuitBreaker, enables Netflix Hystrix, the circuit breaker solution (http://martinfowler.com/bliki/CircuitBreaker.html).

In short, a circuit breaker is something that, when it detects a certain threshold of failure, will open the circuit and prevent any future remote calls for a certain amount of time. The purpose is to prevent cascade failures while giving the remote service an opportunity to heal itself and come back online. Slamming a service in the middle of startup might be detrimental.

For example, if the images microservice's HomeController makes a call to comments, and the system is down, it's possible for the calling thread to get hung up waiting for the request to timeout properly. In the meantime, incoming requests are served by a slightly reduced threadpool. If the problem is bad enough, it can hamper calls coming into the frontend controller, effectively spreading the remote service outage to users.

A side effect when operating multiple instances of such a service is that it can also speed up the failover to an alternate instance of the service.

In exchange for opening the circuit on a service (and failing a call), we can provide a fallback command. For example, if Netflix's recommendation engine happens to be down when a user finishes a show, it will fallback to showing a list of newly released shows. This is definitely better than a blank screen, or, worse, a cryptic stack trace on the website or someone's TV.

In the previous section, we had this fragment of code inside HomeController.index():

    restTemplate.exchange( 
      "http://COMMENTS/comments/{imageId}", 
      HttpMethod.GET, 
      null, 
      new ParameterizedTypeReference<List<Comment>>() {}, 
      image.getId()).getBody()); 

We want to wrap this remote call to the comments system with a circuit breaker/fallback command.

First, we need to move the code into a separate method as follows:

    @HystrixCommand(fallbackMethod = "defaultComments") 
    public List<Comment> getComments(Image image) { 
      return restTemplate.exchange( 
        "http://COMMENTS/comments/{imageId}", 
        HttpMethod.GET, 
        null, 
        new ParameterizedTypeReference<List<Comment>>() {}, 
        image.getId()).getBody(); 
 
    } 

This tiny Hystrix command can be described as follows:

  • This shows the exact same restTemplate call we wrote using Ribbon and Eureka earlier in this chapter
  • @HystrixCommand(fallback="defaultComments") wraps the method with an aspect that hooks into a Hystrix proxy
  • In the event the remote call fails, Hystrix will call defaultComments

What would make a good fallback command? Since we're talking about user comments, there is nothing better than an empty list, so a separate method with the same signature would be perfect:

    public List<Comment> defaultComments(Image image) { 
      return Collections.emptyList(); 
    } 

In this scenario, we return an empty list. But what makes a suitable fallback situation will invariably depend on the business context.

Hystrix commands operate using Spring AOP (Aspect Oriented Programming). The standard approach is through Java proxies (as opposed to AspectJ weaving, which requires extra setup). A well-known issue with proxies is that in-class invocations don't trigger the enclosing advice. Hence, the Hystrix command method must be put inside another Spring bean, and injected into our controller.

There is some classic advice to offer when talking about Hystrix's AOP advice--be careful about using thread locals. However, the recommendation against thread locals is even stronger when we are talking about Reactor-powered applications, the basis for this entire book. That's because Project Reactor uses work stealing, a well-documented concept that involves different threads pulling work down when idle. Reactor's scheduler is thread agnostic, which means that we don't know where the work is actually being carried out. So don't use thread locals when writing Reactor applications. This impacts other areas too such as Spring Security, which uses thread locals to maintain contextual security status with SecurityContextHolder. We'll visit this subject in Chapter 9, Securing Your App with Spring Boot.

The following shows our method pulled into a separate class:

    @Component 
    public class CommentHelper { 
  
      private final RestTemplate restTemplate; 
 
      CommentHelper(RestTemplate restTemplate) { 
        this.restTemplate = restTemplate; 
      } 
 
      // @HystrixCommand code shown earlier 
 
      // fallback method 
    } 

We've already seen the @HystrixCommand code as well as the fallback. The other parts we wrote include:

  • The CommentHelper class is flagged with an @Component annotation, so, it's picked up and registered as a separate Spring bean
  • This component is injected with the restTemplate we defined earlier via constructor injection

To update our HomeController to use this instead, we need to adjust its injection point:

    private final CommentHelper commentHelper; 
 
    public HomeController(ImageService imageService, 
     CommentHelper commentHelper) { 
       this.imageService = imageService; 
       this.commentHelper = commentHelper; 
    } 

The code in HomeController is almost the same, except that instead of injecting a RestTemplate, it injects commentHelper.

Finally, the call to populate comments in the index() method can be updated to use the new commentHelper:

    put("comments", commentHelper.getComments(image)); 

At this point, instead of calling restTemplate to make a remote call, we are invoking commentHelper, which is wrapped with Hystrix advice to handle failures, and, potentially, open a circuit.

Notice earlier that I said, "In the event the remote call fails, Hystrix will call defaultComments.", but didn't mention anything about opening the circuit? Perhaps that's confusing, since this whole section has been about the circuit breaker pattern. Hystrix tabulates every failure, and only opens the circuit when a certain threshold has been breached. One missed remote call isn't enough to switch to an offline state.
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.21.248.162