In many scenarios, when a cloud service is down (or not responding), in order to achieve Resiliency you can prevent an application from continuously re-sending a request to the cloud service. This architectural behavior is achieved via the Circuit Breaker pattern. The state diagram of this pattern is represented in the following figure:
When application A sends requests to cloud service B, we're in the Closed state. In this state, if some error occurs the Circuit Breaker pattern counts these errors and when the predefined error limit is reached, the state becomes Open.
When we're in the Open state, requests from application A to cloud service B are stopped (the pattern could throw an exception or log the error).
After a certain amount of time (defined according to the business scenario), the state will switch to HalfOpen. In the HalfOpen state, the application can again send a request to cloud service B. If the request has a success, the state will become Closed and the communication between A and B will become healthy, otherwise, the state of the circuit is switched back to Open.
As you can see, this pattern prevents a situation where application A continuously sends requests that fail to cloud service B (down), resulting in an unstable system with a high load traffic (or CPU).
An important aspect to remember—the purpose of the Circuit Breaker pattern is different to the Retry pattern. The Retry pattern enables an application to retry an operation in the expectation that it'll succeed. The Circuit Breaker pattern prevents an application from performing an operation that is likely to fail.
To see an example of how to implement the Circuit Breaker pattern, consider an application that sends orders to a remote service (here it is a database) and reads the order status. The following lines of code show a C# console application (available on GitHub) that uses the Circuit Breaker pattern.
The order is represented by the Order object (C# class):
public class Order
{
public int ID { get; set; }
public decimal Value { get; set; }
public override string ToString()
{
return $"{{ Order ID: {ID}, Order Value: '{Value}'}}";
}
}
In the Main function, the console application instantiates the CircuitBreakerRepository class (pattern implementation) that connects to the remote service via the ServiceConnectionFactory class and randomly sends read or write requests to the remote service by calling the ReadOrWrite function:
class Program
{
static void Main(string[] args)
{
int requestToSend = 100;
// Create a circuit breaker repository
IOrderRepository repository = new
CircuitBreakerRepository(ServiceConnectionFactory.Connection);
for (int i = 0 ; i < requestToSend; i++)
{
try
{
ReadOrWrite(repository, i);
}
catch (Exception e)
{
Console.WriteLine($"{ex.GetType().FullName}:
{ex.Message}");
}
Thread.Sleep(1000);
}
}
static void ReadOrWrite(IOrderRepository repository, int i)
{
var random = new Random();
if (random.Next(50) > 25)
{
// make a write
var order = new Order
{
ID = i,
Value = 100 + i
};
repository.Write(order);
Console.WriteLine($"Write Request: {order}");
Console.WriteLine("");
}
else
{
// make a read
Console.WriteLine($"Read Request: {string.Join(", ",
repository.Read())}");
Console.WriteLine("");
}
}
}
The class CircuitBreakerRepository implements the interface IOrderRepository (that contains the Read and Write methods for requests):
public interface IOrderRepository
{
List<Order> Read();
void Write(Order order);
}
The pattern is as follows:
public class CircuitBreakerRepository : IOrderRepository
{
private CircuitBreakerState _state;
private OrderRepository _repository;
public CircuitBreakerRepository(MongoClient client)
{
_state = new CircuitBreakerClosed(this);
_repository = new OrderRepository(client);
}
public List<Order> Read()
{
return _state.HandleRead();
}
public void Write(Order order)
{
_state.HandleWrite(order);
}
private abstract class CircuitBreakerState
{
protected CircuitBreakerRepository _owner;
public CircuitBreakerState(CircuitBreakerRepository owner)
{
_owner = owner;
}
public abstract List<Order> HandleRead();
public abstract void HandleWrite(Order order);
}
private class CircuitBreakerClosed : CircuitBreakerState
{
private int _errorCount = 0;
public CircuitBreakerClosed(CircuitBreakerRepository owner)
:base(owner){}
public override List<Order> HandleRead()
{
try
{
return _owner._repository.Read();
}
catch (Exception e)
{
_trackErrors(e);
throw e;
}
}
public override void HandleWrite(Order order)
{
try
{
_owner._repository.Write(order);
}
catch (Exception e)
{
_trackErrors(e);
throw e;
}
}
private void _trackErrors(Exception e)
{
_errorCount += 1;
if (_errorCount > Config.CircuitClosedErrorLimit)
//Limit of error requests to accept
{
_owner._state = new CircuitBreakerOpen(_owner);
}
}
}
private class CircuitBreakerOpen : CircuitBreakerState
{
public CircuitBreakerOpen(CircuitBreakerRepository owner)
:base(owner)
{
new Timer( _ =>
{
owner._state = new CircuitBreakerHalfOpen(owner);
}, null, Config.CircuitOpenTimeout, Timeout.Infinite);
}
public override List<Order> HandleRead()
{
throw new CircuitOpenException();
}
public override void HandleWrite(Order order)
{
throw new CircuitOpenException();
}
}
private class CircuitBreakerHalfOpen : CircuitBreakerState
{
private static readonly string Message = "Call failed when
circuit half open";
public CircuitBreakerHalfOpen(CircuitBreakerRepository
owner)
:base(owner){}
public override List<Order> HandleRead()
{
try
{
var result = _owner._repository.Read();
_owner._state = new CircuitBreakerClosed(_owner);
return result;
}
catch (Exception e)
{
_owner._state = new CircuitBreakerOpen(_owner);
throw new CircuitOpenException(Message, e);
}
}
public override void HandleWrite(Order order)
{
try
{
_owner._repository.Write(order);
_owner._state = new CircuitBreakerClosed(_owner);
}
catch (Exception e)
{
_owner._state = new CircuitBreakerOpen(_owner);
throw new CircuitOpenException(Message, e);
}
}
}
}
}
public static class Config
{
public static int CircuitOpenTimeout => 5000;
public static int CircuitClosedErrorLimit = 10;
}
In the Main function, the application sends n requests to the remote service (read or write requests are randomly selected). When the remote service is down, the application receives an exception from the remote host until the accepted error limit is reached (check the _trackErrors function).
When this limit is reached (in the preceding example there are 10 error requests), the machine state goes into the opened state. All requests to the remote service are now stopped and the retry will succeed after the CircuitOpenTimeout elapsed time.