Options

The central issue here is that of a transaction. Actually building a system to call these APIs is trivial; however, we have an issue with ensuring that we either book all or none of the selected services.

We also have an issue with scaling – while it may be trivial to call the APIs, we should consider what would happen if we were to get two conflicting bookings: if we book a flight for person A and a hotel for person B and then try to book the flight for person B, we’ll find that we’ve already used that booking up for someone else.

Manual Process

As with previous chapters, let’s investigate how we could achieve this through a manual process to help us better understand the business domain.

That seems a very easy process; however, what would we do if at stage 2, the sales assistant found out that the hospital was too busy on that day? Well, in that case, we would try to cancel the space flight.

Should we have a similar problem with the hotel, then we would need to cancel the two bookings.

Then, if any of these are unavailable, we could get back to the others and ask them to cancel the reservation; otherwise, we could book each service.

This looks like a much better system, but let’s put ourselves in the position of, say, the hospital; they are busy and are essentially tying up one of their bookings until we get back to them; they potentially would need to turn away other bookings for that day, as a result; similarly with the space shuttle company – although the costs involved for the space shuttle company are much larger.

The process that we’re discussing here in various forms is a transaction.

Transactions

If I contact my bank and ask them to transfer this money, the preceding process is a simplified version of what happens: they take the money out of one account, and they put it in another – a two-stage process. Now, what would happen if in between 1 and 2, the system crashed? Without a transaction to wrap around these two activities, Account 1 would still be debited, but the money would simply disappear – it would never make it to Account 2.

A transaction around the two activities ensures that either all of these activities happen, or none do. The valid result states are either the transaction failed and Account 1 still has the £300, or the transaction succeeded and Account 2 now has that money, but Account 1 does not. In order to ensure the validity of this data, most database systems implement a concept of an ACID transaction.

Distributed Transactions

A distributed transaction is a method to provide the functionality that we expect from a standard transaction across multiple systems. Let’s see how that works and whether it meets our requirements.

The main principle here is, what is known as, a two-phase commit; this is exactly what it sounds like: you first announce your intent to commit, and then commit. A distributed transaction across n systems would have n + 1 participants. In our case, we have three systems, and therefore, we have four participants.

Figure 3-1 illustrates the sequence flow for a distributed transaction. The transaction coordinator essentially polls each participant in the transaction and asks if they are ready to begin the transaction. If the coordinator receives a confirmation from every participant, then it issues a commit instruction and, again, expects each participant to confirm that they have committed.

Each participant in this transaction is required to keep a persisted state of the transaction following the lock, until the data is committed, or an abort message is sent. The coordinator would need to persist the state of the transaction after the first commit is sent; this would repeatedly poll the participants until it receives an acknowledgment.

The transaction in the diagram would have a unique reference, which would be distributed during the initial message. From this point on, any communication would refer to this reference.

This seems an eminently usable system; admittedly, it’s a little chatty and likely slow, but we have a guarantee that all the bookings are made, or none are made. However, we should consider how this system deals with failure.

Let’s consider some possible scenarios.

Distributed Transaction with Timeout

We could easily adapt our distributed transaction to do this by simply changing the rules slightly. We could allow the participants in the transaction to unilaterally time out – for example, when a prepare message has been issued, but no commit received. We could then have an error state where the transaction coordinator polled all of the participants at the end to determine the status of the booking. Where one or more of the participants have not secured the booking, we simply attempt to book; if we can’t, then cancel the others.

In Figure 3-2, any of the services can unilaterally decide that they wish to time out during the timeout window. During this time, should they decide to time out, they would issue a message to the controller, who would then bear the responsibility of clean up.

In this case, clean-up is, effectively, attempting to cancel the remaining transactions: if a confirm hasn’t been issued, then by abandoning the transaction; or, where it has, then by instigating a cancellation.

Book and Cancel

Our next option is a little less from a computer science background and more from a business background. We could simply book the resources and, where a single element of the booking is not available, attempt to cancel the other parts. The obvious risk here is that we would not be able to cancel a particular thing; for example, the hotel may refuse the cancellation.

Hold a Booking

This possibility appears at first glance to be the most sensible. Here, we are effectively saying to the service provider that we wish them to reserve a place for us, but not to actually make the booking. We have discussed this kind of scenario previously in this chapter. The issue here is that you are asking a provider to potentially refuse a firm booking in exchange for your potential one. In fact, this could be said to fit into our transaction diagram shown before – we need simply to accept the prepare as a hold instruction and confirm as a firm booking.

Advanced Purchase

This appears to be the approach taken by the bigger players in the travel industry today. Again, it is also not a technical solution to the problem, but a business one. What you do is project how many trips you are likely to sell and then buy that many from the supplier. In our case, we may decide to buy 20 nights in the hotel and book and pre-pay for 20 hospital appointments on the same day. However, in our case, it would not be practical for a simple reason: the space shuttle flight would be prohibitively expensive.

Consequently, this is unlikely to be the best decision (given that only one flight not being re-sold could potentially bankrupt the company).

Business Decision

In fact, as with many such cases, the right answer here is not an architectural one, but a business one. The person making such a decision is going to need to balance risk against potential profit; such decisions are beyond the scope of this book, safe to say that it is very unlikely that you will face any architectural problem that doesn’t contain an element of a business decision.

Having spoken to a representative from the business, they decide to adopt the Book and Cancel option.

Stateful Service

One possibility here is to start to make the bookings and to persist the state of the booking. That is, each time a part of the transaction executes, a flag would be updated in some form of persistence layer. If the service were to crash then, when it came back up, it would check the persisted state and resume where it left off – as described in the earlier section on distributed transactions.

This service itself could then execute based on a message broker, but the responsibility for persistence is within the service. This has the advantage of meaning that the service could be taken and deployed anywhere (as it’s self-contained). However, it does mean that each individual transaction must be managed by a single instance of the coordinator.

Distributed Service

This option would use the message broker itself as a persistence layer. The way that this works is that the service behaves very much as before but simply writes a message back to a queue, and then another instance of the same service picks that up. The advantage here is that the service is more scalable, as a new instance is used for each section of the task. This means that we can increase the number of workers and the new workers will pick up the next part of the transaction.

Target Architecture Diagram

We’ve now covered the principles of a distributed transaction and discussed how we might implement this.

Figure 3-4 shows the target architecture. Essentially, we’re leveraging the message broker in order to orchestrate the transaction coordinator. Each time a transaction is started, then a message is added to the queue to communicate with the hospital API; when we have a successful response, then we’ll communicate with the hotel API and so forth. Each time a step in the transaction is completed, we add a message to the broker; that way, should the transaction coordinator itself fail, we would simply come back up and read the relevant message queue.

Project Structure

Our sample solution will consist of three parts: third-party APIs, a client, and a coordinator.

Figure 3-5 illustrates the project structure for our sample. Each third-party service is represented by an API project. In our example, we’ve made them all broadly REST APIs; however, that is by no means a prerequisite. In fact, since these represent APIs that we have little control over, the code to interact with them should be segregated and abstracted where feasible (we haven’t really done that here, as that is not the purpose of this example).

Before we go about creating the rest of the project, we will need to set up the Azure Service Bus. I’ve covered setting this up via the Azure Portal (https://portal.azure.com); in order to follow along, you’ll need to have an Azure account.

Service Bus Configuration

Although we are using Azure in this example, you could easily substitute any other cloud provider message broker or any message broker at all – for example, RabbitMQ would work fine for this situation.

As we’ve discussed in previous chapters, the architectural decision to make here is whether to use a cloud provider or to try to manage the scale yourself. This particular example – low traffic now, but an expectation to ramp up rapidly – lends itself very well to the cloud model.

The first step in Azure Service Bus is to create a namespace.

In Figure 3-6, we are creating a namespace for our Service Bus. This is effectively a grouping mechanism for all the Service Bus resources.

Within the namespace, we can create Queues and Topics.

For our example, we will create a single queue, from which we will orchestrate everything. Figure 3-7 illustrates how to set up the queue; however, there are other ways to do this. For example, we could generate a queue for each section of the transaction.

The next step is to generate an access policy.

In order to connect to the Service Bus, we need to configure an access policy (shown in Figure 3-8). This allows us to dictate what permissions are given to processes that connect to the Service Bus. In our case, we have an access policy on our queue and an access policy for the namespace. You should always attempt to allocate the least privilege to any resource; in our case, that’s the ability to read and write to the queue, but not to affect the queue (i.e., delete or change).

Now that we have successfully configured our service bus, we can move onto the main part of the project; this is our coordinator.

Coordinator

The coordinator here will simply listen to the Service Bus queue that we’ve created (BookingQueue) and will process each message by its type. The coordinator holds absolutely no state, which means that we can run several of these processes should we encounter a rush in orders.

In Listing 3-1, what we are doing is leveraging the C# LinkedList in order to chain together these endpoints.

In Listing 3-2, we simply detect the type of request; there are four types: the initial booking request initiated by the client, and then the hospital booking request, hotel booking request, and space flight booking request – which are all initiated by the coordinator itself.

You’ll also see from this code that we have a concept of a function; this allows us to traverse through the stack of bookings but, where there has been an error, back again to cancel each in turn. Listing 3-2 shows the top level of this; that is, either the request has just started or it has tried and failed to make the booking.

From Listing 3-3, we can see that there are two basic logical flows within the method: either we are trying to book or we are trying to cancel . Within the logic to book, should the call fail, we initiate a cancel flow by simply sending a message back to the queue with the cancel function and to the previous entity in the linked list. This will then traverse back up the list until the top.

The cancel branch of the code attempts to cancel; and if it can’t, it logs the error and continues on through the list.

The method that books the space flight, shown in Listing 3-4, does not handle the cancel branch. The reason is that this is the top of the list, so it can never be called to cancel. We also established that cancelling this would be prohibitively expensive, and so it is only ever called once we have successfully booked the rest of the trip.