No matter how hard I try I have yet to write any significant block of code that does not contain any errors. Nor have I been very good at predicting the wide range of crazy things users do with my applications. Why would anybody click that link 73 times in a row? I'll never know.
Dealing with failures in a messaging scenario is very easy. The core of the failure strategy is to embrace errors. We have exceptions for a reason and to spend all of our time trying to predict and catch exceptions is counter-productive. You'll invariably spend time building in catches for errors that never happen and miss errors that happen frequently.
In an asynchronous system, errors need not be handled as soon as they occur. Instead, the message that caused an error can be put aside to be examined by an actual human later. The message is stored in a dead letter, or error, queue. From there the message can easily be reprocessed after it has been corrected or the handler has been corrected. Ideally the message handler is changed to deal with messages exhibiting whatever property caused the errors. This prevents future errors and is preferable to fixing whatever generates the message as there is no guarantee that other messages with the same problem aren't lurking somewhere else in the system. The workflow of a message through the queue and error queue can be seen here:
As more and more errors are caught and fixed, the quality of the message handlers increases. Having an error queue of messages ensures that nothing important, such as a BuySimonsBook message is missed. This means that getting to a correct system becomes a marathon instead of a sprint. There is no need to rush a fix into production before it is properly tested. Progress towards a correct system is constant and reliable.
Using a dead letter queue also improves the catching of intermittent errors. These are errors that result from an external resource being unavailable or incorrect. Imagine a handler that calls out to an external web service. In a traditional system, a failure in the web service guarantees failure in the message handler. However, with a message based system, the command can be moved back to the end of the input queue and tried again whenever it reaches the front of the queue. On the envelope we write down the number of times the message has been dequeued (processed). Once this dequeue count reaches a limit, like five, only then is the message moved into the true error queue.
This approach improves the overall quality of the system by smoothing over the small failures and stopping them from becoming large failures. In effect, the queues provide failure bulkheads to prevent small errors from overflowing and becoming large errors that might have an impact on the system as a whole.
When developers are working with a set of messages that produce an error, the ability to reprocess messages is also useful. Developers can take a snapshot of the dead letter queue and reprocess it in debug mode again and again until they have correctly processed the messages. A snapshot of a message can also make up a part of the testing for a message handler.
Even without there being an error, the messages sent to a service on a daily basis are representative of the normal workflows of users. These messages can be mirrored to an audit queue as they enter into the system. The data from the audit queue can be used for testing. If a new feature is introduced, then a normal day's workload can be played back to ensure that there has been no degradation in either correct behavior or performance.
Of course if the audit queue contains a list of every message, then it becomes trivial to understand how the application arrived at its current state. Frequently people implement history by plugging in a lot of custom code or by using triggers and audit tables. Neither of these approaches do as good of a job as messaging at understanding not only which data has changed, but why it has changed. Consider again the address change scenario, without messaging we will likely never know why an address for a user is different from the previous day.
Maintaining a good history of changes to system data is storage intensive but that cost is easily paid by allowing auditors to see how and why each change was made. Well-constructed messages also allow for the history to contain the intent of the user making the change.
While it is possible to implement this sort of messaging system, in a single process it is difficult. Ensuring that messages are properly saved in the event of errors is difficult, as the entire process that deals with messages may crash, taking the internal message bus with it. Realistically if the replaying of messages sounds like something worth investigating then external message busses are the solution.
I mentioned earlier that messages should be considered immutable. This is not to say that messages cannot be rebroadcast with some properties changed or even broadcast as a new type of message. In fact, many message handlers may consume an event and then publish a new event after having performed some task.
As an example, you might consider the workflow for adding a new user to a system:
In this case, the CreateUser command triggers a UserCreated event. That event is consumed by a number of different services. One of these services passes on user information to a select number of affiliates. As this service runs, it publishes its own set of events, one for each affiliate that receives the new user's details. These events may, in turn, be consumed by other services which could trigger their own events. In this way changes can ripple through the entire application. However, no service knows more than what starts it and what events it publishes. This system has very low coupling. Plugging in new functionality is trivial and even removing functionality is easy: certainly easier than a monolithic system.
Systems constructed using messaging and autonomous components are frequently referred to as using Service Oriented Architecture (SOA) or Microservices. There remains a great deal of debate as to the differences, if indeed there are any, between SOA and Microservices.
The altering and rebroadcasting of messages can be thought of as being a pipe or a filter. A service can proxy messages through to other consumers just as a pipe would do or can selectively republish messages as would be done by a filter.
As systems evolve, the information contained in a message may also change. In our user creation example, we might have originally been asking for a name and e-mail address. However, the marketing department would like to be able to send e-mails addressed to Mr. Jones or Mrs. Jones so we need to also collect the user's title. This is where message versioning comes in handy.
We can now create a new message that extends the previous message. The message can contain additional fields and might be named using the version number or a date. Thus a message like CreateUser might become CreateUserV1 or CreateUser20140101. Earlier I mentioned polymorphic messages. This is one approach to versioning messages. The new message extends the old so all the old message handlers still fire. However, we also talked about how there are no real polymorphic capabilities in JavaScript.
Another option is to use upgrading message handlers. These handlers will take in a version of the new message and modify it to be the old version. Obviously the newer messages need to have at least as much data in them as the old version or have data that permits converting one message type to another.
Consider a v1 message that looked like the following:
class CreateUserv1Message implements IMessage{
__messageName: string
UserName: string;
FirstName: string;
LastName: string;
EMail: string;
}Consider a v2 message that extended it adding a user title:
class CreateUserv2Message extends CreateUserv1Message implements IMessage{
UserTitle: string;
}Then we would be able to write a very simple upgrader or downgrader that looks like the following:
var CreateUserv2tov1Downgrader = (function () {
function CreateUserv2tov1Downgrader (bus) {
this.bus = bus;
}
CreateUserv2tov1Downgrader.prototype.processMessage = function (message) {
message.__messageName = "CreateUserv1Message";
delete message.UserTitle;
this.bus.publish(message);
};
return CreateUserv2tov1Downgrader;
})();You can see that we simply modify the message and rebroadcast it.