Chapter 10. Internals

To get a better understanding of how a system works, on various occasions, developers need to dig under the hood in the implementation of that system or at least get a basic overview of its high-level architecture and its most critical components. In this chapter, we will discuss how RabbitMQ is designed and implemented and how to write plugins for RabbitMQ.

The topics that will be covered in the chapter are as follows:

  • High-level architecture for RabbitMQ
  • Overview of RabbitMQ components
  • Developing plugins for RabbitMQ

High level architecture of RabbitMQ

We already discussed a lot of details about how the message broker works. We discussed that RabbitMQ instances are Erlang applications that communicate with each other by means of Erlang message passing with the help of a shared Erlang cookie that is used to allow communication between endpoints. As every system that provides a server and one or more clients for different platforms, we could easily guess that before an AMQP message is sent to the broker, an AMQP client opens a TCP socket using the utilities that are provided by the particular programming language in which it writes the AMQP data. The most interesting part is what happens when the message arrives at the message broker. Once the message broker receives the AMQP message, it needs to parse and process it, accordingly. If we take a closer look at what the rabbitmq-server script executes, we will notice that several actions take place, as follows:

  • The start method from the rabbit_prelaunch module is executed and it performs the basic validation (such as, whether a node with the same name exists and whether the node distribution port is valid) before actually starting the server.

If the checks from the previous step are successful, then the server starts from the rabbit module that implements the application behavior (meaning that the module has certain callback methods that must be implemented by the module).

Before we are to understand how RabbitMQ works, we need to clarify a few important features provided by Erlang and their uses in the implementation of the message broker. A behaviour, in terms of the Erlang programming language, specifies that a module must implement certain methods that give it a certain meaning (behavior). We can compare this with the way in which inheritance works—we can have an abstract class that extracts certain logic and can be extended by different implementations, which on the other hand, can be further extended. In that sense, an Erlang module can either define that it has a certain behavior (and it needs to implement a particular set of callback methods that are defined by that behavior) or the module itself is a behavior that defines a set of methods that must be implemented by other modules that use the behavior. In the following example, we have a sample module that uses the built-in application behavior and defines the sample behavior with two functions: start_sender and start_receiver, as follows:

-module(sample).
-export([behaviour_info/1]).
-behaviour(application).
start(normal, []) -> true.
behaviour_info(callbacks) ->
    [{start_sender,2},
     {start_receiver, 0}
];

The sample module uses the application behavior and needs to provide the implementation of the start(normal, []) method that is executed before the application (this module) is started successfully. On the other hand, the module creates a behavior with the same name as that of the module using the behaviour_info method that specifies the callback functions along with their arity (number of arguments) that must be implemented by the users of the sample behavior. This seemingly simple mechanism lays the basis for creating more complex interactions among the components of an Erlang application. Two of these mechanisms are built in Erlang and used by RabbitMQ, as shown in the following:

  • The supervisor behavior allows the creation of a process tree. The main purpose of this behavior is to allow a parent process to monitor the child processes for failure and restart them, based on a predefined policy in that parent (supervisor) process. This allows a fault-tolerant handling of the failures in the application, which is necessary in the case of RabbitMQ in order to ensure a decent degree of reliability that prevents the broker from failing upon process failure.
  • The gen_event behavior allows the exchange of messages between processes.

The supervisor behavior is essential for Erlang and for the RabbitMQ message broker, in particular. A good understanding of how and why RabbitMQ relies on a supervision tree of processes is necessary in order to understand how the message broker works at the runtime. Consider the following diagram that provides an overview of a sample process tree in terms of Erlang and the supervisor behavior:

High level architecture of RabbitMQ

We have a root process that supervises other processes; if any of them fail, the supervisor is responsible to restart it. The leaves of the tree are the actual processes that are running in the application.

The following diagram provides a high-level overview of the RabbitMQ components and their initialization during the server startup:

High level architecture of RabbitMQ

The rabbit module uses the rabbit.hrl Erlang header file that provides the definitions of all types (such as, queue, exchange, binding, vhost, and so on) that are used in the server and for this reason, the header is included in most of the Erlang sources of the message broker. The start(normal, []) method of the rabbit module triggers the start up of the server. First, the root process supervisor that is provided by the rabbit_sup module is started by invoking the rabbit_sup:start_link() method. Then, a number of boot steps are executed (we will refer to this process as the boot component of the message broker). Many of the boot steps start a child process, which is added to the supervisor tree that has a certain role. The following diagram describes the process tree, one level under the rabbit_sup supervisor:

High level architecture of RabbitMQ

In the next section, we will briefly discuss most of the processes that are mentioned in the preceding diagram, along with the structure of the process subtrees that are provided by the most essential child supervisors, such as rabbit_tcp_client_sup, rabbit_direct_client_sup, and rabbit_amqqueue_sup_sup, and their corresponding Erlang modules. Apart from when networking is started as a part of the boot process on or more tcp_listener_sup supervisor processes are started for each TCP/SSL interface configured for the message broker.

After the boot steps are finished, the RabbitMQ plugins are loaded. At this point, the message broker is ready to accept the connections. Logging in the RabbitMQ components is done by means of the utilities provided by the rabbit_log module.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.225.57.126