To be able to dig into the root cause of a problem requires a good understanding of the Erlang programming language. In this section, we will cover the basics of Erlang and make use of this knowledge in the last chapter of the book, when we discuss how to create a plugin for RabbitMQ and how to implement RabbitMQ.

To begin, you need to add the <erlang_home>in directory to your PATH and execute the following command from the command line:

erl

The command will fire up the Erlang REPL (Read-Eval-Print-Loop) shell, where you can type the Erlang commands. To connect to a particular node that is running on the local workstation, you can provide the domain name of the instance with the –sname option (sname stands for 'short names' and it is the default instance-naming format that RabbitMQ uses), as shown in the following:

erl –sname rabbit@DOMAIN

In order to use the preceding command, you need to stop the rabbit@DOMAIN node first.

You can start by evaluating the following expression using the Erlang interpreter (don't forget the dot at the end of each expression):

(4 + 6) * 2.

Not only can the arithmetic expressions be evaluated. Let's transform the preceding example using two variables, as follows:

X = 4.
Y = 6.
(X + Y) * 2.

If you reassign the X variable to 10, as follows:

X = 10.

You will get an error as shown in the following:

** exception error: no match of right hand side value 10

To reassign the variable, you need to first unbind it using the f() function:

f(X).

Note that you can unbind all variables by simply calling the following function:

f().

The preceding expression is not of much use; therefore, let's make a function out of it from the Erlang shell:

F = fun(X,Y) -> (X + Y) * 2 end.

The fun keyword can be used to define an anonymous function. In the previous case, this function is bound to the F variable. Now, you can evaluate the former expression using the following function:

F(4,6).

Functions in Erlang are typically defined in modules. A module in Erlang is defined as a file with an .erl extension, which is further compiled to an Erlang object file with a .beam extension that represents the actual byte code that is executed by the Erlang virtual machine. You can define the preceding function in a module called sample (saved in a sample.erl file. Please note that the name of the file should match the module declaration):

-module(sample).
-export([double/2]).
double(X,Y) -> (X+Y) * 2.

The –module declaration specifies the name of the module, followed by one or more -export declarations that explicitly specify which functions from the module are exported by the module and can be used by other modules. You should specify the name of the function along with its arity (number of parameters that the function accepts). Functions with the same name but different numbers of parameters are treated as separate function declarations by Erlang. In the module, there is a double function—this declaration is valid only in a module and cannot be executed from the shell—you should use the fun keyword for this, as we saw earlier.

To compile the module, you must first navigate to the directory of your module using the cd() function and then, the c() function, to compile the module to a beam file. Assuming the sample.erl file is created in the D:sources directory, you can execute the following from the Erlang REPL in order to compile the module:

cd('D:/sources').(sample).

If compilation is successful, you will see a message as follows:

{ok,sample}

This is actually a tuple that is returned from the c() function, which indicates a successful status (ok) and the name of the compiled module. A tuple, in Erlang, is a container with a fixed number of elements that can be of different types. In order to invoke the double function from the sample module, you can write the following:

sample:double(6,4).

Use the m() function or the module_info() method (which returns a list with the result) that is available for each Erlang module to check for information, such as available functions, about the module:

m(sample).
sample:module_info().

These can also be pretty useful utilities to inspect the existing modules in a system such as RabbitMQ.

Variable definitions do not specify the type of the variable, it is determined at runtime (as seen in the double function). We have the following types of data:

Erlang uses the concept of pattern matching in order to bind one or more variables to the particular values. It is used to assign variables (denoted by atoms) using more complex expressions that direct assignment. Consider the following examples:

{X,b} = {a,b}.
[10,[Y],15] = [10,[[1,2,3]],15].
{X,X} = {a,b}.
[A,2] = [10].

The first expression binds X to a, the second expressions binds Y to the [1,2,3] list, and the third and fourth expressions result in exceptions as pattern matching fails in these cases. We will briefly cover error handling later in the chapter.

Another useful concept is list comprehensions, where you can iterate over a list and return a modified list using a filter function and a generator for the elements of the new list. Consider the following example:

[X+1 || X <- [4,5,6], X rem 2 == 0].

The result is the [5,7] list, all even elements are filtered and incremented by one in the new list. We can rewrite the preceding example using a recursive function, as Erlang enforces the functional programming style along with idioms derived from languages such as Prolog; the language does not provide a looping construct. The filter_list_sample function implements the same behavior as the list comprehension using an if statement:

filter_list_sample(L) -> filter_list_sample_helper(L, []).
filter_list_sample_helper([], Res) -> Res.
filter_list_sample_helper([X|L], Res) -> 
if 
    X rem 2 == 0 -> 
    filter_list_sample_helper(L, [X+1| Res]).
    true -> 
          filter_list_sample_helper(L, Res)
end.

If you add this to the sample module that we created earlier, export the filter_list_sample function from the module, and recompile it, you can invoke the preceding function with the following:

sample:filter_list_sample([4,5,6]).

The result is returned in reverse order due to the recursion; implement a function that reverses the resulting list as an exercise. Note that if you have multiple definitions of the same function (in this case, filter_list_sample_helper), you should separate them with a semicolon. Multiple expressions in the same function are separated by a comma. You can also use the case expression instead of the if expression in the preceding example, as shown in the following:

filter_list_sample_helper([X|L], Res) -> 
case X rem 2 of
0 -> filter_list_sample_helper(L, [X+1| Res]).
    _ -> filter_list_sample_helper(L, Res)
end.

The underscore (_) indicates any match (in this case, this could be only 1).

There are many scenarios where Erlang may throw an error, and we can differentiate between the three types of runtime errors, as follows:

All the types of errors can be caught using a try … catch block. The following example demonstrates the use of the different types of exceptions in Erlang:

exception_sample(Val) -> 
    case Val of
        1 -> throw("Invalid value: 1").
        2 -> error("Invalid value: 2").
        3 -> exit("Invalid value: 3").
        _ -> "Success"
    end.
    
exception_handler(Val) ->
    try 
        exception_sample(Val)
    catch
        error: Error -> {error, Error}.
        throw: Error -> {throw, Error}.
        exit: Error -> {exit, Error}    
    end.

Export the exception_handler() function as part of the sample module and execute it with different arguments to see how it behaves:

sample:exception_handler(1).
sample:exception_handler(2).
sample:exception_handler(3).
sample:exception_handler(4).

You should receive the following output:

{throw,"Invalid value: 1"}
{error,"Invalid value: 2"}
{exit,"Invalid value: 3"}
"Success"

When an Erlang process exits as a result of an error that is not handled by the process, you will get a result that is in a format similar to the RabbitMQ node crashing as RabbitMQ nodes are started as Erlang processes.

So far, we discussed the basic constructs of the language. However, Erlang excels when it comes to distributed programming. Processes in Erlang are lightweight, they are created by the Erlang VM without actually interacting with the underlying operating system (and creating any OS-level threads or processes). Communication between processes is possible via message passing. The Erlang VM takes the responsibility of handling the process execution underneath on one or more CPUs in the system on which the Erlang VM runs. Thus, reducing context switching' you don't need to go to the kernel scheduler to switch between the currently executing threads. This, and the ability to dynamically allocate process stacks (thus saving the effort to reserve a lot of RAM), provides the possibility of creating thousands of Erlang processes at once. If any two processes need to communicate on the same machine, you can do it directly using the ! and receive expression in order to exchange messages, as demonstrated in the following example:

sample_sender(Pid, Message) -> 
    Pid ! Message.

sample_receiver() -> 
    receive
        Message -> io:format(Message, [])
    end.

start() -> 
    Preceiver = spawn(?MODULE, sample_receiver, []),
    spawn(?MODULE, sample_sender, [Preceiver, "Test message."]).

We create a sender and receiver as separate processes in the start() method using the spawn function that creates a process based on a module function, along with the parameter passed to that function upon process creation. The ?MODULE macros refer to the current module, you can think of the Erlang macros as C++ preprocessor directives. The sample_sender() function sends a message using the ! operator to the process identified by a particular pid (proportional–integral–derivative). The sample_receiver() method uses the receive expression to wait for a message and is blocked until a message is received. The message is printed on the standard output using the built-in io:format Erlang function. You need to export all the three functions from the sample module and run the demo using the following line of code from the Erlang REPL:

sample:start().

In this particular example, the processes run in the same Erlang VM. However, if the processes are started on a remote machine, then several concerns are further raised. The most important issues to solve are as follows:

The answer to the first question is the register() built-in function that allows you to map a symbolic name to a process identifier. This mapping information is stored in an Erlang register, and when a process needs to communicate with another remote process, it must know the address of the machine where the other process resides along with the symbolic name of the remote process. The rest is handled by Erlang behind the scenes.

The answer to the second question is the Erlang cookies that we mentioned in the earlier chapters when we talked about RabbitMQ clustering. Erlang cookies are stored in an .erlang.cookie file and are used by the Erlang processes as a shared secret. A node is not obliged to use the same cookie for all other remote nodes—a different cookie can be specified for communication with a remote node. This can be accomplished using the erlang:set_cookie() method that uses the remote node identifier and Erlang cookie instance as arguments. To retrieve the current cookie used by the node, you can use the erlang:get_cookie() method. In case no cookie is in use, the method will return nocookie.

Our brief primer of the Erlang language should be sufficient in order to make use of the utilities provided by the language for further troubleshooting of your RabbitMQ instances. You can retrieve the name of the current node with the following command:

node().

You can also retrieve the names and the ports of the processes that are registered by the EPMD (Erlang Port Mapper Daemon) process running on the same Erlang VM:

net_adm:names().

Assuming that we have started our three-node cluster on the same machine, we should observe the following output:

{ok,[{"rabbit",25672},
     {"instance1",25701},
     {"instance2",25702}]}

The ports that you see for each node are the ports assigned to the Erlang processes for each RabbitMQ instance (in the previous case, 20000 + the name of the RabbiqMQ instance port).

We can also use the rpc:call function in order to execute a function in a particular local/remote Erlang process (and this could be the process of a RabbitMQ instance). You can also use the different Erlang utilities, such as the rpc:call() function, to execute the commands on remote processes or retrieve the information about these processes.

The Erlang crash dump file is created in the current working directory of a Rabbit instance when it crashes. The crash dump file contains useful statistics that are collected at the time of the crash along with the information about the processes that are affected as part of the crash. The reason for the node failure is indicated by the line starting with the word slogan. For example, the following command indicates that there is a problem with starting up of a node (without providing more details as a part of the reason):

Slogan: init terminating in do_boot ()

You can use the knowledge gained from the previous section to inspect the information that is collected in the crash dump or better, use the Crashdump Viewer GUI utility to inspect the crash dump. To start the utility, invoke the following commnad from the Erlang REPL:

crashdump_viewer:start().

After the tool is started, you will be prompted to select the crash dump file. After the file is selected, the tool will divide the information from the file into proper sections and tables for easier inspection, as follows:

We will expand further on the concept of troubleshooting when we discuss the internal architecture of the message broker. If you get an error that contains: init terminating in do_boot(), then there are several things that might be the root cause of the problem (make sure that you analyze the crash dump for more information on the problem):