The restart specification

The restart tuple, of the format:

{RestartType, MaxRestart, MaxTime}

specifies what happens to the other children in its supervision tree if a child terminates abnormally. By “child” we mean either a worker or another supervisor. Starting with Erlang 18.0, you can also use a map. The map defining the restart specification has the following type definition:

#{strategy  => strategy(), 
  intensity => non_neg_integer(), 
  period    => pos_integer()}

There are four different restart types: one_for_one, one_for_all, rest_for_one, and simple_one_for_one. Under the one_for_one strategy (Figure 8-7), only the crashed process is restarted. This strategy is ideal if the workers don’t depend on each other and the termination of one will not affect the others. Imagine a supervisor monitoring the worker processes that control the instant messaging sessions of hundreds of thousands of users. If any of these processes crashes, it will affect only the user whose session is controlled by the crashed process. All other workers should continue running independently of each other.

Figure 8-7. One for one

Under the one_for_all strategy shown in Figure 8-8, if a process terminates, all processes are terminated and restarted. This strategy is used if all or most of the processes depend on each other. Picture a very complex FSM handling a protocol stack. To simplify the design, the machine has been split into separate FSMs that communicate with each other asynchronously, and these workers all depend on each other. If one terminates, the others would have to be terminated as well. For these cases, pick the one_for_all strategy.

Figure 8-8. One for all

Under the rest_for_one strategy (Figure 8-9), all processes started after the crashed process are terminated and restarted. Use this strategy if you start the processes in order of dependency. In our frequency_sup example, we first start the overload event manager, followed by the frequency allocator. The frequency allocator sends requests to the overload event manager whenever it runs out of frequencies. So if the overload manager has crashed and is being restarted, there is a risk the frequency server might send it requests that get lost. Under such circumstances, we want to first terminate the frequency allocator, and then restart the overload manager and the frequency allocator in that order.

Figure 8-9. Rest for one

If losing the alarms sent to the frequency allocator did not matter (as the requests were asynchronous), we could have used the one_for_one strategy. Or we could have taken it a step further by making the raising and clearing of the alarms to the overload manager synchronous. In this case, if the overload manager had crashed and was being restarted, the frequency allocator would have also been terminated only when trying to make a synchronous call to it. Had the frequency allocator not run out of frequencies, thus not needing to raise or clear alarms, it could have continued functioning. As we have seen, there is no “one size fits all” solution; it all depends on the requirements you have and behavior you want to give your system.

There is one last restart strategy to cover: simple_one_for_one. It is used for children of the same type added dynamically at runtime, not at startup. An example of when we would use this strategy is in a supervisor handling the processes controlling mobile phones that are added to and removed from the supervision tree dynamically. We cover dynamic children and the simple_one_for_one restart strategy later in this chapter.

The last two elements in the restart tuple are MaxRestart and MaxTime. MaxRestart specifies the maximum number of restarts all child processes are allowed to do in MaxTime seconds. If the maximum number of restarts is reached in this number of seconds, the supervisor itself terminates with reason shutdown, escalating the termination to its higher-level supervisor. What is in effect happening is that we are giving the supervisor MaxRestart chances to solve the problem. If crashes still occur in MaxTime seconds, it means that a restart is not solving the problem, so the supervisor escalates the issue to its supervisor, which will hopefully be able to solve it.

Look at the supervision tree in Figure 8-2. What if the phone FSMs under the phone supervisor are crashing because of corrupt data in the frequency handler? No matter how many times we restart them, they will continue to crash, because the problem lies in the frequency allocator, a worker supervised outside of our supervision subtree. We solve cyclic restarts of this nature through escalation. If we allow the phone supervisor to terminate, the top supervisor will receive the exit signal and restart the frequency server and event manager workers before restarting the phone supervisor. Hopefully, the restart can clear the corrupt data, allowing the phone FSMs to function as expected.

The key to using supervisors is to ensure you have properly designed your start order and the restart strategy associated with it. Though you will never be able to fully predict what will cause your processes to terminate abnormally, you can nevertheless try to design your restart strategy to recreate the process state from known-good sources. Instead of storing the state persistently and assuming it is uncorrupted such that it reading it after a crash will correctly restore it, retrieve the various elements that created your state from their original sources.

For example, if the corrupted data causing your worker to crash was the result of a transient transmission error, rereading it might solve the problem. The supervisor would restart the worker, which in turn would successfully reread the transmission and continue operating. And since the system would have logged the crash, the developer could look into its cause, modify the code to handle it appropriately, and prepare and deploy a new release to ensure that future similar transmission errors do not negatively impact the system.

In other cases, recovery might not be as straightforward. More difficult transmission errors might cause repeated worker crashes, in turn causing the supervisor to restart the worker. But since the restarts do not correct the problem, the client supervisor eventually reaches the restart threshold and terminates itself. This in turn affects the top-level supervisor, which eventually reaches its own restart threshold, and by terminating itself it takes the entire virtual machine down with it. When the virtual machine terminates, heart, a monitoring mechanism we cover in Chapter 11, detects that the node is down and invokes a shell script. The recovery actions in this script could be as simple as restarting the Erlang VM or as drastic as rebooting the computer. Rebooting might reset the link to the hardware that is suffering from transmission problems and solve the problem. If it doesn’t, after a few reboot attempts the script might decide not to try again and instead alert an operator, requesting manual intervention.

Hopefully, a load balancer will already have kicked in to forward requests to redundant hardware, providing seamless service to end users. If not, this is when you receive a call in the middle of the night from a panicking first-line support engineer informing you there is an outage. In either case, the crash is logged, hopefully with enough data to allow you to investigate and solve the bug: namely, ensuring that data is checked before being introduced into your system so that data corrupted by transmission errors is not allowed in the first place. We look at distributed architectures and fault tolerance in Chapter 13. For now, let’s stay focused on recovery of a single node. Next in line are child specifications.

The child specification

The child specification contains all of the information the supervisor needs to start, stop, and delete its child processes. The specification is a tuple of the format:

{Name,StartFunction,RestartType,ShutdownTime,ProcessType,Modules}

or, in Erlang 18.0 or newer, a map with the following type specification:

child_spec() = #{id => child_id(),       % mandatory
                 start => mfargs(),      % mandatory
                 restart => restart(),   % optional
                 shutdown => shutdown(), % optional
                 type => worker(),       % optional
                 modules => modules()}   % optional

The elements of the tuple are:

Name

Any valid Erlang term, used to identify the child. It has to be unique within a supervisor, but can be reused across supervisors within the same node.

StartFunction

A tuple of the format {Module, Function, Args}, which, directly or indirectly, calls one of the behavior start_link functions. Supervisors can start only OTP-compliant behaviors, and it is their responsibility to ensure that the behaviors can be linked to the supervisor process. You cannot link regular Erlang processes to a supervision tree, because they do not handle the system calls.

RestartType

Tells the supervisor how to react to a child’s termination. Setting it to permanent ensures that the child is always restarted, irrespective of whether its termination is normal or abnormal. Setting it to transient restarts a child only after abnormal termination. If you never want to restart a child after termination, set RestartType to temporary.

ShutdownTime

ShutdownTime is a positive integer denoting a time in milliseconds, or the atom infinity. It is the maximum time allowed to pass between the supervisor issuing the EXIT signal and the terminate callback function returning. If the child is overloaded and it takes longer, the supervisor steps in and unconditionally terminates the child process. Note that terminate will be called only if the child process is trapping exits. If you are feeling grumpy or do not need the behavior to clean up after itself, you can instead specify brutal_kill, allowing the supervisor to unconditionally terminate the child using exit(ChildPid, kill).

Choose your shutdown time with care, and never set it to infinity for a worker, because it might cause the worker to hang in its terminate callback function. Imagine that your worker is trying to communicate with a defunct piece of hardware, the very reason for your system needing to be rebooted. You will never get a response because that part of the system is down, and this will stop the system from restarting. If you have to, use an arbitrarily large number, which will eventually allow the supervisor to terminate the worker. For children that are supervisors themselves, on the other hand, it is common but not mandatory to select infinity, giving them the time they need to shut down their potentially large subtree.

ProcessType and Modules

These are used during a software upgrade to control how and which processes are being suspended during the upgrade. ProcessType is the atom worker or supervisor, while Modules is the list of modules implementing the behavior. In the case of the frequency server, we would include frequency, while for our coffee machine we would specify coffee_fsm. If your behavior includes library modules specific to the behavior, include them if you are concerned that an upgrade of the behavior module will be incompatible with one of library modules. For example, if you changed the API in the hw interface module as well as the coffee_fsm behavior calling it, you would have to atomically upgrade both modules at the same time to ensure that coffee_fsm does not call the old version of hw. By listing both of these modules, you would be covered. But if you did not list hw, as in our example, you would have to ensure that any upgrade would be backward-compatible and handle both the old and the new APIs. We cover software upgrades in more detail in Chapter 12.

What if you don’t know your Modules at compile time? Think of the event manager, which is started without any event handlers. When you do not know what will be running when you do a software upgrade, set Modules to the atom dynamic. When using dynamic modules, the supervisor will send a request to the behavior module and retrieve the module names when it needs them.

Before looking at the interface and callback details, let’s test our example with what we’ve learned. Looking at their child specifications, we see that both the overload event manager and the frequency server are permanent worker processes given 2 seconds to execute in their terminate functions. We start the supervisor and its children, and see immediately that they have started correctly. In shell command 4, we stop the frequency server, but because it has its RestartType set to permanent, the supervisor will immediately restart it. We verify the restart in shell command 5 by retrieving the pid for the new frequency server process and noting that it differs from the pid of the original server returned from shell command 2. In shell command 6 we explicitly kill the frequency server, and shell command 7 shows that, once again, it restarted:

1> frequency_sup:start_link().
{ok,<0.35.0>}
2> whereis(frequency).
<0.38.0>
3> whereis(freq_overload).
<0.36.0>
4> frequency:stop().
ok
5> whereis(frequency).
<0.42.0>
6> exit(whereis(frequency), kill).
true
7> whereis(frequency).
<0.45.0>
8> supervisor:which_children(frequency_sup).
[{frequency,<0.45.0>,worker,[frequency]},
 {freq_overload,<0.36.0>,worker,[freq_overload]}]
9> supervisor:count_children(frequency_sup).
[{specs,2},{active,2},{supervisors,0},{workers,2}]

In shell command 8, which_children/1 returns a tuple list containing the ChildId its pid, worker or supervisor to denote its role, and the Modules list. Be careful when using this function if your supervisor has lots of children, because it will consume lots of memory. If you are calling the function from the shell, remember that the result will be stored in the shell history and not be garbage collected until the history is cleared.

supervisor:which_children(SupRef) -> [{Id, Child, Type, Modules}]
supervisor:count_children(SupRef) -> [{specs, SpecCount},
                                      {active, ActiveProcessCount},
                                      {supervisors, ChildSupervisorCount},
                                      {workers, ChildWorkerCount}]
supervisor:check_childspecs(ChildSpecs) -> ok
                                           {error, Reason}

The function count_children/1 returns a property list covering the supervisor’s child specifications and managed processes. The elements are:

specs

The total number of children, both those that are active and those that are not

active

The number of actively running children

workers and supervisors

The number of children of the respective type

And finally, check_childspecs/1 is useful when developing and troubleshooting child specifications and startup issues. It validates a list of child specifications, returning an error if any are incorrect or the atom ok if it finds no problems.

Supervisor specifications are easy to write. And as a result, they are also easy to get wrong. Too often, programmers pick configuration values that do not reflect the reality and conditions under which the application is running, or copy specifications from other applications, or, even worse, use the default values from skeleton templates that different editors provide. You must take care to get your supervision structure right when designing your start and restart strategy, and must build in fault tolerance and redundancy. The tasks include starting your processes in order of dependency, and setting restart thresholds that will propagate problems to supervisors higher up in the hierarchy and allow them to take control if supervisors lower down in the supervision tree cannot solve the issue.

Simple one for one

The simple_one_for_one restart strategy is used when there is only one child specification shared by all the processes under a single supervisor. Our phone supervisor example fits this description, so let’s rewrite it using this strategy. In doing so, we have added the detach_phone/1 function, which we explain later. Note how we have moved the hlr:new() call to the supervisor init function:

-module(simple_phone_sup).
-behavior(supervisor).

-export([start_link/0, attach_phone/1, detach_phone/1]).
-export([init/1]).

start_link() ->
    supervisor:start_link({local, ?MODULE}, ?MODULE, []).

init([]) ->
    hlr:new(),
    {ok, {{simple_one_for_one, 10, 3600},
          [{ms, {phone_fsm, start_link, []},
           transient, 2000, worker, [phone_fsm]}]}}.

attach_phone(Ms) ->
    case hlr:lookup_id(Ms) of
        {ok, _Pid}    ->
            {error, attached};
        _NotAttached ->
            supervisor:start_child(?MODULE, [Ms])
    end.

detach_phone(Ms) ->
    case hlr:lookup_id(Ms) of
        {ok, Pid}    ->
            supervisor:terminate_child(?MODULE, Pid);
        _NotAttached ->
            {error, detached}
    end.

If you have looked at the code in detail, you might have spotted a few differences between the simple_one_for_one restart strategy and the one we used earlier for dynamic children. The first change is the arguments passed when starting the children. In the supervisor init/1 callback function, the {phone_fsm, start_link, ChildSpecArgs} in the child specification specifies no arguments (ChildSpecArgs is []), whereas the function phone_fsm:start_link(Args) in the earlier example takes one, Ms. As the children are dynamic, they are started via the function supervisor:start_child(SupRef, StartArgs). This function takes its second parameter, which it expects to be a list of terms, appends that list to the list of arguments in the child specification, and calls apply(Module, Function, ChildSpecArgs ++ StartArgs).

For the phone FSM, ChildSpecArgs in the child specification is empty, so the result of passing [Ms] as the second argument (StartArgs) to supervisor:start_child/2 is that it calls phone_fsm:start_link(Ms). It is also worth noting that we are initializing the ETS tables using the hlr:new() call in the init/1 callback, making the supervisor the owner of the tables.

The second difference is that in the simple_one_for_one strategy you do not use the child’s name to reference it, you use its pid. If you study the detach_phone/1 function, you will notice this. You will also notice in the code that we are terminating the child without deleting it from the child specification list. We don’t have to, as it gets deleted automatically when terminated. Thus, the functions supervisor:restart_child/1 and supervisor:delete_child/1 are not allowed. Only supervisor:terminate_child/2 will work. Testing the supervisor reveals no surprises:

1> frequency_sup:start_link().
{ok,<0.35.0>}
3> simple_phone_sup:start_link().
{ok,<0.40.0>}
4> simple_phone_sup:attach_phone(1), simple_phone_sup:attach_phone(2).
{ok,<0.43.0>}
5> simple_phone_sup:attach_phone(3).
{ok,<0.45.0>}
6> simple_phone_sup:detach_phone(3).
ok
7> supervisor:which_children(simple_phone_sup).
[{undefined,<0.42.0>,worker,[phone_fsm]},
 {undefined,<0.43.0>,worker,[phone_fsm]}]

Once we’ve detached the phone, it does not appear among the supervisor children. This is specific to the simple_one_for_one strategy, because with the other strategies, you need to both terminate and delete the children. Another difference is during shutdown; as simple_one_for_one supervisors often grow to have many children running independently of each other (often a child per concurrent request), when shutting down, they terminate the children in no specific order, often concurrently. This is acceptable, as determinism in these cases is irrelevant, and most probably not needed. Finally, simple_one_for_one supervisors scale better with a large number of dynamic children, as they use a dict key-value dictionary library module to store child specifications, unlike other supervisor types, which use a list. While other supervisors might be faster for small numbers of children, performance deteriorates quickly if the frequency at which dynamic children are started and terminated is high.

This is quite a bit of information to absorb. Before going ahead, let’s review the functional API used to manage dynamic children. Keep in mind that terminate_child/2, restart_child/2, and delete_child/2 cannot be used with simple_one_for_one strategies:

supervisor:start_child(Name, ChildSpecOrArgs)  -> {ok, Pid}
                                                  {ok, Pid, Info}
                                                  {error, already_started | 
                                                          {already_present,Id} |
                                                          Reason}
supervisor:terminate_child(Name, Id)  -> ok
                                         {error, not_found | simple_one_for_one}
supervisor:restart_child(Name, Id)   -> {ok, Pid}
                                        {ok, Pid, Info}
                                        {error, running | restarting |
                                                not_found | simple_one_for_one}
supervisor:delete_child(Name, Id)  -> ok
                                      {error, running | restarting |
                                              not_found | simple_one_for_one |
                                              Reason}

Gluing it all together

Before wrapping up this example, let’s create the top-level supervisor, bsc_sup, which starts both the frequency_sup and the simple_phone_sup functions. We will test the system using the phone.erl phone test simulator, which lets us specify the number of phones and the number of calls each phone should attempt, and then makes random calls, replying to and rejecting calls. The code for the top-level supervisor is as follows:

-module(bsc_sup).
-export([start_link/0, init/1]).
-export([stop/0]).

start_link() ->
    supervisor:start_link({local,?MODULE}, ?MODULE, []).

stop() -> exit(whereis(?MODULE), shutdown).

init(_) ->
    ChildSpecList = [child(freq_overload, worker),
                     child(frequency, worker),
                     child(simple_phone_sup, supervisor)],
    {ok,{{rest_for_one, 2, 3600}, ChildSpecList}}.

child(Module, Type) ->
    {Module, {Module, start_link, []},
     permanent, 2000, Type, [Module]}.

We pick the rest_for_one strategy because if the phones or the phone supervisor terminates, we do not want to affect the frequency allocator and overload handler. But if the frequency allocator or the overload handler terminates, we want to restart all of the phone FSMs. We allow a maximum of two restarts per hour, after which we escalate the problem to whatever is responsible for the bsc_sup supervisor.

Suppose that corrupted data in the frequency server is causing the phone FSMs to crash. After the simple_phone_sup has terminated three times within an hour, thus surpassing its maximum restart threshold, bsc_sup will terminate all of its children, bringing the frequency server down with it. The restart will hopefully clear up the problem, allowing the phones to function normally. We show how this escalation is handled in the upcoming chapters. Until then, let’s use our phone.erl simulator and test our supervision structure and phone FSM by starting 150 phones, each attempting to make 500 calls:

1> bsc_sup:start_link().
{ok,<0.35.0>}
2> phone:start_test(150, 500).
*DBG* <0.107.0> got {'$gen_sync_all_state_event',
                        {<0.33.0>,#Ref<0.0.4.37>},
                        {outbound,109}} in state idle
<0.107.0> dialing 109

...<snip>...

*DBG* <0.92.0> switched to state idle
*DBG* <0.53.0> switched to state idle
3> counters:get_counters(freq_overload).
{counters,[{{event,{frequency_denied,<0.38.0>}},27},
           {{set_alarm,{no_frequency,<0.38.0>}},6},
           {{clear_alarm,no_frequency},6}]}

For the sake of brevity, we’ve cut out all but one of the debug printouts. Having run the test, we retrieve the counters and see that during the trial run, we ran out of available frequencies six times, raising and eventually clearing the alarm accordingly. During these six intervals, 27 phone calls could not be set up as a result. Examining the logs, we can get the timestamps when these calls were rejected. If a pattern emerges, we can use the information to improve the availability of frequencies at various hours.

Before moving on to the next section, if you ran the test just shown on your computer and still have the shell open, try killing the frequency server three times using exit(whereis(frequency), kill). You will cause the top-level supervisor to reach its maximum restart threshold and terminate. Note how, when the phone FSM detaches itself in the FSM terminate function, you get a badarg error as a result of the hlr ETS tables no longer being present. The error reports originate in the terminate function if the supervisor has terminated before the phone FSM, taking the ETS tables with it. These error reports might shadow more important errors, so it is always a good idea within a terminate function to embed calls that might fail within a try-catch and, by default, return the atom ok.

Supervisor bridges

In the mid-1990s, when major projects for the next generation of telecom infrastructure of that time were started at Ericsson, OTP was being implemented. The first releases of these systems, while following many of the design principles, were not OTP-compliant because OTP did not exist. When OTP R1 was released, we ended up spending more time in meetings discussing whether we should migrate these systems to OTP than it would actually have taken to do the job. It is at times like these, when no progress is made, that the supervisor_bridge behavior comes in handy.

The supervisor bridge is a behavior that allows you to connect a non-OTP-compliant set of processes to a supervision tree. It behaves like a supervisor toward its supervisor, but interacts with its child processes using predefined start and stop functions. In Figure 8-11, the right-hand side of the supervision tree consists of OTP behaviors, while the left-hand side of the supervision tree connects the non-OTP-compliant processes.

Figure 8-11. Supervisor bridges

Start a supervisor bridge using the supervisor_bridge:start_link/2,3 call, passing the optional NameScope, the callback Mod, and the Args. This results in calling the init(Args) callback function, in which you start your Erlang process subtree, ensuring all processes are linked to each other. The init/1 callback, if successful, has to return {ok, Pid, State}. Save the State and pass it as a second argument to the terminate/2 callback.

If the Pid process terminates, the supervisor bridge will terminate with the same reason, causing the terminate/2 callback function to be invoked. In terminate/2, all calls required to shut down the non-OTP-compliant processes have to be made. At this point, the supervisor bridge’s supervisor takes over and manages the restart. If the supervisor bridge receives a shutdown message from its supervisor, terminate/2 is also called. While the supervisor bridge handles all of the debug options in the sys module, the processes it starts and is connected to have no code upgrade and debug functionality. Supervision will be limited to what has been implemented in the subtree.

supervisor_bridge:start_link(NameScope, Mod, Args) ->
    {ok, Pid} | ignore | {error, {already_started,Pid}}


Mod:init(Args)               -> {ok,Pid,State} | ignore | {error,Reason}
Mod:terminate(Reason, State) -> term()

Adding non-OTP-compliant processes

Remember that supervisors can accept only OTP-compliant processes as part of their supervision tree. They include workers, supervisors and supervisor bridges. There is one last group, however, that can be added: processes that follow a subset of the OTP design principles, the same ones standard behaviors follow. We call processes that follow OTP principles but are not part of the standard behaviors special processes. You can implement your own special processes by using the proc_lib module to start your processes and handle system messages in the sys module. With little effort, the sys, debug, and stats options can be added. Processes implemented following these principles can be connected to the supervision tree. We cover them in more detail in Chapter 10.