Release Directory Structure

In this section we explore the files needed for a release. Your own releases will have the same directories and file structures as the Erlang root, so we spend some time looking at that. The only differences between the root and your own releases are the applications that are loaded and started, their versions, and the version of the runtime system. This becomes evident in the next few sections, where we create our own base station controller release that follows these very principles.

Four directories are mandatory in every OTP release, as shown in Figure 11-1. We have already looked at lib, which contains all of the applications with their version numbers appended to their directory names. You rummaged through it in “The Application Structure” when reading about applications and their directory structures. After upgrades, you could end up with multiple versions of a single application, differentiated by a version number in the application directory name. With multiple instances, the code search path defined when creating the release usually points to the ebin directory of the latest version of the application.

Figure 11-1. Release directory structure

The erts directory contains binaries for the Erlang runtime system. Even here, if you have at some point upgraded your installation, you might find multiple instances of the directory, distinguished by the erts version number appended to the directory name. In erts the most interesting subdirectory is bin. It contains executables and shell scripts related not only to the virtual machine, but also to all the tools that can be invoked from the shell. Look around in the directory and you will find the following:

erl

A script or program (depending on the target environment) that starts the runtime system and provides an interactive shell.

erlexec

The binary executable called by the erl script.

erlc

A common way to run Erlang-specific compilers. The compiler chosen depends on the extension of the file you are trying to compile.

epmd

The Erlang port mapper daemon. It acts as a name server in distributed Erlang environments, mapping Erlang nodes to IP addresses and port numbers.

escript

Allows you to execute short Erlang programs as if they were scripts, without having to compile them.

start

Starts an embedded Erlang target system in Unix environments. This kind of release runs as a daemon job without a shell window. We look at embedded target systems in “Creating a Release Package”.

run_erl

The binary called by start to start Unix-based embedded systems, where I/O is streamed to pipes.

to_erl

Connects to the Erlang I/O streams with nodes started by run_erl in an embedded target system.

werl

Starts the runtime system in Windows environments, in a separate window from the console.

start_erl

Part of the chain of commands to start embedded target systems, setting the boot and config files in Unix systems. In Windows environments, this is similar to the Unix start command previously described.

erlsrv

Similar to run_erl but for Windows environments, allowing Erlang to be started without the need for the user to log in.

heart

Monitors the heartbeat of the Erlang runtime system and calls a script if the heartbeat is not acknowledged.

dialyzer

A static analysis tool for beam files and Erlang source code. It finds, among other things, type discrepancies and dead or unreachable code. The dialyzer should be part of everyone’s build process.

typer

Infers variable types in Erlang programs based on how the variables are used. It adds type specifications derived from your source code and provides input data to the dialyzer.

Programmers use several of the executables listed in the bin directory when creating and starting an Erlang release. The ones we list are the most important and most relevant to what we cover in more detail later in this chapter. But the list is nowhere near complete, as the full contents depend on the Erlang/OTP version and operating system you are running.

These contents of the erts-version/bin directory are similar to those of bin in the Erlang root directory. The version-specific directory contains links and copies to the scripts and executables of the bin directory of the Erlang runtime version you start by default. This directory is needed because you might have several versions of a release installed and running at any one time. Although typing erl would point to the script in the bin directory, environment variables would redirect it to the erts-version/bin version you are using. Let’s have a look at the contents of the erl script. With release 18.2 on a Mac running OS X Yosemite, it looks like this:

#!/bin/sh
#
# %CopyrightBegin%
#
# Copyright Ericsson AB 1996-2012. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# %CopyrightEnd%
#
ROOTDIR="/usr/local/lib/erlang"
BINDIR=$ROOTDIR/erts-7.2/bin
EMU=beam
PROGNAME=`echo $0 | sed 's/.*///'`
export EMU
export ROOTDIR
export BINDIR
export PROGNAME
exec "$BINDIR/erlexec" ${1+"$@"}

ROOTDIR and BINDIR, along with other environment variables, are set when installing or upgrading Erlang. Note how BINDIR points to the $ROOTDIR/erts-7.2/bin directory we inspected at the beginning of this section and ends up executing erlexec. Look in the erts-7.2/bin directory for erl.src and you will find the source file used to create the erl script. Similar source files exist for start_erl and start. We cover .src files in “Creating a Release”.

If you enter the releases directory, also located in the Erlang root directory, you will find a subdirectory for every release you’ve installed on your machine. There should be a one-to-one mapping to the erts directories, because most new releases come with a new version of the runtime system. Inspect the contents of any of the start_erl.data files and you will see two numbers, the first referring to the emulator version used in the current installation of Erlang and the second referring to the directory of the OTP release being used:

$ cd releases
$ ls
17	18	RELEASES	RELEASES.src	start_erl.data
$ cat start_erl.data
7.2 18
$ ls 18
OTP_VERSION                     start.boot              start_sasl.boot
installed_application_versions  start.script            start_sasl.rel
no_dot_erlang.boot              start_all_example.rel   start_sasl.script
no_dot_erlang.rel               start_clean.boot
no_dot_erlang.script            start_clean.rel

If you list the contents of the directory specified in start_erl.data—we’ve picked version 18 in our example—you will find files with .rel, .script, and .boot extensions. Files with the .rel suffix list the versions of the applications and runtime system for a particular release. The .boot file is a binary representation of the .script file, which contains commands to load and start applications when the system is first started. Enter the subdirectory of the latest release and look at any of the .rel and .script files to get a feel for what they might do. We create our own scripts in an upcoming section.

You can override the default location of the releases directory by setting the sasl application configuration variable releases_dir or the OS environment variable RELDIR. The Erlang runtime system must have write permissions to this directory for upgrades to work, as it updates the RELEASES file in conjunction with upgrades.

Release Resource Files

All your project’s OTP applications, including those that come as part of the standard distribution, and proprietary as well as open source applications, are bundled up in a release specification containing their versions. This specification also includes the system release version and name, together with the version of the runtime system. The build system uses this information to do sanity checks, create the boot files, and create the target directory structure.

The minimal (and default) release consists of the kernel and stdlib applications, but most releases also include and start sasl because it contains all of the tools required for a software upgrade. You might not think about upgrades when creating your first release, but you’ll probably need to do so at a later date. You are given the option of including sasl by default when installing Erlang from source, but if you are using third-party binaries, this choice will have already been made for you.

Let’s look at the rel files more closely. If, from the Erlang root directory, you enter into the releases directory and from there move into any of the subdirectories, you will find at least one file with the .rel suffix. As an example, we’ve picked the releases/18/start_sasl.rel file, stripping out the comments:

{release, {"Erlang/OTP","18"}, {erts, "7.2"},
 [{kernel,"4.1.1"},
  {stdlib,"2.7"},
  {sasl, "2.6.1"}]}.

As we can see, this release will run emulator version 7.2, starting kernel version 4.1.1, stdlib version 2.7, and sasl version 2.6.1. The name of the release is “Erlang/OTP” and its version is “18.” Other examples and versions of the rel files and corresponding boot and script files in the directory specify how other systems are grouped together.

Let’s create a release file named basestation.rel to use in our base station controller example. The release name is “basestation” and we’ve given it version “1.0.” Along with the standard included applications, we’ll include version 1.0 of bsc. The file is fairly straightforward and differs very little from the previous example:

{release,
 {"basestation","1.0"},
 {erts, "7.2"},
 [{kernel, "4.1.1"},
  {stdlib, "2.7"},
  {sasl, "2.6.1"},
  {bsc, "1.0"}]}.

The resource file is by convention named ReleaseName.rel. Following this convention is not mandatory, but doing so makes life easier for those supporting and maintaining your code. The resource file contains a tuple with four elements: the release atom, a tuple of the format {ReleaseName, RelVersion}, a tuple of the format {erts, ErtsVersion}, and a list of tuples containing information about the applications and their versions. The application tuples we’ve seen so far were of the format {Application, AppVersion}, but as the following shows, other formats exist as well:

{release,
 {ReleaseName, RelVersion},
 {erts, ErtsVersion},
 [{Application, AppVersion},
  {Application, AppVersion, Type},
  {Application, AppVersion, IncludedAppList},
  {Application, AppVersion, Type, IncludedAppList}]
}.

All of the version fields for the various elements in the tuple are strings. In your application tuple, you can also add an application Type. You can include the types we covered in “Application Types and Termination Strategies”, as well as load and none:

load

Loads the application but does not start it.

none

Loads the modules in the application, but not the application itself.

permanent

Shuts down the node when the top-level supervisor terminates. When the application terminates, all other applications are cleanly taken down with it. This is the default chosen if no restart type is specified.

transient

Shuts down the node when the top-level supervisor terminates with a non-normal reason. This is useful only for library applications that do not start their own supervision trees, because top-level supervisors will always terminate with the non-normal reason shutdown, yielding the same outcome as a permanent application.

temporary

Applications that terminate, normally or abnormally, are reported in the SASL logs, but do not affect other applications in the release.

Finally, you can specify a list of included applications in IncludedAppList. The list must be a subset of the applications specified in the application app file.

Creating a Release

Having defined what is included in our release, the time has come to create it in a few simple steps, as shown in Figure 11-2:

1. Start by creating a binary boot file, which contains the commands required to load modules and start applications.

2. With your boot file in place, create a directory structure that includes all application directories, release directories, and, if required, the emulator. This package is target independent, but could be OS and hardware specific. Your directory structure must follow the directives described in “Release Directory Structure”, making it compatible with the boot file you created.

3. Create a start script defining your configurations, system limits, code search paths, and other system-specific environment variables, including a pointer to the boot file. Your script will be based on the .src files you saw in the bin directory of the emulator. The script will depend on the directory structure you have created and how you want your target system to behave.

4. With the start script in place, create a deployment package specific to your target environment. It could be a tar file, a Debian or Solaris package, a container, or any other instance that you can configure and deploy with tools of your choice or the hype of the moment.

Figure 11-2. Creating an OTP release

In our example, we keep it simple by creating and deploying a tar file using the systools library that comes as part of the SASL application in the OTP distribution. The typical target directory structure includes all of the applications listed in the release file and, in the majority of cases, the Erlang runtime system. Once we’ve created our tar file, we will want to untar it and fix scripts, configuration files, and other target-specific environment variables before creating the final package. This step could be done manually or as part of your automated build process. It could be done locally on your computer or in your target environment. How you do it depends on the development and target environments as well as the tools you pick. There never has been, and never will be, a “one size fits all” approach.

Creating the Boot File

Let’s start by creating our boot file. To do this, we need the systools:make_script/2 library function. This function creates a binary boot file used by a start script to boot Erlang and your system. To get the start_script/2 function to work, we need to copy the bsc application example, ensuring it follows the directory structure we covered in “The Application Structure”. The structure is available in this chapter’s directory of the GitHub repository. If you download it and recreate the example on your computer, don’t forget to compile the Erlang files and place them into the ebin directory.

The script starts off by looking for the application versions specified in the basestation.rel file. It does so using the code search path, and any other paths you might have included in your {path, PathList} environment variable. In our example, assuming we started Erlang in the same directory as the bsc directory, we would use the [{path, ["bsc/ebin"]}] option or start Erlang using erl -pa bsc/ebin. Remember, PathList is a list of lists, so even if you have only one directory, the directory must be defined in a list: [Dir]. Let’s try it out:

$ erl -pa bsc/ebin/
Erlang/OTP 18 [erts-7.2] [smp:8:8] [async-threads:10] [kernel-poll:false]

Eshell V7.2  (abort with ^G)
1> systools:make_script("basestation", [{path, ["bsc/ebin"]}]).
Duplicated register names:
        overload registered in sasl and bsc
error
ok

Oops, systools detected a problem when building the script. Remember how, in your app file, you specified a list of registered processes? Apparently, there is another process defined in the sasl app file with the name overload. We actually introduced freq_overload in Chapter 7, and before changing it, had it registered with the name overload in Chapter 8. When creating the first app file, we ended up using the wrong name.

If you are running the script on your laptop, you might get errors informing you that the script was unable to find a certain version of the app file, an error that is easily reproducible if you change any of the versions in basestation.rel. This is where version control becomes important. You need to know exactly which module, application, and release versions you are running in production, because your system may be running for years on end and is likely to be managed by other people. Should you get called in to support someone else’s mess, at least you’ll know what version of the mess you have to deal with.

When creating your boot file, sanity checks are run to:

• Check the consistency and dependencies of all applications defined in the rel files. Do all the applications exist, and are there no circular dependencies? Ensure that the versions defined in the app files match those specified in the rel files.

• Ensure that the kernel and stdlib applications of type permanent are part of the release. Warnings will be raised if sasl is not part of the release, but the script and boot file generation will not fail. You can suppress these warnings by passing no_warn_sasl as one of the options when creating the boot file.

• Detect clashes in the registered process names defined in the application app files, ensuring that no two processes are registered with the same name.

• Ensure that all modules defined in the app files have corresponding beam files in the ebin directory. While doing so, the sanity check detects any module name clashes, where the same module (or module name) is included in more than one application. If you want to ensure that the beam files match the latest version of the source code, include src_tests in the options.

As we look at our release, we see that the registered process name clash arises as an error in our app file. Changing overload to freq_overload in the registered process names of the bsc.app file fixes the problem.

When viewing the resulting contents of the directory as shown in the following example, we discover two new files, basestation.script and basestation.boot. Before investigating them further, let’s use the boot file to start the base station release:

1> systools:make_script("basestation", [{path, ["bsc/ebin"]}]).
ok
2> q().
ok
$ ls
basestation.boot	basestation.rel		basestation.script	bsc
$ erl -pa bsc/ebin -boot basestation
Erlang/OTP 18 [erts-7.2] [smp:8:8] [async-threads:10] [kernel-poll:false]


=PROGRESS REPORT==== 25-Dec-2015::20:37:46 ===
          supervisor: {local,sasl_safe_sup}
             started: [{pid,<0.35.0>},
                       {id,alarm_handler},
                       {mfargs,{alarm_handler,start_link,[]}},
                       {restart_type,permanent},
                       {shutdown,2000},
                       {child_type,worker}]


...<snip>...

=PROGRESS REPORT==== 25-Dec-2015::20:37:46 ===
          supervisor: {local,bsc}
             started: [{pid,<0.43.0>},
                       {id,freq_overload},
                       {mfargs,{freq_overload,start_link,[]}},
                       {restart_type,permanent},
                       {shutdown,2000},
                       {child_type,worker}]

=PROGRESS REPORT==== 25-Dec-2015::20:37:46 ===
          supervisor: {local,bsc}
             started: [{pid,<0.44.0>},
                       {id,frequency},
                       {mfargs,{frequency,start_link,[]}},
                       {restart_type,permanent},
                       {shutdown,2000},
                       {child_type,worker}]

=PROGRESS REPORT==== 25-Dec-2015::20:37:46 ===
          supervisor: {local,bsc}
             started: [{pid,<0.45.0>},
                       {id,simple_phone_sup},
                       {mfargs,{simple_phone_sup,start_link,[]}},
                       {restart_type,permanent},
                       {shutdown,2000},
                       {child_type,worker}]

=PROGRESS REPORT==== 25-Dec-2015::20:37:46 ===
         application: bsc
          started_at: nonode@nohost
Eshell V7.2  (abort with ^G)
1> observer:start().
ok

Because the bsc application was not placed in the lib directory, we have to provide the code search path to the .app and .beam files using the -pa directive to the erl command. Note all the progress reports that start appearing as soon as sasl is started (we’ve removed a few in our example). Just to be completely sure that the supervision tree has started, start the observer tool, select the Applications tab (Figure 11-3), and have a look at the bsc supervision tree.

Figure 11-3. The observer Applications tab

This is how we start OTP-compliant simple target systems. Simple target systems are used by several popular open source projects, are more robust than basic target systems, and represent a step in the right direction. But we can (and will) do better! Before discovering how, let’s review in more detail the contents of the files we’ve generated and the parameters we can pass to systools:make_script/2.

Script files

Figure 11-4 shows the basic relationships between files used to build a release. The basestation.boot file is a binary file containing all of the commands executed by the Erlang runtime system and needed to start the release. Unlike other files we look at, the boot file has to be a binary because it contains the commands that load the modules that allow the runtime system to parse and interpret text files. You can find the textual representation of the boot file’s commands in basestation.script. And even better, for those of you who like to tinker, you can edit the file or write your own. (Do this while sparing a thought for those using OTP R1 back in 1996, when make_script/2 had not yet been written.)

Figure 11-4. Creating boot and release files

Have a look at the contents of a script file. It is a file containing an Erlang term of the format {script, {ReleaseName, ReleaseVsn}, Actions}:

{script,
    {"basestation","1.0"},
    [{preLoaded,
         [erl_prim_loader,erlang,erts_internal,init,otp_ring0,prim_eval,
          prim_file,prim_inet,prim_zip,zlib]},
     {progress,preloaded},
     {path,["$ROOT/lib/kernel-4.1.1/ebin","$ROOT/lib/stdlib-2.7/ebin"]},
     {primLoad,[error_handler]},
     {kernel_load_completed},
     {progress,kernel_load_completed},
     {path,["$ROOT/lib/kernel-4.1.1/ebin"]},
     {primLoad, KernelModuleList},  %%
     {path,["$ROOT/lib/stdlib-2.7/ebin"]},
     {primLoad, StdLibModuleList},
     {path,["$ROOT/lib/sasl-2.6.1/ebin"]},
     {primLoad, SASLModuleList},
     {path,["$ROOT/lib/bsc-1.0/ebin"]},
     {primLoad, BscModuleList},
     {progress,modules_loaded},
     {path,
         ["$ROOT/lib/kernel-4.1.1/ebin","$ROOT/lib/stdlib-2.7/ebin",
          "$ROOT/lib/sasl-2.6.1/ebin","$ROOT/lib/bsc-1.0/ebin"]},
     {kernelProcess,heart,{heart,start,[]}},
     {kernelProcess,error_logger,{error_logger,start_link,[]}},
     {kernelProcess,application_controller,
         {application_controller,start, KernelAppFile}}
     {progress,init_kernel_started},
     {apply, {application,load, StdLibAppFile}},
     {apply, {application,load, SASLAppFile}},
     {apply, {application,load, BscAppFile}},
     {progress,applications_loaded},
     {apply,{application,start_boot,[kernel,permanent]}},
     {apply,{application,start_boot,[stdlib,permanent]}},
     {apply,{application,start_boot,[sasl,permanent]}},
     {apply,{application,start_boot,[bsc,permanent]}},
     {apply,{c,erlangrc,[]}},
     {progress,started}]}.

We replaced the kernel, stdlib, sasl, and bsc applications’ module lists and app file contents with variables shown in italics to make the file more book-friendly and readable. The script file starts off by defining any modules that have to be preloaded before any processes are spawned. Let’s step through these commands one at a time. Although you need not understand what they all mean if all you need to do is get a system up and running, having knowledge of the various steps helps when you have to dive into the internals of the kernel or need to troubleshoot why your system is not starting (or even more worrisome, not restarting):

preLoaded

Contains the list of Erlang modules that have to be loaded before any processes are allowed to start. You can find them in the erts application located in the lib directory. Of relevance to this section are the init module, which contains the code that interprets your boot file, and the erl_prim_loader module, which contains information on how to fetch and load the modules.

progress

Lets you report the progress state of your initialization program. The progress state can be retrieved at any time by calling the function init:get_status/0. The function returns a tuple of the format {InternalState, ProgressState}, where InternalState is starting, started, or stopping. ProgressState is set to the last value executed by the script. In our example, the only progress state that matters to the startup procedure is the last one, {progress, started}, which changes InternalState from starting to started. All other phases have no use other than for debugging purposes.

kernel_load_completed

Indicates a successful load of all the modules that are required before starting any processes. This variable is ignored in embedded mode, where loading of the modules happens before starting the system. We discuss the embedded and interactive modes in more detail later in this chapter.

path

A list of directories, represented as strings. They can be absolute paths or start with the $ROOT environment variable. These directories are added to the code search path (together with directories supplied as command-line arguments using -pa, -pz, and -path) and used to load modules defined in primLoad entries. Note how the generated paths—specifically, the one in the bsc application—assume that the beam files of the target environment are located in $ROOT/lib/bsc-1.0/ebin and not bsc/ebin. Note also how the application version numbers in the start scripts have been added to the path, assuming a standard OTP directory structure.

primLoad

Provides a list of modules loaded by calling the erl_prim_loader:get_file/1 function. If loading a module fails, the start script terminates and the node is not started. Modules may fail to be loaded when the beam files are missing, are corrupt, or were compiled by a wrong version of the compiler, or when the code search path is incorrect (e.g., if you forgot to add your application to the lib directory or have omitted the directory version number). In various places throughout this chapter we explain how to troubleshoot startup errors.

{kernelProcess, Name, {M, F, A}}

Starts a kernel process by calling apply(M, F, A). In our file, kernelProcess is used for three modules: heart, error_logger, and application_controller. You already know what the error logger and the application controller do. We look at heart in more detail in “Heart”. Once started, the kernel process is monitored, and if anything abnormal happens to it the node is shut down.

{apply, {M, F, A}}

Causes the process initializing the system to execute the apply(M, F, A) BIF, where the first argument is the module, the second is the function, and the third is a list of arguments for the function. If this function exits abnormally, the startup procedure is aborted and the system terminates. A function started in this manner may not hang and has to return, because starting the node is a synchronous procedure. If an apply does not return, the next command will not be executed.

Now that we are enlightened about each line of the script file, we can follow what is happening in our start script:

1. We start off by preloading all of the modules in the erts application, together with the error_handler in the kernel application. Once they load, we inform the script interpreter with {kernel_load_completed} and issue a progress report.

2. For all applications listed in the release file, we add the path to the end of the code search path and use primLoad to load all of the modules listed in the respective application app files. We then issue a modules_loaded progress report.

3. We start all of the kernel processes, starting with heart, the error_logger, and the application_controller (you already know about the latter two). We issue an init_kernel_started progress report.

4. We call application:load(AppFile) to load all the applications that are part of this release. This loads the four applications listed in our rel file: kernel, stdlib, sasl, and bsc. When complete, we issue an applications_loaded progress report.

5. Now that we’ve started the kernel processes and loaded all of the applications, it is time to start them. Note how, instead of calling application:start/1 in the {apply, {M, F, A}} tuple, we are calling application:start_boot/2. This is an undocumented function that, unlike application:start/2, assumes that the application has already been loaded and asks the application controller to start it.

6. Before issuing the final started progress report, we call c:erlangrc(). This function is not documented, but reads and executes the .erlang file in your home or Erlang root directory. This is a useful place to set code paths and execute other functions.

Be very careful of the code search paths in your target environment. The only reason our example can start the bsc application is that we provide the path to the beam files using -pa in the command-line prompt when starting Erlang. Our base station script expects them to be in $ROOT/lib/bsc-1.0/ebin. When generating the start script for the target environment, all applications are assumed to be in the directory AppName-version within the root directory $ROOT/lib/. This will become evident when we generate the target directory structure and files.

systools:make_script(Name, OptionsList).

Creating a Release Package

Now that we know the ins and outs of creating and starting a simple target system and have a boot file at hand, let’s have a look at how the experts package, deploy, and start their releases. The most solid and flexible way of deploying an Erlang node is as an embedded target system. Unfortunately Erlang/OTP uses the term “embedded” in several contexts, which we explain in this chapter, so please don’t assume it means the same thing each time we use it. Here, by embedded we mean our target system becomes part of a larger package running on the underlying operating system and hardware. It is capable of executing as a daemon job in the background, without the need to start an interactive shell or keep it open all the time, and it typically starts when the operating system is booted. To communicate without a shell, an embedded target system streams all I/O through pipes.

Because target environments differ based on design and operational choices, there is no “one size fits all” solution. The basic steps when creating a release package are as follows, but in practice you will often find the need to tweak them based on the details of what you are trying to achieve:

1. Create a target directory and release file.

2. Create the lib directory with the application versions specified in the rel file.

3. Create the release directory with the boot scripts and the application configuration file.

4. Copy the erts executable and binaries to the target directory.

5. Create a bin directory and copy the configuration files and the start scripts to it.

These steps are, at least in part, usually integrated in an automated build system and the install scripts executed on the target machine or run by one of the many available tools. Because OTP originally did not ship with tools to create target releases, and eventually included a complex tool focused on batch handling, the boundary for what is done by the build environment and what is done by the installation scripts on the target host varies among users. What also vary are the manual versus the automated steps. If you are doing your build on the same hardware and operating system as your target environment, you might be better off getting everything ready in one place. If you do not have this luxury, do not know where (or on what target machine) your deployment will be running, or need other target-specific configuration files created on the fly, parts of the procedure may have to be performed on the target environment.

Now we make our way through the steps required to manually create a target system, assuming that our development and target environments are the same. Based on how you are used to building, deploying, and configuring your target systems, it should be straightforward to understand where you should be drawing the boundary between what you do in your build process and what you do on the target host. We also cover some tools that can be used to automate this process.

We start off by creating the target directory, which we are going to call ernie,¹ and adding the releases and lib directories to it. Along with standard Unix commands, we use the systools:make_tar/2 library function. We start in the same directory as the bsc application directory. The make_tar call also expects the system release and boot files to be located here, alongside a config file.

The configuration file is optional at this stage. You might want to generate target-specific values at install time, overriding those specified in the app files. If you choose to omit it at this stage, you must not forget to add it when installing the system, as otherwise your system will not start. The configuration file must be named sys.config, although you can change the name by tweaking the arguments you pass to the emulator when starting it.

We create our ernie target directory, rename the configuration file sys.config, and place it in the same directory as the bsc application and the rel and boot files. When done, we can create our tar file:

$ mkdir ernie
$ cp bsc.config sys.config
$ erl
Erlang/OTP 18 [erts-7.2] [smp:8:8] [async-threads:10] [kernel-poll:false]

Eshell V7.2  (abort with ^G)
1> systools:make_tar("basestation",
                      [{erts, "/usr/local/lib/erlang/"},
                       {path, ["bsc/ebin"]}, {outdir, "ernie"}]).
ok
2> q().
ok
3>
$ cd ernie
$ ls
basestation.tar.gz
$ tar xf basestation.tar.gz
$ ls
basestation.tar.gz      lib
erts-7.2                releases
$ ls lib/
bsc-1.0         kernel-4.1.1    sasl-2.6.1      stdlib-2.7
$ ls releases/
1.0             basestation.rel
$ ls releases/1.0/
basestation.rel start.boot      sys.config
$ ls erts-7.2/bin/
beam          dialyzer    erl.src     heart           start.src
beam.smp      dyn_erl     erlc        inet_gethost    start_erl.src
child_setup   epmd        erlexec     run_erl         to_erl
ct_run        erl         escript     start           typer
$ rm basestation.tar.gz

Our call to systools:make_tar(Name, OptionsList) generates the basestation.tar.gz package. Name is the name of the release and OptionsList accepts all of the options make_script takes, together with the {erts, Dir} directive. We give this directive if we wish to include the runtime system binaries, resulting in the erts-7.2 directory. That is not always the case, because the runtime system binaries might already be installed on the target machine, or a single version of Erlang might be used to run multiple nodes. Also note that the sys.config file is included in the releases/1.0 directory. If it is in a different directory from the rel file, you have to copy it in a later stage of the installation.

You could deploy basestation.tar.gz to your target machine and run your local configuration scripts when you install the node, or do it in your build environment and create a single tar file for all deployments of this particular node. Keep in mind that your node might run in tens of thousands of independent installations—one for every base station controller your company sells—or, if hosted, in multiple occurrences of the node, all in a single installation. Your configuration parameters will depend on your needs and the type of installation; they might be the same across all tens of thousands of deployments, or may have to be individually customized when installing the software on each target environment. Often, it is a combination of both. Configuration scripts could be proprietary to your system and be included in the tar file, or be managed by third-party deployment and configuration tools such as Chef, Puppet, or Capistrano.

In our example, we untar the basestation.tar.gz file manually. The remaining steps could run either on the target or in our build environment. When untarring the file, we find three new directories: the lib directory containing all of the application directories (including their version numbers), the releases directory, and the erts directory. The erts directory is there because we included the {erts, Dir} directive in the sys_tools:make_tar/2 call.

We already know that Name is the name of the release file:

systools:make_tar(Name, OptionsList).

OptionsList is a list that can be empty or can contain some combination of the following elements:

{dirs, IncDirList}

Copies the specified directories (in addition to the defaults priv and ebin) to the application subdirectories. Thus, to add tests, src, and examples to the release, set the IncDirList to [tests, src, examples].

{path, DirList}

Adds paths to the code search path. This option can be used along with the -pa and -pz parameters passed when starting the Erlang VM that runs the system. You can include wildcards in your path. For instance, ["lib/*/ebin"] will expand to contain all of the subdirectories in lib that contain an ebin directory.

{erts, Dir}

Includes the binaries of the Erlang runtime system found in directory Dir in the target tar file. The version of the runtime system is extracted from the rel file. Make sure that the binaries have been compiled and tested on your target operating system and hardware platform.

{outdir, Dir}

Puts the tar file in directory Dir. If omitted, the default directory is the same directory as that of the rel file.

exref and {exref, AppList}

Tests the release with the Xref cross-reference tool, which looks for calls to undefined and deprecated functions. This is the same test executed by the systools:make_script/2 call when passing the same option.

src_tests

Issues a warning if there are discrepancies between the source code and the beam files. This is the same test executed by the systools:make_script/2 call when passing the same option.

silent

Returns a tuple of the format {ok, ReleaseScript, Module, Warnings} or {error, Module, Error} instead of printing the results to I/O. You can get formatted errors and warnings by calling Module:format_error(Error) and Module:format(Warning), respectively. Use this option if you are integrating systools in your build process; it works in the same way for this as for the systools:make_script/2 call.

Two additional options, {variables,[{Prefix, Var}]} and {var_tar,VarTar}, allow you to change and manipulate the way target libraries and packages are created. Use them when deviating from the standard Erlang way of doing things; for example, if you prefer to deploy your release as deb, pkg, rpm, or other packages or containers. They allow you to override the application installation directory (by default set to lib) and influence where and how the packages are stored. We do not cover these options in this chapter; for more information and some examples, read the systools reference manual page.

Start Scripts and Configuring on the Target

Now that we have our target files in place, we need to configure our start scripts. Here we go through these steps manually, later introducing tools that automate the process:

1. In the target directory (ernie, in our case), create a bin directory in which to place and edit the start scripts that will boot our system.

2. Create the log directory, to which all debug output from the start scripts is sent. It will be one of the first points of call when the system fails to start.

3. Create a file called start_erl.data in the releases directory containing the versions of the Erlang runtime system and its release.

4. If the original tar file did not contain a sys.config file, create one (possibly empty) and place it in the release version directory.

At this point, fingers crossed, everything will start. Let’s go through these steps in more detail, adding and editing files as we go along. All of this is in the ernie directory:

$ mkdir bin
$ cp erts-7.2/bin/start.src bin/start
$ cp erts-7.2/bin/start_erl.src bin/start_erl
$ cp erts-7.2/bin/run_erl bin
$ cp erts-7.2/bin/to_erl bin
$ mkdir log

In our example, we create the bin directory and copy start.src and start_erl.src to it, renaming them start and start_erl, respectively. We also copy over run_erl, which the start scripts expect to be available locally, and to_erl, which we will use to connect to an embedded Erlang shell. The start script initializes the environment for the embedded system, after which it calls start_erl, which in turn starts Erlang via the run_erl script.

Think of start_erl as an embedded version of erl and start as a script you can use and customize as you please. Depending on your needs and requirements, you might also want your own version of the erl and heart scripts and, if running distributed Erlang, the epmd binary. All of these can be copied from the bin directory of the runtime system.

Now that the files and binaries are in place, we need to edit them accordingly. We modify the start file, replacing %FINAL_ROOTDIR% with the absolute path to the new Erlang root directory. In our case, this directory is ernie, and we change the file using perl with its -i in-place modification option, using the value of our shell’s PWD variable for the replacement text. We then show you the before and after versions using the diff command:

$ pwd
/Users/francescoc/ernie
$ perl -i -pe "s#%FINAL_ROOTDIR%#$PWD#" bin/start
$ diff erts-7.2/bin/start.src bin/start
27c27
< ROOTDIR=%FINAL_ROOTDIR%
---
> ROOTDIR=/Users/francescoc/ernie
$ echo '7.2 1.0' > releases/start_erl.data
$ bin/start
$ bin/to_erl /tmp/
Attaching to /tmp/erlang.pipe.1 (^D to exit)

1> application:which_applications().
[{bsc,"Base Station Controller","1.0"},
 {sasl,"SASL  CXC 138 11","2.6.1"},
 {stdlib,"ERTS  CXC 138 10","2.7"},
 {kernel,"ERTS  CXC 138 10","4.1.1"}]
2> [Quit]
$ ls /tmp/erlang.*
/tmp/erlang.pipe.1.r	/tmp/erlang.pipe.1.w

Having modified the start file, we create the start_erl.data file in the releases directory. It contains the version of the Erlang runtime system and the release directory containing all the boot scripts and configuration files for the release. These two items, in our example both numbers, are separated by a space.

We are now able to boot our system with the start command. Notice how, unlike when using the erl command, this release starts as a background job. To connect to the Erlang shell, we use the to_erl command, passing it the /tmp directory where the read and write pipes reside.

BREAK: (a)bort (c)ontinue (p)roc info (i)nfo (l)oaded
       (v)ersion (k)ill (D)b-tables (d)istribution

If you are trying to connect to the pipes on your computer and get an error of the form No running Erlang on pipe /tmp/erlang.pipe: No such file or directory, look in the log directory to find out why your Erlang node failed to start. All start errors in your scripts will be recorded there. Problems might include wrong paths, missing sys.config files, a corrupt boot file, or an incorrectly named binary.

It is good practice to always include the erl command in the bin directory of your target system. This will come as a blessing when, after a failure of some sort, you are unable to restart your node. Your first point of call in these situations will be the SASL report logs, where crash and error reports will in most cases tell you what triggered the chain of errors that caused the node to fail. The last thing you want to do is to have to move the SASL logs to a remote computer every time you want to view them just because your Erlang nodes will not start. Be safe and always generate a second boot file similar to start_sasl.boot that contains the same application versions of kernel, stdlib, and sasl as your system.

In our example, we used the /tmp directory for the read and write pipes, as it is the default directory used by our scripts. If you plan on running multiple embedded nodes on the same machine, though, this will cause a problem. A good practice is to redirect your pipes to a subdirectory of your Erlang root directory in your target structure. This allows multiple node instances to run on the same computer, a common practice in many systems. If you look at the last line of the start script, you will see where to replace /tmp/ with the absolute path of your new pipes in the root directory. You can also redirect all of the logs elsewhere:

$ROOTDIR/bin/run_erl -daemon /tmp/ $ROOTDIR/log "exec ..."

Arguments and Flags

So far, so good. But what if we want to start a distributed Erlang node or add a patches directory to the code search path? Or maybe we have developed a dislike for the sys.config filename and want to retain the original bsc.config file. Or, even more importantly, are there flags we can pass to the emulator that will disable the ability to kill the node via Ctrl-c a?

When starting Erlang, we can pass three different types of arguments to the runtime system. They are emulator flags, flags, and plain arguments. You can recognize emulator flags by their initial + character. They control the behavior of the virtual machine, allowing you to configure system limits, memory management options, scheduler options, and other items specific to the emulator.

Flags start with - and are passed to the Erlang part of the runtime system. They include code search paths, configuration files, environment variables, distributed Erlang settings, and more.

Plain arguments are user-defined and not interpreted by the runtime system. You first came across them in “Environment Variables” to override application environment variables in app and configuration files from the command line. You can use plain arguments in your application business logic.

The following sample command uses several different arguments:

erl -pa patches -boot basestation -config bsc -init_debug +Bc

This starts Erlang with the patches directory added to the beginning of the code search path. It also uses the basestation.boot and bsc.config files to start the system, and sets the init_debug flag, increasing the number of debug messages at startup. The +Bc emulator flag disables the shell break handler, so when you press the sequence Ctrl-c a, instead of terminating the virtual machine you terminate just the shell process and restart it.

Let’s look at some of the emulator flags in more detail. We’ve picked high-level flags and system limit flags that do not deal with memory management, multicore architectures, ports and sockets, low-level tracing, or other internal optimizations. The internal optimizations are outside the scope of this book and should be used with care, only if you are sure of what you are doing. You can read more about the arguments we cover (and those we don’t) in the manual pages for erl. The ones we list are those we have used ourselves in some shape or form before the need to optimize our target systems:

+Bc

It is dangerous to keep the break handler enabled in live systems, as your fingers are often faster than your mind (especially if this is a support call in the middle of the night when your mind is still fast asleep). If you are used to terminating the shell that way, you will be inclined to do it on production systems as well. Using the +Bc flag makes Ctrl-c a terminate the current shell and start a new one without affecting your system. This is the option to enable for all your live systems.

+Bd

This allows you to terminate the Erlang node using simply Ctrl-c, bypassing the break handler altogether.

+Bi

This makes the emulator ignore Ctrl-c, in which case the only way to terminate your Erlang virtual machine is using the shell command q() or the halt() BIFs. This option is dangerous because it does not allow you to recover should an interactive call fail to return, thereby hanging the shell.

+e Num

This sets the maximum number of ETS tables, which defaults to 2,053. With Erlang/OTP R16B03 or newer, you can obtain the value of the maximum number of ETS tables at runtime by calling erlang:system_info(ets_limit).

+P Num

This changes the system limit on the maximum number of processes allowed to exist simultaneously. The limit by default is 262,144, but it can range from 1,024 to 134,217,727.

+Q Num

This changes the maximum number of ports allowed in the system, set by default to 65,536. The allowable range is 1,024 to 134,217,727.

+t Num

This allows you to change the maximum number of allowed atoms, set by default to 1,048,576. These limits are specific to Erlang 17 or newer and to Unix-based OSs. Default values might differ on other operating systems.

+R Rel

This allows your Erlang node to connect using distributed Erlang to other nodes running an older, potentially non–backward-compatible version of the distribution protocol.

Regular flags are defined at startup, retrieved in the Erlang side of the runtime system, and used by standard and user-defined OTP applications alike. Remember that large parts of the Erlang kernel and runtime system are written in Erlang, so how you define and retrieve flags in your application is identical to how Erlang defines and retrieves them in its runtime. Here are the main flags:

-Application Key Value

Sets Application’s environment variable Key to Value. We covered this option in “Environment Variables”.

-args_file FileName

Allows you to list all of the flags, emulator flags, and plain arguments in a separate configuration file named FileName, which is read at startup. The file can also contain comments that start with a # character and continue until the end of the line. Using an arguments file is the recommended approach, so as to avoid the need to mess with the start scripts to set or change arguments. This approach can also allow you keep the arguments file under version control with the rest of your code.

-async_shell_start

Allows the shell to start in parallel with other parts of the system, rather than the default of not processing what you type in the shell until the system has been completely booted. This is useful when you are trying to debug startup issues or figure out where timeouts are occurring.

-boot filename

Sets the name of the boot file to filename.boot. If you do not include an absolute path, the emulator assumes the boot file is in the $ROOT/bin directory.

-config filename

Sets the location and name of the configuration file to filename.config.

-connect_all false

Stops the global subsystem from maintaining a fully connected network of distributed Erlang nodes, in effect disabling the subsystem.

-detached

Starts the Erlang runtime system in a manner detached from the system console. You need this option when running daemons and background processes. The -detached option implies -noinput, which basically starts the Erlang node but not the shell process that runs the read-evaluate loop interpreting all the commands you type. The -noinput option in turn implies the -noshell command, which starts the Erlang runtime system without a shell, potentially making it a component in a series of Unix pipes.

-emu_args

Prints, at startup, all of the arguments passed to the emulator. Keep this on all the time in your production systems, as you never know when you will need access to the information.

-init_debug

Provides you with detailed debug information at startup, outlining every step executed in the boot script. The overheads of using -init_debug and -emu_args are negligible, but the information they provide is priceless when troubleshooting.

-env Variable Value

An alternate (and convenient) way to set host operating system environment variables. It is mainly used for testing, but is also useful when dealing with Erlang-specific values.

-eval

Parses and executes an Erlang expression as part of the node’s initialization procedure. If the parsing or execution fails, the node shuts down.

-hidden

When using distributed Erlang, starts the Erlang runtime system as a hidden node, publishing neither its existence nor the existence of the nodes to which it is connected.

-heart

Starts the external monitoring of the Erlang runtime system. If the monitored virtual machine terminates, a script that can restart it is invoked. We cover heart in detail in “Heart”.

-mode Mode

Establishes how code is loaded in the system. If Mode is interactive, calls to modules that have not been loaded are automatically searched for in the code search path. Your target systems should run in embedded mode, where all modules should be loaded at startup by the boot file, and calls to nonexisting modules should result in a crash. You can still load modules in embedded mode using the l(Module) or code:load_file(Module) calls from the shell.

Running in embedded mode is recommended for all production systems. It ensures that in the middle of a critical call, you do not pause the process while traversing the code search path looking for a module that has not been loaded.

-nostick

Disables a feature that prevents loading and overriding modules located in sticky directories. By default, the ebin directories of the kernel, compiler, and stdlib applications are sticky, a measure intended to prevent key elements of the system from being accidentally corrupted.

-pa and -pz

Add directories containing beam files to the beginning and end of the code search path, respectively. One common use is to add -pa patches to point to a directory used to store temporary patches in between releases.

-remsh node

Starts a shell connected to a remote node using distributed Erlang. This is useful when running nodes with no shells or when you need to remotely connect to a node.

-shutdown_time Time

Specifies the number of milliseconds the system is allowed to spend shutting down the supervision trees. It is by default set to infinity. Use this option with care, though, because it overrides the shutdown values specified in the behavior child specifications.

-name name and -sname name

When working with distributed Erlang, these start distributed nodes with long or short names, respectively. If nodes are to communicate with each other, they must share a cookie, which can be set using the -setcookie directive, and all have either long or short names. Nodes with short and long names cannot communicate with each other.

-s module, -s module function, -s module function args

The first of these forms executes, at startup, module:start(). The second executes module:function(). The third is like the second but includes the argument list to the function. All args are passed as atoms. The -run option works similarly, except that if arguments are defined, they are passed as a list of strings to module:function/1. Functions executed by -run and -s must return, or the startup procedure will hang. If they terminate abnormally, they will cause the node to terminate as well, aborting the startup procedure.

When troubleshooting systems, you can connect to a remote node using distributed Erlang. For instance, assume you want to connect to node foo@ramone, which has cookie abc123. You would do so by starting an Erlang VM with the –remsh flag:  

$ erl -sname bar -remsh foo@ramone -setcookie abc123
Erlang/OTP 18 [erts-7.2] [smp:8:8] [async-threads:10] [kernel-poll:false]

Eshell V7.2  (abort with ^G)
(foo@ramone)1> node().
foo@ramone
(foo@ramone)2> nodes().
[bar@ramone]
(foo@ramone)3>
BREAK: (a)bort (c)ontinue (p)roc info (i)nfo (l)oaded
       (v)ersion (k)ill (D)b-tables (d)istribution
a
$

All commands will be executed remotely in foo, with the results displayed locally. Be careful of how you exit the local shell. Using halt() and q() will terminate the remote node. Always use Ctrl-c a.

Let’s now try using -s, -eval, and -run in the shell to get a feel for how they work:

$ erl -s observer
Erlang/OTP 18 [erts-7.2] [smp:8:8] [async-threads:10] [kernel-poll:false]

Eshell V7.2  (abort with ^G)
1> q().
ok
$ erl -noshell 
-eval 'Average = (1+2+3)/3, io:format("~p~n",[Average]), erlang:halt()'
2.0
$ erl -run io format 1 2 3
Erlang/OTP 18 [erts-7.2] [smp:8:8] [async-threads:10] [kernel-poll:false]

123Eshell V7.2  (abort with ^G)
1> q().
ok

$ erl -s io format 1 2 3
Erlang/OTP 18 [erts-7.2] [smp:8:8] [async-threads:10] [kernel-poll:false]

{"init terminating in do_boot",
 {badarg,[{io,format,[<0.24.0>,['1','2','3'],[]],[]},
          {init,start_it,1,[]},{init,start_em,1,[]}]}}

Crash dump is being written to: erl_crash.dump...done
init terminating in do_boot ()

We successfully use erl -s observer to call observer:start(). You do not see it in the shell output shown here, but it opens up an observer wxWidgets window. This is an efficient way to start debugging tools when starting the emulator. We then use the -eval flag to calculate the average of three integers, print out the result, and stop the emulator, all without starting the Erlang shell. In our third and fourth examples, we use -run io format 1 2 3 to call io:format(["1","2","3"]) and -s io format 1 2 3 to call io:format(['1','2','3']). The latter crashes because it attempts to call io:format/1 with a list of atoms, when it is expecting a string.

When using the -run and -s flags, beware of calling functions such as spawn_link and start_link that link themselves to the initialization process, because the process is there to initialize the system and not act as a parent. Although the process currently continues running after executing the initialization calls, you should not depend on that behavior because it is not documented and might change in a future release.

Applications can use the init:get_arguments() and init:get_argument(Flag) functions to retrieve flags. Flag can be one of the predefined flags root, progname, and home, together with all other command-line user-defined flags.

Plain arguments include all arguments specified before emulator flags and regular flags, after the -extra flag, and in between the -- directive and the next flag. We can retrieve plain arguments using the init:get_plain_arguments/0 call:

$ erl one -two three -pa bin/bsc -- four five -extra 6 7 eight
Erlang/OTP 18 [erts-7.2] [smp:8:8] [async-threads:10] [kernel-poll:false]

Eshell V7.2  (abort with ^G)
1> init:get_plain_arguments().
["one","four","five","6","7","eight"]
2> init:get_argument(two).
{ok,[["three"]]}
3> init:get_argument(pa).
{ok,[["bin/bsc"]]}
4> init:get_argument(progname).
{ok,[["erl"]]}
5> init:get_argument(root).
{ok,[["/usr/local/lib/erlang"]]}
6> init:get_argument(home).
{ok,[["/Users/francescoc"]]}

#!/bin/sh
#Basic Heart Script for the Base Station Controller

ROOTDIR=/Users/francescoc/ernie

$ROOTDIR/bin/start

$ $ cp bsc_heart ernie/bin/.
$ export HEART_COMMAND=/Users/francescoc/ernie/bin/bsc_heart
$ vim bin/start_erl
$ diff erts-7.2/bin/start_erl.src bin/start_erl
47c47
< exec $BINDIR/erlexec ... -config $RELDIR/$VSN/sys ${1+"$@"}
---
> exec $BINDIR/erlexec ... -config $RELDIR/$VSN/sys ${1+"$@"} -heart
$ bin/start
$ bin/to_erl /tmp/
Attaching to /tmp/erlang.pipe.5 (^D to exit)

1> halt().
heart: Sat Aug 23 12:49:47 2014: Erlang has closed.
[End]
$ bin/to_erl /tmp/
Attaching to /tmp/erlang.pipe.5 (^D to exit)

1>
BREAK: (a)bort (c)ontinue (p)roc info (i)nfo (l)oaded
       (v)ersion (k)ill (D)b-tables (d)istribution
a
[End]
$ bin/to_erl /tmp/
Attaching to /tmp/erlang.pipe.5 (^D to exit)

1>

erl -heart -env HEART_BEAT_TIMEOUT 10
 -env HEART_COMMAND boot_bsc

$ erl -pa bsc/ebin
Erlang/OTP 18 [erts-7.2] [smp:8:8] [async-threads:10] [kernel-poll:false]

Eshell V7.2  (abort with ^G)
1> systools:make_script("basestation", [local]).
ok

$ erl -name [email protected] -setcookie cookie
Erlang/OTP 18 [erts-7.2] [smp:8:8] [async-threads:10] [kernel-poll:false]

Eshell V7.2  (abort with ^G)
([email protected])1> erl_boot_server:start([{127,0,0,1}]).
{ok,<0.42.0>}

$ erl -name [email protected] -id foo -hosts 127.0.0.1 
     -loader inet -setcookie cookie 
     -init_debug -emu_args -boot basestation
Executing: /usr/local/lib/erlang/erts-7.2/bin/beam.smp
    /usr/local/lib/erlang/erts-7.2/bin/beam.smp --
    -root  /usr/local/lib/erlang -progname erl --
    -home /Users/francescoc --
    -name [email protected] -id foo -hosts 127.0.0.1
    -loader inet -setcookie cookie
    -init_debug -boot basestation
{progress,preloaded}
{progress,kernel_load_completed}
{progress,modules_loaded}
{start,heart}
{start,error_logger}
{start,application_controller}
{progress,init_kernel_started}

...<snip>....

=PROGRESS REPORT==== 26-Dec-2015::12:59:05 ===
          supervisor: {local,bsc}
             started: [{pid,<0.50.0>},
                       {id,simple_phone_sup},
                       {mfargs,{simple_phone_sup,start_link,[]}},
                       {restart_type,permanent},
                       {shutdown,2000},
                       {child_type,worker}]
{apply,{c,erlangrc,[]}}

=PROGRESS REPORT==== 26-Dec-2015::12:59:05 ===
         application: bsc
          started_at: '[email protected]'
{progress,started}
Eshell V7.2  (abort with ^G)
([email protected])1> application:which_applications().
[{bsc,"Base Station Controller","1.0"},
 {sasl,"SASL  CXC 138 11","2.6.1"},
 {stdlib,"ERTS  CXC 138 10","2.7"},
 {kernel,"ERTS  CXC 138 10","4.1.1"}]

The init Module

The init module is preloaded in the Erlang runtime system. It manages arguments and the startup and shutdown procedures of your release. At startup, it executes all the commands in the boot file. Of interest to us is the ability the module gives us to restart the system, cleanly shut down all applications, and stop the node, as well as the ability to reboot the virtual machine. Here is a list of common uses for the module:

init:restart/0

Restarts the system in the Erlang node without restarting the emulator. Applications are taken down smoothly, modules are unloaded, and ports are closed, after which the boot file is executed again, using the same boot arguments originally provided. You can use the -shutdown_time flag to limit the amount of time spent taking down the applications.

init:reboot/0

Like restart, except that the emulator is also shut down and restarted. Heart, if used, will attempt to restart the system, causing a potential race condition that will resolve itself when it kills the emulator and restarts it. Timeout values set with the -shutdown_time flag will be followed.

init:stop/0

Takes the system down smoothly and stops the emulator. If running, heart is also stopped before any attempts to restart the node are made. This is the correct way to stop running nodes, because it allows the applications to terminate and clean up after themselves and properly shut down. Calling init:stop(Status) has the same effect as calling halt(Status). Timeout values set with the -shutdown_time flag will be followed.

init:get_status()

Determines whether the system is being started, is stopped, or is currently running. It returns a tuple of the format {InternalStatus, ProvidedStatus}, where InternalStatus is one of starting, started, or stopping. When starting the system, ProvidedStatus indicates what part of the boot script init is currently running. It gets Info status from the last {progress, Info} term interpreted by the boot.

We have already covered other useful functions in the init module, including get_arguments/0, get_argument/1, and get_plain_arguments/0, in “Arguments and Flags”.

Generating a Rebar3 Release Project

You can use rebar3 together with an appropriate project template to generate a project skeleton for a system like our base station controller example. Although our example uses only a single user-defined application, bsc, we use an approach that can accommodate multiple apps, since that is typical of most projects.

First, let’s create a new directory, ernie2, and within it use rebar3 to generate a new bsc release project:

$ mkdir ernie2
$ cd ernie2
$ rebar3 new release bsc desc="Base Station Controller"
===> Writing bsc/apps/bsc/src/bsc_app.erl
===> Writing bsc/apps/bsc/src/bsc_sup.erl
===> Writing bsc/apps/bsc/src/bsc.app.src
===> Writing bsc/rebar.config
===> Writing bsc/config/sys.config
===> Writing bsc/config/vm.args
===> Writing bsc/.gitignore
===> Writing bsc/LICENSE
===> Writing bsc/README.md

As the output shows, rebar3 generates a number of directories and files for our release, including skeleton source files under the apps/bsc/src directory, a sys.config file under the config directory, and a rebar.config file. The latter provides directives that supply rebar3 with project-specific details such as compiler flags, release information, and dependencies. Here’s the basic rebar.config that rebar3 generated for our bsc release project:

$ cd bsc
$ cat rebar.config
{erl_opts, [debug_info]}.
{deps, []}.

{relx, [{release, {'bsc', "0.1.0"},
         ['bsc',
          sasl]},

        {sys_config, "./config/sys.config"},
        {vm_args, "./config/vm.args"},

        {dev_mode, true},
        {include_erts, false},

        {extended_start_script, true}]
}.

{profiles, [{prod, [{relx, [{dev_mode, false},
                            {include_erts, true}]}]
            }]
}.

This particular rebar.config file contains four tuples, each described in the following list. You can modify any of these settings or add others as required for your project:

• The erl_opts tuple provides compiler options for the erlc compiler.

• The deps tuple declares dependencies for the project. Fortunately, bsc depends on nothing outside of standard Erlang/OTP.

• The relx tuple provides settings for release generation. Rebar3 uses the relx tool to generate releases. Because our goal in this section is to use rebar3 to generate a bsc release, we investigate these settings in detail later.

• The profiles tuple provides a way of having different settings for different development tasks or roles. The prod profile generated here is, as its name implies, intended to provide settings for generating a production release.

Among the generated source file skeletons, take special note of the application resource file skeleton, apps/bsc/src/bsc.app.src. Rebar3 generates this file rather than creating an actual application resource file because later, as part of its compilation process, it takes the bsc.app.src skeleton, automatically fills in its modules definition with the names of all the application source modules, and generates the bsc.app application resource file from that. We can see this by compiling our newly generated files, after first changing the "0.1.0" version numbers rebar3 generated in the bsc.app.src file and rebar.config to the correct "1.0" bsc version (any text-filtering tool can be used for this purpose; we’ve entered a Perl one-liner here):

$ perl -i -pe 's/0.1.0/1.0/' ./apps/bsc/src/bsc.app.src ./rebar.config
$ rebar3 compile
===> Verifying dependencies...
===> Compiling bsc

and then looking at the _build/default/lib/bsc/ebin/bsc.app file generated by the compilation process:

$ cat _build/default/lib/bsc/ebin/bsc.app
{application,bsc,
             [{description,"Base Station Controller"},
              {vsn,"1.0"},
              {registered,[]},
              {mod,{bsc_app,[]}},
              {applications,[kernel,stdlib]},
              {env,[]},
              {modules,[bsc_app,bsc_sup]},
              {contributors,[]},
              {licenses,[]},
              {links,[]}]}.

As the file contents show, rebar3 created the modules definition for us based on the Erlang modules present in the src directory. When we add more modules, rebar3 automatically adds them to the application resource file for us during its compilation phase, which is much easier than manually editing the resource file ourselves. The only tricky part is that if you want to modify other fields of the application resource file, you have to remember to edit the bsc.app.src file rather than the generated bsc.app file.

To run the skeleton application, we can just start a rebar3 shell, which ensures that all the appropriate project paths are on the Erlang load path. When the shell starts, it also starts our application:

$ rebar3 shell
===> Verifying dependencies...
===> Compiling bsc
Erlang/OTP 18 [erts-7.2] [smp:8:8] [async-threads:0] [kernel-poll:false]

===> Booted bsc
===> Booted sasl

...<snip>...

=PROGRESS REPORT==== 26-Dec-2015::21:58:36 ===
         application: sasl
          started_at: nonode@nohost
Eshell V7.2  (abort with ^G)
1> application:which_applications().
[{sasl,"SASL  CXC 138 11","2.6.1"},
 {bsc,"Base Station Controller","1.0"},
 {inets,"INETS  CXC 138 49","6.1"},
 {ssl,"Erlang/OTP SSL application","7.2"},
 {public_key,"Public key infrastructure","1.1"},
 {asn1,"The Erlang ASN1 compiler version 4.0.1","4.0.1"},
 {crypto,"CRYPTO","3.6.2"},
 {stdlib,"ERTS  CXC 138 10","2.7"},
 {kernel,"ERTS  CXC 138 10","4.1.1"}]

Our generated skeleton application contains no actual code, but still, it starts and runs correctly. Note that a rebar3 shell starts some other applications bsc doesn’t need, such as inets and ssl; if we were to start our application manually, these would not be present.

To fill out our project we can retrieve our original sources by copying our bsc example code, which is available in this chapter’s directory of the book’s GitHub repository:

$ cp -v <path-to-bsc-example-dir>/src/*.erl apps/bsc/src
<path-to-bsc-example-dir>/src/bsc.erl -> apps/bsc/src/bsc.erl
...

When that’s complete, we can again use rebar3 to clean and compile the project:

$ rebar3 do clean, compile
===> Cleaning out bsc...
===> Verifying dependencies...
===> Compiling bsc

If we again start a rebar3 shell, we can see that our application runs as expected:

$ rebar3 shell
...<snip>...
1> application:which_applications().
[{sasl,"SASL  CXC 138 11","2.6.1"},
 {bsc,"Base Station Controller","1.0"},
 {inets,"INETS  CXC 138 49","6.1"},
 {ssl,"Erlang/OTP SSL application","7.2"},
 {public_key,"Public key infrastructure","1.1"},
 {asn1,"The Erlang ASN1 compiler version 4.0.1","4.0.1"},
 {crypto,"CRYPTO","3.6.2"},
 {stdlib,"ERTS  CXC 138 10","2.7"},
 {kernel,"ERTS  CXC 138 10","4.1.1"}]

Creating a Release with Rebar3

The rebar3 tool uses relx, rather than the standard Erlang/OTP reltool facility, in an effort to make it easier for developers to create releases, due to reltool being widely viewed as being difficult to configure and use correctly.

Creating a release with rebar3 is straightforward:

$ rebar3 release
===> Verifying dependencies...
===> Compiling bsc
===> Starting relx build process ...
===> Resolving OTP Applications from directories:
          /Users/francescoc/ernie2/bsc/_build/default/lib
          /Users/francescoc/ernie2/bsc/apps
          /usr/local/lib/erlang/lib
===> Resolved bsc-1.0
===> Dev mode enabled, release will be symlinked
===> release successfully created!

Once we’ve generated the release, we can verify that it works as expected:

$ _build/default/rel/bsc/bin/bsc console
Exec: /usr/local/lib/erlang/erts-7.2/bin/erlexec -boot ...
Root: /Users/francescoc/ernie2/bsc/_build/default/rel/bsc
/Users/francescoc/ernie2/bsc/_build/default/rel/bsc
Erlang/OTP 18 [erts-7.2] [smp:8:8] [async-threads:30] [kernel-poll:true]


=PROGRESS REPORT==== 27-Dec-2015::11:37:56 ===
          supervisor: {local,sasl_safe_sup}
             started: [{pid,<0.49.0>},
                       {id,alarm_handler},
                       {mfargs,{alarm_handler,start_link,[]}},
                       {restart_type,permanent},
                       {shutdown,2000},
                       {child_type,worker}]

...<snip>...

=PROGRESS REPORT==== 27-Dec-2015::11:37:56 ===
         application: sasl
          started_at: bsc@francescoc
Eshell V7.2  (abort with ^G)
(bsc@francescoc)1> application:which_applications().
[{sasl,"SASL  CXC 138 11","2.6.1"},
 {bsc,"Base Station Controller","1.0"},
 {stdlib,"ERTS  CXC 138 10","2.7"},
 {kernel,"ERTS  CXC 138 10","4.1.1"}]

Instead of the console argument to _build/default/rel/bsc/bin/bsc shown in this example, which starts the application and gives us an Erlang shell, you can instead specify start to start the release in the background, attach to get a shell attached to an already-started release, or stop to stop an already-started release. Run the command _build/default/rel/bsc/bin/bsc with no arguments to see a list of all its arguments and options.

Because of the default release settings in the relx tuple in rebar.config, this generated release is intended for development, not production. The default configuration sets dev_mode to true, which means application source files used to create the release are links to sources under the apps/bsc/src directory. The dev_mode setting also sets include_erts to false, which keeps the Erlang runtime from being included in the release. These settings are handy for development because they allow developers to edit their files under the apps area and have those changes instantly available for either building into a release or recompiling and reloading into an already-running release. The settings also allow developers to use the Erlang installation on the system rather than having to build one into each release, which enables quick testing of the release against multiple runtime versions.

Fortunately, though, building a production release is easy, even with these default settings in place, thanks to rebar3 profiles. The profiles tuple in rebar.config includes a profile named prod that sets dev_mode to false and include_erts to true. To use the prod profile, we just specify it using the rebar3 as directive on the command line:

$ rebar3 as prod release
===> Verifying dependencies...
===> Compiling bsc
===> Starting relx build process ...
===> Resolving OTP Applications from directories:
          /Users/francescoc/ernie2/bsc/_build/default/lib
          /Users/francescoc/ernie2/bsc/apps
          /usr/local/lib/erlang/lib
===> Resolved bsc-1.0
===> Including Erts from /usr/local/lib/erlang
===> release successfully created!

The as directive instructs rebar3 to run the specified commands using the given profile. Pay particular attention to the text in bold near the bottom of this example; it shows that relx includes the Erlang runtime system this time, as directed by the prod profile. And because the prod profile sets dev_mode to false, if you look under _build/prod/rel/bsc/lib/bsc-1.0/src you’ll see that the source files have been copied into the release, rather than linked back to the apps source area as with the default release.

The rebar3 tar directive makes it trivial to create a tar file containing a release:

$ rebar3 as prod tar
===> Verifying dependencies...
===> Compiling bsc
===> Starting relx build process ...
===> Resolving OTP Applications from directories:
          /Users/francescoc/ernie2/bsc/_build/prod/lib
          /Users/francescoc/ernie2/bsc/apps
          /usr/local/lib/erlang/lib
          /Users/francescoc/ernie2/bsc/_build/prod/rel
===> Resolved bsc-1.0
===> Including Erts from /usr/local/lib/erlang
===> release successfully created!
===> Starting relx build process ...
===> Resolving OTP Applications from directories:
          /Users/francescoc/ernie2/bsc/_build/prod/lib
          /Users/francescoc/ernie2/bsc/apps
          /usr/local/lib/erlang/lib
          /Users/francescoc/ernie2/bsc/_build/prod/rel
===> Resolved bsc-1.0
===> tarball /Users/francescoc/ernie2/bsc/_build/prod/rel/bsc/bsc-1.0.tar.gz
         successfully created!

Rebar3 Releases with Project Dependencies

So far our rebar3 example has been limited to including only a single application, bsc, which has no dependencies, but in practice Erlang applications often depend on other applications. Fortunately, rebar3 is able to fetch such dependencies and compile them together with the application that depends on them.

Let’s assume we decide to change bsc logging using the popular open source lager framework so that our logfiles can work with existing log rotation tools, and so that we can count on lager to protect our application from running out of memory should it attempt to emit a storm of log messages because of some unexpected persistent error condition. Adding a dependency on lager to the bsc application is easy—we just specify it in the deps tuple in the rebar.config file:

{deps, [{lager, {git, "git://github.com/basho/lager.git",
                      {tag, "3.0.2"}}}]}.

This directive tells rebar3 that lager is a source dependency, with the git tuple telling rebar3 the location from which it can fetch the lager source code and the tag tuple indicating the version of lager on which the bsc application depends.

With this directive in place, we can ask rebar3 what our dependencies are:

$ rebar3 deps
lager* (git source)

Asking rebar3 to compile causes it to fetch the source for the lager dependency as well as the sources for any dependencies lager itself has:

$ rebar3 compile
===> Verifying dependencies...
===> Fetching lager ({git,"git://github.com/basho/lager.git",
                                 {tag,"3.0.2"}})
===> Fetching goldrush ({git,"git://github.com/DeadZen/goldrush.git",
                                    {tag,"0.1.7"}})
===> Compiling goldrush
===> Compiling lager
===> Compiling bsc

This compilation occurs under the default profile, so if we look under _build/default/lib after it completes, we see directories for bsc, for lager, and also for goldrush, a dependency of lager:

$ ls _build/default/lib
bsc		goldrush	lager

To build a release including lager, we first need to modify apps/bsc/src/bsc.app.src to add lager into the applications list, following kernel and stdlib. With these changes in place, we can build a release under the default profile:

$ rebar3 release
===> Verifying dependencies...
===> Compiling bsc
===> Starting relx build process ...
===> Resolving OTP Applications from directories:
          /Users/francescoc/ernie2/bsc/_build/default/lib
          /Users/francescoc/ernie2/bsc/apps
          /usr/local/lib/erlang/lib
          /Users/francescoc/ernie2/bsc/_build/default/rel
===> Resolved bsc-1.0
===> Dev mode enabled, release will be symlinked
===> release successfully created!

If we look at the contents of the _build/default/rel/bsc/lib directory, we can see that rebar3 built all the applications necessary to include in the release:

$ ls _build/default/rel/bsc/lib
bsc-1.0		goldrush-0.1.7	lager-3.0.2

We can then run our application and see that all the applications we expect to see are indeed running:

$ _build/default/rel/bsc/bin/bsc console
...<snip>....
(bsc@francescoc)1> application:which_applications().
[{sasl,"SASL  CXC 138 11","2.6.1"},
 {bsc,"Base Station Controller","1.0"},
 {lager,"Erlang logging framework","3.0.2"},
 {goldrush,"Erlang event stream processor","0.1.7"},
 {compiler,"ERTS  CXC 138 10","6.0.2"},
 {syntax_tools,"Syntax tools","1.7"},
 {stdlib,"ERTS  CXC 138 10","2.7"},
 {kernel,"ERTS  CXC 138 10","4.1.1"}]

Not only are bsc, lager, and goldrush running, but the standard compiler and syntax_tools applications were started as well because goldrush uses them, which you can see by examining the applications list in the _build/default/lib/goldrush/src/goldrush.app.src file.

There is much more to rebar3 than what our bsc application requires. It has a plug-in system that makes it extensible and customizable, ties into Erlang’s common-test, dialyzer, and eunit facilities to support testing and code coverage and analysis, and supports publishing packages into the Erlang/Elixir hex package management system. And as we show next, in Chapter 12, rebar3 also supports release upgrades.