Introspecting Applications with Observer

Every time we start our Rumbl application, we have multiple applications running side by side. Each of our dependencies is in fact its own application. Phoenix and Elixir itself are applications too! An Application in Elixir is a runtime concern with these responsibilities:

• Applications package our code. Every time we compile our rumbl project, Mix prints “Generated rumbl app”. Open the file _build/dev/lib/rumbl/ebin/rumbl.app. It mostly contains metadata about the application, such as its modules, processes, a description and more.

• Supervisors can start and stop applications as a unit. An application may have a supervision tree, which defines exactly which services to start when the application starts, and which services to shutdown when the application shuts down.

• Applications provide unified configuration. Each application has its own environment, which is a key-value store to host application configuration.

All of those responsibilities may feel a bit abstract. Let’s open up the fantastic Observer, a tool shipping with Erlang, to see how Applications look in practice. To take it for spin, start a new iex -S mix session and run this command:

That command opens up an graphical user interface that looks like the screenshot.

Observer is a great tool for understanding all running processes for your application. When you open it up, the initial tab gives you general information about Erlang and also statistics about your system and your application. The tabs let you access charts to visualize different aspects of your system, check out memory allocation, look at running processes, and the like.

You Might Not Have Observer Installed
	Some package managers like to break the Erlang standard library into multiple packages. If :observer.start doesn’t work, you might be missing the erlang-observer (or similar) package.

Consider the Processes tab. You can see a list of all running processes in your system, providing a tremendous amount of visibility into the system. Remember that in Elixir, almost all state exists in your processes. With Observer, we can see the state of our entire system and who’s responsible for each piece. The process tab also includes the Message Queue (MsgQ) for each process. If a process has a very large message queue, it is likely that it is a bottleneck in your system. Therefore, ordering the processes by the Message Queue size can be a great way to spot bottlenecks, and that’s exactly what the Phoenix team did when optimizing their channels implementation to support more than 2 million connections on a single node.^[31]

You won’t explore all tabs now, but let’s look at one more in particular: Applications. There, you can see all of the applications that run on your system as well as each application’s supervision tree. Click the Applications tab and explore some of the applications on the left-side panel. When you are ready, click the Rumbl entry. You can see something like the figure.

That’s the rumbl supervision tree, more or less. Because we started iex -S mix and not iex -S mix phx.server, the server is missing from the tree. Still, there is a lot for us to explore. We can see the database connection pool, the PubSub system, and more. Inspecting our supervision trees is a great way to analyze how complex our systems are. If a supervision tree is growing too big or too wide, you can use Observer as a tool to help break the system apart.

You May Have a Different Supervision Tree

By the time we finished this book, the Phoenix team was already working on new features and enhancements to provide a more granular supervision tree. So if you are running on a Phoenix version later than Phoenix v1.4, you may see a slightly different tree than the one we showed here. We will talk about what’s coming in Chapter 14, ​What’s Next?​. Regardless of the shape of the tree, all of the points made in this discussion still apply.

Observer also allows us to trigger failures. You can right-click a process in the tree, such as Rumbl.Repo and send it a kill signal, which will cause it to terminate. You’ll find a crash report in the terminal. However, since our services are supervised, the supervisor will notice the failure and start a new instance of the same service in its place.

With a more solid understanding of what constitutes each application, let’s consider whether we might break our existing application into two smaller ones. Let’s be clear here: the main benefit is better boundaries. If all of the code belongs to a single application, then it gets harder to visualize how all of the modules in the same application depend on each other as the application grows.

For instance, while we expect our RumblWeb modules to call into our backend, we have an implicit understanding that it would be highly unexpected for Rumbl to call into RumblWeb. After all, one of the intents behind Phoenix contexts is to allow us to write our business rules without strongly coupling them to our web frontend, be it HTML, JSON, or channels. However, those rules are implicit in our Rumbl app today. By moving to an umbrella with two separate applications underneath, we can keep the web and the backend as two distinct applications which can only use each other if they have explicit dependencies between them. For example, the web appplication will have to explicitly declare that it depends on the backend. For some teams, this may be a small benefit, but for others that enjoy strong boundaries, it makes a drastic difference.

You may have tried doing something similar to this in your previous work. You had a large application, you broke it apart into different Git repositories, and versioned them separately. Initially you were proud of the boundaries you were able to define but after working for a couple months under this new schema, you noticed you got less productive. A lot of time was spent navigating and reviewing code between the different repositories. While each project had their own version, their features were often developed together, and that required you and your team to constantly match and upgrade many packages whenever a new version of any given package was out. Umbrella projects provide an alternative to this. Instead of breaking applications into multiple distinct source-code repositories, which would add too much overhead to our development workflow, the applications in an umbrella are managed and versioned together, under the same repository.

Let’s work on separating rumbl from rumbl_web now. We’re going to extract all of the web functionality to its own application. When we’re done, we’ll effectively have two isolated applications, :rumbl and :rumbl_web, in the same umbrella project. That approach will let us deploy, build, test, and package Rumbl as a whole.