Data Comes from Everywhere

Users and applications are constantly generating an enormous amount of data. Once your applications or APIs start living on the Web, or are being consumed by a number of other services or users, you will have to start struggling through logs and user-generated events. The problem with these streams of data is that they are actually extremely valuable at a macro level, but probably difficult to interpret individually.

Think for example of a simple log entry for a timeout. If you spotted it, you could interpret it as a simple warning of a certain error that happens every now and again in your application.

Now imagine that you could easily correlate that timeout message with certain events happening in one or some of your services. You would be able to identify patterns, and perhaps find out that a certain user is accessing expensive methods or performing resource-consuming actions.

Building a strong analytics service will help you:

• Scale your systems and infrastructure where and when this is needed the most.
• Improve overall service availability and maintainability for your infrastructure, applications, and networks.
• Diagnose all your environments from one application.

Being able to rapidly search, filter, and visualize actions and events can significantly reduce the time required for root cause analysis, by providing insights into how your web application is being used and how it performs, so that you can focus your optimization efforts where they are needed.

Tools combining analytics, log management, and event tracking will parse all requests in the application’s logs, create events on specific actions, create time series of various important metrics, and also send all the information gathered to a database so you can build up and keep historical data.

There are different services and tools available if you wish to develop an analytics platform. Some of these are open source, and you just need to install them. Others are provided as SaaS solutions that you can easily integrate into your application with an API.

What solution you should adopt depends, as usual, on a number of factors. Your decision would specifically result from how you answer the following questions:

• Can you maintain your analytics platform or would you like to focus only on developing your product?
• Can you maintain the analytics infrastructure (application server), or would you prefer not to worry about this aspect?
• Can you dedicate some time or resources to defining the events you would like to track, or would you prefer to work with a predefined set of events?
• Do you want to keep your event data, or do you not mind hosting it elsewhere?

Once you have decided to either build your own solution or use a framework or third-party service that handles some of the work for you, you can start planning your analytics and reporting architecture.

Monolithic Versus Microapplication Architectures

In a monolithic application all your data will be stored in one database, so to have meaningful insights into that data you will just have to query that database and retrieve the information you need. It is common practice, though, not to run the reporting application out of the same database as the main application. Instead, you duplicate the application database completely, to keep the reporting and the production environment separate (see Figure 14-1).

Figure 14-1. A monolithic application where the app DB is replicated in the reporting DB

There are a number of problems that such an architectural choice could bring. Firstly, your reporting application will run on “old” data. Typically the reporting database is updated at certain intervals, so you are never working on real-time data. Secondly, every time the databases need to be completely replicated or even updated, some stress will be generated on both systems. Thirdly, the reporting application might need data to be structured differently than your production environment, so you might end up having to duplicate some of the data, or store all the data plus some other information. You might then start to lose control of the extra data on the reporting database. Imagine when you need to track different events; you might end up without historical data about those events being directly available. In such a situation you might have to run expensive queries on your production database to find the data you actually need and have it available to your reporting service.

There is also a fourth issue that you might easily encounter when creating an external reporting system. If you would like your users to have access to some reporting data, they will have to go through the system. Therefore, you will either have to call it from your main application or have them access your reporting service through a different application. You can already see how this architecture in many ways encourages replication and complexity.

In a microapplication architecture, data is instead stored in a number of different databases and is shared through RESTful APIs. In this environment it makes sense that when an interesting event is triggered, the services or applications concerned make requests to the analytics service themselves.

Therefore, it is not up to the analytics and reporting service to run the queries on each database to pull the data. Each service or application instead “pushes” the data to the analytics service, through REST API calls, whenever an interesting event is triggered.

Interesting events are configured at the single application and service level, but all the statistics and reporting logic remains with the analytics service.

A microapplication approach will also allow you to easily integrate reporting tools and third-party services into your infrastructure through the analytics microservice. The analytics service is itself another application sharing and offering data in the architecture (see Figure 14-2). Therefore, if your users will need to access reporting information or logs, you will just have to map the appropriate API calls. Also, if you are missing historical data, you will not have to stress your entire system: only the application containing the data you need will have to sync with the analytics service.

Figure 14-2. A microapplication architecture, where analytics is just another service in the platform

With this approach you can also decide if you would like some events to be sent to your reporting app as they happen. Here is what happens in this case:

1. An event is triggered in an application or service.
2. Data is sent with a REST call.
3. The analytics service processes the event.

Alternatively, the events may be scheduled to be sent at a later time. In this case it is often said that data is pumped to the reporting service at regular intervals. The workflow is then:

1. An event is triggered.
2. The event is scheduled to be sent in the next pump.
3. The pump is sent to the analytics service.
4. The analytics service processes the batch of events that it has just received.

In the next sections we are going to see what tools we have available at the moment and how we can configure events in our application.

Monitor, Optimize, and Enhance

Monitoring a system in production can become a complex task when the system is composed of a multitude of small services and applications.

With monolithic applications the ultimate source of errors is always the application itself, but in a microapplication environment any system component could be a source of potential problems; therefore, each microservice and microapplication needs to be monitored.

In addition, in a microservices application several third-party services are probably integrated within the application to solve additional issues. These calls need to be monitored as well.

One of the first objections against a microservices-oriented architecture is that you have to analyze multiple log sources in order to debug an issue. While it is true that microapplications and microservices certainly introduce more complexity when it comes to monitoring, it is also true that since the system is clearly split into logical units performing simple actions, it is possible to clearly understand which actions we need to monitor.

So, while microapplication systems can generate an enormous amount of log data that could easily become unmanageable, with the right tools in place we can easily isolate issues to the microservice or microapp level, without having to debug an entire monolithic application code base.

If we analyzed the architecture of a hypothetical monolithic app, we would find the same components as in a microservices architecture. These components wouldn’t be services on their own, but instead would be libraries or classes in the app. Also, some functions in the app could be scattered across several classes, and each of these classes could integrate different libraries. In such a situation, finding the culprit behind a slow call would probably be quite difficult. Of course, each approach has its pros and cons, but it is clear that using efficient statistics and correctly configuring reporting systems will make a microservices architecture easier to debug, by cutting maintenance complexity and isolating issues to a single service.

To understand how a microservices architecture should be monitored, we should think in terms of the single logical units first. Each microservice would ideally correspond to a logical unit, performing a single action.

Let’s go back to the Citywalks API. This application integrates two other APIs, the WikiCat API and the Wikipin API (Figure 14-3). In such a scenario, to begin with, we would probably want to monitor that our calls to all the APIs we are integrating are not timing out.

There are a number of indicators of how your Rails application is performing that you should monitor. We will take a look at these in order. In general, there are two simple rules when it comes to monitoring: if you can measure it you can improve it, and if you want to improve it you need to measure it first.

Application Logs

You can start analyzing a Rails application via its log files. Logs are stored by the Rails server in the /log directory, where each runtime environment has its separate log file.

Logs monitor application errors and general runtime messages and warnings, depending on your configuration.

Rails uses the standard Ruby logger to log information, but you can substitute another logger if you wish (such as Log4r) by specifying the alternative logger in your environment.rb or any environment file:

Rails.logger = Logger.new(STDOUT)
Rails.logger = Log4r::Logger.new("Application Log")

Note

Log4r is a logging library for Ruby applications.

When a certain action or error is logged, it gets printed to the corresponding log file (so long as the log level of the message is equal to or higher than the configured log level of the app). By default, each log is created under Rails.root/log/ and the log file name is <environment_name>.log.

The standard Rails logger offers five log levels: :debug, :info, :warn, :error, and :fatal, corresponding to the log level numbers from 0 to 4, respectively. To change the default log level, use:

config.log_level = :debug # In any environment initializer, or
Rails.logger.level = 0    # at any time in your code

Figure 14-3. Monitoring the Citywalks API

The default Rails log level is info in production mode and debug in development and test modes.

To write in the current log, use the appropriate logger.(debug|info|warn|error|fatal) method from within a controller, model, or mailer:

logger.debug "Resource attributes hash: #{@resource.attributes.inspect}"
logger.info "Processing the request..."
logger.fatal "Terminating application, raised unrecoverable error!!!"

Adding extra logging messages makes it easy to monitor unexpected or unusual behavior in your application. If you decide to produce extra logging messages, though, be sure to adjust the logging levels accordingly to avoid filling your production logs with noise.

Monitor Processes

As part of your deployment process you should also keep your server processes and tasks running smoothly, without consuming too many resources.

There has been a long-running discussion in the Rails community about how Ruby on Rails does multithreading, how efficient this is (or isn’t), and whether it consumes too many of our server resources.

Consider two Ruby threads: thread A is waiting and does not need to use the CPU, while thread B is waiting but does need to use the CPU. Ruby specifically switches to another thread when it needs to block for I/O. That is, whenever a Ruby thread needs to wait and it is not using the CPU, like with thread A, Ruby allows for another waiting thread to take over the CPU—so in this case, thread B takes over the CPU, while A waits.

This means that in the unlikely situation that one of your web requests uses the CPU for 30% of the time and waits for I/O the rest of the time, you can ideally serve three requests in parallel, almost maximizing usage of your CPU resources. Figure 14-4 illustrates this scenario.

Figure 14-4. Simple multithreading in Ruby

Nevertheless, you might consider using a monitoring framework to monitor your server’s performance.

God is a monitoring framework completely written in Ruby. God performs actions such as keeping processes up or restarting processes when resource utilization exceeds your desired specifications. It is possible to configure many different kinds of conditions to have your processes restarted when memory or CPU usage is too high, when disk usage is above a threshold, when a process returns an HTTP error code on a specific URL, and more.

In addition, you can write your own custom conditions and use them in your configuration files. And because the config file is written in actual Ruby code, you can use Ruby language syntax to define conditions in your configuration, thus eliminating duplication. Many different lifecycle controls are available, alongside a sophisticated and extensible notification system.

To understand how God works, let’s imagine writing a simple process:

loop do
  puts 'Hello'
  sleep 1
end

Let’s save it as server.rb.

Now we can write a God config file to monitor our process. We’ll call it monitor.god:

God.watch do |w|
  w.name = "god-monitor"
  w.start = "ruby /full/path/to/server.rb"
  w.keepalive
end

This is the bare minimum God configuration, declaring a God.watch block (a watch represents a process that you want to watch and control). At minimum, for each watch you must provide a unique name and a command that tells God how to start the process. The keepalive declaration tells God to keep the process alive. If it’s not running when God starts, it will be started, and if it dies it will be restarted.

For more configuration possibilities, see the God website.

Threads (in Ruby): Enough Already

Although it’s now a bit old, I recommend this great blog post from Yehuda Katz about threads in Ruby.

Monitor Your Server Diagnostics

Monitoring a server is about making sure that the system is healthy. This applies for your web server, your middleware server, and your database server.

Depending on the configuration and the tasks the server executes, there are a number of metrics that may need to be monitored. Memory usage and access to I/O are only some of these. You might also want to monitor average CPU utilization across all CPUs or cores, as well as on a per-CPU or per-core basis. You can set alerts to let you know when CPU utilization reaches a certain level or when your application is overloading your server memory. You might also want to keep control of how many times the server was unavailable or has restarted in the last time interval.

A list of common metrics to monitor includes:

• Uptime
• Active users
• Server time
• Average CPU usage
• Memory usage
• Resource usage

Let’s look at each of these in turn. The uptime command:

$ uptime

gives a one-line display indicating the current time, how long the system has been running, how many users are currently logged on, and the system load averages for the past 1, 5, and 15 minutes.

You can see which users are currently running processes on your system by running the following:

$ ps aux | awk '{ print $1 }' | sed '1 d' | sort | uniq

This uses the ps command, with a few filters: we get rid of everything except the first column of output (the usernames), remove the header, sort the output, and eliminate duplicate lines, displaying only the users list.

You could also filter out all system users and only show the names of regular users (and/or the root user) with active processes:

$ ps aux | awk '{ print $1 }' | sed '1 d' | sort |
  uniq | perl -e 'for (<>) { chomp; $u = ( getpwnam($_) )[2];
  print $_, "
" if ( ( $u >= 1000 || $u == 0 ) &&
  ( $_ =~ /[[:alpha:]]/ && $_ ne "nobody" ) ) }'

This would probably output just your system name and the root user.

To see the server’s local time and date, use:

$ date

What if you would like to know the number of CPUs available in your system? You can check it on your instance by running the following command:

$ grep processor /proc/cpuinfo | wc -l

To see how much memory your server is using you can use the free command, which displays RAM details in *nix machines:

$ free

       total      used       free      shared   buffers   cached
Mem:   7513644    7115076    398568    52424    381372    1134788
-/+ buffers/cache:    5598916    1914728
Swap:  50331640   6500400    43831240

The first line gives details about memory: total RAM, used RAM, free RAM, shared RAM, RAM used for buffers, and RAM used for caching content. Line 2 indicates the actual total RAM used (including by buffers and the cache) and actual total RAM available, while line 3 indicates the total swap memory available as well as how much is used and how much is free.

Resources usage

Using the top command you can get an overview in real time of which processes are using most of the system resources on your server:

$ top

Figure 14-5 shows some sample output.

Figure 14-5. System statistics with top

The first line of top output gives the load and uptime values, followed by the number of users and the load average. The next lines summarize the task list (which will depend on what operations your server is performing at the moment), the load on the CPUs, and memory and disk swap statistics.

In top you can also customize the output in order to filter the processes. If you type n and then the number of processes you want to look at—for example, 5—you will see only the top 5 processes.

If you press Shift-M (for “memory”), you can order the processes by memory consumption.

If you want to check the CPU usage—more specifically, what processes are using the CPU and how much—you proceed by choosing which field (column) to study by pressing Shift-O (the letter O, not the number 0). To concentrate on CPU usage, press k—note in Figure 14-6 how the asterisk (*) moves next to the “k: %CPU” row, indicating the CPU field has been selected—and then press Enter.

Return to the processes page, and you will see that the %CPU column is the ordered field. From the interface page (Shift-O) you can choose to monitor different metrics to know exactly what is happening on your server. Also, as top by default automatically updates every 3 seconds, it gives a real-time overview of the status of your server.

Figure 14-6. The top settings interface shows what processes are available for inspection; the columns show a letter, the column code, and a summary of that code

Actions and Events

There is a subtle relationship between data and events. Generally speaking an event is a data point resulting from something that has simply happened, and where different information has been recorded. An event can be the result of a user action, or of a state change in your system.

Events are a unifying paradigm crossing different fields of study, and how you define events in your service will inherently depend on the kind of problem that your business or application is trying to solve.

Events that we will consider valuable to record are those that we are interested in. Other types of events still occur, but we may decide to ignore them.

Imagine that you have deployed an application suggesting videos based on certain information about the user. Imagine now that you also wish to offer the user an interesting new video every time she logs in.

In order to recommend something that your users will find interesting, though, you have to learn something about them. This is usually accomplished by creating a system that is able to collect users’ preferences and then make some predictions based on those preferences. For example, if a user has shown interest in basketball, it is likely that he might enjoy a video like “Best game winner in NBA history.” The resulting predictions can be used for a variety of services, from search engines to resource suggestions and targeted advertising. The system’s functionality thus relies on users implicitly or explicitly revealing their activity and personal preferences, which are ultimately used to generate personalized recommendations.

In order to create profiles of your users’ activities, you might start to collect different information. The kind of information that you might be interested in includes not only what videos they have actually watched, but also if they have decided to watch certain videos again, what time of the day they are most likely to log in at, and which days of the week they are most likely to share videos.

As you start considering which events to store and to analyze, you might find out more about the users of your platform. For example, you might notice that some of your users are more likely to watch a video from the beginning to the end, while others will tend to skip part of it or even close it before it has finished. Some users are more likely to share on other platforms what they have watched, while others will just ignore the platform recommendations.

As you start analyzing raw event data, you might start to ask yourself some questions:

• What patterns can you isolate about events in your platform or your users’ behavior?
• What is the average activity on your platform? What does this look like?
• What raw information can you translate into a predictive model?
• What raw information can be used to create useful metrics about your platform?

As you start asking yourself questions about your data, you will be able to create more meaningful data points that correspond to particular actions happening on your platform, and display aggregated data about these actions.

In our example scenario, users would be presented with a video that had been recommended to them. They could watch the video, share it, or just skip it by moving to the next video.

Each user can therefore perform three possible actions:

• Watch
• Share
• Skip

The three actions do not need to be exclusive. This means every combination of the three actions is possible in our app. Users can watch a video, then share it, then skip it and move on to the next video. They can also share it without watching it, or watch it for a while and then skip it, and so on.

Each action can also be analyzed from different points of view. For example, we can count how many videos were watched or shared in a certain time frame, or how many users skipped a certain video.

Once you start aggregating data, you will discover what metrics make the most sense for your platform. But once again, as you start counting and measuring you will start asking yourself some new questions:

• If you are measuring the number of videos watched, are you counting unique views or total number of views?
• If you are measuring the number of shares, are you counting the number of videos that were shared or the total number of sharing actions performed?
• What time ranges are you choosing for your aggregated metrics? Why?
• Are your metrics localized on your time zone or on the user’s time zone?
• What models, algorithms, and analyses should you consider and implement?

How you answer all these questions depends on a number of factors that are mostly dictated by your application problem and real-world context. 

[(path, (timestamp, value)), ...]

Wrapping Up

In this chapter we learned about data analytics and how to manage different data sources in our applications or API platforms. In the next chapter we will see how we can scale Rails applications gracefully.