Instrumenting Your Apps with Metrics

One of the most powerful things you can do in monitoring is also one of the most overlooked things: instrumenting your own applications. Why would you want to do this? Easy: your apps have a ton of valuable information about their performance, many of which will allow you to move from reactive to proactive in maintaining that performance.

One common fear I’ve heard is that it’s difficult and time-consuming to add metrics to apps, but I’m here to tell you neither are true. As with all things, starting simple is the key: what about timing how long database queries take? Or how long some external vendor API takes to respond? Or how many logins happen throughout the day?

Once you start instrumenting your app, it becomes addictive. App metrics are so useful for a variety of things, you’ll wonder why you didn’t get started sooner.

I know I’ve said that I wouldn’t be talking about specific tools in this book, but there are exceptions. StatsD is one of those due to how easy it is to use in a variety of situations. It also perfectly illustrates how simple it is to instrument your apps.

StatsD is a tool used to add metrics inside of your code. Created by Etsy in 2011, StatsD has become a staple in any modern monitoring stack due to its ease of use and flexibility. Even if you don’t use StatsD, the takeaways are still valuable. StatsD was originally designed for a Graphite backend, so the metric names are dot-delimited (e.g., my.cool.metric).

I think explaining the value by example is the best way to really get a sense of the benefits here. Let’s take this simple login function:

def login():
    if password_valid():
        render_template('welcome.html')
    else:
        render_template('login_failed.html', status=403)

If the password is valid, we return the welcome page (and an implicit HTTP 200), but if the login fails, we return the login_failed.html page and an HTTP 403 Access Denied. Wouldn’t it be nice if we knew how often each happened? I’m so glad you asked:

import statsd
statsd_client = statsd.StatsClient('localhost', 8125)

def login():
    statsd_client.incr('app.login.attempts')
    if password_valid():
        statsd_client.incr('app.login.successes')
        render_template('welcome.html')
    else:
        statsd_client.incr('app.login.failures')
        render_template('login_failed.html', status=403)

I’ve added three metrics to this function now: how many times a login is attempted, how many logins succeed, and how many logins fail.

StatsD also supports timing how long something takes, and this is where we get into interesting things. What if our login service was a separate microservice, or even external? How would you know if things were about to go wrong?

import statsd
statsd_client = statsd.StatsClient('localhost', 8125)

def login():
    login_timer = statsd_client.timer('app.login.time')
    login_timer.start()
    if password_valid():
        render_template('welcome.html')
    else:
        render_template('login_failed.html', status=403)
    login_timer.stop()
    login_timer.send()

In this example, we’re setting up a timer to record how long the function takes, in effect, telling us how long a login takes. Once you know how long it normally takes to process a login, the question of “Do logins seem slower to you?” is immediately answered with data instead of instinct and a shrug.

And that’s only the beginning of what you can do with instrumenting your application with StatsD!

5
9
30
25
7
3
2
15
17
80

mean_90: 10.37
upper_90: 25
sum_90: 83
upper: 80
lower: 2
sum: 193
mean: 19.3
count: 10

Monitoring Build and Release Pipelines

Monitoring a build/release pipeline or procedure is an oft-overlooked aspect of the build-release process. Monitoring the process yields so much more insight and information into your app and infrastructure and helps you spot regressions and other problematic areas. You might be asking, “What could you possibly be monitoring here? A deploy either works or it doesn’t!” That’s mostly true. The real benefit here is in the meta-information (when did the deploy start, when did it end, what build deployed, who triggered the deploy) being available in the same place as your other app and infrastructure metrics.

Etsy popularized this concept in its seminal blog post, “Measure Anything, Measure Everything”, resulting in many different new ideas and tools for making it work even better. These days, most moderns metrics tools (both SaaS and on-premise) contain some way to implement this functionality (usually called events, annotations, or most aptly, deployments).

You’re probably wondering why this is useful at all, so let’s look at Figure 7-1: deployment events overlaid on API error rates.

Figure 7-1. Deployment marker and API errors

The graph clearly shows a correlation between a recent deploy and significantly lowered API success rates. While correlation does not equal causation, this graph shows a strong case for the deploy having caused a problem somewhere. Recording information about deploy timing, the build data, and the deployer gives us more useful information for troubleshooting. Recording this information isn’t often helpful by itself; but by combining this information with other metrics, we gain a whole new perspective and understanding of what our app and infrastructure is doing.

Health Endpoint Pattern

Oddly enough, this concept has been around for ages, but no one seems to have settled on an official name for the pattern. I’ve taken to calling it the /health endpoint pattern, while there are also some articles online referencing the concept as a canary endpoint. Some just call it a status endpoint.

Whatever you call it, the concept is straightforward: an HTTP endpoint in your app that tells you the health of the app, and sometimes includes some basic information about the app (such as deployed version, status of dependencies, etc.). Underlying the endpoint is separate code that pulls information about the app’s health and state. The implementation can range from simple to incredibly complex, as we’ll soon see.

Why would you want to use this pattern? After all, we just spent several pages talking about how great push-based performance data is, so why would you want to use something that requires regularly pulling data?

There are several benefits to this pattern that you can’t get with metric-based approaches:

• This endpoint can be used as the health check for a load balancer or for service discovery tools.

• The endpoint is helpful for debugging: exposing build information in the endpoint helps with determining what is running in the environment easily.

• Increasing the depth of the health check eventually yields an app that is aware of its own health.

Of course, it’s not an either-or situation: you can implement both patterns, emitting metrics and providing a /health endpoint. Many teams do exactly that, depending on their needs.

Let’s look at a simple implementation to illustrate. Consider a simple Django (a Python framework) app with a database dependency. I’m glossing over the underlying Django configuration, which I’m leaving as an exercise to you; this code is simply to illustrate how the health check might work, not how to use Django. You’ll want to configure your app to call health() at whatever route you use (such as /health):

from django.db import connection as sql_connection
from django.http import JsonResponse


def health():
    try:
        # Connect to a SQL database and select one row
        with sql_connection.cursor() as cursor:
            cursor.execute('SELECT 1 FROM table_name')
            cursor.fetchone()
        return JsonResponse({'status': 200}, status=200)
    except Exception, e:
        return JsonResponse({'status': 503, 'error': e}, status=503)

This example reuses the existing database configuration for the app, which is a great idea so that you don’t end up in the situation of the /health endpoint and the app using different credentials. This example just does a very simple query to return a single row. If the connection is successful, an HTTP 200 is returned, while an HTTP 503 is returned if it fails.

What if this app had more dependencies than just MySQL? What if it also depended on, say, Redis?

from django.db import connection as sql_connection
from django.http import JsonResponse
import redis


def check_sql():
    try:
        # Connect to a SQL database and select one row
        with sql_connection.cursor() as cursor:
            cursor.execute('SELECT 1 FROM table_name')
            cursor.fetchone()
        return {'okay': True}
    except Exception, e:
        return {'okay': False, 'error': e}


def check_redis():
    try:
        # Connect to a Redis database and retreive a key
        redis_connection = redis.StrictRedis()
        result = redis_connection.get('test-key')

        # Compare the key's value against a known value
        if result == 'some-value':
            return {'okay': True}
        else:
            return {'okay': False, 'error': 'Test value not found'}
    except Exception, e:
        return {'okay': False, 'error': e}


def health():
    if all(check_sql().get('okay'), check_redis().get('okay')):
        return JsonResponse({'status': 200}, status=200)
    else:
        return JsonResponse(
            {
                'mysql_okay': check_sql().get('okay'),
                'mysql_error': check_sql().get('error', None),
                'redis_okay': check_redis().get('okay'),
                'redis_error': check_redis().get('error', None)
            },
            status=503
        )

In this example, we’ve moved the two health checks into their own functions, then call them both from the health() function. If both return that they’re working, the code returns an HTTP 200. However, if either one (or both) return otherwise, the code returns an HTTP 503.

If your service depends on other services, you could use this health check to check those too. For example, if your service has a hard dependency on an external API, why not check that?

from django.http import JsonResponse
import requests


def health():
    r = requests.get('https://api.somesite.com/status')
    if r.status_code == requests.codes.ok:
        return JsonResponse({'status': 200}, status=200)
    else:
        return JsonResponse({'status': 503, 'error': r.text}, status=503)

You aren’t restricted to read-only operations with this, either—feel free to write data and test that. Let’s see how that might look with our preceding Redis example:

from django.http import JsonResponse
import redis

redis_connection = redis.StrictRedis()


def write_data():
    try:
        # Connect to Redis and set a key/value pair
        redis_connection.set('test-key', 'some-value')
        return {'okay': True}
    except Exception, e:
        return {'okay': False, 'error': e}


def read_data():
    try:
        # Connect to Redis and retrieve the key/value we set
        result = redis_connection.get('test-key')
        if result == 'some-value':
            return {'okay': True}
        else:
            return {'okay': False, 'error': 'Redis data does not match'}
    except Exception, e:
        return {'okay': False, 'error': e}


def health():
    if not write_data().get('okay'):
        return JsonResponse({'status': 503, 'error': write_data().get('error')},
                            status=503)
    else:
        if read_data().get('okay'):
            # Clean up the old data before returning the HTTP responses
            redis_connection.delete('test-key')
            return JsonResponse({'status': 200}, status=200)
        else:
            # Clean up the old data before returning the HTTP responses
            redis_connection.delete('test-key')
            return JsonResponse(
                {
                    'status': 503,
                    'error': read_data.get('error')
                },
                status=503)

In this example, we first write a Redis key-value pair, then read the data back. If the value matches what we set (which means all connections worked well), then we return HTTP 200, and HTTP 503 if any part of the test fails.

As you can see, after only a few iterations of this pattern, the health check has become much more complex than when we first started. Using this pattern extensively in a distributed microservice architecture for every microservice would ultimately result in all your services knowing their own health at all times. Essentially, you would have just automated constant testing of the entire environment.

It is worth noting, however, that some teams have found that overly complex /health endpoints actually make it harder to debug problems when the endpoint says there’s an issue, as well as the endpoint becoming needlessly noisy due to all of the dependencies it might be checking. It’s not hard to imagine a scenario where a highly interconnected service has so many health checks that it would become hard to determine where problems actually lie.

One question people often have is whether the endpoint should be just another route in the app, or a separate app entirely. You want to do the former, so that your monitoring is shipped right alongside the app. Otherwise, you’re defeating the point of this pattern.

One important thing I’ve found when using this pattern that is easy to overlook: you should be using the proper HTTP return codes. If everything is fine, return HTTP 200. If things aren’t fine, return something other than HTTP 200 (HTTP 503 Service Unavailable is a good one to use here). Using the correct HTTP response codes makes it easy to determine when things are working, without having to parse any text that is returned.

Speaking of returning text, depending on what you’re using this pattern for, it can be useful to return some data in the response. I’m a big fan of returning data as JSON, but feel free to use any other structured format—I don’t recommend using unstructured formats, as they’re more difficult to parse by machines. If your endpoint’s implementation is relatively simple, then there’s really no need to return data—just return an HTTP code and call it done.

There are downsides to this pattern, too. The biggest issue is that it’s a lot more engineering work to implement than a simple metrics-based approach (that is, push-based) requires. You also need tooling that can consistently check the endpoint. If you come from metrics-based culture, you may not have this sort of infrastructure available to you.

So there you have it, the /health endpoint pattern. I’m rather fond of the pattern, but it certainly has its challenges and hurdles. In my opinion, the pattern is still useful, even if you aren’t in a microservice environment, simply for being able to easily run a sanity check on the app.

Application Logging

Metrics can only tell you so much about what your application is up to, which is why it’s important to also be logging behavior and actions from your applications.

If you recall back from Chapter 2, we talked about the importance of using structured logs over unstructured logs. While using structured logs in some server-side applications (e.g., Apache) requires real configuration that could be non-trivial for some people, emitting structured logs from your applications is super easy thanks to direct access to the data structures. Since JSON is simply a dictionary/hash, you can build up log data using those structures, then output using the logging library in your language of choice. Of course, many languages and frameworks have specialized libraries for emitting JSON-structured logs, which makes this even easier (to name a few, Python has structlog, Rails has lograge, and PHP has monolog).

Serverless / Function-as-a-Service

Consider this problem: a serverless app (which I’ll refer to as a function) exists only as long as there is work to do. It’s invoked, does a job, then ceases to exist. The entire execution time? Under a second. Sometimes way under.

How in the world do you monitor such a thing? Traditional polling models don’t work: the polling interval isn’t short enough.

Many serverless platforms are already recording some metrics for you, such as execution time, number of invocations, and error rates. But, you’d probably like to know about what’s happening inside the function too.

The answer is quite simple: StatsD. Have a look back at the beginning of this chapter for more on that.

Also, don’t forget: your functions are probably making use of other established services with metrics and logs already available (e.g., AWS S3, AWS SNS, etc.), so be sure to check on those too.

Of course, monitoring one or two functions is different from monitoring an entire architecture built around them. If you’re in that situation, you are probably going to be very interested in distributed tracing. Let’s take a look at that now.

Monitoring Microservice Architectures

In a world where microservices are taking over everything, great monitoring becomes a must-have. Whether you’ve got three or four microservices or a hundred (or more!), understanding the interactions between these services can get complex, which makes monitoring challenging.

Consider the case of the monolith application in Figure 7-2:

Figure 7-2. A simple monolith app architecture

It’s easy and straightforward to understand the requests coming in, the results going out, and what’s happening between. And remember, a monolith need not be run on a single server—this example shows a monolith application scaled horizontally across four nodes.

What happens if we have even a small microservice environment? Imagine the preceding architecture—except repeated multiple times and abstracted away as a stand-alone service. Suddenly, you’re no longer certain where a request started, where it ended up, and where it might have gone wrong (Figure 7-3). Latencies are hidden in a microservice architecture unless you have rather robust and mature monitoring practices.

Figure 7-3. Microservice app architecture

Suddenly, understanding what happens with user requests is much more challenging. Enter distributed tracing.

Distributed tracing is a methodology and toolchain for monitoring the complex interactions inherent in a microservice architecture. Popularized by the Google Dapper paper and first implemented outside of Google with Zipkin, distributed tracing is becoming an integral component of the monitoring toolset for teams running microservice architectures.

How it works is straightforward: for every request that comes in, “tag” it with a unique request ID. This request ID stays with the request and resulting requests throughout its life, allowing you to see what services a request touches and how much time is spent in each service. One important distinction of tracing versus metrics is that tracing is more concerned with individual requests than the aggregate (though it can also be used for that). Figure 7-4 shows an example of a trace.

Figure 7-4. An example trace¹

Distributed tracing is far-and-away the most challenging and time-consuming monitoring technique to implement properly, not to mention only being useful for a small segment of the industry. Distributed tracing is not for the faint of heart or the understaffed-and-overworked engineering team. If you’ve already got all the metrics and logs you want, but still find yourself struggling to understand inter-service performance and troubleshooting in a distributed architecture, then distributed tracing might be for you (and the same goes for those of you with significant serverless infrastructures). Otherwise, effectively instrumenting your apps with metrics and logs is going to result in much better (and quicker!) outcomes.²

Wrap-Up

To summarize what we’ve learned in this chapter on application monitoring:

• Instrumenting your application with metrics and logs is one of the most impactful things you can do for increasing your ability to understand and troubleshoot the performance of your applications.

• Tracking releases and correlating with performance in your apps and infrastructure.

• The /health endpoint pattern is pretty neat, though really only useful for certain architectural designs.

• Serverless and microservice monitoring really isn’t that different from any other application, unless you have a sizable deployment. Then distributed tracing is probably where you want to begin investing time and effort.

Application code runs on servers somewhere, of course, and the performance of those servers has huge impact on the performance of your application. Let’s go look at what it means to monitor your server infrastructure.