Test-driven development

Today’s gold standard for testing software prior to its release in production is Test-Driven Development (TDD).² Within the last two decades, TDD is arguably one of the more successful practices to take hold across the software development industry. TDD has provided a very useful framework for shift-left testing that catches, and prevents, many potential problems long before they reach production. Widely adopted across wide-swaths of the software development industry, TDD should be credited with having uplifted the quality level for code running production services.

TDD is a powerful practice that provides engineers with a clear way to think about software operability. Applications are defined by a deterministic set of repeatable tests that can be run hundreds of times per day. If these repeatable tests pass, then the application must be running as expected. Before changes to the application are actually produced, they start as a set of new tests that exist to verify that change would work as expected. A developer can then begin to write new code in order to ensure the test passes.

TDD is particularly powerful because tests run the same way every time. Data typically doesn’t persist between test runs, it gets dropped; erased and recreated from scratch for each run. Responses from underlying or remote systems are stubbed or mocked. With TDD, developers are tasked with creating a specification that precisely defines the expected behaviors for an application in a controlled state. The role of tests is to identify any unexpected deviations from that controlled state and deal with them immediately. In doing so, TDD removes guesswork and provides consistency.

But that very consistency and isolation also limits what TDD can reveal about what is happening with your software in production. Running isolated tests doesn’t reveal whether or not customers are having a good experience with your service. Nor does passing those tests mean that any errors or regressions could be quickly and directly isolated and fixed before releasing that code back into production.

Any reasonably experienced engineer responsible for managing software running in production can tell you that production environments are anything but consistent. Production is full of interesting deviations that your code might encounter out in the wild, but that have been excised from tests because they’re not repeatable, they don’t quite fit the specification, or they don’t go according to plan. While the consistency and isolation of TDD makes your code tractable, it does not prepare your code for the interesting anomalies that should be surfaced, watched, stressed, and tested because they ultimately shape how your software behaves when real people start interacting with it.

Observability can help you write and ship better code even before it lands in source control—because it’s part of the set of tools, processes, and culture that allow engineers to find bugs in their code quickly.

Observability in the development cycle

Catching bugs cleanly, resolving them swiftly, and preventing them from becoming a backlog of technical debt that weighs down the development process relies on a team’s ability to find those bugs quickly. Yet, software development teams often hinder their ability to do so for a variety of reasons.

For example, consider organizations where software engineers aren’t responsible for operating their software in production. Engineers merge their code into master, cross their fingers hoping that this change won’t be one that breaks prod, and they essentially wait to get paged if a problem occurs. Sometimes they get paged soon after deployment. The deployment is then rolled back and the triggering changes can be examined for bugs. But more likely, problems wouldn’t be detected for hours, days, weeks, or months after that code had been merged. By that time, it becomes extremely difficult to pick out the origin of the bug, remember the context, or decipher the original intent behind why that code was written or why it shipped.

Resolving bugs quickly critically depends on being able to examine the problem while the original intent is still fresh in the original author’s head. It will never again be as easy to debug a problem as it was right after it was written and shipped. It only gets harder from there: speed is key. At first glance, the links between observability and writing better software may not be clear. But it is this need for debugging quickly that deeply intertwines the two.

Determining where to debug

Newcomers to observability often make the mistake of thinking that observability is a way to debug your code, similar to using highly verbose logging. While it’s possible to debug your code using observability tools, that is not the primary purpose of observability. Observability operates on the order of systems, not on the order of functions. Emitting enough detail at the lines level to reliably debug code would emit so much output that it would swamp most observability systems with an obscene amount of storage and scale. It would simply be impractical to pay for a system capable of doing that because it would likely cost somewhere in the ballpark of 1X-10X as much as your system itself.

Observability is not for debugging your code logic. Observability is for figuring out where in your systems to find the code you need to debug. Observability tools help you swiftly narrowing down where problems may be occuring. From which component did an error originate? Where is latency being introduced? Where did a piece of this data get munged? Which hop is taking up the most processing time? Is that wait time evenly distributed across all users, or is it only experienced by a subset thereof? Observability helps your investigation of problems pinpoint likely sources.

Often, observability will also give you a good idea of what might be happening in or around an affected component, what the bug might be, or even provide hints as to where the bug is actually happening: your code, the platform’s code, or a higher-level architectural object.

Once you’ve identified where the bug lives and some qualities about how it arises, then observability’s job is done. From there, if you want to dive deeper into the code itself, the tool you want is a good old-fashioned debugger (for example, gdb). Once you suspect how to reproduce the problem, you can spin up a local instance of the code, copy over the full context from the service, and continue your investigation. While they are related, the difference between an observability tool and a debugger is an order of scale; like a telescope and a microscope, they may have some overlapping use cases, but they are primarily designed for different things.

This is an example of different paths you might take with debugging vs observability:

• You see a spike in latency. You start at the edge; break down by endpoint, calculate average, 90th, and 99th percentile latency; identify some cadre of slow requests; trace one of them; it shows the timeouts begin at service3. You copy the context from the traced request into your local copy of service3 binary and attempt to reproduce it in the debugger.

• You see a spike in latency. You start at the edge; break down by endpoint, calculate average, 90th, and 99th percentile latency; notice that only endpoints which are write-only are the ones that are suddenly slower. You break down by db destination host, and note that it is distributed across some, but not all, of your db primaries. For those db primaries, this is only happening to ones of a certain instance type or in a particular AZ. You conclude the problem is not a code problem, but one of infrastructure.

Shifting observability left

While test-driven development ensures developed software adheres to an isolated specification, observability-driven development ensures that software works in the messy reality that is a production environment; strewn across a complex infrastructure, at any particular point in time, experiencing fluctuating workloads, with particular users.

Building instrumentation into our software early in the development lifecycle allows engineers to more readily consider and more quickly see the impact that small changes truly have in production. By focusing on just adherence to an isolated spec, teams inadvertently create conditions where they cease to have visibility into the chaotic playground where that software comes into contact with messy and unpredictable people problems. As we’ve seen in previous chapters, traditional monitoring approaches only reveal an aggregate view of measures that were developed in response to triggering alerts for known issues. Traditional tools provide very little ability to accurately reason about what happens in complex modern software systems.

As a result, teams will often approach production as a glass castle; a beautiful monument to their collective design over time, but one they’re afraid to touch because any unforeseen movement could shatter the entire structure. Developing the engineering skills to write, deploy, and use good telemetry to understand behaviors in production, teams become empowered to reach further and further into the development lifecycle to consider the nuances of detecting unforeseen anomalies that could be roaming around their castles unseen.

Observability-driven development is what allows engineering teams to turn their glass castles into playgrounds. Production environments aren’t immutable, they’re full of action, and engineers should be empowered to confidently walk into any game and score a win. But that only happens when observability isn’t considered solely the domain of SREs, infrastructure engineers, or operations teams. Software engineers must adopt observability and work it into their development practices in order to unwind the cycle of fear they’ve developed over making any changes to production.