Chapter 4. Principles of Testing
We’re human. We make mistakes. When organizations punish people for these mistakes, it can feel senseless as outcomes from this punishment increase errors. Fear of repercussions increases the risk of errors and slows down releases. Tests help us change how to talk about the work and reduce that fear. Rather than expecting individuals to be perfect all the time and work from a place of fear, as a team, we build safety nets that help us work from confidence.
Often overlooked or misunderstood, testing is a critical set of work for system administrators. Testing helps you deliver a working product, eliminates single points of knowledge, and increases confidence that problems won’t easily make it to end-users.
There is also an expectation for every engineer to have fundamental knowledge about all areas of the product lifecycle. For companies to adopt continuous testing, you must do your part in applying testing principles and practices in your scope of work.
Testing is a broad subject, so in this chapter, I focus on general testing principles. By the end of this chapter, you’ll know the difference between linting, unit, integration, end-to-end, and exploratory testing and how to assess a project’s overall testing strategy.
Later chapters will address specific types of testing work that sysadmins do, including:
-
Infrastructure Testing,
-
Testing in production,
-
Failover and capacity testing, and
-
Security and compliance testing.
Why should Sysadmins Write Tests?
Often we find ourselves all agreeing that tests are necessary “but”
-
Tests are expensive to write,
-
Tests are hard to write,
-
I’m not a/the tester.
So with these challenges, why should you write tests?
-
Write tests to increase your team’s confidence in your code.
-
Write tests to speed up delivery of working tools and infrastructure with increased confidence.
-
Write tests to eliminate yourself as a single point of failure in your organization so you can tackle new projects.
Differentiating the Types of Testing
Software engineers or testers implement different types of tests for projects. It benefits you to know what tests are completed against a service or product to understand the quality level of the product and identify how costly it may be to maintain and support. Some tests may have tools or patterns in use that can help you to eliminate manual processes.
Having a solid foundation in testing is crucial to help you write more effective tools and infrastructure code. Understanding the different types of testing, including their benefits and drawbacks, help you create the appropriate level of testing.
Note
There is no exact definition of these test types, so depending on the team, there may be slightly different test implementations. Just as I recommend folks on a team come together with a common understanding, I’m defining these test types to help you read the next chapter and use testing in practice.
Check out this example of how some teams at Google reframed how they categorized tests with sizes instead.
Linting
Linters are a testing tool for static analysis of code to discover problems with patterns or style conventions. Linting can help identify issues with code early. It can uncover logic errors that could lead to security vulnerabilities. Linting is distinct from just formatting changes to code because it analyzes how code runs, not just its appearance.
Tip
There’s no tests to write with linting so you don’t have to learn an extra language. Adopt linting as an initial additional testing practice in a build pipeline to level up the quality of code.
Note that folks should never make sweeping changes without talking to the team. There can be undocumented reasons why a team has adopted a particular style or convention.
There are three primary reasons to adopt linting in your development workflow: bug discovery, increased readability, and decreased variability in the code written.
-
Bug discovery helps identify quick problems that could impact the underlying logic of code. If you discover a problem while you are writing the code, it’s easier to fix. Because the code is fresh in your mind, you still have the clarity about what you intended to write. Additionally, for new functionality, it’s less likely that others will have dependencies on the code that you are writing.
-
Increasing readability helps to understand the code more quickly. Code context ages quickly; remembering what code was intended to do can be difficult. Debugging hard to read old code is even more difficult, which then makes it harder to maintain, fix, and extend functionality in the code.
-
Decreasing variability in code helps a team to come to a set of common standards and practices for a project. It helps a team ensure cohesiveness of code. Encoded style prevents arguments over team conventions, also known as bikeshedding.
You can add linting plugins for many languages to your editor for near-instantaneous feedback about potentially problematic code. This allows you to fix potential issues as they arise instead of after putting together your code into a commit and submitting a pull request for review.
You can also configure the linter to ignore rules or modify the defaults of the rules that the team doesn’t want to adopt with a configuration file.
For example, your team may want to enable longer length lines in a ruby project. Rubocop, the ruby language linter is configured with a rubocop.yml file that is stored in version control with the source code of the project.
Example 4-1.
Metrics/LineLength:Max:99
While individuals may have preferences about whether to use 2 or 4 spaces, or whether to use tabs instead of spaces within their code, common practice within a project can be identified and resolved within the editor. This helps make the project more readable as well as conform to any language requirements. The editor Visual Studio Code with the appropriate plugins automatically highlights problematic issues.
Unit Tests
Unit tests are small, quick tests that verify whether a piece of code works as expected. They do not run against an actual instance of code that is running. This makes them super helpful for quick evaluation of code correctness because they are fast (generally taking less than a second to run). With unit tests, you aren’t checking code on real instances, so you don’t receive insight into issues that are due to connectivity or dependency issues between components.
Unit tests are generally the foundation of a testing strategy for a project as they’re fast to run, less vulnerable to being flakey or noisy, and isolate where failures occur. They help answer questions about
-
design,
-
regressions in behavior,
-
assumptions about the intent in code, and
-
readiness to add new functionality.
It’s essential when unit testing your code that you aren’t checking the software that you are using, just the code that you are writing. For example, with Chef code that describes infrastructure to build, don’t write tests that test whether Chef works correctly (unless working on the Chef code itself). Instead, write tests that describe the desired outcomes and check the code.
Examples of a unit in infrastructure code might be a managed file, directory, or compute instance. The unit test to verify the example units of infrastructure code describes the file, directory, or compute instance requirements including any specific attributes. The unit test describes the expected behavior.
Integration Tests
Integration tests check the behavior of multiple objects as they work together. The specific behavior of integration tests can vary depending on how a team views “multiple objects.” It can be as narrow as 2 “units” working together, or as broad as different, more significant components working together. this doesn’t test each component of the project; it gives insight into the behavior of the software at a broader scope.
Failing integration tests aren’t precise in problem determination.
In general, an integration test should run in minutes. This is due to their being increased complexity in setting up potential infrastructure dependencies as well as other services and software.
End-to-End Tests
End-to-end tests check if the flow of behavior of an application functions as expected from start to finish. An end-to-end test tests all the application and services that were defined by the infrastructure code on the system and how they worked together.
Three reasons end-to-end testing should be minimal in their implementation are that they are sensitive to minor changes in interfaces, and take significant time to run, write and maintain.
End-to-end testing can be very brittle or weak in response to changes. End-to-end tests fail and builds break when interfaces at any point in a stack change.
Documentation about the interfaces can be wrong or out-of-date. The implementation may not function as intended or change unexpectedly.
End-to-end testing increases our confidence that the complete system, with all of its dependencies are functioning as designed.
Additionally, end-to-end test failure is not isolated and deterministic. Test failure may be due to dependent service failure. End-to-end tests checking specific function output require more frequent changes to the test code.
For example, a test environment located in an availability zone on Amazon with network issues may have intermittent failures. The more flakey the tests, the less likely individuals will spend effort maintaining those tests, which leads to lower quality in the testing suite.
End-to-end tests can also take a long time to implement. These tests require the entire product to be built and deployed before the tests can be run. Add on the test suite, and they can be quite lengthy to complete.
Even with these challenges, end-to-end tests are a critical piece of a testing strategy. They simulate a real user interacting with the system. Modern software can be comprised of many interconnected subsystems or services that are being built by a different team inside or outside of an organization. Organizations rely on these externally built systems rather than expending resources into building them in house(which incidentally has even higher risk). System administrators often manage the seams where different systems built by different people are connecting.
Examining the Shape of Testing Strategy
We can examine the tests that exist for a project and qualify the strategy as a shape based on the number of tests. This informs us of potential gaps where additional testing is needed or tests that need to be eliminated.
One of the recommendations in software development is that the quantity of testing should look very much like a pyramid. Mike Cohn described the Test Automation Pyramid in his 2009 book Succeeding with Agile.
Figure 4-1. Recommended Test Strategy with Shape of Pyramid
Over time the pyramid has been modified and updated, but the general premise remains. The pyramid codifies what is needed when it comes to testing, and stresses the importance of the different types of tests while recognizing that tests have different implementation times and costs. Over time, this pyramid has been adopted and shared widely with the occasional minor modifications. In the pyramid model, approximately 70% of the volume is taken up by unit tests, 20% for integration tests, and 10% for end-to-end.
A good rule is to push tests as far down the stack as possible. The lower in the stack it is, the faster that it will run, and the faster it will provide feedback about the quality and risk of the software. Unit tests are closer to the code testing specific functions, where end-to-end is closer to the end-user experience, hence the pyramid shape based on how much attention and time we are spending writing the particular type of tests.
Figure 4-2. Test Strategy with Shape of Square
If a project’s tests resemble a square meaning, there are tests equally at every level, that may be an indication that there are overlapping testing concerns. In other words, there is testing of the same things at different levels. This may mean longer test times and delayed delivery into production.
Figure 4-3. Test Strategy with Shape of Inverted Pyramid
If a project’s tests resemble an inverted pyramid meaning there are more end to end tests, and fewer unit tests, that may be an indication that there is insufficient coverage at the unit and integration test level. This may mean longer test times, and delayed code integration as it will take longer to verify that code works as expected. Increasing the unit test coverage will increase the confidence of changes in code and reduce the time it takes to merge code leading to fewer conflicts!
Figure 4-4. Test Strategy with Shape of Hourglass
If a project’s tests resemble an hourglass meaning there are more end to end tests compared to integration tests, that may be an indication that there is insufficient integration coverage, or that there are more end-to-end tests than are needed. Remember that end to end tests are more brittle so they will require more care and maintenance with changes.
Having an hourglass or inverted pyramid will also indicate the potential that more time is spent on maintenance of tests rather than developing new features.
Understanding these shapes of testing strategies can help you understand how much invisible work is being passed on to the system administration team to support a project.
That said, infrastructure code testing does not always follow these patterns. Specifically, infrastructure configuration code testing does not benefit from unit tests except when checking for different paths of configuration. For example, a unit test is beneficial when there are differences in platform requirements due to supporting different operating systems. It can also be beneficial where there are differences in environments between development, testing and production or making sure that production API keys don’t get deployed in development and testing.
Remember, push tests as far down the stack as possible. With infrastructure, due to its nature, integration testing might be as far down as it makes sense.
Existing Sysadmin Testing Activities
Have you ever run through installing a set of software on a non-production or non-live system, watching to see how the system responded and whether there were any gotchas that could be user impacting?
This process is a type of testing that sysadmins get good at without thinking about it as an “official” type of testing. This manual testing approach is known as exploratory testing. The goal with exploratory testing is to help discover the unknown by experimenting with the system looking at areas of the system that may need more subjective analysis as to whether they are in a good state. In Exploratory Testing Explained, James Bach defined exploratory testing as “simultaneous learning, test design, and test execution.”
Sysadmins can level up exploratory testing skills by adopting more rigor in their analysis. This means defining testing objectives with short feedback loops. Working with software engineers and testers, sysadmins can help shape the testing so that the team eliminates some of the manual testing.
Note
It’s helpful to have new team members explore products and processes as part of their onboarding. They can bring unbiased insight to level up quality to correct problems with the product and processes and clear up misunderstanding and inconsistencies that may already exist within the team.
Wrapping Up
In this chapter, I shared general testing principles. You should now have a solid understanding of how to distinguish linting, unit, integration, end-to-end, and exploratory testing.
In the next chapter, it’s time to apply those principles to writing tests to help make your work more predictable, repeatable, and collaborative.