Local environment

A local environment is a single developer’s machine (laptop or desktop). One of the advantages of developing and running code locally is that you don’t need the Internet to run your software. The phrase “Works on my machine!” is spoken by a developer who has functionality on their computer even though the code may break in another environment. This discrepancy can happen because environments can have vast differences in technical dependencies, data, and other resources.

Require developers to write unit tests to accompany each component they write. Depending on the feature and how much it interacts with other components (in your system or third-party services), integration tests with stubbed responses may also be written and run locally.

Sometimes you need to interact with other services and tools through HTTP requests when working on your local machine. If you need to work offline, those responses can usually be stubbed. In other words, you can trick your algorithm into thinking that it received a response. Stubbing or mocking is especially important to use in your automated unit tests (refer to “Going beyond the Unit Test,” later in this chapter) to speed the time that tests take to run and to ensure a consistent response for the code to ingest. Remember to update your stubbing if the API you’re calling changes its response!

Development environment

The development environment is where the first phase of testing for new code takes place. This environment is often referred to as “DEV.” After developers know that a feature works on their local machine, they deploy new code to DEV to test it there.

When the code is in DEV, engineers run unit tests and integration tests to ensure that the new code still works as expected when merged into the main master or trunk branch in git. Developers often also play around manually with the new functionality to double-check that it’s ready for a code review by a peer and deployment to the testing environment. In other words, the development environment is where developers can determine whether they think they’ve accomplished what they needed to do or they need to rework it.

The development, or DEV, environment is the least stable environment in the release pipeline. Changes are constantly being integrated by developers working on multiple areas of the codebase. Developers must confirm that the code works and the tests pass consistently before passing it on to the next environment.

Testing environment

This stage is sometimes referred to as quality assurance (QA). Traditionally, after a developer felt confident in their work, they would submit a pull request to check in their code, undergo a code review, and then hand the code over to the QA team to test it in the testing environment. But that’s not very DevOps-like. In DevOps, people work together and share responsibility.

Depending on how far along in your DevOps transformation you are, the QA team may still “own” the testing environment. Although this situation isn’t ideal, it’s a fine place to start. QA teams commonly fear automating themselves out of a job. Reframe the opportunity to show the QA team how they’ll transition from the reactive and rote toil of manual testing to becoming experts in automated testing and continuous integration.

DevOps fundamentally changes the role of QA on an engineering team. No longer does a QA engineer “own” the testing environment, test code, and then pass it off to operations after it’s deemed functional. Instead, DevOps empowers people in QA to act more like engineers. Today, QA teams assist in writing automated tests and serve as experts in testing practices, procedures, and approaches.

The DevOps emphasis on automation and continuous improvement make the hand-off to QA more nuanced. As you consider how DevOps will impact your testing practices, take time to think through what your QA team might look like in the next year. How will you level up your QA engineers? And how will you take advantage of their unique knowledge to teach developers how to write better, more reliable tests?

No matter who deploys the code to the testing environment — or whether deployment happens automatically in a CI/CD setup — it’s a slightly more robust environment than development (more resources and data) in which additional tests are run. Although unit tests can verify functionality in logic, they lack the whole picture. The testing environment is an ideal place to start running user interface tests and security challenges.

Tests may be run in two ways: serially, with each test being run sequentially, one at a time; or in parallel. A parallelized testing environment is advanced but is a differentiator between high-velocity engineering teams and those with slower software delivery.

Staging environment

The staging environment should be a mirror of the production environment. These two environments should have data and resource parity (or as close as you can get) so that you can confirm that the infrastructure does not have an unexpected impact on the code being released. The only difference between staging and production is that staging does not serve customer traffic. This approach enables you to ensure that the code is performant, and you can check for potential bugs with external services and database interactions. In addition to being the place for final testing, staging is where certain configuration or migration scripts can be run.

Although staging should mirror production as much as possible, it will never fully emulate the production environment because it lacks customer interaction and usage. Different approaches to testing in production and releasing software (discussed in the next chapter) have evolved from this fact. Testing isn’t fail-safe, but it will give you and your team confidence in your software and limit the blast radius of potential failures.

Production environment

The production environment is the final stage for your code, and it’s the one in which you have the most to lose. Your production environment serves customer traffic. After a build is released to production, it’s supposed to work as expected. Of course, in the real world, things go wrong all the time. As long as you have a way of handling rollbacks or deploying in a phased manner, you should be fine. (I discuss deployment approaches in Chapter 11.)

Being notified by customers of an incident is not ideal because it damages trust. Application insights, monitoring, logging, and telemetry are all tools that provide you with information on your system’s on performance, server load, and memory consumption. Ideally, your incident alerting system (discussed in Chapter 17) brings issues to your attention before your customers reach out. Even so, make sure that your customers can easily get your attention when they’re impacted.

Unit tests: It’s alive!

Developers write unit tests as they work to test the functionality of the logic they just built. A single function may have a dozen associated tests. Just as functions should do only one thing, so, too, should tests. Each test should ensure that the algorithm works as expected through a variety of scenarios.

Unit tests give developers immediate feedback and eliminate multiple loops of the traditional development life cycle. Instead of writing code, passing it to the QA team, and having them kick it back repeatedly, an engineer can check their work within seconds.

Unit tests are cheap, meaning that they require fewer dependencies (they test the functionality of only one piece of code) and they run quickly. A unit test can run in milliseconds, as compared to certain user interface or end-to-end tests that, depending on the complexity of the component, can take minutes to run.

Code coverage refers to how much of your codebase is “covered” by tests. Many tools evaluate your codebase against your test suite and give you a percentage of coverage, but that approach is flawed because it doesn’t measure the quality of those tests and is easily gamified. I think code coverage is more useful as a data point for stubborn executives than as a real measure of the efficacy of your engineering team. Trust them to write quality tests and to verify that work in code reviews. Provide continuing education opportunities for engineers to share knowledge and learn how to write better tests, not just more tests.

Integration tests: Do all the pieces work together?

Integration tests are typically the most useful in staging (see the “Staging environment” section, earlier in this chapter), where the application has access to the network, databases, and file systems. Unlike unit tests that validate functionality of a single piece of logic, integration tests confirm that multiple components communicate as expected.

Though a bit more complex than other tests to set up, integration tests catch bugs that are hard to track down. Not only do all the pieces of code need to work together, but they have to work with the rest of the environment as well. In integration testing, you are looking for all the little variables that can make things go awry. How does the code work with real data? What about with heavy user traffic? Do problems arise when the code interacts with mail servers?

Stubs are snippets of code that mimic a user action in a test. Drivers, on the other hand, mimic a server response.

Regression tests: After changes, does the code behave the same?

Regression testing verifies that after you make changes to the code, key metrics for how your application works and runs haven’t changed as well. This verification includes previous functionality. Have old bugs resurfaced? Did a new change impact a previous version of an API?

This testing might check that the accuracy or precision hasn’t degraded. Sometimes regression tests are as simple as ensuring that a simple CSS color change didn’t make the site a different color or cause a link to break.

Visual tests: Does everything look the same?

Visual testing is relatively new and fascinating. It’s essentially automated testing for the user interface (UI) and ensures that the application appears the same to users (tailored to specific browsers and devices) — down to the pixel. Every other kind of test verifies an expected function. Visual tests are unique in that they test the UI for consistency. I highly recommend that you don’t roll your own visual testing tool and instead opt for one of the dozens of open source or enterprise tools available.

Visual testing works by establishing a visual baseline through a screenshot, which serves as the expected display. When you merge a change into the master code branch, the testing library will take a screenshot of the new results and compare it to the baseline. If the test detects differences, the test fails. Some tools even go so far as to highlight the differences so that you can see exactly what changed — which is a front-end developer’s dream.

Performance testing

Performance tests verify the overall application performance. Is the app responsive? Stable? Does it scale as expected and use a reasonable amount of resources? Performance testing can also include security tests and load tests. Security tests verify that no known vulnerabilities were introduced in the latest build, and load tests mimic a large number of users or data that will stress the system.

Don’t forget security! Security tests should look at network security and system security as well as client-side and server-side application security. The world of security testing is vast and deep. I highly recommend the Open Web Application Security Project (OWASP) testing guide found at https://www.owasp.org/index.php/Category:OWASP_Testing_Project.