Chapter 17. Performance and Stress Testing

If you know neither the enemy nor yourself, you will succumb in every battle.

—Sun Tzu

After peripherally mentioning performance and stress testing in previous chapters, we now turn our full attention to these tests. In this chapter, we discuss how these tests differ in purpose and output as well as how they impact scalability. Whether you use performance tests, stress tests, neither, or both, this chapter should give you some fresh perspectives on the purpose and viability of testing that you can use to either revamp or initiate a testing process in your organization.

As is the case with the quality of your product, scalability is something that must be designed early in the development life cycle. Testing, while a necessary evil, is an additional cost in our organizations meant to uncover problems and oversights with our designs.

1. Performance testing. Wikipedia. http://en.wikipedia.org/wiki/Performance_testing.

According to our definition, the goal of performance testing is to identify, document, and, where possible, eliminate bottlenecks in the system. This is done through a strict controlled process of measurement and analysis. Load testing is utilized as a method in this process.

When replacing systems, the benchmarks of the old (to-be-replaced system) are often used as a starting point for the expectations of the new system. Usually such replacements are predicated on creating greater throughput for the purpose of reducing costs at equivalent transaction volumes or to allow the company to grow more cost-effectively in the future.

The performance testing environment should mimic the production environment to the greatest extent possible because environmental settings, configurations, different hardware, different firewall rules, and much more can all dramatically affect test results. Even different patch versions of the operating system, which might seem a trivial concern, can have dramatically different performance characteristics for applications. This does not mean that you need a full copy of your production environment; although that would be nice, few companies can afford such a luxury. Instead, make wise tradeoffs but stick to the same basic architecture and implementation as much as possible. For example, pools of servers that in production include 40 servers can be scaled down in a test environment to only two or three servers. Databases are often very difficult to scale down because the amount of data affects the query performance. In some cases, you can “trick” the database into believing it has the same amount of data as the production database to ensure the queries execute with the same query plans. Spend some time pondering the performance testing environment, and discuss the tradeoffs that you are making. If you can sufficiently balance the cost with the effectiveness, you will be able to make the best decisions in terms of what the environment should look like and how accurate the results will be.

An additional reason to create a separate test environment may arise if you are practicing continuous delivery or have plans to do so. For your automated systems to be able to easily schedule delivery of packages, they ideally should run those packages through stages of environments, where each stage is focused on some aspect of the potential quality issues (such as a performance testing environment in this case). The more constrained or shared your environments, the more changes that will become backed up waiting for automated environment reconfiguration to perform automated testing. Splitting out these environments helps ensure a fluid pipeline with fastest possible delivery (assuming no major blocking issues) into the production environment.

When you define the tests, be sure to include tests of various types. Some types or categories of tests include endurance, load, most used, most visible, and component (app, network, database, cache, and storage). An endurance test is used to ensure that a standard load experienced over a prolonged period of time does not have any adverse effects due to such problems as memory leaks, data storage, log file creation, or batch jobs. A normal user load with as realistic traffic patterns and activities as possible is used here. It is often difficult to come up with actual or close-to-actual user traffic. A minimum substitute for this input is a series of actions—such as a signup process followed by a picture upload, a search for friends, and a logout—written into a script that can be executed over and over. A more ideal scenario is to gather actual users’ traffic from a network device or app server and replay it in the exact same order while varying the time period. That is, first you can run the test over the same time period in which the users generated the traffic, and then you can increase the speed and ensure the application performs as expected with the increased throughput.

Remember to be mindful of the test definition as it relates to both continuous integration and—more importantly—continuous delivery (refer back to Chapter 5, Management 101, for a definition or refresher on this topic). To be successful with continuous delivery, we need to ensure that the tests we define can be automated and that the success criteria for them can be evaluated by the automation.

In the text execution step, you work through the test plan, executing the tests methodically in the environment established for this testing and recording various measurements such as transaction times, response times, outputs, and behavior. Gather everything that you can. Data is your friend in performance testing. It is important to keep this data from release to release. As described in the next step, comparison between various releases is critical to understanding the data and determining if the data indicates normal operating ranges or the potential presence of a problem.

In organizations practicing continuous delivery, there are a few ways to think about how to execute performance tests. The first is to have the performance testing occur nearly continuously and outside the critical path to delivery into your production environment. This approach has the beneficial effect of not stalling releases waiting for past submissions to the performance test environment to complete. An unfortunate side effect is that should any release identify a significant performance problem that may cause availability problems, you will not find it prior to release. As a result, outages may increase and availability decrease in exchange for the benefit of decreasing time to market.

A second approach is to have releases move through the performance environment sequentially prior to release. While this protects you against potential performance-related outages, it can significantly decrease your delivery velocity. Imagine that you have several releases in the automated delivery queue. If you will perform endurance tests including overnight batch testing, each may need to wait for its own cycle. The result is that an approach meant to be a faster and low-risk introduction to a production environment starts to slow down relative to your old way of doing things.

A hybrid approach is likely to work best, with some level of testing (exercising the code for a short period of time) done in series prior to release and longer-endurance testing happening with batches of releases once a day. This approach allows you to mitigate much of the risk associated with outages, even as you continue to enjoy the time-to-market benefits of continuous delivery. When practicing the hybrid approach, you will likely need at least two performance testing environments: one for sequential testing (in-line and prior to release) and one for prolonged-endurance testing (post release).

A more detailed analysis involves graphing the data for visual reference. Sometimes, it is much easier when data is graphed on line, bar, or pie charts to recognize anomalies or differences. Although these may or may not be truly significant, such graphs are generally quick ways of making judgments about the release candidate.

A further detailed analysis involves performing statistical analysis on the data. Statistical tests such as control charts, t-tests, factor analysis, main effects plots, analysis of variance, and interaction plots can all be helpful. These tests help identify the factors causing the observed behavior and help to determine whether you should be concerned about their overall effect.

In the case of continuous delivery, failure of the automated performance test cases (e.g., on a percentage of capacity loss for a given attribute such as queries per second) should either stall the release for evaluation or deliver the release but open a work ticket for analysis. You might decide that you are willing to accept an automated release if the change in performance falls below one threshold (e.g., 2%) and that the release should be stalled for evaluation above that threshold.

The goal of sharing is to ensure that each item or anomaly from the report gets worked to closure. Closure may occur in one of at least two ways. The first case is to identify the anomaly as an expected outcome of the changes. In this case, the engineer responsible for the explanation should be able to support why the performance deviation is not only expected but actually warranted (as in the case where the increase in revenue will offset the resulting increase in cost). The second case is for a bug to be filed so that the engineering team can investigate the issue further and ideally fix it. It is entirely possible that more tests (with the help of engineering) may need to be run to make a solid business case for no action being taken or to fix a possible bug. In the case of continuous delivery workflows, all reporting should be automated.

Performance testing covers a broad range of testing evaluations, but they share a focus on the necessary characteristics of the system rather than on the individual materials, hardware, or code. Concentrating on ensuring the software meets or exceeds the specified requirements or service level agreements is what performance testing is all about.

Although several different methods of stress testing exist, the two most commonly used are positive testing and negative testing. In positive testing, the load is progressively increased to overwhelm the system’s resources. In negative testing, resources such as memory, threads, or connections are taken away. Besides determining the exact point of demise or (in some instances) the degradation curve of the application, a major purpose of stress testing is to drive the application beyond its capacity to make sure that when it fails, it can recover gracefully. This approach tests the application’s recoverability.

An extreme example of negative stress testing is Netflix’s Chaos Monkey. Chaos Monkey is a service that runs on Amazon Web Services (AWS). It seeks out auto-scaling groups (ASGs) and terminates virtual machines (instances within AWS) for each of the groups. Netflix has taken it a step further with Chaos Gorilla, another service that shuts down entire Amazon Availability zones to make sure healthy zones can successfully handle system load with no impact to customers. The company does this to understand how such a loss of resources will impact its solution. According to the Netflix blog, “Failures happen and they inevitably happen when least desired or expected.”² The idea is that by scheduling the Chaos Monkey or Chaos Gorilla to “work” during the normal business day, the team can respond to, analyze, and react to issues that would otherwise catch them by surprise in the middle of the night.

2. Chaos Monkey released into the wild. Netflix Techblog, July 30, 2012. http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html.

Why do we call this example extreme? The folks at Netflix run it in their production environment! Considered one way, this is really a parallel evolution of continuous delivery practices. To be successful in emulating Netflix, a company first needs to ensure that it has all of the incident and crisis management procedures identified earlier in this book nailed. The good news is that should you be interested, the kind folks at Netflix have released the Chaos Monkey into the wild under the project Simian Army on GitHub. Go check it out!

Stress testing can help identify baselines, ease of recoverability, and system interactions, in addition to the results of negative testing. Broadly speaking, ease of recoverability and baselines are considered positive testing. Stress testing to establish a baseline helps to identify the peak utilization possible or degradation curve of a product. Recoverability testing helps to understand how a system fails and recovers from that failure. Testing systems’ interactions attempts to ensure that some given functionality continues to work when one or more other services are overloaded.

The first factor to use in determining which services should be selected for stress testing is the criticality of each service to the overall system performance. If there is a central service such as a database abstraction layer (DAL) or user authorization, it should be included as a candidate for stress testing because the stability of the entire application depends on this service. If you have architected your application into fault-tolerant “swim lanes” (discussed in depth in Chapter 21, Creating Fault-Isolative Architectural Structures), you are likely to still have core services that have been replicated across the lanes.

The second consideration for determining services to stress test is the likelihood that a service will affect performance. This decision will be influenced by knowledgeable engineers but should also be somewhat scientific. You can rank services by the usage of processes such as synchronous calls, I/O, caching, locking, and so on. The more of these higher-risk processes that are included in the service, the more likely they are to have an effect on performance.

The third factor for selecting services to be stress tested is identification of services during load testing as a bottleneck. With any luck, if a service has been identified as a bottleneck, this constraint will have already been fixed—but you should recheck it during stress testing.

Collectively, these three factors should provide you with strong guidelines for selecting the services on which you should focus your time and resources to ensure you get the most out of your stress testing.

The important thing is to be methodical, record as much data as possible, and create a significant failure of the service. Stress can be placed upon the service in a variety of manners, such as by increasing the number of requests, shortening any delays, or reducing the hardware capacity. An important factor to remember is that loads, whether identified in production or in load testing, should always be scaled to the appropriate level based on the differences in hardware between the environments.

It is important to spend some time pondering the appropriate stress testing environment, just as you did for the performance testing environment. Understand the tradeoffs that you are making with each difference between your production and testing environments. Balance the risks and rewards to make the best decisions in terms of what the environment should look like and how useful the tests will be. Unlike with performance testing, you need not be concerned about how the environment affects continuous delivery with stress testing. Generally stress testing is a point-in-time phenomenon and need not be performed prior to each release.

Some items that might be important to consider as potential data points are the results or behavior of the service, response time, CPU load, memory usage, disk usage, thread deadlocks, SQL count, transactions failed, and so on. The results of the service are important in the event that the application provides erroneous results. Comparison of the expected and actual results should be considered as a very good measure of the behavior of the service under load.

The other significant determinant in how the data should be analyzed is the objectives or goals determined in step 1. If the objective is to establish a baseline, little analysis needs to be done—perhaps just enough to validate that the data accurately depicts the baseline, that it is statistically significant, and that it has only common cause variation. If the objective is to identify the failure behavior, the analysis should focus on comparing the results from the case where the load was below the breaking point and the case where the load was above it. This will help identify warning signs of an impending problem as well as the emergence of a problem or inappropriate behavior of the system at certain loads. If the objective is to test for the behavior when the resource is removed completely from the system, the analysis will probably want to include a comparison of response times and other system metrics between various resource-constrained scenarios and post load to ensure that the system has recovered as expected. For the interactivity objective, the data from many different services may have to examined together. This type of examination might include multivariate analyses such as principal component analysis or factor analysis. The objective identified in the very first step will be the guidepost for this analysis.

A successful analysis will meet the objectives set forth for the tests. If a gap in the data or missing test scenario prevents you from completing the analysis, you should reexamine your steps and ensure you have accurately followed the eight-step process outlined earlier.

We need to take a break in our description and praise of the stress testing process to discuss the downside of such testing. Although we encourage the use of stress testing, it is admittedly one of the hardest types of testing to perform properly—and if you don’t perform it properly, the effort is almost always wasted. As we discussed in step 4 about setting up the proper environment, if you switch classes of storage or processor speeds, these changes can completely destroy the validity of the test results. Unfortunately, establishing the appropriate environment is a relatively easy step to get correct, especially when compared to the sixth step, creating the load. Load creation is by far the hardest task and the most likely place that you or your team will mess up the process and cause erroneous or inaccurate results. It is very, very difficult to accurately capture and replay real user behavior. As discussed earlier, doing so often necessitates synchronization of data within caches and stores, such as databases or files, because inconsistencies will exercise different code paths and render inaccurate results. Additionally, creating a very large load itself can often be problematic from a capacity standpoint, especially when you’re trying to test the interactivity of multiple services.

Given these challenges, we caution you about using stress testing as your only safety net. As we will discuss in the next chapter on go/no-go decisions and rollback, you must have multiple relief valves in the event that problems arise or disaster strikes. We will also cover this subject more fully in Part III, “Architecting Scalable Solutions,” when we discuss how to use swim lanes and other application-splitting methods to improve scalability and stability.

As we stated at the beginning of this section, the purpose of stress testing is to determine an application’s stability when subjected to above-normal loads. It is clearly differentiated from load testing, where the load is only as much as specified; in stress testing, we go well beyond this level to the breaking point and watch the failure and the recovery of the service or application. We recommend an eight-step stress testing process starting with defining objectives and ending with analyzing the data. Each step in this process is critical to ensuring a successful test yielding the results that you desire. As with our other processes, we recommend starting with this one intact and adding to it as necessary for your organization’s needs.

Additionally, stress testing is necessary to ensure that the expected breakpoint or degradation curve is still at the same point as previously identified. It is possible to leave the normal usage load unchanged but decrease the total load capacity through new code paths or changes in logic. For instance, an increase in a data structure lookup of 90 milliseconds would likely go unnoticed if included in the total response time for a user’s request. If this service is tied synchronously to other services, however, as the load builds, hundreds or thousands of 90-millisecond delays will add up and decrease the peak capacity that services can handle.

When we talk about change management, as defined in Chapter 10, Controlling Change in Production Environments, we are really discussing more than the lightweight change identification process for small startup companies. That is, we are referring to the fuller-featured process by which a company attempts to actively manage the changes that occur in its production environment. We have previously defined change management as consisting of the following components: change proposal, change approval, change scheduling, change implementation and logging, change validation, and change efficacy review. Performance testing and stress testing augment this change management process by providing a practice implementation and—most importantly—a validation of the change. You would never expect to make a change without verifying that it actually affected the system the way that you think it should, such as by fixing a bug or providing a new piece of functionality. As part of performance and stress testing, we validate the expected results in a controlled environment prior to production. This additional step helps ensure that when a change is made in production, it will also work as it did during testing under varying loads.

The most significant factor that we should consider when relating performance testing and stress testing to scalability is the management of risk. As outlined in Chapter 16, Determining Risk, risk management is one of the most important processes when it comes to ensuring your systems will scale. The precursor to risk management is risk analysis, which attempts to calculate the amount of risk associated with various actions or components. Performance testing and stress testing are two methods that can significantly decrease the risk associated with a particular service change. For example, if we were using a failure mode and effects analysis tool and identified a failure mode of a particular feature as being an increase in query time, the mitigation recommended could be to test this feature under actual load conditions, as with a performance test, to determine the actual behavior. This could also be done with extreme load conditions, as with a stress test, to observe behavior above normal conditions. Both of these tests would provide much more information with regard to the actual performance of the feature and, therefore, would lower the amount of risk. Clearly, these two testing processes are powerful tools when it comes to reducing, and thereby managing, the amount of risk within the release or the overall system.

From these three areas—headroom, change control, and risk management—we can see the inherent relationship between successful scalability of a system and the adoption of the performance and stress testing processes. As we cautioned previously in the discussion of stress testing, the creation of the test load is not easy, and if done poorly can lead to erroneous data. However, this challenge does not mean that it is not worth pursuing the understanding, implementation, and (ultimately) mastery of these processes.

For the stress testing process, we defined an eight-step process. These were the basic steps we felt necessary to have a successful process. You can add other steps as necessary to ensure a proper fit of this process within your organization.

We concluded this chapter with a discussion of how performance testing and stress testing fit with scalability. Based on the relationship between these testing processes and three factors—headroom, change control, and risk management—these processes also are directly responsible for scalability.

• The goal of performance testing is to identify, document, and (where possible) eliminate bottlenecks in the system.

• Load testing is a process used in performance testing.

• Load testing is the process of putting load or user demand on a system so as to measure its response and stability.

• The purpose of load testing is to verify that the application can meet a desired performance objective, often one specified in a service level agreement.

• Load and performance testing are not substitutes for proper architecture.

• The seven steps of performance testing are as follows:

1. Establish the criteria expected from the application.

2. Establish the proper testing environment.

3. Define the right tests to perform.

4. Execute the tests.

5. Analyze the data.

6. Report to the engineers, if they are not organized into Agile teams.

7. Repeat as necessary.

• Stress testing is a process that seeks to determine an application’s stability when subjected to above-normal loads.

• Stress testing, as opposed to load testing, goes well beyond the normal traffic—often to the breaking point of the application—and observes the behaviors that occur.

• The eight steps of stress testing are as follows:

1. Identify the objectives of the test.

2. Choose the key services for testing.

3. Determine how much load is required.

4. Establish the proper test environment.

5. Identify what must be monitored.

6. Create the actual test load.

7. Execute the tests.

8. Analyze the data.

• Performance testing and stress testing impact scalability through the areas of headroom, change control, and risk management.