Benchmarking is relevant in software development on all levels, from OEM or Java Virtual Machine vendors to developers of standalone Java applications. It is too often the case in software development that while the functionality goals of an application are well specified, no performance goals are defined at all. Without performance goals and benchmarks in place to track the progress of those goals, the end result may be stable but completely unusable. During our careers we have seen this many times, including during the development of business critical systems. If a critical performance issue is discovered too late in the development cycle, the entire application may need to be scrapped.

Typically, in order to avoid these kinds of embarrassments, performance must be a fundamental requirement of a system throughout the entire software development process, and it needs to be verified with regular benchmarking. The importance of application performance should never be underestimated. Inadequate performance should be treated as any other bug.

An application that is developed without a good Quality Assurance (QA) infrastructure in place from day one is likely to be prone to bugs and instabilities. More specifically, an application without good functional unit tests that run whenever new code is checked in, is likely to break, no matter how well the new code has been reviewed. This is conventional wisdom in software engineering.

The first and foremost purpose of regression testing is to maintain stability. Whenever a new bug is discovered along with its fix, it is a good practice to check-in a regression test in the form of a reproducer, possibly based on code left over from debugging the problem. The ideal reproducer is a program with a few lines of code in a main function that breaks whatever was wrong with the system, but reproducers can also be more complex. It is almost always worth it to spend time turning a more complex reproducer into a regression test. Keeping the regression test running upon new source code check-ins will prevent the particular problem from recurring, potentially saving future hours of debugging an issue that has been fixed at least once already. Naturally, all functionality tests don't easily break down into simple regression tests or self-contained reproducers. Extensive runs of large, hard-to-setup applications are still often needed to validate stability.

The other side of regression testing is to maintain performance. For some reason this has not been as large a part of the conventional wisdom in software engineering as functionality testing, but it is equally important. New code might as easily introduce performance regressions as functional problems. The former is harder to spot, as performance degradations normally don't cause a program to actually break down and crash. Thus, including performance tests as part of the QA infrastructure makes a lot of sense. Discovering too late that application performance has gone down requires plenty of detective work to figure out exactly where the regression happened. Possibly this involves going over a large number of recent source code check-ins, recompiling the application at different changes and rerunning the affected application until the guilty check-in is spotted. Thus, integrating simple performance regression tests into the source code check-in criteria to guard against regressions makes as much sense as with unit tests or regression tests for functionality.

Performance regression testing is also good for detecting unexplained performance boosts. While this might seem like a good thing, sometimes this indicates that a bug has been introduced, for example important code may no longer be executed. In general, all unexpected performance changes should be investigated and performance regression tests should trigger warnings both if performance unexpectedly goes up as well as if it goes down.

A cardinal rule when regression testing performance is to have as many points of measurements as possible, in order to quickly detect regressions. One per source control modification or check-in is ideal, if the infrastructure and test resources allow for it.

Let us, for a moment, disregard large and complex tests in the QA infrastructure and concentrate on simple programs that can serve as unit tests. While a unit test for functionality usually takes the form of a small program that either works or doesn't, a unit test for performance is a micro benchmark. A micro benchmark should be fairly simple to set up, it should run only for a short time, and it can be used to quickly determine if a performance requirement is reached. We will extensively discuss techniques for implementing micro benchmarks as well as more complex benchmarks later in this chapter.

Another reason for keeping a collection of relevant benchmarks around is that performance is a difficult thing to quantify—are we talking about high throughput or low latency? Surely we can't have both at the same time, or can we? If so, how can we ensure that our application is good enough in both these problem domains?

While running an application with a large set of inputs and workloads is a good idea for the general QA perspective, it might provide too complex a picture to illustrate where the application performs well and where it requires improvements.

A lot of trouble can be avoided if a program can be broken down into sub-programs that can be treated as individual benchmarks and if it can be made sure that all of them perform well. Not only is it easier to understand the different aspects of the performance of the program, it is also easier to work on improving these aspects on a simpler problem domain that only measures one thing at a time. Furthermore, it is also simpler to verify that code optimizations actually result in performance improvements. It should be common sense for an engineer that the fewer factors that are affected at once, the easier it is to measure and draw conclusions from the results.

Also, if a simple and self-contained benchmark correctly reflects the behavior of a larger application, performance improvements to the benchmarks will most likely be applicable to the larger application as well. In that case, working with the benchmark code instead of with the complete application may significantly speed up the development process.

Finally, a large number of industry-standard benchmarks for various applications and environments exist on the Internet. These are useful for verifying and measuring performance in a specific problem domain, for example, XML processing, decoding mp3s, or processing database transactions.

Industry benchmarks also provide standards with which to measure the performance of an application against competing products. Later in this chapter, we will introduce a few common industrial benchmarks, targeting both the JVM itself as well as Java applications on various levels in the stack.

Being the world leader on a recognized benchmark makes for good press release material and excellent marketing.