There’s no universally correct set of release criteria for all projects. Your release criteria must correlate with the project’s success criteria (see Chapter 1). That is, whatever indicators you choose to tell you when the product is ready to ship must provide some confidence that the project will be an overall success. To able to measure and monitor the release criteria during development, you need to identify surrogate indicators of the product’s ultimate performance in the field. You might define separate release criteria for a series of releases that contain increasingly demanding functionality, reliability, or performance requirements.
Note
Writing weak release criteria that don’t align with achieving the project’s business objectives. These can give a false sense of confidence that the project will be a success.
By way of illustration, Table 5-1 lists release criteria for two products described in the public domain. One is the next release of a compiler and the second is a revised internetwork operating system that runs on a family of routers. These criteria reflect important ideas, but some of them are vague: "extensive customer exposure," "high level of customer satisfaction." Such fuzzy statements are subject to individual interpretation and can’t be evaluated precisely to judge whether they’re satisfied. Some other ambiguous words to avoid are: acceptable, adequate, appropriate, comprehensive, extensive, high, improved, low, reasonable, robust, seamless, and sufficient (Wiegers 2003).
Table 5-1. Some Actual Product Release Criteria
Internetwork Operating System | |
|---|---|
|
|
Consultant Johanna Rothman recommends we write product release criteria that are SMART (Rothman 2002). This acronym reminds us that the criteria should be:
Specific (not vague or imprecise)
Measurable (not qualitative or subjective)
Attainable (not unrealistic stretch goals that you’ll never achieve)
Relevant (related to customer needs and your business objectives)
Trackable (something we can monitor throughout the project)
These keywords are slightly different from the SMART criteria we encountered for business objectives in Chapter 4, but the basic idea is the same. Try applying these five criteria to the two examples shown in Table 5-1. You’ll see that the release criteria for the internetwork operating system don’t make the grade for several of the SMART conditions. I can anticipate energetic debates of whether the product has received enough customer exposure, what constitutes a "serious" defect, or whether the internal testing was sufficiently comprehensive. The criteria shown for the new compiler are easier to evaluate, provided the meaning of the word "supported" is made clear in every case. In general, I avoid using the word "support" in any kind of requirement—and release criteria define a type of requirement—because it’s so ambiguous and context-dependent.
Table 5-2 shows a third example, release criteria for an open-source personal information management system called Chandler, developed by the Open Source Applications Foundation (http://wiki.osafoundation.org/Projects/OsafQAProcess). Most of these are clear enough that an objective observer could determine whether each was satisfied.
Table 5-2. Actual Release Criteria for a Personal Information Management System
Area | Release Criteria for the Area |
|---|---|
Features | All features scheduled for the release are code complete and all feature development is frozen. |
Quality Assurance | Functional and integration tests have run successfully. |
Certification | Testing building Chandler fully from the sources on all approved platforms, including software and hardware. Also testing the debug and end user’s distributions on all the approved platforms. |
Bugs | No blocker, critical or major bugs outstanding in Bugzilla [a defect-tracking system] for the target milestone set for this release. |
Documentation | Full set of documentation for all the features delivered in the release. Also documentation of the features that were originally in the design but didn’t make it in the release. |
Performance Certification | For the performance criteria decided for the release, making sure the product performance conforms to that. |
Consider the perspectives of various stakeholder groups to avoid disconnects and conflicts. User-developed acceptance criteria provide a valuable input to the release criteria you select. Developers often don’t know what is really important to the user, so work with key user representatives to make sure their perspectives are incorporated.
Note
Conflicting release criteria. All team members need to work toward a common set of goals and make the appropriate tradeoff decisions.
The following sections suggest some possible release criteria, grouped into several categories. Select conditions from these lists that will help you determine when it’s time to take your product out of the oven and serve it. Of course, not all of these will apply to any single project and you might well have other criteria that are pertinent to your situation. Tailor these generic examples to suit your own project. Keep the SMART characteristics in mind as you write them, so your release criteria are as precise and meaningful as possible.
Everyone wants high-quality software, but the definition of "quality" has been debated for decades. Quality is a complex and multidimensional set of product characteristics. Releasing a buggy product prematurely can lead to high operational costs, user disappointment, poor product reviews, excessive maintenance costs, product returns, and even lawsuits. As one indicator of quality, you can monitor the number and types of defects discovered during development and testing. Of course, you don’t know how many defects you haven’t discovered yet. You only know how many you’ve found so far, how many you’ve fixed, and how fast the team is discovering new ones. It would be nice to estimate the likely frequency of software failure once the product is in the users’ hands. However, the number of pre-release defects found is not a reliable predictor of the number of post-release failures to be expected (Sassenburg 2003).
You’re never going to get perfect software, so you need some indicators that will suggest when the software is good enough to deliver. Trends in defect discovery rates can provide an indication of how many defects are likely to remain undetected in the product. The field of software reliability engineering provides some assistance. Various models have been developed to estimate the number of remaining defects from the rate at which new ones are being discovered through testing (Musa, Iannino, and Okumoto 1987; Walsh 1993). Some of the remaining defects will be inconsequential but others could be showstoppers. You just don’t know.
If quality is a success driver for your project, consider adapting defect-related release criteria from the following general statements:
There are no open defects with severity 1 or 2 on a 4-level severity scale.
The number of open defects has decreased for X weeks and the estimated number of residual defects is acceptable.
All errors and warnings reported by the compiler, source code static analyzers, and run-time analyzers have been corrected.
Specific known problems from previous releases are corrected and no additional defects were introduced while making the corrections.
Most software teams rely heavily on testing of various types to discover defects, although other quality practices, such as peer reviews (Wiegers 2002) and code analyzer tools, are also valuable. I’m not sure that most testers really know when they’re done testing or can describe the state of the product’s quality when they declare testing complete. People sometimes stop testing because they’re out of testing time. If you have historical data about the typical defect densities (defects per thousand lines of code, or KLOC) from previous projects you can estimate the number of defects likely to be hiding in your next product. You might decide to stop testing when the estimated number of undiscovered defects is acceptable, when testing has continued for a predefined number of hours with no failures, and when testing has continued for at least two weeks after reaching the specified estimate for remaining defects (Grady and Caswell 1987). Some other testing-related release criteria are:
All code compiles, builds, and passes smoke tests on all target platforms.
100 percent of integration and system tests were passed.
Specific functionality has passed all system and user acceptance tests (for example, the normal flows and associated exception handlers for the most commonly executed use cases).
Test case execution specified in the test plan is complete for all documented functional requirements.
Predetermined testing coverage targets for code or requirements (for example, functional requirements, use case flows, or product features) have been achieved.
Combining a consideration of testing-related and defect-related factors, one author proposes that your software is ready to ship when the following conditions are all satisfied (Foody 1995):
You have a full regression test suite that covers 100 percent of the functionality and 80 percent of the branches.
You have no known severity 1 or severity 2 defects.
The density of known remaining defects is less than 0.5 defects per KLOC.
The rate of discovery of new defects is less than 40 defects per 1000 hours of testing.
The mean time between failures is at least 100 hours. (That is, your interactive application runs continuously for at least 100 hours without something going wrong.)
You’ve completed stress, configuration, installation, localization, usability, and naïve-user testing.
These numbers might not be exactly appropriate for your situation, but the idea of quantifying several quality-related criteria provides valuable input into determining the right time to release. It’s important to have multiple, complementary criteria. As an illustration, if no tests are executed in a certain period, obviously no defects will be found through testing. So the rate of defect discovery alone is not a dependable indicator of quality.
Another way to think about software quality is in terms of various attributes that characterize the product’s behavior. Often called "the ilities" for short, these quality attributes include reliability, security, integrity, usability, portability, maintainability, efficiency, robustness, interoperability, and others (Wiegers 2003). You could assess the product’s readiness for release partly based on whether it satisfies its critical specified quality attribute requirements. Here are some examples:
Quantitative performance goals are satisfied on all platforms.
Reliability goals are satisfied. For example, the mean time between failures is at an acceptable level and is increasing with each build that goes through system test.
Relevant corporate security policies and requirements have been satisfied.
Specified conditions have been met that will enable the product to pass a necessary certification audit or evaluation.
Every product requires a minimal set of functionality that provides appropriate value to its customers. Careful prioritization lets your team deliver a useful product as quickly as possible, deferring less urgent and less important requirements to later releases. Consider the following requirements-related release criteria:
All high-priority requirements committed for this product release are implemented and functioning correctly.
Specified customer acceptance criteria are satisfied.
Any requirements for accessibility by disabled users are satisfied.
All localization and globalization tests to ensure that the software runs properly in different languages were passed.
Specified legal, contractual, standards-compliance, or regulatory goals are met.
All functional requirements trace to test cases.
The discipline of configuration management is akin to managing the parts list for your product. Configuration management includes activities to uniquely identify the components that are assembled into the product, to manage and control versions of these components, and to assemble (build) them properly into the deliverable files and documents. The following configuration management-related release criteria apply to virtually every software project:
The product can be built reproducibly on all target platforms.
A physical configuration audit confirmed that the correct versions of all components are present.
The product installs correctly on all target platforms, including reinstall, upgrade from a previous release, uninstall, and recovery from aborted or failed installation attempts.
The release image and media have been scanned for viruses and other malware.
You might think the software is done but that doesn’t mean the world is ready for it. Make sure you’ve lined up the other elements necessary for a smooth rollout and implementation. Think about factors such as these:
Release notes have been prepared that identify defects corrected in this release, enhancements added, and any capabilities removed.
Software release and support procedures are published and understood by affected stakeholders.
Known uncorrected defects are logged in the project’s defect-tracking system.
The support function is ready to receive and respond to customer problem reports.
The operating environment has all necessary infrastructure in place to execute the software.
Software manufacturing and distribution channels are ready to receive the product. Manufacturing is especially important when the product includes software embedded into hardware components.