4 Test Policy and Test Handbook
The purpose of the test policy and test handbook is to provide a test definition that can be applied to all projects in an organization. Their purpose is to standardize and simplify the creation of test plans and test schedules, gather best practices, and provide them to future projects.
4.1 Quality Policy and Test Policy
Usually, an organization producing (software) systems puts certain demands on the quality of its products. These demands are based on the desire to capture market share, win customers, and push sales—ideally, in order to increase profit. Often the quality demands that a company expects of itself are molded into a publicly available, documented quality policy.
The →quality policy expresses the particular importance that an organization attaches to “quality” and the demands it makes on the quality of its products, services, and processes. It forms the basis from which it derives its guidelines, specifications, and instructions for the implementation of these demands and the practices for the verification of quality actually achieved. Furthermore, the quality policy is expected to contain the organization’s commitment to continuous improvement.
Quality policy
Consequently, the quality policy constitutes the basis for the work of the quality management (QM) staff or department and of all the quality assuring bodies within the company. It goes without saying that an organization’s senior management must bear full responsibility for its corporate quality policy as it is the only authority that can ensure the policy’s appropriateness to the organization’s business objectives and its proper dissemination, comprehension, and compliance within the entire organization.
Senior management must also ensure that the policy is reviewed to confirm its continuing appropriateness [ISO 9001].
Excerpt from the quality policy of ARM [URL: ARM]
ARM Quality Policy:
Customer Focused
Satisfy our customers’ needs and expectations
Make commitments we fully understand and believe we can meet
Meet all commitments to customers on time
Performance Driven
Verify that our products and services meet agreed requirements
Monitor, benchmark and continuously improve our business, products and services, organization and employees’ performance
Achieve ARM’s Mission and Goals
Sustain and develop business growth and Intellectual Property
Consequently, the testing of software-based products or processes as a quality assurance method must likewise be performed based on the organization’s quality policy. With regard to the test process, this corporate philosophy is called →test policy.
The test policy comprises the following components:
Contents of a test policy
The definition of the term “test” in the organization: Which problems are supposed to be solved by testing? What relative importance is given to test in the overall development process? How is the process delimited against other quality assurance measures?
A description of the test processes: Which phases and subtasks does the test process consist of? Which roles are involved, which entry and exit documents are associated with which subtasks, and which test levels need to be considered?
Specifications for test evaluation: Which metrics are to be used to ensure test effectiveness in the project?
The quality level to be achieved: Which quality criteria are to be tested and which quality level is the system required to achieve prior to its release with regard to these criteria?
An approach to test process improvement: At what times and by which means is the quality of the test process to be evaluated and what are the techniques to recognize improvement potentials and derive improvement actions? Which objectives are being pursued?
Like the quality policy, the test policy is to have validity across the entire organization. It is typically drawn up and maintained by a central authority with test competence, such as the quality management staff, →test center, or IT department. It comprises requirements concerning testing of new systems as well as requirements concerning testing within the context of system maintenance. Moreover, in addition to being a corporate directive, the test policy has a second, equally important role:
Testing is a spot-check-oriented technique supposed to create confidence in the quality of the tested systems. Such confidence is increased if the basis and objectives of testing are known, documented, easy to understand, and consistent with corporate quality goals.
Since the test policy is often part of the quality policy, it may be difficult to maintain it as a separate entity, and keeping both together makes as documents the relationship between the two easier to see. In practice, though, rather than keeping quality and test policy as one single document, they are often found under different names or spread over several other, “regulative” corporate documents such as the quality manual according to ISO 9001:2000, process definitions, procedures, and guidelines.
Quality and test policy as documents
The following excerpts from a genuine test policy lead us to ask what is it that makes a “good” test policy:
Excerpts from a test policy
On the significance of testing: “Testing is performed in the organization to verify and validate developed systems. Test activities in the area of validation (testing the system’s usability for the purpose it was designed for) are usability tests and alpha and beta tests. For verification, testable system requirements are defined whose correct implementation can be tested through system tests. Besides testing, the QM manual describes additional quality assurance activities used for validation and verification purposes.”
On the targeted quality level: “For each system requirement we define explicit acceptance criteria ... The task of verification is to ensure that a released system satisfies each functional acceptance criterion fully and all nonfunctional acceptance criteria at least 95% with regard to efficiency (compare with the definition in ISO 9126).”
On the approach to test process improvement: “Test processes are audited as part of our regular project and process audits under ISO 9000:2000 and the following pages to monitor test effectiveness and continuous process improvement. Identified improvement potentials are included in our regular improvement projects based on the TMM. The long-term process improvement objective is to cut down on follow-up defect and quality assurance costs spent on software systems developed in the organization.”
4.2 Bring the Test Policy to Life
The basic structure of the test policy as we have just discussed forms a skeleton to be fleshed out according to the organization’s specific needs. Useful extensions are, for instance, a general statement of corporate attitude toward qualification and further training of testers and the latter’s share in the responsibility for achieving the quality goals.
An essential aspect is the acceptance of the policy by those affected and the feasibility to put the principles defined in it into practice. Thus, a viable test policy is a critical factor for the success of testing. To ensure that this happens, the test policy must, in addition to those components mentioned above, meet the requirements outlined in the following paragraphs.
Relevance to Business Objectives
Implementation of the test policy is largely a matter of motivation. If the requirements defined in the test policy are directly related to the organization’s business objectives, it is much easier for those implementing the policy to fend off the inevitable arguments and discussions about cost and necessity.
Features of a “good” test policy
Be Realistic
Unrealistic requirements or expectations may crop into a test policy if they are not defined by testers. The following is an example of a definition raising exaggerated expectations as to the importance of testing in an organization:
“In our company, testing serves as proof that our products are free from defects at market launch.”
Avoid exaggerated expectations
First of all, testing is no method of proof, and second, testing is based on samples and can therefore never prove the absence of defects. But even technically justified requirements such as “the test must achieve 100% branch coverage” may turn out to be unrealistic in the organizational context and should not be used as a yardstick in the test policy.
Adequate Maturity
No matter how ambitious a test policy may be, it is bound to be ineffective and will fail if chances for its implementation in the organization are slim. A common cardinal error is an overprecise definition of the test processes—for instance, by defining phases and roles that cannot be enforced or efficiently organized. Many organizations introduce into their test policy a test process that relies on test automation as a means of cost reduction without taking into account that the test process is predominantly resting in the hands of testers who aren’t technically trained.
Measurability
During the definition stage, when evaluation criteria are set up and the quality level is determined, the same care must be taken, as with the definition of any other metric, to define criteria (ideally with little effort) that can actually be measured.
The following serves as a negative example of the definition of a quality target: “At the time of its release to the market, a system must not contain any functional defects.”
It is impossible to algorithmically calculate the number of defects in a system; they can at best be statistically estimated. A suitable, because it’s measurable, definition of the targeted quality level would be as follows: “For a newly released system, no more than 100 high priority defects shall be found by customers.” Based on this target, we can estimate how many defects must be found by system test or other test levels before the product can be released.
Liveliness
A test policy is not carved in stone but lives from “feedback” reflecting the success or failure of its implementation. A good test policy incorporates in its test process all necessary feedback mechanisms whose constant application ensures—for instance, in its approach to test process improvement—that the test policy is regularly reviewed and improved in a goal-oriented fashion.
4.3 Test Policy and Test Handbook
The test policy has corporate validity across all departments, projects, test levels, and test phases. As such, it needs to be kept at a relatively abstract level. However, in order to be able to support the implementation of the test policy in concrete cases, a further level of detail is useful: the test handbook.
Test handbook: a concretization of the test policy
The →test handbook takes up the demands set up in the quality policy and addresses those quality risks that may be, and should be, reduced through testing. When a test manager plans his tests in a new project, he can, ideally, draw upon the test handbook as a catalogue for the test activities to be performed.
The concrete realization of these activities is then documented in the project’s test plan (see chapter 5).
First, the test handbook describes all possible test levels and outlines the test activities for each applicable test level. For each test level, the following aspects are considered:
Contents of a test handbook
The entry criteria for the start of the test level (may be identical to the exit criteria of the previous test level; see below)
The test procedure (e.g., top-down, bottom-up, priority driven)
The test specification techniques used to derive the test cases, specifying the system’s quality characteristics to be verified as described, for example, in [ISO 9126]
Test completion criteria and their specifications (i.e., suitable metrics and threshold values)
Exit conditions for process phases (for instance, process documentation to be created and criteria for its release)
Degree of independence of the testing (i.e., whether testing is performed by developers or an independent team)
Standards to be complied with; can be standards pertaining to the area of software testing itself or and more general standards (see chapter 13)
The environment in which software tests are performed; descriptions of reference hardware and software as well as setup procedures in preparation for the tests
The approach toward test automation (e.g., suitable tools and automation methods and decision criteria regarding test automation, such as feasibility and benefit)
The approach to reusability of testware at successive test levels
The approach to retesting and regression test planning (criteria, scope, selection of tests, etc.)
The detailed definition of the test process (outlined in the test policy, including documents and test results such as, for example, test reports)
The measures and metrics to be recorded
The approach for incident management (incident status model, involved roles, incident report format, tools to be used)
The art of preparing a test handbook is in keeping it concrete enough to be of benefit to the user (for instance, by making it easier for him to structure tasks or make decisions) and at the same time not letting it become project specific.
Balance between reusability and concreteness
A pragmatic approach would be to generalize from a test plan that has already proved its worth in a project and make it available to other projects in a first version. Applying the handbook in other projects will then quickly reveal content matter that cannot be applied generally so that it can be deleted.
Another possibility to succeed in this balancing act is to draw up several test handbooks with decreasing scope of application and increasing degree of detail.
For instance, it is possible to write a generic corporate test handbook and then tailor it into further test handbooks for different product ranges with greatly differing requirements regarding test intensity. The motivation factor for this kind of split could be differences in the criticality of the systems developed in the different units.
Like the test policy, a test handbook must regularly be aligned with the evolving test plans and adapted to changing circumstances.
Updating the test handbook
One well-proven approach is to plan regular project audits at the end of test projects during which the effectiveness and implementability of test policy and test handbook in the test plan are systematically evaluated. If necessary, improvement actions are initiated. It is best to define this feedback loop as a fixed component in the test policy and to integrate it in the test handbook as part of the process improvement program.
4.4 Summary
To establish software testing in a company strategically, test policy and test handbook can be used to regulate common aspects of the software testing process.
The strategic approach to testing begins with the definition of an implementable test policy. It is a component of the general corporate quality policy.
The test handbook is derived from the test policy, which contains the generic test requirements of an organization. It addresses the risks, shows the relationship between testing and risk reduction, and describes a process that addresses these risks in compliance with the test policy.
The test handbook serves as a starting point for each test plan and consequently for the implementation of all test activities. It contains descriptions of all applicable test levels.
A mature test handbook also incorporates the approach toward test process improvement in the organization.
The test policy and test handbook are translated into practice by the users, i.e., all those involved in testing. Both documents therefore require feedback and regularly derived improvements. With time, they evolve into valuable means of interproject knowledge transfer.
The test handbook does not necessarily exist as a single document but may be integrated in the QM system and process documentation, or it may be distributed over several documents like a corporate test handbook and some additional test handbooks for specific product ranges.
The latter makes sense if different application areas in the organization require very different test regulations, such as, for instance in sections of the organization that develop safety critical applications.