2 Test Process and Test Tools
This chapter introduces the fundamental test process, its associated activities, and appropriate tool support1.
2.1 Test Process Fundamentals
In order to perform structured tests, a general description of the task as found in most development models (see chapter 3) is not sufficient. Besides integrating testing into the development process it is also necessary to provide a detailed test procedure (see figure 2-1). The development task consists of the process phases test planning and control, analysis and design, implementation and execution, evaluation of the test exit criteria and reporting, as well as test completion activities. Although the presentation and description of the individual tasks suggest a sequential procedure they may of course overlap or be performed in parallel.
2.1.1 Test Planning and Control
Planning a comprehensive task such as testing ought to start as soon as possible in the initial stages of software development.
Resource planning
The role and purpose of testing must be defined as well as all the necessary resources, including staff for task execution, estimated time, facilities, and tools.
The associated specifications are to be documented in the test plan. An organizational structure including test management needs to be in place and ought to be adapted, if necessary.
Figure 2–1 Fundamental test process
Test management is responsible for the administration of the test process, the test infrastructure, and testware. Regular control is necessary to see if planning and project progress are in line. This may result in the need for updates and adjustments to plans to keep the test process under control. The basis for controlling the test process is either staff reporting or relevant data and its evaluation by appropriate tools.
Since exhaustive testing is impossible, priorities must be set. Depending on the risks involved, different test techniques and test exit criteria must be specified when establishing a test strategy. Critical system components must be intensively tested. However, in the case of less critical components a less comprehensive test may suffice or testing may even be waived. The decision must be very well-founded to achieve the best possible allocation of the tests to the “important” parts of the software system.
Determining the test strategy
→Test intensity is determined by the test methods employed and by the intended degree of coverage when executing the test cases. The degree of coverage is one of several criteria for deciding when a test is completed.
Determining test exit criteria
Software projects are often under pressure of time, a fact which must be anticipated during planning. Prioritizing tests causes the most critical software components to be tested first in case not all planned tests can be performed due to time or resource constraints.
Prioritizing tests
Without adequate tools the test process cannot be sufficiently performed. If tools are missing, their selection and procurement must be initiated early in the process.
Tool support
Moreover, parts of the test infrastructure themselves often need to be established, for instance the test environment, in which system components can be executed. They need to be put in place early so that they are available when coding of the test objects is completed.
2.1.2 Test Analysis and Design
The test strategy developed during planning defines the test design techniques to be used. As a first step of test analysis, the test basis needs to be checked to see if all required documents are detailed and accurate enough to be able to derive the test techniques in agreement with the test strategy2. The specification of the test object determines its expected behavior. The test designer uses it to derive the prerequisites and requirements of the test cases.
Verification of the test basis
Depending on the analysis results it may be necessary to rework the test basis so that it can serve as a starting point for the test design techniques techniques defined in the test strategy. For example, if a specification is not accurate enough it may need to be improved. Sometimes it is the test strategy itself which may need to be changed, for instance, if it turns out that the selected test design techniques cannot be applied to the test basis.
During test design, test techniques are applied to identify the respective test cases, which are then documented in the test specification. Ultimately, the →test project or test schedule determines the timing of the test execution sequence and the assignment of the test cases to the individual testers.
When specifying test cases, logical test cases must be defined first. Once this is done, concrete, i.e., actual input and expected output, values may be defined.
Logical and concrete test cases
However, this is done during implementation, which is the next step of the fundamental test process.
Logical test cases can be identified based on the specification of the test objects (black box techniques) or based on program text (white box techniques). Thus, the specification of the test cases may take place at quite different times during the software development process (before or after or parallel to coding, depending on the test techniques selected in the test strategy). In this connection, many of the life cycle models described in the next chapter show only test execution phases (e.g., the general V-model). Test planning and specification activities can and should take place concurrently with earlier development activities, as explicitly pointed out in the W-model or in extreme programming.
Black box and white box techniques
During test case specification the particular starting situation (precondition) must be described for each test case. Test constraints to be observed must be clearly defined. Prior to test execution it needs to be defined in the post-condition which results or which behavior is expected.
Test cases comprise more than just test data
In order to determine the expected results a test oracle is queried which predicts the expected outcomes for every test case. In most cases the specification or the requirements are used as the test oracle to derive the expected results from individual test cases.
Test oracle
Test cases can be distinguished according to two criteria:
Positive and negative test cases
Test cases for testing specified results and reactions to be delivered by the test object (including treatment of specified exceptional and failure situations)
Test cases for testing the reaction of the test objects to invalid or unexpected inputs or other conditions for which “exception handling” has not been specified and which test the test object for robustness
The required test infrastructure to run the test object with the specified test cases is to be established in parallel to the other activities so as to prevent delays in the execution of the test cases. At that point the test infrastructure should be set up, integrated, and also tested intensively.
Setting up the infrastructure
2.1.3 Test Implementation and Execution
In this step of the test process, concrete test cases must be derived from the logical test cases, and executed. In order to run the tests, test infrastructure and test environment must both be implemented and in place. The individual test runs are to be performed and logged.
The actual tests are to be run observing the priorities that we defined earlier. It is best to group individual test cases into test sequences or test scenarios in order to allow for the tests to be run efficiently and to gain a clear structure of the test cases.
Timing and test case sequence
The required test harness must be installed in the test environment before the test cases can be executed.
At the lower test levels, component and integration testing, it makes sense to run automated rather than manual tests (e.g., using JUnit [URL: JUnit])
During test execution an initial check is done to see if the test object is, in principal, able to start up and run. This is followed by a check of the main functions (“smoke test” or acceptance test during entry check of the individual test levels).
Checking main function completeness
If failures occur already at this stage further testing makes little sense.
Test execution must be logged accurately and completely. Based on test protocols, test execution must be traceable and evidence must be provided that the planned test strategy has actually been implemented. The test protocol also contains details concerning which parts were tested when, by whom, to what extent, and with what result.
Tests without a test protocol are useless
With each failure recorded in the test log a decision needs to be made whether its origin is thought to lie inside or outside the test object. For instance, the test framework may have been defective or the test case may have been erroneously specified.
Evaluating the test protocols
If a failure exists it needs to be adequately documented and assigned to a incident class.
Based on the incident class the priority for defect removal is to be determined. Successful defect correction needs to be ascertained: has the defect been removed and are we sure that no further failures have occurred?
Correction successful?
The earlier made prioritization has the effect that the most important test cases are executed first and that serious failures can be detected and corrected early.
Most important tests come first!
The principle of equal distribution of limited test resources over all test objects of a project is of little use since such an approach leads to equally intensive testing of critical and non-critical program parts.
2.1.4 Test Evaluation and Test Report
It needs to be checked if the test exit criteria defined in the plan have been met. This check may lead to the conclusion that test activities may be considered completed but may also show that test cases were blocked and that not all planned test cases could be executed. It may also mean that additional test cases are required to meet the criteria.
Test completion reached?
Closer analysis, however, may reveal that the necessary effort to meet all exit criteria is unreasonably high and that further test cases or test runs had best be eliminated. The associated risk needs to be evaluated and taken into account for the decision.
If further tests are necessary, the test process must be resumed and the step has to be identified from where test activities are to be resumed. If necessary, planning must be revised as additional resources are required.
Besides test coverage criteria, additional criteria may be used to determine the end of the test activities (see chapter 11).
Test cycles develop as a result of observed failures, their correction, and necessary retesting. Test management must take such correction and test cycles into account in their planning (see also the W-model in section 3.4). Otherwise, project delays are the rule. It is rather difficult to calculate the effort needed for the test cycles in advance. Comparative data from earlier, similar projects or from already completed test cycles may help.
Allow for several test cycles
In practice, time and cost often determine the end of testing and lead to the termination of test activities.
Exit criteria in practice: time and cost
Even if during testing there is more budget spent than planned, testing as a whole does cause savings through the detection of failures and subsequent correction of software defects. Defects not detected here usually cause considerably higher cost when found during operation.
At the end of this activity of the test process, a summary report must be prepared for the decision makers (project manager, test manager, and customer, if necessary) (see also [IEEE 829]).
Test report
2.1.5 Completing the Test Activities
Unfortunately, in practice, the closing phase of the test processes is mostly neglected. At this stage, the experiences gained during the test process should be analyzed and made available to other projects. In this connection, the presumed causes of differences between planning and implementation are of particular interest.
Learning from experience
A critical evaluation of the activities performed in the test process, taking into account effort spent and the achieved results, will definitely reveal improvement potential. If these findings are documented and applied to subsequent projects in an understandable manner, continuous process improvement has been achieved. Chapter 7 will take a closer look at the models for analysis, evaluation, and test process improvement.
A further finishing activity is the “conservation” of the testware for future use. During the operational use of software systems, hitherto undetected failures will occur despite all previous testing, or customers will require changes. In both cases this will lead to revised versions of the program and require renewed testing. If testware (test cases, test protocols, test infrastructure, tools, etc.) from development is still available, test effort will be reduced during the maintenance or operational phases of the software.
Testware “conservation”
2.2 Test Tools
The following section provides an overview over different types of test tools3. Tool types are comprehensively described in Software Testing Foundations ([Spillner 07, chapter 7]). A closer look is taken at tools that support test management. It is particularly important for the test manager to learn how to select and use such tools (see chapter 12).
Short overview
There are many tools supporting or automating test activities, all of which are known as CAST tools (Computer Aided Software Testing), analgous to CASE tools (Computer Aided Software Engineering).
CAST tools
Depending on which activities or phases in the test process are supported, we may distinguish between different tool types.
As a rule, not all available test tools are applied in a project. However, the test manager should know available tool types in order to be able to decide if and when to use a tool efficiently in a project.
The following sections describe the various functions provided by the different tool types.
2.2.1 Tools for Management and Test Control
Planning and control are the first activities in the test process. The respective test management tools offer mechanisms for easy capturing, prioritizing, cataloguing, and administration of test cases. They allow test case status tracking, i.e., they document and evaluate if, when, how often, and with which result a test case has been executed. Moreover, they may be used to support resource and schedule planning for the tests. The test manager can plan the tests and remain informed at all times about the status of hundreds or thousands of test cases.
Test management tool
In addition to these core tasks, test management tools offer support for tasks and activities such as:
Support for advanced management tasks
Requirements-based testing: system requirements are linked with those tests that check the corresponding requirement. Different consistency checks can be performed, for instance to see if for each requirement at least one test case has been planned.
Defect management: tool support is indispensable for the management of problem or incident reports. Capturing, administration, and statistical evaluation of incident reports should not be done manually, since this is simply too error-prone. Tools help the test manager to be kept informed about the actual project at all times.
Preparing test reports and test documents: both test management and incident management tools provide extensive analysis and reporting features, including the possibility to generate complete test documentation (test plan, test specification, test report) from their data repository.
2.2.2 Tools for Test Data and Test Script Specification
So-called test (data) generators can support the test designer in generating test data. There are several approaches, depending on the test basis used:
Test data and test script generators
Database-based test data generators create test data on the basis of database schemas or database content.
Code-based test data generators analyze the source code to derive the test data. Target or expected values, however, cannot be derived.
Interface-based test data generators derive test data through identification of interface parameter domains (for example, using equivalence class partitioning and boundary value analysis). Here, too, the problem exists that target values cannot be generated.
Specification-based test data generators derive test data and associated target values from a formal specification.
Model-based test generators derive test scripts from formal models, starting, for instance, from a UML sequence diagram specifying the call sequences of methods.
2.2.3 Tools for Static Testing
Static checks can be carried out on design documents, given the availability of a formal notation, and on (also only partially available) source code, i.e., even before executable program (parts) are available. Tools supporting the static test help to detect defects and discrepancies already in the early phases of the development process. Therefore a test manager should consider using these tools.
Static analysis
Static analyzers provide measures of miscellaneous characteristics of the program code which can be used to identify complex and hence defect-prone or risky code sections. Violations of programming guidelines, broken or invalid links in website contents and many other anomalies or discrepancies can be analyzed statically.
Model checkers analyze specifications if they are available in a formal notation or as a formal model. For example, they can find missing states, missing state transitions, and other inconsistencies in the state model to be checked.
Furthermore, there are tools to support reviewing and which help in the planning, execution, and evaluation of review results.
2.2.4 Tools for Dynamic Testing
Test tools for automating test execution relieve the tester from carrying out unnecessary mechanical tasks. The tools supply the test object with test data, record the test object’s reactions, perform a comparison with the expected reactions, and log the test execution.
Tool support for functional tests
In the narrow sense, a debugger is not a test tool but is very useful for root cause analysis and for enforcing exception handling in the program code.
Test drivers and test bed generators offer a mechanism to address test objects via their application programming interface (API), or via another interface not directly accessible to the end user, such as, for example, the Ethernet, serial interface, and so on.
Simulators can be used to emulate an operating environment. They are particularly used for testing embedded software if the target system is not yet available, or if testing in the target system is very expensive or if it requires a disproportionally high effort.
Test robots (capture and replay tools) record all input that is manually performed inputs (keyboard inputs, mouse clicks) during a test session and save them in a test script. The recorded test can be automatically repeated by replaying the test script as often as one likes.
Comparators are used to identify differences between expected and actual results. They are typically included in other tools.
Dynamic analyzers acquire additional information during program execution, for instance on allocation, usage, and release of memory (memory leaks, pointer allocation, pointer arithmetic problems, and so on).
Coverage analyzers provide structural test coverage values measured at code level during test execution (see also chapter 11).
Besides tools that support functional testing there are also tools for testing non-functional features of the test objects:
Tool support for non-functional tests
Load and performance test tools generate a synthetic load, e.g., database queries, user transactions, or network traffic, for the execution of volume, stress, and performance tests.
Monitors are used to support tests and analysis in that they identify and evaluate necessary data. They are typically integrated in load and performance test tools.
Tools for checking access and data security analyze possible security gaps in the test object.
2.2.5 Constraints to be Considered
Creative test activities can be supported by tools. The mechanical test execution can be automated, reducing the test effort or allowing the execution of more test cases without spending any additional effort. More test cases, however, do not necessarily mean better tests.
Tool use and test process
Without a well-established test process and adequate test methods, tools cannot achieve the desired cost reduction. The introduction and efficient use of tools requires a thorough evaluation of the test processes and accompanying process improvement activities (see also chapters 7 and 12).
On the other hand, the economic execution of the test processes can only be achieved with appropriate tool support; for instance, to be able to execute and evaluate many test cases within an adequate time frame.
The test manager must be aware of all these constraints and must act accordingly.
2.3 Summary
Testing must be divided into individual process steps. A fundamental test process is divided into the following steps:
Test planning and control: Define required resources (staffing, schedule, tools), define the test strategy together with the selection of the test methods to be used, the respective coverage criteria, and prioritization of the tests. Also, determine the sequence of test execution in the test schedule. Intervene to control during the whole test processes if there are any deviations from the plan.
Test analysis and design: Check the test basis for completeness and sufficient accuracy. Design logical test cases using the test methods of the test oracle. Begin creating the test infrastructure.
Test implementation and execution: Draw up test cases and group them to test sequences or scenarios, and complete the test infrastructure. The first step in the execution is to check that the test object is executable and that calling up main functions does not cause any serious failures. All test runs are to be recorded and evaluated in detail.
Test evaluation and report: Show that the test exit criteria have been satisfactorily fulfilled. If not, decide if further tests are to follow or if the test process may be finished nonetheless. Draw up a summary test report.
Completion of the test activities: The main task of this last activity of the test process is to learn from experience and to provide the testware needed for maintenance.
For each phase of the test process tools are available that help the test manager and the tester to qualitatively improve their test activities.
The use of test tools is only of advantage if the test process is a controlled and defined process.