6.7. Test Execution

We have planned for testing and have created test classes thus far. Test execution is the actual execution of test cases against the given software. This requires the implementation of all planning done so far. Herein we describe the general approach to test execution.

6.7.1. Getting Ready

Preparation is the initial part of executing tests. All test cases need some preliminary preparations before they can be successfully executed. This includes the preparedness of the modules that are to be tested, the availability of test data, and the availability of sample test results against which the results of the test execution can be measured.

Getting ready for test execution also includes familiarization of the tester with the test cases and the input and output of the test cases. While test data is prepared when the test designs are finalized and the test cases are written, it is not unusual for the test data to be incomplete during the initial cycles of testing. In preparing for the test execution, this data may have to be augmented by the results of the previous cycle, or from any other relevant source. In preparing for the tests, it is necessary for the tester to familiarize himself with the associated test cases.

Test cases that test the remaining functions within a class, or other classes within the component, will have some bearing on the current test case. An overall idea of the test designs is essential for all testers before the formal tests are executed. Finally, administrative procedures like backing up and re-creating test data and test cases also need to be put in place before the test execution begins. It is a part of the getting-ready procedure.

6.7.2. Acceptance Criteria

It is important to understand the criteria that will decide whether the test is a success or not. These criteria may range from the single criterion for a unit test to a broad description of acceptance of the system at the integrated-test level. Although the expected results are a part of the acceptance criteria, the acceptance criteria are more than just the results. A user may accept the results as valid, but may point out a change in input data in practical usage of the system. The user may also accept some results as valid but with additional conditions. Also, the acceptance of a test as a pass in the initial cycle of testing may not hold in the subsequent cycles of testing. Therefore, it is essential to understand what constitutes a pass for a test from the user's perspective.

Acceptance criteria determine the success or failure of a test case. If the results of the test case satisfy the acceptance criteria, then the test case is successful. The criteria itself need not be positive. The criteria may specify the requirement of a failure (negative), and accordingly the test case should produce a negative result. For example, if invalid logon is supposed to reject the logon attempt, then a failure to logon is a success as far as the test criteria are concerned. Acceptance criteria also specify more than the simple pass or fail. They specify the understanding of the user of the test case. Thus, acceptance criteria can be specified at the unit-test level to indicate a unit level pass or fail. More importantly, they can also be specified at the system or integration test level, wherein the user or business sponsor may formally state what would be an acceptance of the system.

Some of the test criteria that can determine the acceptance of the system follow:

• Results validation. All actual results match up with the expected results. However, specifying correct expected results is the responsibility of the user.

• Logic validation. In addition to verifying the results, users may also want a proof of the logic, by doing a white box test for the same. Users may state that they will accept the system only after its logic has been validated.

• Error handling. A system may be accepted only after it proves proper error handling. Users may decide not to accept a system that performs all calculations correctly when correct input data is provided, but crashes if the user enters invalid data. Informative error messages and graceful error handling may be valid acceptance criteria. This assures that the system is able to recover from a crash—an important acceptance criterion even for routine transaction-based banking and financial applications.

• Operational requirements. Performance, scalability, and security are vital criteria for user acceptance of the system. For example, various attempts at logins, physical access to the system, software aided attempts to get in the system, and so on form part of the security criteria. Furthermore, the ability of the system to operate under different configurations and versions of the environment (such as operating systems and databases) is also part of the operational acceptance criteria of the system.

• Help, CBT, and user documentation. Context-sensitive help, associated computer-based training (CBT), and supporting documentation on how to install and operate a system are, more often than not, necessary for a system's acceptance.

6.7.3. Execute Test Suites

Once the test cases are designed and understood, and their dependencies worked out, it is time to conduct the actual testing. This requires executing the set of test cases and comparing the results in order to record the success or failure of the test cases. The testing follows the test plans and uses the test cases within the test designs to test the executable components of the system. Depending on the architecture of testing, some test cases are executed by actual execution of components; others can be white box walkthroughs. Also, depending on the testing cycle, some test cases will be broad-brush runs, whereas the same test cases during the major iteration of testing will be thorough and will provide a wide coverage of data and functionality.

6.7.4. Record Incident Reports

The results of the test cases are normally categorized as pass or fail. However, it is essential to categorize the results in further detail. This facilitates the analysis and understanding of not only the system being tested, but also the testing process itself. The results of the test execution are called incidences within the resurrection process. Recording and analyzing these results is described in the next section.