6.3. Testing Architecture

The testing approaches feed into the testing architecture. The architecture, as shown in Figure 6.4, depicts the various types of tests and how they combine to provide a comprehensive testing strategy for any software system being tested. The testing architecture decides the level at which tests are conducted (for example, unit versus system), and the frequency, intensity, and repetition of tests. The influence of the previous discussion on testing approaches affects all elements of good test architecture.

6.3.1. Unit Test

The unit test is the most immediate testing of the class or program and is usually conducted by the programmer herself. It deals with an individual class or program and is more focused on the correctness or accuracy of execution of a program, rather than its meaning within the overall system. Unit tests benefit by manually creating test harnesses (writing a small piece of code and attaching it to the class) and a very basic set of data. It is always a good idea to store test harnesses and/or test scripts together with the source code being tested. Alternatively, a complete test environment can be created and classes tested under this environment every time the programmer makes a change to the classes. In either case, it is important to modify the test script/harness together with the modification to the actual source code.

Unit testing continues until the developer is satisfied that the results from the test match the expected results and that all the functionality that the class is supposed to satisfy is indeed satisfied. During unit testing the programmer may also check the implemented structure of the class against its intended design structure. From this description it is obvious that unit testing is restricted to the performance of an individual class or program and, therefore, the detailed functionality of the rest of the system may not make much sense at this stage. Recording the results and retesting the code may also be informal at this stage. This is because at the end of a test the programmer may realize the error, correct it, and retest—without ever recording the error. However, depending on the importance of the class, formal recording of errors and fixes at unit test level may be carried out.

The testing approaches discussed in the previous section have a bearing on the unit tests. For example, white box testing is very common when unit tests are conducted. The programmer may step through the code herself and inspect it for statement syntax, branching logic, and adequate coverage of all alternatives. While statement coverage only inspects the statements that are executed, the inspection of branching logic involves each of the conditional statements within the program and the results appearing as true or false at the end of each test. Black box tests occur when the GUI classes are tested. In these unit tests the programmer wants to ensure that the data content and the display from the class she has written satisfies the requirements of the class.

6.3.2. Component Test

A component test is conducted when a few classes are ready to be tested together. Thus, this test not only concentrates on the output from the given classes, but also on how the classes behave in relation with each other. For example, a Client component that is made up of ten classes related to various types of clients will have to be tested for their implementation together, in a component. The concept of coupling between the component under test and other components, as well as the concept of cohesion for classes within the component, comes into play during the component test. Once again, a component test can be a white box manual test wherein we are stepping through the designs of the component, or a black box test wherein we are writing test harnesses, which run a large amount of data through the component in order to test every possible variation to the input.

6.3.3. System Test

The system test is still within the technical domain and is involved with detailed testing of the system not only with respect to its functionality, but also with respect to its performance, installation, backup and recovery, security, and stability. System testing involves end-to-end testing of all components within the system in an integrated fashion. System testing involves both black and white box testing approaches, and the test suites comprise test harnesses to pass a large amount of data through the system.

The functionality of the system is tested by using the test designs and the test cases within the test designs. These test designs are created by using the use cases and scenarios in the requirements-modeling activity. The performance is tested by simulating the real environment in which the system has to operate. This involves creating a large amount of test data and loading the system with a large number of transactions by using an automated test simulator. Installing the application (if physically deployed) on a new machine tests out the installation and deployment of the new application. Sociological aspects (like graceful recovery of software from a crash) are also tested within the system test. Saving these test scripts and their results is of immense value when future modifications are made to the system throughout its lifecycle.

Garbage collection should be tested by running the systems many times over, and over a long time. Memory leaks and related garbage can be tested not only by using memory testers, but also by physically running the system over substantial periods. Dangling pointers left in the memory by unsuspecting programmers should be detected during testing—better still, should be prevented from occurring by good quality modeling in the MOSS.

Exception handling can be a project or an organizational level issue. It can be an internal standard.

6.3.4. Acceptance Test

During acceptance testing the system moves from the technical domain into the user domain. It is up to the users to accept (or reject) the system by testing its functionality against the specified requirements. This test decides whether the system is ready for deployment or not. Therefore, it is important that the users of the system test out the system using all relevant approaches to testing. Testing and verification of the specified help system, the accompanying user guides, and so forth are also carried out in this test.

Users may need technical support at this stage. This support involves creating databases, installing the application, and providing help in using test harnesses and/or automated testing tools, if relevant.

The acceptance criteria for this testing (acceptance testing) have some influence on the way the test cases are designed. For example, if one of the important acceptance criteria is system performance (3-second response times under all conditions) then the acceptance test cases must not be limited to testing the functionality of the system only. They should also test the system operationally. In those situations, it is also important to note the differences in the capabilities of the testing hardware and the production hardware.

6.3.5. Regression Test

Regression tests are conducted in the later iterations of a development project. After the system has undergone all the formal testing processes and the errors are recorded and fixed, it is essential to ensure that not only are the fixes sufficiently working, but also that by fixing the errors we have not introduced additional errors in other parts of the system. This flow-on effect of errors is minimized by the object-oriented designs—wherein it is easier to locate the errors because of encapsulation of classes. However, regression tests are still essential after the errors have been fixed.

They can be performed by means of the automated tools and may be black box tests—unless a design or code inspection is deemed necessary, in which case they should be white box manual tests. Regression tests are facilitated by the cyclic approach to testing, as the experience and the results from the first cycle of testing can be reused in the next cycle. If the results from the initial cycle of the overall tests are minor errors, then a smaller number of samples may be selected from within the equivalence partitions. However, it is recommended that all boundary values should be selected in all passes of the regression tests.

6.3.6. Operational Testing

Operational tests can be applied to system, integration, and acceptance testing, as shown in Figure 6.4. Since they are tests of the system in operation they make more sense later in the lifecycle. Some of these tests include performance, security, and scalability as discussed in the subsequent subsections.

6.3.7. Performance (Stress and Volume) Testing

The performance test deals with the capacity of the database as well as its speed of response. It is important to test out the ability of the database to respond to the transaction hits when the system goes into production. In order to test this it is necessary to load the database with dummy records. Tests can then be carried out on the speed of searches on the database and the corresponding response times. The capacity of the database to store records is its volume. The performance test ensures the adequacy of the database's speed of response and its capacity to hold data.

6.3.8. Security Testing

This test ensures that there is no unauthorized access to the system or any of its components. Security-related technologies (for example, 128-bit encryption) are investigated, incorporated, and tested in this exercise. It should be noted that software security is important but so is the physical security of the system. Most modern hardware is sophisticated enough to continue to support the system even if the permanent storage is removed. Therefore, in addition to software security, the physical security of the hardware must be considered in this testing. Finally, the system should be secured from various virus attacks by using the latest anti-virus software together with the latest anti-virus file definitions.

6.3.9. Scalability Testing

This test deals with the potential of the system to grow as the demands increase. This is an important test that ensures that once the system is in operation and the demands on the system resources start growing, the system is geared to handle the necessary upgrades to both the hardware and the software. The potential increase in demand and the corresponding support by hardware and software should be tested before the system is released.