7.2.1 Primary Attributes

We consider four primary attributes: the scale, the goal, the property, and the method. We review these in turn in the following:

• The Scale: We consider three possible values for this attribute:
  • A unit: This represents a programming unit that implements a data abstraction or a functional abstraction. As such, it can be a class in an object-oriented language, the implementation of an abstract data type, or a routine.
  • A subsystem: This represents a component in an integrated software system. To test such a component in a credible manner, we may have to run the whole system and observe the behavior of this particular component within the system. This situation arises in the context of maintenance, for example, where we may change a component and then test the whole system to check whether the changes are satisfactory, and whether the new subsystem works smoothly within the overall system.
  • A system: This represents a complete autonomous software system.
• The Goal: This is perhaps the most important attribute of a test. The values it may take include the following:
  • Finding and removing faults: The most common goal of software testing is to observe the behavior of the program on test data, and to diagnose and remove faults whenever the program fails to satisfy its specification.
  • Proving the absence of faults/certifying compliance with quality standards: While in practice it is virtually impossible to test a program on all possible combinations of inputs and configurations, this possibility cannot be excluded in theory. Also, we can imagine cases where the set of possible input data is small, or cases where we can design a test data D such that if the program runs successfully on D, then it runs successfully on all the input space S.
  • Estimating fault density: Software testing can be used to estimate the fault density of a product, as will be discussed in Chapter 13. This is done by seeding faults, then running a test to compute how many seeded faults have been recovered and how many unseeded faults have been uncovered; fault density can then be estimated by interpolation.
  • Estimating the frequency of failures: While the previous goals are concerned with faults, this and the next goal are concerned with failures instead; there is a sound rationale for focusing on failures rather than faults, because it is better to reason about observable/relevant effects than about hypothetical causes. To pursue this goal, we run the software product under conditions that simulate its operational environment, and estimate its failure rate; the estimate is reliable only to the extent that the test environment is a faithful reflection of the operating environment, and that the test data reflects, in its distribution, the usage pattern of the software product. It is only under such conditions that the failure rate observed during testing can be borne out during field usage. We identify three possible instances of this goal, depending on whether we want to estimate the reliability, the safety, or the security of the product:
    • Estimating reliability
    • Estimating safety
    • Estimating security

      We get one instance or another, depending on the oracle that we use for the test: To estimate reliability, we use the functional specifications of the software product; to estimate safety, we use the safety-critical requirements of the product; to estimate security, we use the security requirements of the product.

  • Ensuring the infrequency of failures: As an alternative to proving the absence of faults, we may want to prove that faults are not causing frequent failures; after all, a program may have faults and still be reliable, or more generally experience infrequent failure. Also, as an alternative to estimating the frequency of failures, we may simply establish that the frequency of failure is lower than a required threshold. Whereas estimating the frequency of failure is a mere analytical process, that analyzes the product as it is, ensuring the infrequency of failures may involve diagnosing and removing faults until the frequency of failures of the software product is deemed to be lower than the mandated threshold. As we argued in the last item, in order for the estimate of the failure rate to be reliable, the software product must be tested in an environment that mimics its operating environment, and the test data must be distributed according to the usage pattern of the product in the field. Also, this goal admits three instances, depending on what oracle is used in the test:
    • Certifying reliability
    • Certifying security
    • Certifying safety

      We get one instance or another, depending on the oracle that we use for the test.

• Method: Given a limited amount of resources (time, manpower, and budget), we cannot run the software product on the set S of all possible input data; as a substitute, we want to run the software product on a set of test data, say D (a subset of S), that is large enough to help us achieve our goal, yet small enough to minimize costs. The test data generation method is the process that enables us to derive set D according to our goal; the criterion that we use to derive D from S depends on the goal of the test, as follows:
  • If the goal of the test is to diagnose and remove faults, then D should cause the maximum number of failures, that is, uncover/sensitize the maximum number of faults.
  • If the goal of the test is to prove correctness, then D should be chosen in such a way that if the program runs successfully on D, we can be reasonably confident (or assured) that it runs successfully on all of S.
  • If the goal of the test is to estimate the product’s failure rate or to ensure that the product’s failure rate exceeds a mandated threshold, then D must be a representative sample of the usage pattern of the product.

Test data generation methods are usually divided into three broad families:

• Structural methods, which generate test data by analyzing the structure of the software product and targeting the data in such a way as to exercise relevant components of the product.
• Functional methods, which generate test data by analyzing the specification of the software product or its intended function, and targeting the data in such a way as to exercise all the services or functionalities that are part of the specification or intended function.
• Random, with respect to a usage pattern. This method generates test data in such a way as to simulate the conditions of usage of the software product in its operating environment.

It is possible to map the goal of the test to the test data generation method in a nearly deterministic manner, as shown in the following table:

Target Attribute: Most typically, one tests a software product to affect or estimate some functional quality of the product, such as correctness, reliability, safety, security, and so on; but as the following list indicates, one may also be interested in testing the product for a broad range of attributes.

• Functionality: Testing a software product for functional properties such as correctness, reliability, safety, security, and so on is the most common form of testing, and is the default option is all our discussions.
• Robustness: Whereas correctness mandates that the software product behaves according to the specification for all inputs in the specified domain, robustness further mandates that the program behaves reasonably (whatever that means: we can all recognize unreasonable program behavior when we see it) outside of the specification domain, that is, on inputs or situations for which the specification made no provisions.
• Design and structure: In integration testing, the focus of the test is on ensuring that the parts of the software system interact with each other as provided by the design; here the attribute we want to test or ensure is the proper realization of the design.
• Performance: We may want to test a software product for the purpose of empirically analyzing its performance under normal usage conditions (e.g., normal workload).
• Graceful degradation: In the same way that we distinguish between correctness (functional behavior for normal inputs) and robustness (functional behavior for exceptional or unplanned inputs), we distinguish between performance (operational behavior under normal workloads) and graceful degradation (operational behavior under excessive workloads). To test a software product for graceful degradation, we operate it under excessive workloads and observe whether its performance decreases in a continuous, acceptable manner.

In Section 7.2.2, we review the secondary attributes and discuss how they are affected by the choices made for the primary attributes reviewed in this section.

7.2.2 Secondary Attributes

We consider the secondary attributes listed earlier and review the set of values that are available for each attribute, as well as how these values are impacted by the primary attributes.

The Oracle: If the target attribute of the test is an operational attribute, such as the response time of the product under normal workloads, or under exceptional workloads, then the oracle takes the form of an operational condition (a response time, or a function plotting the response time as a function of the workload). If the target attribute of the test is functional, then the oracle depends on whether the goal of the test is to find faults or to certify failure freedom. The following table highlights these dependencies.

The Test Life Cycle: Whereas in Chapter 3 we have presented a generic test lifecycle, we can imagine three variations thereof, which we present in the following text:

• A sequential life cycle, which proceeds sequentially through three successive phases of test data generation, test execution, and test outcome analysis. An algorithmic representation of this cycle may look like this

{testDataGeneration(D);          // D: test data set;
   T=empty;                     // T: reportwhile (not empty(D)) 
   {d=removeFrom(D);
   d’=P(d);
   if (not(oracle(d,d’)) {add(d,T;}}analyze(T);}

In this cycle, the phases of test data generation, test execution, and test analysis take place sequentially.

• A semisequential life cycle, where the execution of tests pauses whenever a failure is observed; this life cycle may be adopted if we want to remove faults as the test progresses. An algorithmic representation of this cycle may look like this:

     {testDataGeneration(D); // D: test data set;
      while (not empty(D))  
       {repeat {d=removeFrom(D); d’=P(d);}
      until not(oracle(d,d’));
        offLineAnalysis(d); // fault diagnosis and removal
      }
    }

• An iterative life cycle, which integrates the test data generation into the iteration. An algorithmic representation of this cycle may look like this:

   {while (not completeTest())  
     {d=generateTestData();
      d’=P(d);
     if (oracle(d,d’)) {successfulTest(d);}
     else {unsuccessfulTest(d);}
     }
   }

The following table shows how the value of this attribute may depend on the primary attributes of goal and method.

Test Assumptions: A test can be characterized by the assumptions it makes about the product under test and/or about the environment in which it runs. As such, this attribute can take three values, depending on the scale of the product being tested, as shown in the following table.

Test Completion: Test completion is the condition under which the test activity is deemed to achieve its goal. Such conditions are as follows:

• The software product has passed the certification standard.
• The software product has performed to the satisfaction of the user.
• It is felt that all relevant faults have been diagnosed and removed.
• The reliability of the software product has been estimated.
• The reliability of the software product has grown beyond the required threshold.
• The test data generated for the test have been exhausted, and so on.

The following table illustrates how this attribute depends on the goal of the test and the test data generation method.

Required Artifacts: Many artifacts may be needed to conduct a test, including any combination of the following artifacts:

• The source code
• The executable code
• The product specification
• The product’s intended function
• The product’s design
• The signature of the software product (i.e., a specification of its input space)
• The usage pattern of the software product (i.e., a probability distribution over its input space)
• The test data generated for the test

This attribute depends on virtually all four primary attributes; for the sake of parsimony, we only show its dependence on the goal of testing and on the test data generation method.

Stakeholders: A stakeholder in a test is a party that has a role in the execution of the test, or has a role in the production of the software asset being tested, or has a stake in the outcome of the test. Possible stakeholders include the product developer, the product specifier, the product user, the quality assurance team, the verification and validation team, the configuration management team, and so on. The following table shows how this attribute depends on the goal of the test and the scale of the asset.

Test Environment: The environment of a test is the set of interfaces that the product under test interacts with as it executes. The following table shows the different values that this attribute may take, depending on the goal of the test and the scale of the software product under test.

Position in the Life Cycle: As we have seen in Chapter 3, several phases of the software life cycle include a testing activity. The software activity at each phase can be characterized by primary attributes; the following table shows how the goal of testing and the scale of the product under test determine the phase at which each test activity takes place.

7.3.1 Unit-Level Testing

We distinguish between two types of unit-level testing:

• Unit-Level Fault Removal: This test is carried out by the unit’s developer as part of the coding and unit testing phase of the software life cycle; its purpose is to detect, isolate, and remove faults as part of the development life cycle.
• Unit-Level Certification: This test is carried out by the configuration management/quality assurance team for the purpose of ensuring that the unit under test meets the quality standards mandated for the project.

The following table illustrates how these two tests differ from each other, by comparing and contrasting their attributes.

7.3.2 System-Level Testing

We consider three system-level tests:

1. Integration test, which arises at the end of the programming phase, when programming units that have been developed, tested, and filed into the product configuration are combined according to the product design to produce an integrated product.
2. Reliability test, which arises as the end of the software development project, prior to product delivery, to evaluate the reliability of the product (and eventually, ascertain that the product reliability exceeds the product’s required reliability).
3. Acceptance test, which is conducted jointly by the development team and the user team to check that the software product meets its requirements.

Even though these tests are all at the same scale (system-wide), they differ from each other in significant ways, as we see in the table below.