Contents

Preface to the Second Edition

Preface to the First Edition

Acknowledgements

PART I: PRELIMINARIES

1. Preliminaries: Software Testing

1.1. Humans, Errors, and Testing

1.1.1 Errors, faults, and failures

1.1.2 Test automation

1.1.3 Developer and tester as two roles

1.2. Software Quality

1.2.1 Quality attributes

1.2.2 Reliability

1.3. Requirements, Behavior, and Correctness

1.3.1 Input domain

1.3.2 Specifying program behavior

1.3.3 Valid and invalid inputs

1.4. Correctness Versus Reliability

1.4.1 Correctness

1.4.2 Reliability

1.4.3 Operational profiles

1.5. Τesting and Debugging

1.5.1 Preparing a test plan

1.5.2 Constructing test data

1.5.3 Executing the program

1.5.4 Assessing program correctness

1.5.5 Constructing an oracle

1.6. Test Metrics

1.6.1 Organizational metrics

1.6.2 Project metrics

1.6.3 Process metrics

1.6.4 Product metrics: generic

1.6.5 Product metrics: OO software

1.6.6 Progress monitoring and trends

1.6.7 Static and dynamic metrics

1.6.8 Testability

1.7. Software and Hardware Testing

1.8. Testing and Verification

1.9. Defect Management

1.10. Test Generation Strategies

1.11. Static Testing

1.11.1 Walkthroughs

1.11.2 Inspections

1.11.3 Software complexity and static testing

1.12. Model-Based Testing and Model Checking

1.13. Types of Testing

1.13.1 Classifier: C1: Source of test generation

1.13.2 Classifier: C2: Life cycle phase

1.13.3 Classifier: C3: Goal-directed testing

1.13.4 Classifier: C4: Artifact under test

1.13.5 Classifier: C5: Test process models

1.14. The Saturation Effect

1.14.1 Confidence and true reliability

1.14.2 Saturation region

1.14.3 False sense of confidence

1.14.4 Reducing ∆

1.14.5 Impact on test process

1.15. Principles of Testing

1.16. Tools

Summary

Exercises

2. Preliminaries: Mathematical

2.1. Predicates and Boolean Expressions

2.2. Control Flow Graph

2.2.1 Basic blocks

2.2.2 Flow graphs

2.2.3 Paths

2.2.4 Basis paths

2.2.5 Path conditions and domains

2.2.6 Domain and computation errors

2.2.7 Static code analysis tools and static testing

2.3. Execution History

2.4. Dominators and Post-Dominators

2.5. Program Dependence Graph

2.5.1 Data dependence

2.5.2 Control dependence

2.5.3 Call graph

2.6. Strings, Languages, and Regular Expressions

2.7. Tools

Summary

Exercises

PART II: TEST GENERATION

3. Domain Partitioning

3.1. Introduction

3.2. The Test Selection Problem

3.3. Equivalence Partitioning

3.3.1 Faults targeted

3.3.2 Relations

3.3.3 Equivalence classes for variables

3.3.4 Unidimensional partitioning versus multidimensional partitioning

3.3.5 A systematic procedure

3.3.6 Test selection

3.3.7 Impact of GUI design

3.4. Boundary Value Analysis

3.5. Category-partition Method

3.5.1 Steps in the category-partition method

Summary

Exercises

4. Predicate Analysis

4.1. Introduction

4.2. Domain Testing

4.2.1 Domain errors

4.2.2 Border shifts

4.2.3 ON-OFF points

4.2.4 Undetected errors

4.2.5 Coincidental correctness

4.2.6 Paths to be tested

4.3. Cause-effect Graphing

4.3.1 Notation used in cause-effect graphing

4.3.2 Creating cause-effect graphs

4.3.3 Decision table from cause-effect graph

4.3.4 Heuristics to avoid combinatorial explosion

4.3.5 Test generation from a decision table

4.4. Tests Using Predicate Syntax

4.4.1 A fault model

4.4.2 Missing or extra Boolean variable faults

4.4.3 Predicate constraints

4.4.4 Predicate testing criteria

4.4.5 BOR, BRO, and BRE adequate tests

4.4.6 BOR constraints for non-singular expressions

4.4.7 Cause-effect graphs and predicate testing

4.4.8 Fault propagation

4.4.9 Predicate testing in practice

4.5. Tests Using Basis Paths

4.6. Scenarios and Tests

Summary

Exercises

5. Test Generation From Finite State Models

5.1. Software Design and Testing

5.2. Finite State Machines

5.2.1 Excitation using an input sequence

5.2.2 Tabular representation

5.2.3 Properties of FSM

5.3. Conformance Testing

5.3.1 Reset inputs

5.3.2 The testing problem

5.4. A Fault Model

5.4.1 Mutants of FSMs

5.4.2 Fault coverage

5.5. Characterization Set

5.5.1 Construction of the k-equivalence partitions

5.5.2 Deriving the characterization set

5.5.3 Identification sets

5.6. The W-Method

5.6.1 Assumptions

5.6.2 Maximum number of states

5.6.3 Computation of the transition cover set

5.6.4 Constructing Ζ

5.6.5 Deriving a test set

5.6.6 Testing using the W-method

5.6.7 The error detection process

5.7. The Partial W-Method

5.7.1 Testing using the Wp-method for m = n

5.7.2 Testing using the Wp-method for m > n

5.8. The Uio-sequence Method

5.8.1 Assumptions

5.8.2 UIO sequences

5.8.3 Core and non-core behavior

5.8.4 Generation of UIO sequences

5.8.5 Explanation of gen-uio

5.8.6 Distinguishing signatures

5.8.7 Test generation

5.8.8 Test optimization

5.8.9 Fault detection

5.9. Automata Theoretic Versus Control-Flow Based Techniques

5.9.1 n-switch-cover

5.9.2 Comparing automata theoretic methods

5.10. Tools

Summary

Exercises

6. Test Generation from Combinatorial Designs

6.1. Combinatorial Designs

6.1.1 Test configuration and test set

6.1.2 Modeling the input and configuration spaces

6.2. A Combinatorial Test Design Process

6.3. Fault Model

6.3.1 Fault vectors

6.4. Latin Squares

6.5. Mutually Orthogonal Latin Squares

6.6. Pairwise Design: Binary Factors

6.7. Pairwise Design: Multi-Valued Factors

6.7.1 Shortcomings of using MOLS for test design

6.8. Orthogonal Arrays

6.8.1 Mixed-level orthogonal arrays

6.9. Covering And Mixed-level Covering Arrays

6.9.1 Mixed-level covering arrays

6.10. Arrays of Strength > 2

6.11. Generating Covering Arrays

6.12. Tools

Summary

Exercises

PART III: TEST ADEQUACY ASSESSMENT AND ENHANCEMENT

7. Test Adequacy Assessment Using Control Flow and Data Flow

7.1. Test Adequacy: Basics

7.1.1 What is test adequacy?

7.1.2 Measurement of test adequacy

7.1.3 Test enhancement using measurements of adequacy

7.1.4 Infeasibility and test adequacy

7.1.5 Error detection and test enhancement

7.1.6 Single and multiple executions

7.2. Adequacy Criteria Based on Control Flow

7.2.1 Statement and block coverage

7.2.2 Conditions and decisions

7.2.3 Decision coverage

7.2.4 Condition coverage

7.2.5 Condition/decision coverage

7.2.6 Multiple condition coverage

7.2.7 Linear code sequence and jump (LCSAJ) coverage

7.2.8 Modified condition/decision coverage

7.2.9 MC/DC adequate tests for compound conditions

7.2.10 Definition of MC/DC coverage

7.2.11 Minimal MC/DC tests

7.2.12 Error detection and MC/DC adequacy

7.2.13 Short-circuit evaluation and infeasibility

7.2.14 Basis path coverage

7.2.15 Tracing test cases to requirements

7.3. Concepts from Data Flow

7.3.1 Definitions and uses

7.3.2 C-use and p-use

7.3.3 Global and local definitions and uses

7.3.4 Data flow graph

7.3.5 Def-clear paths

7.3.6 Def-use pairs

7.3.7 Def-use chains

7.3.8 A little optimization

7.3.9 Data contexts and ordered data contexts

7.4. ADequacy Criteria Based on Data Flow

7.4.1 c-use coverage

7.4.2 p-use coverage

7.4.3 All-uses coverage

7.4.4 k-dr chain coverage

7.4.5 Using the k-dr chain coverage

7.4.6 Infeasible c- and p-uses

7.4.7 Context coverage

7.5. Control Flow Versus Data Flow

7.6. The “Subsumes” Relation

7.7. Structural and Functional Testing

7.8. Scalability of Coverage Measurement

7.9. Tools

Summary

Exercises

8. Test Adequacy Assessment Using Program Mutation

8.1. Introduction

8.2. Mutation and Mutants

8.2.1 First-order and higher order mutants

8.2.2 Syntax and semantics of mutants

8.2.3 Strong and weak mutations

8.2.4 Why mutate?

8.3. Test Assessment Using Mutation

8.3.1 A procedure for test adequacy assessment

8.3.2 Alternate procedures for test adequacy assessment

8.3.3 “Distinguished” versus “killed” mutants

8.3.4 Conditions for distinguishing a mutant

8.4. Mutation Operators

8.4.1 Operator types

8.4.2 Language dependence of mutation operators

8.5. Design of Mutation Operators

8.5.1 Goodness criteria for mutation operators

8.5.2 Guidelines

8.6. Founding Principles of Mutation Testing

8.6.1 The competent programmer hypothesis

8.6.2 The coupling effect

8.7. Equivalent Mutants

8.8. Fault Detection Using Mutation

8.9. Types of Mutants

8.10. Mutation Operators for C

8.10.1 What is not mutated?

8.10.2 Linearization

8.10.3 Execution sequence

8.10.4 Effect of an execution sequence

8.10.5 Global and local identifier sets

8.10.6 Global and local reference sets

8.10.7 Mutating program constants

8.10.8 Mutating operators

8.10.9 Binary operator mutations

8.10.10 Mutating statements

8.10.11 Mutating program variables

8.10.12 Structure Reference Replacement

8.11. Mutation Operators for Java

8.11.1 Traditional mutation operators

8.11.2 Inheritence

8.11.3 Polymorphism and dynamic binding

8.11.4 Method overloading

8.11.5 Java specific mutation operators

8.12. Comparison of Mutation Operators

8.13. Mutation Testing within Budget

8.13.1 Prioritizing functions to be mutated

8.13.2 Selecting a subset of mutation operators

8.14. Case and Program Testing

8.15. Tools

Summary

Exercises

PART IV: PHASES OF TESTING

9. Test Selection, Minimization, and Prioritization for Regression Testing

9.1. What is Regression Testing?

9.2. Regression Test Process

9.2.1 Revalidation, selection, minimization, and prioritization

9.2.2 Test setup

9.2.3 Test sequencing

9.2.4 Test execution

9.2.5 Output comparison

9.3. Regression Test Selection: the Problem

9.4. Selecting Regression Tests

9.4.1 Test all

9.4.2 Random selection

9.4.3 Selecting modification traversing tests

9.4.4 Test minimization

9.4.5 Test prioritization

9.5. Test Selection Using Execution Trace

9.5.1 Obtaining the execution trace

9.5.2 Selecting regression tests

9.5.3 Handling function calls

9.5.4 Handling changes in declarations

9.6. Test Selection Using Dynamic Slicing

9.6.1 Dynamic slicing

9.6.2 Computation of dynamic slices

9.6.3 Selecting tests

9.6.4 Potential dependence

9.6.5 Computing the relevant slice

9.6.6 Addition and deletion of statements

9.6.7 Identifying variables for slicing

9.6.8 Reduced dynamic dependence graph

9.7. Scalability of Test Selection Algorithms

9.8. Test Minimization

9.8.1 The set cover problem

9.8.2 A procedure for test minimization

9.9. Test Prioritization

9.10. Tools

Summary

Exercises

10. Unit Testing

10.1. Introduction

10.2. Context

10.3. Test Design

10.4. Using JUnit

10.5. Stubs and Mocks

10.5.1 Using mock objects

10.6. Tools

Summary

Exercises

11. Integration Testing

11.1. Introduction

11.2. Integration Errors

11.3. Dependence

11.3.1 Class relationships: static

11.3.2 Class relationships: dynamic

11.3.3 Class firewalls

11.3.4 Precise and imprecise relationships

11.4. OO versus Non-OO Programs

11.5. Integration Hierarchy

11.5.1 Choosing an integration strategy

11.5.2 Comparing integration strategies

11.5.3 Specific stubs and retesting

11.6. Finding a Near-optimal Test Order

11.6.1 The TD method

11.6.2 The TJJM method

11.6.3 The BLW method

11.6.4 Comparison of TD, TJJM, and the BLW methods

11.6.5 Which test order algorithm to select?

11.7. Test Generation

11.7.1 Data variety

11.7.2 Data constraints

11.8. Test Assessment

11.9. Tools

Summary

Exercises