1 Introduction
Like never before, everyday life has become dependent on software and software-based systems. Most of today’s appliances, machines, and devices are completely or at least partly controlled by software. Administrative proceedings in state agencies and industry, too, rely to a large extent on highly complex IT systems. Examples are the management of insurance policies, inventory control systems, biometric characteristics in passports and ID cards, and the electronic health chip card.
High dependency on software
This strong dependency on software requires ever higher investments in quality assurance activities to enable IT systems to perform reliably. Software testing is developing toward a specialized, independent field of study and professional discipline within the computer sciences.
Software testing – a professional discipline in its own right
Within the discipline of software testing, “test management” is of particular importance. Test management comprises classical methods of project and risk management as well as knowledge of the appropriate use of well-defined test methods. With this stock-in-trade, the test manager1 can select and purposefully implement appropriate measures to ensure that a defined basic product quality will be achieved. In doing so, the test manager adopts an engineering approach.
Test management
Whereas today’s project management training is well established, and while there are a tremendous number of study courses, training programs, and specialist literature to choose from, there has, until recently, been hardly any attempt at defining or standardizing the contents of training programs for the “software test manager”. In view of the increasing responsibility assumed by test managers in the execution of their job, this has been an unsatisfactory situation.
Training for test managers
With the ISTQB Certified Tester – Advanced Level – Test Manager we have, for the first time, developed an internationally recognized training and qualification scheme that defines training contents and qualification modules for the tasks of the test manager. This book sets out to convey the associated teaching contents and may be read as a textbook in preparation for the exams.
ISTQB Certified Tester – Advanced Level – Test Manager
The “ISTQB Certified Tester” qualification scheme consists of three levels. The basics of software testing are described in the syllabus for the Foundation Level ([URL: ISTQB] -Syllabi), whereas the corresponding subject matter is explained in detail in Software Testing Foundations [Spillner 07].
Foundation Level
The Advanced Level curriculum ([URL: ISTQB] -Syllabi) defines advanced proficiency skills in software review and testing and shows possible opportunities for specialization:
Advanced Level
Exhaustive treatment of different black box and white box test methods in the Advanced Level, Technical Tester, and Functional Tester modules
Extensive, in-depth presentation of test management methods and techniques in the Test Manager module2
Since the “Advanced Level” syllabus is very comprehensive, it will not be treated in its entirety in this book; instead, we shall concentrate exclusively on the “Advanced Level – Test Manager module”. The topic of “reviews”3, however, will be excluded.
The third level, the “Expert Level”, is in the process of being defined by expert groups and comprises topics such as the specific characteristics of object-oriented software testing, advanced knowledge in Testing & Test Control Notation (TTCN-3, [URL: TTCN-3]), advanced knowledge in test process improvement methodology, and various other areas of expertise associated with software testing.
Expert Level
The “ISTQB” [URL: ISTQB] provides for the homogeneity and comparability of the teaching and examination contents of all participating countries.
International Software Testing Qualifications Board (ISTQB)
Today, the ISTQB has become an affiliation of more than 25 national initiatives and associations worldwide (see figure 1-1). More national boards will follow.
Figure 1–1 Structure of the ISTQB
As independent expert bodies, the national testing boards are responsible for the provision of training (accreditation of providers of proficiency testing schemes) and examinations (certification by an independent institution) in their respective countries and native languages and for ensuring compliance with ISTQB standards.
National Testing Boards
The three levels of the ISTQB qualification scheme build on one another. This book, Software Testing Practice: Test Management, presumes that the reader is familiar with the subject matters dealt with in the Foundation Level.
Knowledge of software testing foundations required
Readers new to software testing are advised to first acquire knowledge of the content of the Foundation Level, either by attending an accredited provider’s seminar or by working through the book Software Testing Foundations [Spiller 07]. The present book contains only a brief recapitulation of the most important basic principles.
1.1 Software Testing Foundations – Condensed
This section provides a brief summary of the Foundation Level syllabus and of the book Software Testing Foundations.
There is a multitude of approaches and proposals available on how to improve software quality through preventive (constructive) actions and the use of verifying (analytical) methods. The following measures are among the most importìant to improve software quality:
Measures to improve software quality
Defined software development processes that contribute to a structured and traceable development of software systems
A well-defined test process and controlled change and incident management as a requirement for the efficient and effective execution of test activities
The application of metrics and quality data to objectively evaluate software products and development processes, to detect improvement potentials, and to verify the effectiveness of correction and improvement activities
The use of formal methods that allow for the precise formulation of development documents and their verification or evaluation by tools
Methods for the systematic identification and execution of test cases that allow efficient detection of defects and anomalies in the developed programs
Methods for static testing, primarily reviews through which defects and anomalies are detected in design documents at an early stage
Test managers must master or at least be familiar with these methods, techniques, and processes in order to be able to select and apply appropriate measures during the course of the project.
Quality goals and quality attributes
The suitability of quality assurance measures, however, also depends on the defined quality goals. The required quality level can thereby be defined based on different quality attributes. A catalogue of such quality attributes (e.g., functionality, reliability, and usability) is defined by the [ISO 9126] standard.
When do we speak of a defect or an error and what do we actually mean when we use these terms? A situation or result can only be classified as faulty, defective, or erroneous if we have previously defined what the expected, correct situation or result is supposed to look like. If the actual software behavior deviates from the expected behavior, we use words such as defect, fault, bug, and anomaly.
Test oracle
In order to establish expected values or expected behavior, a so-called test oracle is required that serves the tester as a source of information. Requirements documents, formal specifications, and the user guide are examples of such information sources.
The term “error” is actually rather imprecise. We do in fact need to distinguish between error, fault, and failure (including their synonyms). For example, a developer’s error while programming leads to a fault in the software that may (but not necessarily) result in a visible failure. In most cases, the impact of a fault or defect only shows itself in uncommon situations; for instance, the erroneous calculation of the leap year becomes effective only on February 29.
Error terminology
Figure 1-2 illustrates the relationship between error, fault, and failure and shows which countermeasures or methods may be used for their detection.
Figure 1–2 Relationship between the different terms used to denote errors
Similar to the term error, the word “test” has different meanings.
Test terminology
Testing frequently denotes the entire process of systematically checking a program to gain confidence in the correct implementation of the requirements4 and to detect failures. It is also a generic term for all the activities and (test) levels in the test process. Each individual execution of a test object under specified conditions to verify the correctness of the expected results is also called testing.
Testing includes many individual activities. The following basic test process is defined in the Foundation Level syllabus, comprising the following activities:
Test planning and control
Test analysis and test design
Test implementation and test execution
Test evaluation of the test exit criteria
Post testing activities
During testing, the product under test (the test object) can be considered at different levels of abstraction or on the basis of different documents and development products. The corresponding term is test level. We distinguish between the different levels of component test, integration test, system test, and acceptance test.
Test levels
Each test level has is own characteristic test objectives, test methods, and test tools.
Furthermore, we distinguish between different →test types: functional test, nonfunctional test, structural test, change-related test, and regression testing. [Spillner 07, section 3.7]
Test types
During testing, we distinguish whether testing is performed by execution of the test object or whether it is done “only” on the associated program text or underlying specification or documentation.
In the first case, we have the so-called dynamic tests, represented by black box and white box test methods [Spillner 07, chapter 5], and in the second case, we talk about static tests, represented, among other things, by different types of reviews [Spillner 07, chapter 4].
Static and dynamic testing
Regardless of which method is used for testing, it is essential that in terms of organization, development/programming and testing are kept separate and that they are performed independently from each other. A developer testing his own code will be blind toward his own mistakes and not very keen on detecting them himself.
Independence between test and development
There are many supporting tools in use for software testing. Depending on their intended use, we distinguish between different tool classes, such as, for instance, tools for test management and control, tools for test specification, and tools for static, dynamic and nonfunctional testing [Spillner 07, chapter 7].
Test tools
In our discussion of the Foundation Level syllabus, we reviewed the test management fundamentals. In addition to test planning, test control, and test reporting, test management includes topics such as change and configuration management as well as the economy of testing [Spillner 07, chapter 6]. This book will cover these test management tasks in more detail.
Test management
For illustration purposes, we shall continue the case study example introduced in the book Software Testing Foundations:
Case study “VirtualShowRoom” (VSR)
A car manufacturer develops a new electronic sales support system called VirtualShowRoom (VSR). The final version of this software system will be installed at every car dealership worldwide. Customer interested in buying a new car will be able to configure their favorite model (model, type, color, extras, etc.) at the terminal with or without the guidance of a salesperson.
The system shows all possible models and combinations of extra equipment and instantly calculates the accurate price of the configured car.
This functionality will be implemented by a subsystem called DreamCar.
When customers make up their mind, they will be able to calculate the most suitable payment method (EasyFinance) as well as place an order online (JustInTime). Of course, they will have the opportunity to sign up for the appropriate insurance (NoRisk).
Personal information and contract data about the customer is managed by the ContractBase subsystem.
Figure 1-3 shows the general architecture of this software system.
Figure 1–3 Architecture of the VSR-System
Each subsystem will be designed and developed by separate developer teams.
Altogether, about 50 developers and additional employees from the respective user departments are involved in working on this project. External software companies are also involved.
In Software Testing Foundations, we described the different →test design techniques used for in-depth testing of the VSR system before putting it into operation.
VSR-2 development follows an iterative development process. Based on the current VSR-1 system, VSR-2 is supposed to be developed in four successive iterations. The planned schedule is one year, with approximately one increment per quarter. Each new increment is expected to provide the full functionality of the previous version; it may, however, be based on a better or more efficient implementation. In addition, each increment introduces a set of new functionalities.
The product manager expects two things from the test manager:
First, the test team must ensure that the old functionality is correctly retained in each intermediary VSR-2 version.
Second, the test team should fairly quickly be able to judge objectively if, and how well, a new feature has been implemented.
The tasks that the test manager has to perform regarding such issues will be discussed and illustrated in the following chapters.
1.2 Software Testing Practice: Test Management – Overview
The topics of the book and the contents of each chapter are briefly described here.
Software testing practice – overview
Chapter 2 discusses the fundamental test process and the types of tools that can be used to support it. Both topics were covered in Software Testing Foundations.
Chapter 3 explains how testing is related to the software life cycle. It discusses different life cycle models used in software development and evaluates the particular importance given to testing in each model.
The significance given to testing in an organization is of great importance to the test manager. The organization’s quality and test policy must be defined by management. Chapter 4 deals with these issues.
Chapter 5 takes a closer look at test planning, one of the important, if not most important, tasks of the test manager.
Planning must be adjusted during the project’s life cycle. Control of the test process based on test progress reports is an essential task for the test manager in order to perform successful testing. This aspect is addressed in chapter 6.
The development and test processes themselves can be evaluated and improved. Chapter 7 describes which techniques and processes are to be applied to accomplish this improvement.
How do we deal with deviations and failures detected during testing? Chapter 8 provides some answers.
Risk evaluation and risk-based tests are important instruments for the test manager to allocate limited test resources. They are used to control the test project with minimized risk. Chapter 9 contains advice on how to proceed.
Without qualified and skilled staff – that is, without consideration of the human factor – the test manager will not be able to succeed. Chapter 10 describes what needs to be considered when selecting a test team.
Test metrics help in defining test exit criteria and in finding evidence on the quality of the test object. Chapter 11 will provide some examples.
In most cases, the test process can be performed more efficiently with adequate tool support. Chapter 12 describes how the test manager selects and introduces such tools.
The last chapter, chapter 13, presents relevant standards.
The glossary contains all the terms that are newly mentioned in this book, the first occurence of which will be preceded by an arrow “→”. All glossary terms used in Software Testing Foundations [Spillner 07] can also be found at [URL: ISTQB] -download.