3
Continuous Process Improvement
The constant improvement of processes aims to reduce design – and test – costs as much as possible and to improve products while respecting existing constraints (e.g. quality, costs and deadlines). These improvements are found in all organizational models (ITIL, COBIT, PMBOK, PRINCE2, CMMI, SPICE, etc.). We will focus on improving the processes related to testing.
Improving the quality of the systems produced – whether software or hardware – requires the systematic analysis of failures and the implementation of process improvement actions to avoid the introduction of such failures. Many safety-critical domains, such as aviation, implement this systematic analysis of failures to prevent their recurrence. This results in a high level of air transport reliability. To obtain reliable data for studying the causes of failures, manufacturers add “black boxes” (painted orange for easy identification) which record various flight parameters (FDR – Flight Data Recorder and CVR – Cockpit Voice Recorder). This data is strongly protected to remain readable and analyzable after a crash. Implementing similar solutions in the case of software is one solution. However, it is also necessary to be concerned with process failures and the need to analyze – objectively and without a priori – the processes to see whether they can avoid the reproduction of failures in the future.
Process improvement can only work effectively if the defects and anomalies identified are considered as learning opportunities and not as elements to be avoided or behaviors to be punished.
3.1. Modeling improvements
Improvement is a continuous and iterative activity. Three elements are essential:
– measurement, because it is impossible to improve if we do not measure the level where we are currently, if impartial and complete data are not available;
– planning, in order to identify the improvement actions to be carried out, and to prioritize them in order to obtain systematic and repeatable improvements;
– implementation and monitoring, to ensure that the planned improvements are actually implemented, and bring – by measurement – the desired improvements.
An improvement is not a criticism of the current or past functioning of the processes, but an identification of the elements of the process that can be optimized. Similarly, contractual or hierarchical pressures can have a negative effect on the identification of failures and/or the estimation of their impacts. For more information see Syed (2015).
3.1.1. PDCA and IDEAL
Popularized by Deming, the PDCA model (Plan, Do, Check, Act) is a non-prescriptive iterative model.
PDCA provides four repeating phases:
– a phase to identify and plan the improvements;
– a phase to do improvements;
– a phase to check the effectiveness of the improvements;
– a phase of finer adaptation of the improvements.
This method is simple, easy to understand and known for many years. It is the source of many other methods of improvement.
The IDEAL model extends the PDCA model by adding an initialization phase where the context, the sponsors and the infrastructure are defined. The following phases are:
– identify and diagnose the actual and desired practices;
– define priorities and implementation approaches;
– act where the solution is created, refined and implemented;
– learn where solutions are analyzed and validated and where future solutions are proposed.
More information is available in Bath and van Veenendaal (2014).
3.1.2. CTP
Figure 3.1 The four phases of CTP
Presented in Black (2003), this technique includes 12 critical testing processes in four phases: Plan, Prepare, Perform, Perfect. Each of these processes is evaluated according to a questionnaire and the results can be represented in a radar-type graph.
– Plan is made up of six processes:
- the test process;
- the context discovery process;
- the quality risk analysis process;
- the test effort estimation process;
- the test planning process.
– Prepare covers:
- the test team building process;
- the test system design and implementation process.
– Perform covers:
- the delivery to the test process;
- the test execution process.
– Perfect focuses on:
- the defect reporting process;
- the test result reporting process;
- the change management and improvement process.
Figure 3.2 Example of CTP radar reporting
Each of these four phases builds on the previous and prepares the following ones.
The advantage of the CTP method is that it allows us to analyze our test processes based on the objectives of our organization instead of comparing them to a hypothetical company that would have all the “best practices”.
The initial result of an analysis with the CTP method can be viewed in the form of a radar graph (see Figure 3.2). The broken line represents the current state of the processes, and the aim is to have the widest possible coverage. In the example, we see that processes 7, 8, 11 and 12 are significantly less well rated than the others. Improving these processes will add value faster and more efficiently than any investment in other processes.
3.1.3. SMART
Another acronym used in process improvements is SMART:
– S: specific;
– M: measurable;
– A: attainable;
– R: realizable;
– T: traceable (sometimes also considered as reachable in time).
This acronym applies to all improvements, in order to ensure effective and measurable implementation. The same acronym can be used to assess the quality of requirements and user stories.
3.2. Why and how to improve?
Recall the beginning of our studies or our career. Tasks that seem simple to us now seemed more complicated back then and took longer to complete. In addition, the level of quality was lower than what we are able to achieve now. Training and experience are factors that significantly improve the effectiveness and efficiency of individuals in carrying out their tasks. The most significant improvement comes from experience, which allows us to put training into practice and continue to improve it over time.
The difference in efficiency between a beginner and an expert is of the order of 1–10: the expert will be 10 times faster and 10 times more efficient than the beginner.
It is also possible to improve the effectiveness and efficiency of the processes by using techniques or methods more suited to the objectives to be achieved. This requires identifying the level of efficiency of the processes and considering their improvement according to the results obtained.
In his works (Jones 2000, 2007, 2008; Jones and Bonsignour 2012), Capers Jones demonstrates that some techniques are more effective than others in detecting defects in deliverables. We will therefore want to ensure the current level of efficiency of our processes to identify those that need to be improved.
3.3. Improvement methods
The ISTQB Advanced Level offers various improvement methods based on external benchmarks or on an analysis of the organization’s results.
Improvement against an external benchmark is based on the existence of a set of best practices (the external benchmark) that apply to all organizations and guarantee the ability to deliver quality products. One of the best-known benchmarks is CMMI, which was designed by the SEI (Software Engineering Institute of Carnegie Mellon University) at the request of the DoD (U.S. Department of Defense) to assess the ability of their subcontractors. The external benchmark defines processes, metrics and measures, and assumes that every business implementing the processes will be able to generate quality products. Therefore, by measuring the difference between the organization and the repository, it is possible to identify the improvement actions to be implemented and the order of these improvements.
Improvement in relation to an analysis of the organization’s results is based on an identification – by the organization – of its shortcomings and failures, in order to determine the improvement actions to be put in place to reduce these failures and make up for these shortcomings. This solution requires analyzing the organization’s processes based on a reduced number of processes, the deliverables generated and their use, in order to identify areas for improvement. An evaluation of the benefits linked to the improvement of each process is carried out, and the improved processes in the order of priority are decided.
In the context of this book, we limit ourselves to the improvement of the test processes, but – the test being intimately linked to the development and the design – the improvement of these processes is also necessary.
3.3.1. External/internal referential
Improvements fall into two broad categories:
– Based on definitions from external repositories, “best practices” or standards, which would apply to all organizations, whatever they may be. At this point, it is essential to ensure that the reference system used is adapted to the profile of the organization.
– Based on an identification of gaps or points of attention internal to the organization.
Regardless of the benchmark – external or internal – used, it is necessary to have a picture of the state of the organization (of its processes, shortcomings and strengths), which will serve as a reference against which improvements will be made or measured.
3.3.1.1. Improving with an external referential
Many repositories and groups of “best practices” exist, often based on models such as CMMI. Among these repositories, we have TMMI, TMMI-Next, TPI-Next and TMap-Next. The common denominator of these repositories is that they are based on a definition of processes and best practices that may not be applicable in our field. Note that this reflection also applies to the use of standards.
The advantage of using an external repository – or a standard – is that it includes “good practices” and which can – probably – apply to our organization. The proposed standards must be analyzed to see their applicability to our environment and our organization, as well as to the organization put in place to create – and qualify – the system-of-systems in progress. The implementation of a repository is often easier to gain acceptance and implement because the number of individuals familiar with these repositories is high. However, their level of knowledge may vary greatly (e.g. implementation of CMMI V2.0 compared to the previous version).
3.3.1.2. Improving with an internal referential
Various improvement methods with an internal repository exist: PDCA by Deming, STEP by Craig and Jaskiel (2007) and CTP (Critical Test Processes) are some of them. Unlike improvements based on external benchmarks, process evaluation requires stakeholders with a high level of professionalism – real experts in software testing – which is not always available.
CTP, proposed in Black (2003), is a content-based test process improvement model. Instead of relying on a list of “best practices” from an external repository, CTP identifies 12 processes considered “critical” and measures these processes to identify the improvements to be made.
CTP makes it possible to obtain rapid assessments and improvements based on the feeling and the current need of the organization, without having to refer to an external reference which may not correspond to our own organization. However, CTP uses and emphasizes the importance of metrics and measurements (Bath and van Veenendaal 2014), and quickly adapts to almost any organization and testing process.
3.3.1.3. Improving from defects
As Eleanor Roosevelt said, “Learn from the mistakes of others. You cannot live long enough to do them all yourself.” This way of seeing defects as a way to improve without stigmatizing their authors can – and must – also apply to software defects.
At first glance, software faults seem to appear randomly. Further study may suggest a Machiavellian intelligence of defects (Murphy’s law), which tends to hide in the system and act at the worst time for users (sometimes with severe consequences). All software faults are manifestations of failures in the processes, whether design or testing. Indeed, the design process aims to generate defect-free code and the testing process aims to find all residual defects (among others). If defects appear during testing or use, they demonstrate that the processes did not work as expected.
A detailed analysis of the causes at the origin of the defects (also called “root causes”) should make it possible to identify the design and test processes that have failed to improve them. A statistical analysis of these root causes will determine the frequency with which these processes impact the results, and thus the processes to be improved in priority.
Root cause analysis (RCA) of a failure is a common but relatively expensive technique identifying the (usual) causes involved in the occurrence of a failure.
During the 1980s, IBM research teams developed a simple method called ODC (Orthogonal Defect Classification), associating triggers (types of defects) with design or test processes that generate defects of this type. This technique is a simplification of root cause analysis in that it assigns a single cause to a failure and not multiple causes combined. Below is a description of the triggers and processes involved (D, C, U, F, S).
We can adapt the triggers (the types of failures) and the processes according to the failures identified in our organizations and the processes in place in our organizations. Adaptation requires finely identifying the objectives of each process and the defects that they can introduce or must detect.
| Trigger | D | C | U | F | S |
| Design conformance | X | X | |||
| Logic/flow | X | X | |||
| Backward compatibility | X | X | |||
| Lateral compatibility | X | X | |||
| Concurrency | X | X | |||
| Internal document | X | X | |||
| Language dependency | X | X | |||
| Side effects | X | X | |||
| Rare situation | X | X | |||
| Simple path | X | ||||
| Complex path | X | ||||
| Coverage | X | ||||
| Variation | X | ||||
| Sequencing | X | ||||
| Interaction | X | ||||
| Workload stress | X | ||||
| Recovery | X | ||||
| Start-up/restart | X | ||||
| Hardware configuration | X | ||||
| Software configuration | X | ||||
| Blocked test/normal mode | X |
D: design review;
C: code review;
U: unit test;
F: functional test;
S: system test.
Implementing the ODC method is simple and requires adding eight fields to the fault descriptions: three fields are to be filled in by the discoverer (often the tester), which are the activity in progress when the fault is discovered, the trigger which has made it possible to highlight the defect and the impact that the defect may have, and five fields are to be completed by the corrector (the developer): the target or what has been corrected, the type or nature of the defect, the qualifier of the defect, the source causing the defect and the age of the defect, that is, is it from a new development or is it old and a technical debt? The information to be introduced is generally available to discoverers and correctors, so this does not increase the cost of dealing with anomalies.
As in all process improvement actions, ODC will only work if the identification of failures is done for the purpose of learning and continuous improvement rather than in a toxic environment where errors are “punished”.
The statistical information obtained relates to the maturity of the processes and the product.
3.4. Process quality
The point of measuring the quality of processes is to assess whether they produce the expected level of fault detection, and whether they are effective and efficient. This means defining, for each process, its objectives and metrics to identify whether or not these objectives have been achieved. If the objectives are not achieved, it will be necessary to improve these processes. This evaluation of the quality of the processes is critical in the case of automated processes since there is a tendency to blindly trust these processes.
The need for process trust is critical in fully automated continuous delivery systems (CI/CD – Continuous Integration/Continuous Delivery) insofar as blind trust in the process is made. It is therefore necessary to ensure that this confidence is justified.
3.4.1. Fault seeding
In the case of automated test processes, it is possible to carry out fault seeding. This consists of injecting into the object to be tested several known faults that the process is supposed to detect and checking whether these faults are detected. If they are not, the process needs to be improved. In all cases, it is imperative to ensure that all injected defects have been correctly removed. This method makes it possible to ensure the confidence that can be placed in the process.
This technique also makes it possible to ensure the effectiveness of automated test processes, whether static or dynamic.
3.4.2. Statistics
A second way to ensure the quality of the processes is to measure their effectiveness in terms of achieving their objectives against statistical data from either our organization’s previous projects or other sources. The objective is to see whether the charges, deadlines and other information seem – or not – within the range of data initially envisaged. This of course implies having past measurements against which to compare current measurements.
3.4.3. A posteriori
Another way to ensure – less finely – the quality of the processes is to look at the results of these processes in terms of workload (including the number of test cases planned and executed) and defects found according to the volumetric components to be tested. A comparison of the measurements against previous measurements or industry measurements such as Capers Jones (Jones 2007, 2008; Jones and Bonsignour 2012) allows us to compare the efficiency of our processes against other organizations in the industry.
In Agile methodologies, measuring the traversal velocity of a Kanban board column makes it possible to obtain a measure of the quality of the processes by measuring the number of times a task had to return to the previous column.
The FTR (First Time Right) metric measures this level of quality by ensuring that a component, product or system does not have to undergo corrections. See also section 6.3.1.
3.4.4. Avoiding introduction of defects
To improve the quality of the products delivered, an obvious solution is to avoid introducing defects in the elements used as input to the design, production or testing processes of the systems constituting the system-of-systems.
The continuous search for defects must apply to all design processes, from the identification of needs, until the delivery of the finished product, including testing activities (testers can also make defects). These research activities are not, strictly speaking, testing activities, but they help to identify defects, as well as to verify and validate deliverables.
Various areas of reflection are possible to identify defects early or to prevent their introduction, such as:
– the implementation of reviews from the definition of requirements and for all deliverables designed and used;
– the implementation of defensive programming (verification of input data, use of checklists, provision of error codes in all cases to inform the functioning of the component, the implementation of Poka-Yoke processes for software, for example, with well-designed TDD, etc.);
– formal milestones to ensure that the risks are limited at each of these milestones. This solution is associated with sequential development cycles, but can impose itself in Agile developments such as SAFe;
– analysis of anomalies detected in order to identify their root causes and their introduction phase. A statistical analysis of this information makes it possible to propose improvements to the processes of the phases generating the most frequent anomalies.
3.5. Effectiveness of improvement activities
Some quality improvement and testing activities are intended to prevent the introduction of defects, and others are intended to identify failures. The work of Jones identifies the causes of the introduction of defects according to the size of the software (in function points) and according to the type of software. His works measure the effectiveness of activities in terms of prevention and detection of defects. The table below allows us to view the average values of prevention and removal of anomalies for each of the activities considered. Nothing obliges us to implement all the proposed activities, but we will be able to identify the most or less effective.
In the book by Jones and Bonsignour (2012), numerous activities are identified that impact the efficiency of defect removal in software. We have pre-test activities such as:
– formal design inspections (from 65 to 97% efficiency);
– formal inspection of the code (from 60 to 96%);
– formal inspections of requirements (from 50 to 90%);
– informal peer reviews (from 35 to 60%);
– proofreading (from 25 to 50%).
| Activity | Prevention | Removal |
| 01 Formal requirements inspection | 0.00% | 78.00% |
| 02 Prototyping | -20.00% | 20.00% |
| 03 Architecture | -44.00% | 85.00% |
| 04 Project planning | 0.00% | 3.00% |
| 05 Initial design | 0.00% | 25.00% |
| 06 Detailed design | -16.00% | 45.00% |
| 07 Design reviews (inspections) | 0.00% | 87.00% |
| 08 Coding and static code analysis | -44.00% | 85.00% |
| 09 Reuse of learning | 0.00% | 3.00% |
| 10 Purchase of packages | 0.00% | 3.00% |
| 11 Code inspection | 0.00% | 85.00% |
| 12 IVV (Independent Verification and Validation) | 0.00% | 20.00% |
| 13 Change management | -15.00% | 3.00% |
| 14 Formal integration | 0.00% | 50.00% |
| 15 Automatic user documentation | -8.00% | 10.00% |
| 16 Unit tests | 0.00% | 35.00% |
| 17 Functional tests | 0.00% | 40.00% |
| 18 Integration tests | 0.00% | 50.00% |
| 19 System tests | 0.00% | 42.00% |
| 20 Beta tests | 0.00% | 40.00% |
| 21 Acceptance tests | 0.00% | 35.00% |
| 22 Independent tests | 0.00% | 35.00% |
| 23 Quality assurance and process | -15.00% | 55.00% |
| 24 Installation and training | 0.00% | 3.00% |
| 25 Project management and milestone reviews | -5.00% | 20.00% |
And the following test activities:
| Test activities | From | To |
| Experience-based testing | 60% | 85% |
| Risk-based testing | 55% | 80% |
| Security testing | 50% | 80% |
| Tests of subroutines | 27% | 60% |
| System testing | 27% | 55% |
| External testing/beta testing | 30% | 50% |
| Performance tests | 30% | 45% |
| Delivery testing | 20% | 20% |
| Cloud testing | 25% | 45% |
| Function tests | 33% | 55% |
| Independent verification | 20% | 47% |
| Clean room testing | 20% | 50% |
| Acceptance tests | 15% | 40% |
| Independent testing | 15% | 42% |
3.6. Recommendations
We all tend to hide our mistakes because they could influence our perception of ourselves or the perception of others towards us. This leads to a systematic under-representation of defects:
– Designers will simply remove defects they identify instead of notifying and then removing them. This will result in the impossibility of detecting failures in specifications or of identifying complex or prone components with many failures (EPM – Error-Prone Modules).
– Feedback meetings – sprint retrospectives in the agile world – do not focus on small failures, but only on important concerns. As a result, small failures (many) are not treated and will recur in the future.
– Efficiency and effectiveness measures are not taken, nor compared to the initial estimates, which will imply errors in the load estimate in the following projects.
It may seem pointless or a waste of time to focus on small improvements, but it is the sum of these small improvements that – in the long run – will significantly improve the quality of systems and software.