After completing this chapter, you will be able to:

• – understand what is meant by verification and validation (V&V);
• – understand the benefits and costs of applying V&V techniques;
• – understand the traceability technique and its usefulness;
• – learn about the IEEE 1012 V&V standard and models used in industry;
• – understand the processes and activities of V&V;
• – understand the activities of the software validation phase;
• – learn how to develop and use a V&V checklist for your project;
• – understand how to write a V&V plan for your project;
• – learn about the V&V tools available;
• – understand the relationship between V&V and the software quality assurance plan.

7.1 Introduction

In an article about safety, Leveson [LEV 00] brilliantly explains the dangers of modern software-based systems. The following text box summarizes her thinking.

The introduction of innovative products associated with software that contain an increasing number of functions often involves increasing the number of computer processing units and size of software. For example, in the automobile sector, over 60 small computers (named electronic control units (ECUs)) with more than 100 million lines of code from different suppliers are used in many car models [REI 04]. These networked ECUs control, amongst others, ignition, braking, the entertainment system, and now autopilot and self-parking. Failure of some units will result in recalls, dissatisfaction of customers, or deterioration in the performance of the vehicle, whereas a failure with the autopilot, direction, acceleration, or braking system could cause an accident, injury, or death.

According to the IEEE 1012 standard for V&V [IEE 12], the goal of V&V in software projects is to help the organization incorporate quality in the software throughout the software life cycle. The V&V process provides an objective evaluation of software products and processes. It is a simple question of addressing quality during development as opposed to trying to add quality to a product after its been built.

Often it is not possible, and perhaps seldom reasonable, to inspect all the fine details of all the software products created during the development and maintenance life cycle, particularly because of a lack of time and budget. Therefore, all organizations have to make certain compromises and this is where the software engineer is expected to follow a rigorous process of justification and selection. Software teams must establish V&V activities during the project planning phase, so as to choose the techniques and approaches that will allow the products to have a proper level of V&V. The choice of these activities and their priorities is based on assessing risk factors and their potential impact. These V&V activities need to be added to the development process of the project in order to reduce risks to an acceptable level.

In this chapter, we present V&V for software. First, we explain the concepts of V&V and the intended benefits and costs. We then present the international standards and models that define V&V activities or impose them in certain situations. We continue with the presentation of an inventory of the various V&V techniques available as well as their utility to address the particular concerns of the software engineer. We then present the typical contents of a V&V process. This is followed by a brief discussion on the importance of independent V&V (IV&V) in critical projects. We then provide more detail for one of the V&V activities often required for safety: traceability. Finally, we explain how to develop and use checklists.

Verification aims to show that an activity was done correctly (doing it right), in accordance with its implementation plan and has not introduced defects in its output. It can be done on successive intermediate states of a product that is the outcome of an activity.

Validation is composed of a series of activities which start, early in the development life cycle, with the validation of customer requirements. End-users or their representatives will also evaluate the behavior of the software product in the target environment, either real, simulated, or on paper.

Validation helps minimize the risk of developing the wrong items by ensuring that the requirements are adequate and complete. Subsequently, it will be ensured that these validated requirements are developed during the following phase, notably, the specifications. Validation also ensures that the software does not do what it should not do. This means that no unintended behavior should arise from it.

If quality assurance is the poor cousin of software development, validation has the same relationship with V&V. While verification practices, such as testing, have a very important place in academia and industry, we cannot say the same for validation techniques. Validation techniques are often absent or ignored by developers and the mandated development processes. Some organizations validate requirements early in a project. Unfortunately, they will carry out some validation only at the very end. Occasionally, we find validation practices embedded at different stages of the development cycle (as shown by Figure 7.1). The lines at the top of this figure indicate the development cycle phases where validation activities can be performed.

Some organizations explicitly have a phase called software validation in their software development process, and if they produce software that is integrated into a system, they will also explicitly have a system validation phase. Figure 7.2 illustrates this with a software life cycle named the V development life cycle process. It illustrates, along the center arrow, that system and software validation plans that will be executed during the validation phase originate from the system and software specification phases. These plans will be updated throughout the development phases and will be used during the validation phases on the ascending line of activities shown on the right hand side of the figure.

One reason for preparing these plans early in the development life cycle is that validation activities may require special equipment or environments to perform the validation of the system that includes software. It is possible that a testing environment also needs to be established. For example, the validation of an air traffic control system that needs to operate in conditions where dozens of aircraft are in the air at the same time may require validation of the system near a busy airport. This would enable the validation of several functional and non-functional requirements of the system.

Figure 7.2 shows that system and software validation plans are developed during the descending part of the development cycle in the V-shaped diagram. These plans will be used during the subsequent phases of development to validate, among other things, the system requirements and software requirements against the needs and during the ascending part of the development cycle in the V-shaped diagram to validate the software during the validation phase. Since the software is a component of a system, it will be integrated with hardware or other software and subject to system validation activities.

After identifying risks and the required V&V techniques, the V&V activities need to be planned. In some projects, V&V activities are planned by a team belonging to different parts of an organization, for example, system engineers, software developers, supplier personnel, a risk manager, V&V or software quality assurance (SQA), software testers, a configuration manager, etc. The main objective of the V&V activity will be to develop a detailed V&V plan for the project.

7.2 Benefits and Costs of V&V

As we have mentioned above, the goal of V&V is to build quality into the software early during its construction and not just try to fix this at the testing stage. Figure 7.3 shows an example of the software processes, in an American company, where defects are injected. The figure shows that a large percentage of defects, approximately 70%, are injected even before any line of code has been produced. It is therefore necessary that we include techniques in the software life cycle that will allow for the early detection and removal of these defects as close as possible to when they are created. Additionally, good detection techniques will greatly reduce the costly rework associated with corrections, which is an important cause of schedule delays.

Figure 7.4 describes the defect detection effectiveness in an American company [SEL 07]. These results originate from Northrop Grumman who collected data for 14 systems where 3418 defects where detected during 731 reviews. These systems contained between 25,000 and 500,000 lines of code and the corresponding teams ranged from 10 to 120 developers. This study shows that not only is it possible to detect errors, but also to eliminate them in the same phase where they were produced. For example, Figure 7.3, shows that 50% of the defects where injected in the requirements phase. Figure 7.4 also shows that 96% of these defects where eliminated in the same phase.

What can be learned from this figure is that it is possible to estimate the percentage of injected defects and to detect and correct a high percentage of them such that they will not propagate from one phase to another. This process has been named the error containment process. It is therefore possible to describe, in the quality plan of the project, quantitative quality criteria concerning defect removal objectives at each phase of the project.

Why is it important to correct defects where they are injected? As shown in Chapter 2 (Figure 2.2), there is a compounding cost of a defect that is not corrected in the phase where it is injected. For example, a defect that arises during the assembly phase will cost three times more to fix than one that is corrected during the previous phase (during which we should had been able to find it). It will cost seven times more to fix the defect in the next phase (test and integration), 50 times more in the trial phase, 130 times more in the integration phase, and 100 times more when it is a failure for the client and has to be repaired during the operational phase of the product.

7.3 V&V Standards and Process Models

The most important standards and process models that describe required processes and practices for V&V are presented next: the ISO 12207 [ISO 17], the IEEE 1012 [IEE 12], and the CMMI^®. Some standards will go as far as to recommend, for critical software, that some programming languages be avoided. For example, a railway control software standard forbids that programmers use the “GoTo” programming instruction and that they remove “dead code” before the final delivery of the product. In the next section, the IEEE 1012 standard is explained, then other standards will be briefly covered.

7.4 V&V According to ISO/IEC/IEEE 12207

The ISO 12207 [ISO 17] standard also presents the requirements for V&V processes. We will not describe all the details here but provide a high level view of the V&V processes, their purpose, and outcomes.

7.5 V&V According to the CMMI Model

Another perspective of V&V can be seen in process models like the CMMI. The staged representation of the CMMI for Development [SEI 10a] has two process areas, at maturity level 3, dedicated to V&V. Preparation for verification is the first step suggested by the CMMI. It consists of selecting the life cycle phase outputs and the methods chosen for each product, in order to prepare the verification activity, or environment, depending on the specific needs of the project. It also suggests that verification success criteria and an iterative procedure be put in place, in parallel to the product design activities.

The purpose of verification is to ensure that selected work products meet their specified requirements. The verification process area includes the following specific goals (SG) and specific practices (SP) [SEI 10a]:

• SG 1 Prepare for verification
  • SP 1.1 Select work products for verification,
  • SP 1.2 Prepare the verification environment,
  • SP 1.3 Establish verification procedures criteria;
• SG 2 Perform peer reviews
  • SP 2.1 Prepare for peer reviews,
  • SP 2.2 Conduct peer reviews,
  • SP 2.3 Analyze peer review data;
• SG 3 Verify selected work products
  • SP 3.1 Perform verification,
  • SP 3.2 Analyze verification results.

CMMI-DEV recommends inspections and walk-throughs for peer reviews as they have been described in a previous chapter.

The purpose of validation is to demonstrate that a product or product component fulfills its intended use when placed in its intended environment. The validation process area includes the following SG and SP [SEI 10a]:

• SG 1 Prepare for validation
  • SP 1.1 Select products for validation,
  • SP 1.2 Establish the validation environment,
  • SP 1.3 Establish validation procedures and criteria;
• SG 2 Validate product or product components
  • SP 2.1 Perform validation,
  • SP 2.2 Analyze validation results.

Validation can be applied to all aspects of the product within its target operational environment: operation, training and maintenance and support. The validation should be executed in a real operational environment with actual data volumes.

V&V activities are often executed together and can use the same environment. End-users are usually invited to conduct the validation activities.

7.6 ISO/IEC 29110 and V&V

The ISO 29110 standard for very small entities has already been introduced. Elements of ISO 12207 V&V processes have been used to develop ISO 29110 standards and guides. This section shows how these very small organizations can conduct V&V using one of the four recommended profiles: the ISO 29110 Basic profile. This profile describes two processes: a project management (PM) process and a software implementation (SI) process.

One of the seven objectives of a PM process is to prepare a project plan describing the activities and tasks for the development of a software for a specific customer. Required tasks and resources are sized and estimated early on. In this plan, V&V tasks are described and are reviewed between the development team and the customer, and then approved.

One of the objectives is that the V&V tasks, for each identified work product, be done according to stated exit criteria to ensure the coherence between the outputs and the inputs of each development task. Defects are identified and corrected and the quality records stored in the V&V report.

Table 7.6 lists the V&V tasks. The table shows the role of the person executing the task, a brief description of the task, the input and output, and their states (in brackets). The following acronyms are used for roles: TL for technical lead, AN for analyst, PR for programmer, CUS for customer, and DES for designer. Table 7.6 is limited to describing only the first task in detail and then lists only the subsequent task names.

ISO 29110 Basic profile imposes a minimum number of V&V tasks to ensure that the end product will meet the requirements and needs of the customer even with a small budget.

ISO 29110 also suggests that a verification result file be updated in order to record the V&V activity results. Table 7.7 shows an example of the proposed format for this important project quality record.

7.7 Independent V&V

IV&V are V&V activities conducted by an independent organization. This can be used to supplement internal V&V and is often used for very critical software: medical devices, metro and railway control, and airplane navigation systems.

Technical independence requires that the V&V effort use personnel who are not involved in the development of the system or its elements. Managerial independence requires that the responsibility for the IV&V effort be vested in an organization separate from the development and program management organizations. Financial independence requires that control of the IV&V budget be vested in an organization independent of the development organization.

7.7.1 IV&V Advantages with Regards to SQA

SQA and V&V are the main organizational processes, that is, the “watchdogs,” put in place to ensure process, product, and service quality. Since software development is under pressure to deliver, there is a need to counter balance the situation so that quality is not forgotten. Internal politics can interfere with these processes and this is why IV&V can be useful.

Given that SQA is part of the development organizational process, this function sometimes has very little influence when there are schedule and cost pressures. The IV&V process is like an external watchdog representing the client's interests and not those of the developers.

Figure 7.5 describes the relationships between customer, supplier, and IV&V.

7.8 Traceability

Software traceability is a simple V&V technique that ensures that all the user requirements have been:

• – documented in specifications;
• – developed and documented in the design document;
• – implemented in the source code;
• – tested;
• – delivered.

Traceability facilitates the development of test plans and test cases. It ensures that the resulting tests have covered all the approved requirements. With traceability, we focus on detecting the following situations: a need without a specification, a specification without a design element, or a design element without source code or tests.

7.9 Validation Phase of Software Development

In some organizations, validation activities have been regrouped into a single development life cycle phase. It is often located at the end of the process. The objective of this last phase is to prove that the software meets the initial requirements, for example, that the right product was developed. The software is tested by the end-users to ensure it is fit for use in a real environment. The validation plan scenarios and test cases are developed and baselined during the integration and test phase.

Figure 7.6 presents a validation process using the Entry-Task-Verification-eXit (ETVX) notation presented earlier in this book. In certain situations, the validation phase will be split into many steps [CEG 90]:

• – testing in the presence of the customer or its representative;
• – installation of the software in the operational environment;
• – user acceptance testing, where the software is either accepted as is or accepted providing defects are corrected or not accepted. In the case where the software is accepted providing defects are corrected, the errors detected during the tests must be corrected and the software tested again before the customer accepts this software;
• – end-user trial testing: use, in production, of the software in trial mode;
• – warranty period, where the system is delivered and used, defects are corrected and change requests are processed;
• – software final acceptance.

The validation phase is very important for the organization. Indeed, the success of this phase will lead to the transfer of the software to the client and, more importantly, to payment to the supplier when a contract is involved. For the developer it is often followed by a final project review where lessons learned are compiled to be used for process improvement.

The end of the validation also leads to the use, in production, of the software and the start of the support phase. The transition to maintenance is also an important phase of the life cycle. Even if there are still minor defects, they will be addressed during the maintenance phase.

During the validation phase, a series of tests are performed. It is not uncommon for anomalies to be detected and that minor changes are required. In addition, corrections or changes must be made, testing the corrected components as well as regression testing must be performed, and the configuration management process must be used to ensure that the changes are reflected in all of the documentation. At this time, the traceability matrix is used to ensure that all documents in the process have been corrected.

Validation can also lead to product qualification or even external certification in certain domains. For example, the Food and Drug Administration (FDA) requires a pre-market submission to the FDA before the release of the software in some situations.

7.9.1 Validation Plan

A software validation plan, written by the project manager, lists the organization and resources required to validate software. It should be approved during the software specification review and it describes:

• – the validation activities planned as well as the roles, responsibilities, and resources assigned;
• – the grouping of the test iterations, steps, and objectives.

To develop this plan, the project manager can use the following source documents: contractual documents, project plan, specifications document, system validation plan (if applicable), software quality plan, and the organization template for the validation plan and the validation plan checklist. Figure 7.7 describes a typical table of contents for the validation plan.

The many roles and responsibilities of the individuals involved in creating this plan can be summarized as follows (adapted from [CEG 90]):

• – the project manager:

• write the validation plan;
• get the approval of the client during a review that takes place at the end of the software specification phase;
• update the plan as required during the subsequent phases;
• supervise the execution of the validation plan;

• – the tester:

• execute the validation plan;
• organize and lead test iterations;
• produce the test iteration report;
• raise defect reports and agree on defect severity;

• – test execution support personnel:

• prepare and configure the test environment;
• get the testing documents from configuration management;
• execute test procedures;
• find defects;
• correct defects;
• correct any documentation impacted by the correction of defects;

• – customer:

• approve and sign the software validation plan;
• approve and sign the test iteration minutes;
• approve the defect correction list;

• – SQA personnel:

• review the software validation plan and provide comments;
• verify that the right versions of documents are used;
• assist with testing iterations;
• assist the project team during the lessons learned review;

• – configuration management personnel:

• provide the latest approved versions of documents required for tests;
• assist the testing team when an error is found or with a minor modification request;
• prepare deliverables identified in the contract and project plan;
• archive project artifacts according to guidelines.

The validation plan does not necessarily need to be a document of its own. The information presented here can also be a section within the SQA plan or of the project plan depending on the size of the project.

7.10 Tests

Tests are central to the V&V of a software. There are four major categories of tests: development tests, qualification, acceptance, and operational tests. The following text box provides their definitions.

7.11 Checklists

A checklist is a tool that facilitates the verification of a software product and its documentation. It contains a list of criteria and questions to verify the quality of a process, product, or service. It also ensures the consistency and completeness of the execution of tasks.

An example of a checklist is one that helps detect and classify a defect (e.g., an oversight, a contradiction, or an omission). A checklist can also be used to ensure that a list of tasks to be accomplished was completed, like a “to do list.” Elements of a checklist are specific to the document, activity, or process. For example, a verification checklist to review a plan is different than a code review checklist. In this section, the following topics are presented:

• – how to develop a checklist;
• – how to use a checklist;
• – how to improve and manage a checklist.

We also provide examples of different types of checklists. Following is the description of an anecdote about the creation of the first checklist.

7.11.1 How to Develop a Checklist

There are two popular approaches used in the development of a checklist. The first is to use an existing list, such as the ones available in this book or those found on the Internet, and adapt them to your needs. The second approach is to develop a checklist from a list of errors, omissions, and problems that were already noted during document reviews and lessons learned reviews. We will see how to improve these lists below.

According to Gilb, checklists are developed according to some rules [GIL 93]:

• – a checklist must be derived, among others things, from process rules or from a standard;
• – a checklist should include a reference to the rule it is inspired from and that it is interpreting;
• – a checklist should not exceed one page because it is difficult to memorize and effectively use a list containing more than twenty items to be checked;
• – a keyword should describe each item in the list, this facilitates its retention;
• – a checklist must include a version number and the date of the last update;
• – the checklist items can be stated using a sentence structure that responds in the affirmative if the condition is satisfied. For example, regarding the clarity of a requirement: “the requirement is clear” and not “the requirement is not ambiguous”;
• – a checklist can contain a classification, for example, the severity of defects: major or minor;
• – a checklist should not contain all possible questions or details, as a concise list should focus on key issues and steps that need to be executed sequentially;
• – a checklist should be kept updated to reflect the experience gained by the organization and its developers.

Lastly, a checklist should be included in the training of the individual user. It does not replace the knowledge required to perform the tasks listed. Table 7.9 describes a checklist used to classify defects.

Defect class number	Defect type	Description
10	Documentation	Comments, messages
20	Syntax	Spelling, punctuation, instruction format
30	Build, package	Change management, library, version management
40	Assignment	Declaration, name duplication, scope, limits
50	Interface	Procedure call, input/output (I/O), user format
60	Validation checks	Error messages, inadequate validation
70	Data	Structure, content
80	Function	Logic, pointers, loops, recursion, calculations, function call defect
90	System	Configuration, timing, memory
100	Environment	Design, compilation, test, other system support problems

Figure 7.8 shows an example of a checklist to verify software requirements, hence the abbreviation used for this list is REQ. Note, that for each item of the checklist, a keyword has been added. Keywords greatly facilitate the memorization of the items of the checklist.

7.12 V&V Techniques

Tools and techniques can be used to help perform V&V activities. Using these tools is highly dependent on the integrity level of the applications, the maturity of the product, the corporate culture, and the type of development, modeling, and simulation paradigm of individual projects.

The degree to which verification activities can be automated directly influences the overall efficiency of V&V efforts. As there is no formal process for selecting tools, it is important to select the right tool. Ideally, modeling and simulation tools used during the design and the development phases should be integrated with the verification tools. Moreover, validation does not permit a detailed match with modeling and simulation processes.

The market for verification tools is large. It is easy to find a list of at least a hundred vendors on the Internet today. These tools fall into the following two categories:

• – generic tools supporting data results from validation:

• database management systems;
• data manipulation tools;
• data modeling tools;

• – formal methods:

• formal language;
• mechanized reasoning tools (automated theorem proofs);
• model verification (checker) tool.

7.13 V&V Plan

The V&V plan essentially answers the following questions: what do we verify and/or validate? How and when, and by whom will the V&V activities be performed and what level of resources will be required?

IEEE 1012 specifies that the V&V effort starts with the production of a plan that addresses the following list of elements. If there is an irrelevant item that the project should not cover in its plan, it is better to state that “This section is not applicable to this project” instead of removing the item from the plan. This allows the SQA to clearly see that the item was not forgotten by the project team. Of course, additional topics may be added. If elements of the plan are already documented in other documents, the plan should refer to it instead of repeating it. The plan must be maintained throughout the software life cycle. The V&V plan proposed by IEEE 1012 includes the following (without listing the system and hardware V&V elements) [IEE 12]:

1. Purpose
2. Referenced documents
3. Definitions
4. V&V overview
   1. 4.1 Organization
   2. 4.2 Master schedule
   3. 4.3 Integrity level scheme
   4. 4.4 Resources summary
   5. 4.5 Responsibilities
   6. 4.6 Tools, techniques, and methods
5. V&V processes
   1. 5.1 Common V&V processes, activities, and tasks
   2. 5.2 System V&V processes, activities, and tasks
   3. 5.3 Software V&V processes, activities, and tasks
      1. 5.3.1 Software concept
      2. 5.3.2 Software requirements
      3. 5.3.3 Software design
      4. 5.3.4 Software construction
      5. 5.3.5 Software integration test
      6. 5.3.6 Software qualification test
      7. 5.3.7 Software acceptance test
      8. 5.3.8 Software installation and checkout (Transition)
      9. 5.3.9 Software operation
      10. 5.3.10 Software maintenance
      11. 5.3.11 Software disposal
   4. 5.4 Hardware V&V processes, activities, and tasks
6. V&V reporting requirements
   1. 6.1 Task reports
   2. 6.2 Anomaly reports
   3. 6.3 V&V final report
   4. 6.4 Special studies reports (optional)
   5. 6.5 Other reports (optional)
7. V&V administrative requirements
   1. 7.1 Anomaly resolution and reporting
   2. 7.2 Task iteration policy
   3. 7.3 Deviation policy
   4. 7.4 Control procedures
   5. 7.5 Standards, practices, and conventions
8. V&V test documentation requirements

7.14 Limitations OF V&V

No technique can prevent all errors or defects. Regarding V&V, we note the following limitations [SCH 00]:

• – Impracticability of testing all the data: for most programs, it is virtually impossible to try to review the program with all possible inputs, due to the multitude of possible combinations;
• – Impracticability of testing all the branch conditions: for most programs, it is impractical to try to test all the possible execution paths of a software. This is also due to the multitude of possible combinations;
• – Impracticability of obtaining absolute proof: there is no absolute proof of correctness of a software based system unless formal specifications can prove it to be correct and accurately reflect user expectations.

It is not uncommon for test plans to be designed by the developer of the system and then approved by the V&V staff. This practice is far from ideal to guarantee a high level of quality. Although the V&V role should not be part of the development team, sometimes the developer becomes the evaluator of his own software. It is therefore important that the V&V role consist of people who have good knowledge and experience of systems in order to provide sound evaluations of the quality of the resulting product.

7.15 V&V in the SQA Plan

The IEEE 730 standard discusses V&V and starts with a statement that SQA activities need to be coordinated with the verification, validation, review, audit, and other life cycle processes needed to ensure the conformity and quality of the final product. There is no need to duplicate efforts here. The standard asks the project team to ensure that the V&V concerns have been well explained in the V&V or SQA plan.

For verification activities, the standard lists the following questions that the project team members should ask themselves [IEE 14]:

• – has verification between the system requirements and the system architecture been performed?
• – have verification criteria for software items been developed that ensure compliance with the software requirements allocated to the items?
• – has an effective validation strategy been developed and implemented?
• – have appropriate criteria for validation of all required work products been identified?
• – have verification criteria been defined for all software units against their requirements?
• – has verification of the software units compared with the requirements and the design been accomplished?
• – have adequate criteria for verification of all required software work products been identified?
• – have required verification activities been performed adequately?
• – have results of the verification activities been made available to the customer and other involved parties?

For the validation of the software, the standard recommends that the tools that will be used for validation be chosen and assessed based on product risk and that the project team evaluate whether these tools need validation. If they are validated, they are to keep the records of this validation. It also asks the team to answer each of the following questions [IEE 14]:

• – have all tools that require validation been validated before using them?
• – has an effective validation strategy been developed and implemented?
• – have appropriate criteria for validation of all required work products been identified?
• – have required validation activities been performed adequately?
• – have problems been identified, recorded, and resolved?
• – has evidence been provided that the software work products as developed are suitable for their intended use?
• – have results of the validation activities been provided to the customer and other involved parties?

Of particular interest in the SQA plan, the project team will pay special attention to the acceptance process and how to classify defects until an exit criterion is met. It is important to clarify the final testing process stage of a project as it is the last line of defense before going to production.

7.16 Success Factors

The execution of the V&V practices can be helped or slowed down depending on a number of organizational factors. The next text box lists some of these factors.