The purpose of Acquisition Verification (AVER) is to ensure that selected work products meet their specified requirements.
Tip
AVER involves ensuring that the evolving work products of the acquisition project meet specified requirements for those products. The work products covered by AVER are those “owned” by the acquisition organization, not those “owned” by the supplier. The suppliers’ verification activities with their work products are covered by VER in the CMMI-DEV model and remain the responsibility of the supplier.
Acquisition verification addresses whether acquirer work products properly reflect specified requirements.
The Acquisition Verification process area involves the following activities:
• Preparing for verification
• Performing verification
• Identifying corrective action
Verification is inherently an incremental process because it occurs throughout the acquisition of the product or service, beginning with verification of requirements and plans, progressing through the verification of evolving work products such as design and test results, and culminating in the verification of the completed product.
X-Ref
The acquirer reviews the supplier verification activities and evaluates the results in ATM. Verification expectations of the supplier should be covered in the supplier agreement in SSAD.
The specific practices of this process area build on each other in the following way:
• The Select Work Products for Verification specific practice enables the identification of work products to be verified, methods to be used to perform the verification, and documented requirements to be satisfied by each selected work product.
• The Establish the Verification Environment specific practice enables the selection or creation of the environment to be used to carry out the verification.
• The Establish Verification Procedures and Criteria specific practice enables the development of verification procedures and criteria that are aligned with selected work products, requirements, methods, and characteristics of the verification environment.
• The Prepare for Peer Reviews, Conduct Peer Reviews, and Analyze Peer Review Data specific practices enable the performance of peer reviews, an important type of verification that is a proven mechanism for effective defect removal.
Tip
Peer reviews also focus on getting the work product “right” and on obtaining the data necessary to prevent defects and improve the process.
• The Perform Verification specific practice enables the conduct of verification according to available methods, procedures, and criteria.
• The Analyze Verification Results specific practice enables analysis of verification results against established criteria.
Refer to the Acquisition Validation process area for more information about confirming that a product or product component fulfills its intended use when placed in its intended environment.
Refer to the Acquisition Requirements Development process area for more information about generating and developing customer and contractual requirements.
Refer to the Acquisition Technical Management process area for more information about evaluating supplier work products.
Refer to the Requirements Management process area for more information about managing requirements.
X-Ref
Many books and other sources describe testing and peer reviews (sometimes known as inspections). For a summary of software testing principles and terminology, see the entry for “Software Testing” on the Wikipedia site, www.wikipedia.org.
Understand that supplier wok product verification activities are governed by the supplier agreement (SSAD), reviewed and evaluated in ATM, but conducted using VER practices in CMMI-DEV. For complex systems that demand access to environments controlled by the acquirer or customer, collaborative approaches to verification may be essential.
Specific Goal and Practice Summary
SG 1 Prepare for Verification
SP 1.1 Select Work Products for Verification
SP 1.2 Establish the Verification Environment
SP 1.3 Establish Verification Procedures and Criteria
SG 2 Perform Peer Reviews
SP 2.1 Prepare for Peer Reviews
SP 2.2 Conduct Peer Reviews
SP 2.3 Analyze Peer Review Data
SG 3 Verify Selected Work Products
SP 3.1 Perform Verification
SP 3.2 Analyze Verification Results
Preparation for verification is conducted.
Up-front preparation is necessary to ensure that verification provisions are embedded in contractual requirements, constraints, plans, and schedules. Verification includes the selection, inspection, testing, analysis, and demonstration of acquirer work products.
Hint
Identify which work products put the project at the highest risk. These work products are often the required elements that must be shared with the supplier and confirmed with the customer.
Verification methods include, but are not limited to, inspections, peer reviews, audits, walkthroughs, analyses, simulations, testing, and demonstrations. Practices related to peer reviews as a verification method are included in specific goal 2.
Hint
Don’t forget to verify work products important to all phases of the product lifecycle, such as maintenance documentation, installation services, and operator training. Often, these are the responsibility of the acquisition organization under an agreement with the customer rather than provided by the suppliers under the agreements the acquirer administers with them.
Preparation for verification includes the definition of support tools, test equipment and software, simulations, prototypes, and facilities.
Select work products to be verified and verification methods to be used.
Acquirer work products are selected based on their contribution to meeting project objectives and requirements, and to addressing project risks.
Tip
Reverification is not called out separately in this process area, since it is actually an iteration of the verification process. However, reverification should be considered when planning verification activities. For example, when fixing a requirements defect, the project team will need to inspect (reverify) other acquisition documents such as systems engineering plans, test plans, transition plans, and so forth to ensure that the project baselines remain consistent.
Selection of verification methods typically begins with the definition of requirements to ensure that the requirements are verifiable. Reverification should be addressed by verification methods to ensure that rework performed on work products does not cause unintended defects. Suppliers should be involved in this selection to ensure that the project’s methods are appropriate for the supplier’s environment.
- Lists of work products selected for verification
- Verification methods for each selected work product
Tip
Methods used for each work product may be shown in a table with columns identifying the work product to be verified, requirements to be satisfied, and verification methods to be used. As mentioned in AVAL, some activities across the various agreements with customers and suppliers may allow a combination of verification and validation activities for synergy.
- Identify acquirer work products for verification.
- Identify requirements to be satisfied by each selected work product.
A traceability matrix is a useful tool for identifying requirements for each selected work product, as it typically includes information that relates requirements to work products. When identifying requirements for each selected work product, consult the traceability matrix maintained as part of managing requirements for the project.
Refer to the Maintain Bidirectional Traceability of Requirements specific practice in the Requirements Management process area for more information about tracing requirements to work products.
- Identify verification methods available for use.
- Define verification methods to be used for each selected work product.
- Identify verification activities and methods to be used in the project plan.
Refer to the Project Planning process area for more information about coordinating with project planning.
Establish and maintain the environment needed to support verification.
Tip
Some work products and verification methods may require special facilities and tools. These should be identified and obtained in advance.
An environment must be established to enable verification to take place. The type of environment required depends on the work products selected for verification and the verification methods used. A peer review may require little more than a package of materials, reviewers, and a room. A product test may require simulators, emulators, scenario generators, data reduction tools, environmental controls, and interfaces with other systems.
Tip
In the case of peer reviews, a co-located team might meet in a room where the document being peer-reviewed can be displayed. Remote team members might participate through teleconferencing and use a Web-based collaboration tool that allows them to see the document while hearing the discussion.
The verification environment may be acquired, developed, reused, modified, or a combination of these, depending on the needs of the project.
- Verification environment
Tip
The organization’s information technology (IT) or facilities group, or perhaps other projects, might have some of the verification resources that a project might need. In some cases, domain-specific models, simulators, environmental labs, or antenna ranges are common resources used by multiple projects and must be reserved for use and should be verified as adequate for use.
- Identify verification environment requirements.
- Identify verification resources that are available for reuse or modification.
- Identify verification equipment and tools.
- Acquire verification support equipment and an environment (e.g., test equipment and software).
Establish and maintain verification procedures and criteria for the selected work products.
Hint
The bottom line is to determine verification methods, environments, and procedures as early in the project as is practical.
Verification criteria are defined to ensure that work products meet their requirements.
Tip
In the case of engineering artifacts (e.g., architectures, designs, and implementations), the primary source for verification criteria is likely to be the requirements assigned to the work product being verified. The acquiring organization may need to verify that the customer’s environment can actually support the system being acquired from the supplier.
- Verification procedures
- Verification criteria
- Generate a set of comprehensive, integrated verification procedures for work products and commercial off-the-shelf products, as necessary.
- Develop and refine verification criteria as necessary.
- Identify the expected results, tolerances allowed, and other criteria for satisfying the requirements.
- Identify equipment and environmental components needed to support verification.
Peer reviews are performed on selected work products.
Tip
Peers are generally coworkers from your project that have interest in the item under peer review. Peer review participants should generally not include your management because that could restrain open and beneficial dialog. It may even be valuable to consider peers outside the project as suitable participants for specific items under review.
The intent is for the acquisition project to use peer reviews on selected products (e.g., solicitation documents, systems engineering plans, test plans) they produce internally to find and remove defects and to ensure compliance to acquisition standards. Many work products produced by the acquisition project set the stage for project success and are critical to supplier performance.
Peer reviews provide opportunities to learn and share information across the team.
Peer reviews are an important part of verification and are a proven mechanism for effective defect removal. An important corollary is to develop an understanding of work products and the processes that produced them to help prevent defects and identify opportunities.
Peer reviews are applied to acquirer-developed work products. These reviews involve a methodical examination of work products by the acquirer’s peers to identify defects for removal and to recommend other changes that are needed. Example work products to be peer-reviewed include the solicitation package and the supplier agreement.
Prepare for peer reviews of selected work products.
Preparation activities for peer reviews typically include identifying the staff to be invited to participate in the peer review of each work product; identifying key reviewers who must participate in the peer review; preparing and updating materials to be used during peer reviews, such as checklists and review criteria; and scheduling peer reviews.
- Peer review schedule
- Selected work products to be reviewed
Hint
Use peer reviews not just for contractual artifacts but also for project management artifacts (e.g., plans), process management artifacts (e.g., process descriptions), and support artifacts (e.g., measure definitions).
- Determine the type of peer review to be conducted.
- Establish and maintain checklists to ensure that work products are reviewed consistently.
- Distribute the work product to be reviewed and related information to participants early enough to enable them to adequately prepare for the peer review.
- Assign roles for the peer review, as appropriate.
Conduct peer reviews of selected work products and identify issues resulting from these reviews.
One of the purposes of conducting a peer review is to find and remove defects early. Peer reviews are performed incrementally as work products are being developed. These reviews are structured and are not management reviews.
X-Ref
Peer reviews are used extensively in software development projects to great success. The principles can be easily applied in other environments (e.g., systems engineering, acquisition, and operations). For a historical perspective of peer review principles and terminology, see the entry for “Software Inspections” on the Wikipedia site, www.wikipedia.org.
- Peer review results
- Peer review issues
- Peer review data
- Perform the assigned roles in the peer review.
- Identify and document defects and other issues in the work product.
- Record results of the peer review, including action items.
Hint
A checklist identifies the classes of defects that commonly occur for a type of work product. By using the checklist, you’re less likely to overlook certain classes of defects. Also, some classes of defects might be assigned to different peer review participants, helping to ensure adequate coverage of the checklist.
- Collect peer review data.
Refer to the Measurement and Analysis process area for more information about data collection.
- Identify action items and communicate issues to relevant stakeholders.
- Conduct an additional peer review if needed.
Analyze data about the preparation, conduct, and results of the peer reviews.
Tip
A classic problem that arises in low-maturity organizations is that participants skip preparation when under schedule pressure. One way to address this is to incorporate the peer review schedule into the acquisition project plans and schedule to help ensure adequate allocation of time. Another way is to postpone the peer review if participants have not prepared.
Refer to the Measurement and Analysis process area for more information about obtaining and analyzing data.
Typical Work Products
- Peer review data
- Peer review action items
- Record data related to the preparation, conduct, and results of peer reviews.
- Analyze the peer review data.
Selected work products are verified against their specified requirements.
Tip
The ratio often reported that compares the cost to find a defect late in the project to the cost to find a defect early in the project is 100:1.
Verification methods, procedures, criteria, and the environment are used to verify selected work products and associated maintenance, training, and support services. Verification activities should be performed throughout the project lifecycle. Practices related to peer reviews as a verification method are included in specific goal 2.
Hint
Provide a nonthreatening environment so that open discussions can occur. Once you start to include nonpeers (e.g., management, customers, and suppliers) in peer reviews, the effectiveness of peer reviews decreases dramatically.
Perform verification on selected work products.
Verifying work products incrementally promotes early detection of problems and can result in the early removal of defects. The results of verification save the considerable cost of fault isolation and rework associated with troubleshooting problems.
Hint
During peer review training, train the staff in all of the roles necessary to conduct the peer reviews. These roles can vary from one peer review to the next.
- Verification results
- Verification reports
- Demonstrations
- As-run procedures log
Tip
Defects and issues are often recorded in a table with columns for such things as location of defect, defect description (including type and origin), discussion, and action items.
- Perform the verification of selected work products against their requirements.
- Record the results of verification activities.
- Identify action items resulting from the verification of work products.
- Document the “as-run” verification method and deviations from available methods and procedures discovered during its performance.
Tip
The analysis can help us answer questions such as what types of defects we are encountering, their severity, in which phases they are being injected and detected, the peer review “yield” (percent of defects detected), the review rate (pages per hour), and the cost (or hours expended) per defect found.
Analyze results of all verification activities.
Actual results must be compared to established verification criteria to determine acceptability.
The results of the analysis of verification results are recorded as evidence that verification was conducted. The acquirer might consult supplier work product verification results and reports to conduct verification activities of acquirer work products.
X-Ref
Some acquisition projects conduct “murder boards” on critical acquisition work products such as a Request for Proposal (RFP) or Statement of Work (SOW). A murder board is a comprehensive, line-by-line walkthrough of a document by all relevant stakeholders.
For more information on murder boards, see A Survival Guide for Project Managers by James Taylor (AMACOM).
Refer to the Acquisition Technical Management process area for more information about evaluating supplier work products and reviewing verification results.
For each work product, all available verification results are incrementally analyzed and corrective actions are initiated to ensure that documented requirements have been met. Corrective actions are typically integrated into project monitoring activities. Since a peer review is one of several verification methods, peer review data should be included in this analysis activity to ensure that verification results are analyzed sufficiently.
Analysis reports or “as-run” method documentation may also indicate that bad verification results are due to method problems, criteria problems, or a verification environment problem.
Refer to the Project Monitoring and Control process area for more information about corrective actions.
Hint
Sometimes the verification procedure cannot be run as defined (e.g., incorrect assumptions were made as to the nature of the work product or verification environment). If so, record any deviations.
- Analysis report (e.g., statistics on performance, causal analysis of nonconformances, comparison of the behavior between the real product and models, and trends)
- Trouble reports
- Change requests for verification methods, criteria, and the environment
Typical Supplier Deliverables
- Verification results
- Verification reports
Hint
When piloting a new verification tool, analyze results to help identify ways to adjust the process or tool to increase the effectiveness of verification activities.
- Compare actual results to expected results.
- Based on the established verification criteria, identify products that do not meet their requirements or identify problems with methods, procedures, criteria, and the verification environment.
- Analyze the defect data.
- Record all results of the analysis in a report.
- Provide information on how defects can be resolved (including verification methods, criteria, and verification environment) and formalize it in a plan.
Tip
The verification criteria established in SP 1.3 play an important role in determining where problems are.