Acquisition Verification: An Acquisition Engineering Process Area at Maturity Level 3
Purpose
The purpose of Acquisition Verification (AVER) is to ensure that selected work products meet their specified requirements.
Tip
AVER involves ensuring that the evolving work products of the acquisition project meet specified requirements for those products. The work products covered by AVER are those “owned” by the acquisition organization—not those “owned” by the supplier.
Introductory Notes
Acquisition verification addresses whether acquirer work products properly reflect specified requirements.
The Acquisition Verification process area involves the following activities:
• Preparing for verification
• Performing verification
• Identifying corrective action
Verification is inherently an incremental process because it occurs throughout the acquisition of the product or service, beginning with verification of requirements and plans, progressing through the verification of evolving work products, and culminating in the verification of the completed product.
X-Ref
Understand that supplier work product verification activities are governed by the supplier agreement (SSAD), reviewed and evaluated in ATM, but conducted using VER practices in CMMI-DEV. For complex systems that demand access to environments controlled by the acquirer or customer, collaborative approaches to verification may be essential.
The specific practices of this process area build on each other in the following way:
• The Select Work Products for Verification specific practice enables the identification of work products to be verified, methods to be used to perform the verification, and documented requirements to be satisfied by each selected work product.
• The Establish the Verification Environment specific practice enables the selection or creation of the environment to be used to carry out the verification.
• The Establish Verification Procedures and Criteria specific practice enables the development of verification procedures and criteria that are aligned with selected work products, requirements, methods, and characteristics of the verification environment.
• The Prepare for Peer Reviews, Conduct Peer Reviews, and Analyze Peer Review Data specific practices enable the conduct of peer reviews, an important type of verification that is a proven mechanism for effective defect removal.
• The Perform Verification specific practice enables the conduct of verification according to established methods, procedures, and criteria.
• The Analyze Verification Results specific practice enables analysis of verification results against established criteria.
Tip
Peer reviews also focus on getting the work product “right” and on obtaining the data necessary to prevent defects and improve the process.
Related Process Areas
Refer to the Acquisition Requirements Development process area for more information about developing and analyzing customer and contractual requirements.
Refer to the Acquisition Technical Management process area for more information about evaluating the supplier’s technical solution and managing selected interfaces of that solution.
Refer to the Acquisition Validation process area for more information about demonstrating that an acquired product or service fulfills its intended use when placed in its intended environment.
Refer to the Requirements Management process area for more information about managing requirements.
Specific Practices by Goal
SG 1 Prepare for Verification
Preparation for verification is conducted.
Up-front preparation is necessary to ensure that verification provisions are embedded in contractual requirements, constraints, plans, and schedules. Verification includes the selection, inspection, testing, analysis, and demonstration of acquirer work products.
Verification methods include but are not limited to inspections, peer reviews, audits, walkthroughs, analyses, architecture evaluations, simulations, testing, and demonstrations. Practices related to peer reviews as a verification method are included in specific goal 2.
Hint
Don’t forget to verify work products that are important to all phases of the product lifecycle.
Preparation for verification includes the definition of support tools, test equipment and software, simulations, prototypes, and facilities.
SP 1.1 Select Work Products for Verification
Select work products to be verified and verification methods to be used.
Acquirer work products are selected based on their contribution to meeting project objectives and requirements, and to addressing project risks.
Hint
Identify which work products put the project at the highest risk. These work products are often the required elements that must be shared with the supplier and confirmed with the customer.
Tip
Reverification is not called out separately in this process area; however, reverification should be considered when planning verification activities. For example, when fixing a requirements defect, the project team will need to inspect (reverify) other acquisition documents, such as systems engineering plans, test plans, transition plans, and so forth, to ensure that the project baselines remain consistent.
Selection of verification methods typically begins with the definition of requirements to ensure that the requirements are verifiable. Re-verification should be addressed by verification methods to ensure that rework performed on work products does not cause unintended defects.
Example Work Products
1. Lists of work products selected for verification
2. Verification methods for each selected work product
1. Identify acquirer work products for verification.
2. Identify requirements to be satisfied by each selected work product.
A traceability matrix is a useful tool for identifying requirements for each selected work product, as it typically includes information that relates requirements to work products. When identifying requirements for each selected work product, consult the traceability matrix maintained as part of managing requirements for the project.
Refer to the Maintain Bidirectional Traceability of Requirements specific practice in the Requirements Management process area for more information about tracing requirements to work products.
3. Identify verification methods available for use.
4. Define verification methods to be used for each selected work product.
5. Identify verification activities and methods to be used in the project plan.
Refer to the Project Planning process area for more information about developing a project plan.
Tip
By incorporating such a matrix into the project plan (perhaps by reference), resources can be provided and commitments made to perform the appropriate verification activities.
Tip
Some work products and verification methods may require special facilities and tools. These items should be identified and obtained in advance.
SP 1.2 Establish the Verification Environment
Establish and maintain the environment needed to support verification.
An environment should be established to enable verification to take place. The type of environment required depends on the work products selected for verification and the verification methods used. A peer review may require little more than a package of materials, reviewers, and a room. A product test can require simulators, emulators, scenario generators, data reduction tools, environmental controls, and interfaces with other systems.
Tip
In the case of engineering artifacts (e.g., architectures, designs, implementations), the primary source for verification criteria is likely to be the requirements assigned to the work product being verified. The acquiring organization may need to verify that the customer’s environment can actually support the system being acquired from the supplier.
The verification environment can be acquired, developed, reused, modified, or obtained using a combination of these activities, depending on the needs of the project.
Example Work Products
1. Verification environment
Tip
In peer reviews, a team might meet in a room where the document being peer-reviewed can be displayed. Remote team members might participate using teleconferencing and a web-based collaboration tool.
Subpractices
1. Identify verification environment requirements.
2. Identify verification resources that are available for reuse or modification.
3. Identify verification equipment and tools.
4. Acquire verification support equipment and an environment (e.g., test equipment, software).
Tip
The organization’s information technology (IT) or facilities group, or perhaps other projects, might have some of the verification resources that a project needs. In some cases, domain-specific models, simulators, environmental labs, and antenna ranges are common resources used by multiple projects; if so, they may need to be reserved for use and should be verified as adequate for use.
SP 1.3 Establish Verification Procedures and Criteria
Establish and maintain verification procedures and criteria for the selected work products.
Verification criteria are defined to ensure that work products meet their requirements.
Hint
The bottom line: You should select verification methods, environments, and procedures as early in the project as is practical.
Example Work Products
1. Verification procedures
2. Verification criteria
Subpractices
1. Generate a set of comprehensive, integrated verification procedures for work products and commercial off-the-shelf products, as necessary.
2. Develop and refine verification criteria as necessary.
3. Identify the expected results, tolerances allowed, and other criteria for satisfying the requirements.
4. Identify equipment and environmental components needed to support verification.
Hint
Often, there isn’t a single right result, but rather a range of results that might be acceptable. In such instances, specify how much variability from the expected answer is considered acceptable.
SG 2 Perform Peer Reviews
Peer reviews are performed on selected work products.
Peer reviews are an important part of verification and are a proven mechanism for effective defect removal. An important corollary is to develop an understanding of work products and the processes that produced them to help prevent defects and identify opportunities.
Tip
Peer reviews provide opportunities to learn and share information across the team.
Peer reviews are applied to acquirer developed work products. These reviews involve a methodical examination of work products by the acquirer’s peers to identify defects for removal and to recommend other changes that are needed.
Tip
The acquisition project uses peer reviews on selected internally generated products (e.g., solicitation documents, systems engineering plans, test plans) to find and remove defects and to ensure compliance with acquisition standards. Many work products produced by the acquisition project set the stage for project success and are critical to supplier performance.
SP 2.1 Prepare for Peer Reviews
Prepare for peer reviews of selected work products.
Preparation activities for peer reviews typically include identifying the staff to be invited to participate in the peer review of each work product; identifying key reviewers who should participate in the peer review; preparing and updating materials to be used during peer reviews, such as checklists and review criteria; and scheduling peer reviews.
Example Work Products
1. Peer review schedule
2. Selected work products to be reviewed
Hint
Use peer reviews not just for contractual artifacts, but also for project management artifacts (e.g., plans), process management artifacts (e.g., process descriptions), and support artifacts (e.g., measure definitions).
Subpractices
1. Determine the type of peer review to be conducted.
2. Establish and maintain checklists to ensure that work products are reviewed consistently.
3. Distribute the work product to be reviewed and related information to participants early enough to enable them to adequately prepare for the peer review.
4. Assign roles for the peer review as appropriate.
Hint
Using a checklist that identifies the classes of defects that commonly occur for a type of work product makes it less likely that defects will be overlooked. Also, some classes of defects might be assigned to different peer review participants.
SP 2.2 Conduct Peer Reviews
Conduct peer reviews of selected work products and identify issues resulting from these reviews.
One of the purposes of conducting a peer review is to find and remove defects early. Peer reviews are performed incrementally as work products are being developed. These reviews are structured and are not management reviews.
Hint
To maximize the effectiveness of the peer review, participants need to prepare prior to the meeting.
Example Work Products
1. Peer review results
2. Peer review issues
3. Peer review data
Tip
The cost of finding a defect late instead of early in the project is often reported as 100:1.
1. Perform the assigned roles in the peer review.
2. Identify and document defects and other issues in the work product.
3. Record results of the peer review, including action items.
4. Collect peer review data.
Refer to the Measurement and Analysis process area for more information about obtaining measurement data.
Hint
Train the staff in all of the roles necessary to conduct the peer reviews. These roles can vary from one peer review to the next.
5. Identify action items and communicate issues to relevant stakeholders.
6. Conduct an additional peer review if needed.
Tip
The analysis can help answer questions about the types of defects being encountered, their severity, the phases in which they are being injected and detected, the peer review “yield” (i.e., the percentage of defects detected), the review rate (pages per hour), and the cost (or hours expended) per defect found.
SP 2.3 Analyze Peer Review Data
Analyze data about the preparation, conduct, and results of the peer reviews.
Refer to the Measurement and Analysis process area for more information about obtaining measurement data.
Example Work Products
1. Peer review data
2. Peer review action items
Subpractices
1. Record data related to the preparation, conduct, and results of the peer reviews.
2. Analyze the peer review data.
SG 3 Verify Selected Work Products
Selected work products are verified against their specified requirements.
Verification methods, procedures, criteria, and the environment are used to verify selected work products and associated maintenance, training, and support services. Verification activities should be performed throughout the project lifecycle. Practices related to peer reviews as a verification method are included in specific goal 2.
X-Ref
Some acquisition projects conduct “murder boards” on critical acquisition work products such as a Request for Proposal (RFP) or Statement of Work (SOW). A murder board is a comprehensive, line-by-line walkthrough of a document by all relevant stakeholders. For more information on murder boards, see A Survival Guide for Project Managers by James Taylor (AMACOM).
SP 3.1 Perform Verification
Perform verification on selected work products.
Verifying work products incrementally promotes early detection of problems and can result in the early removal of defects. The results of verification save the considerable cost of fault isolation and rework associated with troubleshooting problems.
1. Verification results
2. Verification reports
3. Demonstrations
4. As-run procedures log
Subpractices
1. Perform the verification of selected work products against their requirements.
2. Record the results of verification activities.
3. Identify action items resulting from the verification of work products.
4. Document the “as-run” verification method and deviations from established methods and procedures discovered during its performance.
Hint
Sometimes the verification procedure cannot be run as defined—for example, if incorrect assumptions were made as to the nature of the work product or verification environment. If so, record any deviations.
SP 3.2 Analyze Verification Results
Analyze results of all verification activities.
Actual results should be compared to established verification criteria to determine acceptability.
The results of the analysis of verification results are recorded as evidence that verification was conducted. The acquirer might consult supplier work product verification results and reports to conduct verification activities of acquirer work products.
Tip
Analysis helps to identify areas (or risks) on which to focus limited resources.
Refer to the Acquisition Technical Management process area for more information about evaluating the supplier’s technical solution and managing selected interfaces of that solution.
Hint
When piloting a new verification tool, analyze results to help identify ways to adjust the process or tool to increase the effectiveness of verification activities.
For each work product, all available verification results are incrementally analyzed and corrective actions are initiated to ensure that documented requirements have been met. Corrective actions are typically integrated into project monitoring activities. Since a peer review is one of several verification methods, peer review data should be included in this analysis activity to ensure that verification results are analyzed sufficiently.
Tip
Peer reviews are addressed in SG 2; this specific practice addresses all other forms of verification.
Analysis reports or “as-run” method documentation may also indicate that bad verification results are due to method problems, criteria problems, or verification environment problems.
Refer to the Project Monitoring and Control process area for more information about taking corrective action.
Example Work Products
1. Analysis report (e.g., statistics on work product performance, causal analysis of nonconformances, comparison of the behavior between the real product and models, trends)
2. Trouble reports
3. Change requests for verification methods, criteria, and the environment
Example Supplier Deliverables
1. Verification results
2. Verification reports
Subpractices
1. Compare actual results to expected results.
2. Based on the established verification criteria, identify products that do not meet their requirements or identify problems with methods, procedures, criteria, and the verification environment.
3. Analyze the defect data.
4. Record all results of the analysis in a report.
5. Provide information on how defects can be resolved (including verification methods, criteria, and verification environment) and formalize it in a plan.
Tip
The verification criteria established in SP 1.3 play an important role in determining where problems are.