The purpose of Acquisition Validation (AVAL) is to demonstrate that an acquired product or service fulfills its intended use when placed in its intended environment.
Validation demonstrates that the acquired product or service, as provided, will fulfill its intended use. In other words, validation ensures that the acquired product or service meets stakeholders’ needs and customer requirements.
Validation activities are performed early and incrementally throughout the project lifecycle. These activities can be applied to all aspects of the product and its components in any of their intended environments, such as operations, training, manufacturing, maintenance, and support services. (Throughout the process areas, where we use the phrase product and product component, the intended meaning encompasses services and their components.)
The product or product components that are selected to be validated by the acquirer vary depending on project attributes. Methods used to conduct validation also can be applied to selected acquirer work products (e.g., customer requirements) and supplier deliverables (e.g., prototypes, simulations, and demonstrations). Method selection is based on which methods best predict how well the acquired product or service will satisfy stakeholder needs.
Tip
Validation activities can be applied to all aspects of the product or service in any of its intended environments, such as operations, training, manufacturing, maintenance, and disposal.
Whenever possible, validation should be conducted using the product or product component operating in its intended environment. Either the entire environment or part of it can be used.
When validation issues are identified, these issues are referred to processes associated with the Acquisition Requirements Development or Project Monitoring and Control process areas for resolution.
Hint
If you wait until the acceptance test to find issues, you may be in big trouble with your customers and end users.
The specific practices of this process area build on each other in the following way.
• The Select Products for Validation specific practice enables the identification of the product or product component to be validated and methods to be used to perform the validation.
• The Establish the Validation Environment specific practice enables the determination of the environment to be used to carry out the validation.
• The Establish Validation Procedures and Criteria specific practice enables the development of validation procedures and criteria that are aligned with the characteristics of selected products, customer constraints on validation, methods, and the validation environment.
• The Perform Validation specific practice enables the performance of validation according to methods, procedures, and criteria.
• The Analyze Validation Results specific practice enables the analysis of validation results against criteria.
Hint
A well-verified work product should be used in validation; otherwise, you may lose valuable time with disruptions or rediscovering requirements you already knew. Conversely, validation helps you to uncover missing needs that you can incorporate into requirements.
Refer to the Acquisition Requirements Development process area for more information about requirements validation.
Refer to the Agreement Management process area for more information about accepting the acquired product or service.
Refer to the Acquisition Technical Management process area for more information about evaluating supplier work products.
Tip
Early AVAL activities may lead to clarification of requirements passed to the supplier. Additional activities may also be demanded of the acquisition organization after acceptance from the supplier and before fully satisfying the end user.
Specific Goal and Practice Summary
SG 1 Prepare for Validation
SP 1.1 Select Products for Validation
SP 1.2 Establish the Validation Environment
SP 1.3 Establish Validation Procedures and Criteria
SG 2 Validate Selected Products and Product Components
SP 2.1 Perform Validation
SP 2.2 Analyze Validation Results
Preparation for validation is conducted.
Validation preparation activities include selecting products and product components for validation and establishing and maintaining the validation environment, procedures, and criteria. Items selected for validation may include only the product or it may include appropriate levels of product components used to build the product. Any product or product component may be subject to validation, including replacement, maintenance, and training products, to name a few.
Hint
Any product or product component can benefit from validation. What you select to validate should depend on the issues relating to user needs that pose the highest risk to project success and available resources.
The environment required to validate the product or product component is prepared. The environment may be purchased or may be specified, designed, and built. Environments used for verification may be considered in collaboration with the validation environment to reduce cost and improve efficiency or productivity.
Tip
Integration tests can address validation-type activities (with an end user present to evaluate the integrated product under different scenarios). Thus, for some product components, product integration, verification, and validation activities may be addressed together. Stakeholders may come from the customer community, other acquisition organizations needing functionality for their systems, and the suppliers covered by the acquisition agreement.
AVAL activities should be designed to reduce interference with supplier and end-user performed activities and to reduce duplication of the validation efforts.
Select products and product components to be validated and validation methods to be used.
Products and product components are selected for validation based on their relationship to user needs. For each product or product component, the scope of the validation (e.g., operational behavior, maintenance, training, and user interface) should be determined.
Tip
Validation can be an expensive activity. It takes good judgment to select (and limit) what to validate. Less formal and more frequent “customer use checks” can augment the more formal validation effort.
Validation methods should be selected early in the life of the project so they are clearly understood and agreed to by relevant stakeholders.
Hint
When you seek to validate a product component that has not yet been built, consider developing a prototype based on the component’s requirements and design. The timely end-user feedback you obtain may more than compensate for the expense of the validation exercise.
Validation methods address the development, maintenance, support, and training for the product or product components, as appropriate.
X-Ref
Customer constraints on the conduct of validation are described as part of the development of customer requirements in ARD SP 1.2. Design constraints are described in ARD SG 2 and SG 3.
Hint
The product (or prototype) may need special interfaces and functionality to properly interact with elements of the validation environment (e.g., data recording equipment). Develop these requirements and incorporate them with other product requirements.
Typical Work Products
- Lists of products and product components selected for validation
- Validation methods for each product or product component
- Requirements for performing validation for each product or product component
- Validation constraints for each product or product component
Tip
Because validation generally involves stakeholders external to the project, it is important early in the project to identify and communicate with them about validation methods so that appropriate preparations can begin.
Subpractices
- Identify the key principles, features, and phases for product or product component validation throughout the life of the project.
- Determine the customer requirements to be validated.
The product or product component must be maintainable and supportable in its intended operational environment. This specific practice also addresses the actual maintenance, training, and support services that may be delivered with the product.
- Select the product and product components to be validated.
- Select the evaluation methods for product or product component validation.
- Review the validation selection, constraints, and methods with relevant stakeholders.
Hint
Items selected for validation might be shown as a table with columns identifying items to be validated, issues to be investigated, related requirements and constraints, and validation methods. The table might also list the work products to be verified by the supplier (under VER in CMMI-DEV), those to be verified by the acquisition organization (under AVER in this model), and the verification methods to be used. Using one table to address all of these work products for verification and validation may lead you to discover opportunities to combine verification and validation efforts.
Establish and maintain the environment needed to support validation.
The requirements for the validation environment are driven by the product or service selected, type of work products (e.g., design, prototype, and final version), and validation methods. These selections may yield requirements for the purchase or development of equipment, software, or other resources. The validation environment may include the reuse of existing resources. In this case, arrangements for the use of these resources must be made.
Tip
Validation is not applied only to discover missing functionality and is not limited to the end-user operational environment. Other features, environments, and categories of user needs should be considered.
Hint
How might a “shrink-wrapped” product be validated? You can observe users with a prototype in their operational environment; bring users to a special testing laboratory; or release a beta version for end-user testing and feedback.
Early selection of products or product components to be validated, work products to be used in validation, and validation methods is needed to ensure that the validation environment will be available when necessary.
Hint
Preparing for and conducting validation requires coordination with many external groups. Obtain commitment from these groups to support the planned validation efforts.
The validation environment should be carefully controlled to provide for replication, results analysis, and revalidation of problem areas.
- Validation environment
Tip
When validation environments are shared across multiple programs such as ranges or high-fidelity test beds, the acquirer must proactively plan for the use, evolution, and control of these environments.
Subpractices
- Identify requirements for the validation environment.
- Identify customer-supplied products.
- Identify reuse items.
- Identify validation equipment and tools.
- Identify validation resources that are available for reuse and modification.
- Plan the availability of resources in detail.
Establish and maintain procedures and criteria for validation.
Validation procedures and criteria are defined to ensure the product or product component will fulfill its intended use when placed in its intended environment. Acceptance test cases and procedures may be used for validation procedures.
Tip
This practice helps to answer questions such as how you will exercise the product prototype to better understand a particular issue (validation procedures) and how you will know whether the performance is acceptable (validation criteria).
The validation procedures and criteria include validation of maintenance, training, and support services.
These procedures also address the validation of requirements and the acquired product or service throughout the project lifecycle. Typically, formal acceptance testing procedures and criteria are established to ensure the delivered product or service meets stakeholder needs before it is deployed in the intended environment.
The validation procedures and criteria applicable to the supplier are typically referenced in the solicitation package and supplier agreement.
Typical Work Products
- Validation procedures
- Validation criteria
- Test and evaluation procedures for maintenance, training, and support
Subpractices
- Review the requirements to ensure that issues affecting validation of the acquired product or service are identified and resolved.
- Document the environment, operational scenario, procedures, inputs, outputs, and criteria for the validation of the acquired product or service.
- Assess the product or service as it matures in the context of the validation environment to identify validation issues.
Selected products and product components are validated to ensure they are suitable for use in their intended operating environment.
The validation methods, procedures, criteria, and the environment are used to validate the selected products and product components and associated maintenance, training, and support services. Validation activities are performed throughout the project lifecycle.
Validation activities are performed by the acquirer, the supplier, or both parties in accordance with the supplier agreement.
Perform validation on selected products and product components.
To be acceptable to stakeholders, a product or product component must perform as expected in its intended operational environment.
Validation activities are performed and the resulting data are collected according to established methods, procedures, and criteria.
The as-run validation procedures should be documented and the deviations occurring during the execution should be noted, as appropriate.
Typical Work Products
- Validation reports
- Validation results
- Validation cross-reference matrix
- As-run procedures log
- Operational demonstrations
Analyze results of validation activities.
The data resulting from validation tests, inspections, demonstrations, or evaluations are analyzed against defined validation criteria. Analysis reports indicate whether needs were met. In the case of deficiencies, these reports document the degree of success or failure and categorize probable causes of failure. The collected test, inspection, or review results are compared with established acceptance criteria to determine whether to proceed or to address requirements or design issues with the supplier.
Hint
Validation activities are expensive and it is important to maximize learning. Therefore, analyzing the results of validation activities may help you to discover missing requirements, features in the product that delight the customer, lingering issues, and risks. In a system–of-systems environment, results may also suggest needed changes in adjacent systems to maximize the capability being delivered.
Analysis reports or as-run validation documentation may also indicate that bad test results are due to a validation procedure problem or a validation environment problem.
Typical Work Products
- Validation deficiency reports
- Validation issues
- Procedure change request
Subpractices
- Compare actual results to expected results.
- Based on the established validation criteria, identify products and product components that do not perform suitably in their intended operating environments, or identify problems with methods, criteria, or the environment.
- Analyze validation data for defects.
- Record results of the analysis and identify issues.
- Use validation results to compare actual measurements and performance to the intended use or operational need.
- Identify, document, and track action items to closure for work products that do not pass their validation.
Refer to the Project Monitoring and Control process area for more information about tracking action items.
Hint
If requirements are missing, you must revisit your acquisition processes. If there are problems with the validation methods, environment, procedures, or criteria, you must revisit project activities that correspond to the specific practices of SG 1.