Acquisition Validation: An Acquisition Engineering Process Area at Maturity Level 3
Purpose
The purpose of Acquisition Validation (AVAL) is to demonstrate that an acquired product or service fulfills its intended use when placed in its intended environment.
Tip
Validation is a series of evaluations in which end users, the product, and other elements of the intended environment interact to determine whether the product will fulfill its intended use. Results may increase confidence in the product or identify issues to be resolved. In contrast with AVER, both the supplier and the acquirer may ensure the utility of the systems in the field.
Introductory Notes
Validation demonstrates that the acquired product or service, as provided, fulfills its intended use. In other words, validation ensures that the acquired product or service meets stakeholders’ needs and customer requirements.
Validation activities are performed early (concept/explorations phases) and incrementally throughout the project lifecycle (including transition to operations and sustainment). These activities can be applied to all aspects of the product and its components in any of their intended environments, such as operations, training, manufacturing, maintenance, and support services. (Throughout the process areas, where the terms “product” and “product component” are used, their intended meanings also encompass services, service systems, and their components.)
Tip
Validation can be applied to all aspects of the product or service in any of its intended environments (e.g., operations, training, manufacturing, maintenance, disposal).
The product or product components that are selected to be validated by the acquirer vary depending on project attributes. Methods used to conduct validation also can be applied to selected acquirer work products (e.g., customer requirements) and supplier deliverables (e.g., prototypes, simulations, demonstrations). Method selection is based on which methods best predict how well the acquired product or service will satisfy stakeholder needs.
Hint
Use well-verified work products in validation; otherwise, you may lose time to disruptions or rediscovery of requirements. Conversely, validation helps you to uncover missing requirements.
Whenever possible, validation should be conducted using the product or product component operating in its intended environment. Either the entire environment or part of it can be used.
When validation issues are identified, these issues are referred to processes associated with the Acquisition Requirements Development or Project Monitoring and Control process areas for resolution.
The specific practices of this process area build on each other in the following way:
• The Select Products for Validation specific practice enables the identification of the product or product component to be validated and methods to be used to perform the validation.
• The Establish the Validation Environment specific practice enables the determination of the environment to be used to carry out the validation.
• The Establish Validation Procedures and Criteria specific practice enables the development of validation procedures and criteria that are aligned with the characteristics of selected products, customer constraints on validation, methods, and the validation environment.
• The Perform Validation specific practice enables the validation to be conducted according to methods, procedures, and criteria.
• The Analyze Validation Results specific practice enables the analysis of validation results against criteria.
Tip
Early AVAL activities may lead to clarification of requirements passed to the supplier. Additional activities may also be demanded of the acquisition organization after acceptance from the supplier and before fully satisfying the end user.
Hint
If you wait until the acceptance test to find issues, you may be in big trouble with your customers and end users.
Related Process Areas
Refer to the Agreement Management process area for more information about accepting the acquired product.
Refer to the Acquisition Requirements Development process area for more information about validating requirements.
Refer to the Acquisition Technical Management process area for more information about evaluating the supplier’s technical solution and managing selected interfaces of that solution.
Specific Practices by Goal
SG 1 Prepare for Validation
Preparation for validation is conducted.
Validation preparation activities include selecting products and product components for validation and establishing and maintaining the validation environment, procedures, and criteria. Items selected for validation may include only the product or it may include appropriate levels of product components used to build the product. Any product or product component may be subject to validation, including replacement, maintenance, and training products, to name a few.
Hint
Any product, product component, or service can benefit from validation. What you select to validate should depend on the issues relating to user needs that pose the highest risk to project success and available resources.
The environment required to validate the product or product component is prepared. The environment can be purchased or can be specified, designed, and built. Environments used for verification can be considered in collaboration with the validation environment to reduce cost and improve efficiency or productivity.
Tip
Integration tests can address validation-type activities (with an end user present to evaluate the integrated product under different scenarios). Thus, for some product components, product integration, verification, and validation activities may be addressed together. Stakeholders may come from the customer community, other acquisition organizations needing functionality for their systems, and the suppliers covered by the acquisition agreement.
Tip
AVAL activities should be designed to reduce interference with supplier and end-user performed activities and to reduce duplication of the validation efforts.
SP 1.1 Select Products for Validation
Select products and product components to be validated and validation methods to be used.
Products and product components are selected for validation based on their relationship to end-user needs. For each product or product component, the scope of the validation (e.g., operational behavior, maintenance, training, user interface) should be determined.
Tip
Validation can be an expensive activity. It takes good judgment to select (and limit) what to validate. Less formal and more frequent “customer use checks” can augment the more formal validation effort.
Hint
When you want or need to validate a product component that has not yet been built, consider developing a prototype. The timely end-user feedback you get from prototyping may more than compensate for the expense of validation.
Validation methods should be selected early in the life of the project so they are clearly understood and agreed to by relevant stakeholders.
Validation methods address the development, maintenance, support, and training for the product or product components as appropriate.
X-Ref
Customer constraints on the conduct of validation are described as part of the development of customer requirements in ARD SP 1.2. Design constraints are described in ARD SG 2 and SG 3.
Tip
Because validation generally involves stakeholders external to the project, it is important early in the project to identify and communicate with them about validation methods so that appropriate preparations can begin.
Example Work Products
1. Lists of products and product components selected for validation
2. Validation methods for each product or product component
3. Requirements for performing validation for each product or product component
4. Validation constraints for each product or product component
Subpractices
1. Identify the key principles, features, and phases for product or product component validation throughout the life of the project.
2. Determine the customer requirements to be validated.
The product or product component should be maintainable and supportable in its intended operational environment. This specific practice also addresses the actual maintenance, training, and support services that can be delivered with the product.
3. Select the product and product components to be validated.
4. Select the evaluation methods for product or product component validation.
5. Review the validation selection, constraints, and methods with relevant stakeholders.
Hint
How might a “shrink-wrapped” product be validated? You can observe users with a prototype in their operational environment, bring users to a special testing laboratory, or release a beta version for end-user testing and feedback.
SP 1.2 Establish the Validation Environment
Establish and maintain the environment needed to support validation.
The requirements for the validation environment are driven by the product or service selected, type of work products (e.g., design, prototype, final version), and validation methods. These selections can yield requirements for the purchase or development of equipment, software, or other resources. The validation environment can include the reuse of existing resources. In this case, arrangements should be made for the use of these resources.
Hint
Preparing for and conducting validation requires coordination with many external groups. Obtain commitment from these groups to support the planned validation efforts.
Hint
The product (or prototype) may need special interfaces and functionality to properly interact with elements of the validation environment (e.g., data recording equipment). Develop these requirements and incorporate them with other product requirements.
Early selection of products or product components to be validated, work products to be used in validation, and validation methods is needed to ensure that the validation environment will be available when necessary.
Tip
When validation environments are shared across multiple programs such as ranges or high-fidelity test beds, the acquirer must proactively plan for the use, evolution, and control of these environments.
The validation environment should be carefully controlled to provide for replication, results analysis, and revalidation of problem areas.
Example Work Products
1. Validation environment
Subpractices
1. Identify requirements for the validation environment.
2. Identify customer supplied products.
3. Identify validation equipment and tools.
4. Identify validation resources that are available for reuse and modification.
5. Plan the availability of resources in detail.
Tip
Because validation resembles a controlled experiment, and because the validation environment must maintain fidelity with the operational environment, many tools, simulations, computers, networks, and skilled people may need to be involved. Thus validation planning may itself be challenging.
SP 1.3 Establish Validation Procedures and Criteria
Establish and maintain procedures and criteria for validation.
Validation procedures and criteria are defined to ensure the product or product component will fulfill its intended use when placed in its intended environment. Acceptance test cases and procedures can be used for validation procedures.
Tip
This practice helps to answer questions such as how you will exercise the product or service prototype to better understand a particular issue (validation procedures) and how you will know whether the performance is acceptable (validation criteria).
The validation procedures and criteria include validation of maintenance, training, and support services.
These procedures also address the validation of requirements and the acquired product or service throughout the project lifecycle. Typically, formal acceptance testing procedures and criteria are established to ensure the delivered product or service meets stakeholder needs before it is deployed in the intended environment.
X-Ref
ARD addresses requirements validation. Requirements validation determines the adequacy and completeness of the requirements.
The validation procedures and criteria applicable to the supplier are typically referenced in the solicitation package and supplier agreement.
1. Validation procedures
2. Validation criteria
3. Test and evaluation procedures for maintenance, training, and support
Hint
Items validated might be shown as a table with columns identifying items to validate, issues to investigate, related requirements and constraints, and validation methods. The table might also list work products to be verified by the supplier (under VER in CMMI-DEV), the work products to be verified by the acquisition organization (under AVER in this model), and the verification methods to be used. Using one table for verification and validation may lead you to discover opportunities to combine verification and validation efforts.
Subpractices
1. Review the requirements to ensure that issues affecting validation of the acquired product or service are identified and resolved.
2. Document the environment, operational scenario, procedures, inputs, outputs, and criteria for the validation of the acquired product or service.
3. Assess the product or service as it matures in the context of the validation environment to identify validation issues.
SG 2 Validate Selected Products and Product Components
Selected products and product components are validated to ensure they are suitable for use in their intended operating environment.
The validation methods, procedures, criteria, and the environment are used to validate the selected products and product components and associated maintenance, training, and support services. Validation activities are performed throughout the project lifecycle.
Tip
Validation is not applied solely to discover missing functionality, and it is not limited to the end user’s operational environment. Other features, environments, and categories of user needs should be considered.
Validation activities are performed by the acquirer, the supplier, or both parties in accordance with the supplier agreement.
SP 2.1 Perform Validation
Perform validation on selected products and product components.
To be acceptable to stakeholders, a product or product component must perform as expected in its intended operational environment.
Hint
The bottom line: You must determine whether the product will perform as expected.
Validation activities are performed and the resulting data are collected according to established methods, procedures, and criteria.
The as-run validation procedures should be documented and the deviations occurring during the execution should be noted as appropriate.
Tip
The validation environment may support the automatic collection of much of the data.
Example Work Products
1. Validation reports
2. Validation results
3. Validation cross reference matrix
4. As-run procedures log
5. Operational demonstrations
SP 2.2 Analyze Validation Results
Analyze results of validation activities.
The data resulting from validation tests, inspections, demonstrations, or evaluations are analyzed against defined validation criteria. Analysis reports indicate whether needs were met. In the case of deficiencies, these reports document the degree of success or failure and categorize probable causes of failure. The collected test, inspection, or review results are compared with established acceptance criteria to determine whether to proceed or to address requirements or design issues with the supplier.
Hint
Validation activities are expensive, so it is important to maximize learning during this process. Analyzing the results of validation activities may help you to discover missing requirements, features in the product that delight the customer, lingering issues, and unmitigated risks. In a system-of-systems environment, results may also suggest needed changes in adjacent systems to maximize the capability being delivered.
Analysis reports or as-run validation documentation may also indicate that bad test results are due to a validation procedure problem or a validation environment problem.
Example Work Products
1. Validation deficiency reports
2. Validation issues
3. Procedure change request
Subpractices
1. Compare actual results to expected results.
2. Based on the established validation criteria, identify products and product components that do not perform suitably in their intended operating environments, or identify problems with methods, criteria, or the environment.
3. Analyze validation data for defects.
4. Record results of the analysis and identify issues.
5. Use validation results to compare actual measurements and product performance to the intended use or operational need.
6. Identify, document, and track action items to closure for work products that do not pass their validation.
Refer to the Project Monitoring and Control process area for more information about managing corrective action to closure.
Hint
If problems with the validation methods, environment, procedures, or criteria arise, you must revisit project activities that correspond to the specific practices of SG 1.