Acquisition Technical Management: An Acquisition Engineering Process Area at Maturity Level 3
Purpose
The purpose of Acquisition Technical Management (ATM) is to evaluate the supplier’s technical solution and to manage selected interfaces of that solution.
Tip
The focus of ATM is on the various suppliers’ products as they evolve into a usable solution; the focus of AVER is on the acquirer’s work products.
Introductory Notes
The Acquisition Technical Management process area focuses on the following:
• Conducting technical reviews of the supplier’s technical solution
• Analyzing the development and implementation of the supplier’s technical solution to confirm technical progress criteria or contractual requirements are satisfied
• Managing selected interfaces
• Typically, these activities interactively support one another to gauge technical progress and allow effective management of project technical risks. Different levels of detailed analysis, depending on the development progress and insight required, may be needed to conduct technical reviews to the acquirer’s satisfaction. Prototypes, simulations, and technology demonstrations created by the supplier can be used to help gauge technical progress, gauge readiness for technical reviews, assess technical risks, and gain knowledge needed to manage selected interfaces.
Tip
Many problems encountered in the integration of product or service components or the product or service transition to operations are caused by interface incompatibilities. Thus ATM covers interface management.
In some acquisitions, the acquirer assumes the role of overall systems engineer, architect, or integrator for the product. In these acquisitions, the Technical Solution process area of CMMI-DEV should also be used. Technical Solution in CMMI-DEV includes additional information about designing, developing, and implementing solutions, including the design approaches, design concepts, and alternative solutions for which an acquirer can have varying degrees of responsibility.
Acquisition Technical Management activities involve measuring technical progress and the effectiveness of plans and requirements. Activities include the ones associated with technical performance measurement and the conduct of technical reviews. A structured review process should demonstrate and confirm completion of required accomplishments and exit criteria as defined in project planning and technical plans (e.g., the systems engineering management plan). Acquisition Technical Management activities discover deficiencies or anomalies that often result in corrective action.
Acquisition Technical Management should be performed with other technical and agreement management processes, such as requirements management, risk management, configuration management, data management, and agreement management.
X-Ref
ATM is driven by the contractual requirements defined by ARD, managed by REQM, and established contractually in SSAD. The technical management in ATM has to coordinate with the supplier agreement management in AM. The processes associated with these process areas interact significantly to accomplish their purposes.
For acquisition of product lines and standard services, the practices described in this process area (as well as Acquisition Requirements Development practices) can also be applied to identify and acquire core assets used in developing, acquiring, or customizing products or service systems. (See the definition of “product line” in the glossary.)
When the supplier is using an Agile method, end users or their proxies are typically involved in the supplier’s development processes. In such a situation, the acquirer also needs to be involved to ensure that needed changes to requirements and supplier agreements are acceptable given the constraints of the acquisition, and also to incorporate the changes into the supplier agreements.
Refer to the Plan Stakeholder Involvement specific practice in the Project Planning process area for more information about supporting the use of Agile methods.
Related Process Areas
Refer to the Agreement Management process area for more information about satisfying supplier agreements.
Refer to the Acquisition Requirements Development process area for more information about allocating contractual requirements and establishing operational concepts and scenarios.
Refer to the Configuration Management process area for more information about controlling configuration items.
Refer to the Decision Analysis and Resolution process area for more information about establishing evaluation criteria and selecting alternative solutions.
Refer to the Plan Data Management specific practice in the Project Planning process area for more information about managing data.
Refer to the Requirements Management process area for more information about managing requirements.
Refer to the Risk Management process area for more information about mitigating risks.
Specific Practices by Goal
SG 1 Evaluate Technical Solutions
Supplier technical solutions are evaluated to confirm that contractual requirements continue to be met.
Tip
SG 1 and SG 2 of ATM apply at any phase in the lifecycle of a product or service if a supplier is delivering a technical solution to meet contractual requirements in a supplier agreement.
Technical reviews (e.g., architectural evaluations) are performed throughout the project lifecycle to gain confidence that the requirements, architecture, and supplier technical solutions are capable of guiding a development that results in a product or service that provides the required capability. This activity should be integrated with risk management activities. Mature organizations typically perform technical reviews using different proven techniques depending on the type of review. They broaden the basis of the review to include other stakeholder needs, expectations, and constraints.
Refer to the Establish the Acquisition Strategy specific practice in the Project Planning process area for more information about specifying technical performance measures and their threshold values.
Tip
It is best to evaluate a range of candidate solutions. Input from stakeholders with diverse skills and backgrounds can help teams identify and address assumptions, constraints, and biases that may be exhibited in the supplier’s candidate designs. Also, consider the “ilities” addressed in the contractual requirements (e.g., usability, reliability, maintainability, interoperability, security).
This specific goal focuses on the following:
• Selecting supplier technical solutions (i.e., preliminary designs, detailed designs, design implementations, delivered increments of component functionality) based on sound decision-making criteria
• Analyzing selected supplier technical solutions
• Conducting technical reviews using results of the analysis
X-Ref
DAR supports the selection of supplier technical solutions for analysis.
SP 1.1 Select Technical Solutions for Analysis
Select supplier technical solutions to be analyzed and analysis methods to be used.
The supplier technical solutions are typically in one of the following three stages:
• Candidate solutions or preliminary designs (includes consideration of design approaches, design concepts, architectures, architectural styles and patterns, product partitions, major interfaces, system states and modes) that potentially satisfy an appropriate set of allocated functional and quality attribute requirements
• Detailed designs for selected solutions (i.e., containing all the information needed to manufacture, code, or otherwise implement the design as a product or product component)
• Implemented designs (i.e., the product or service)
Tip
When you consider the successful products or services you have acquired, ask what made them successful. Was it features, usability, cost, customer support, response time, reliability, or something else? These characteristics were achieved through careful evaluation of technical solutions.
Tip
Complexity is a “double-edged sword.” Sometimes today’s high-performance solution becomes tomorrow’s high-maintenance challenge.
Depending on where in the acquisition lifecycle the highest risks occur, the acquirer selects supplier technical solutions for analysis to reduce those risks. Analysis methods are selected based on the type of technical solution being analyzed and the nature of the risk.
Hint
When evaluating a supplier’s alternative solution, treat its components together, not individually. For example, do not rush to evaluate a promising COTS component or new technology without considering the impacts and risks of its integration with the rest of the solution.
Hint
Consider the entire life of the product when evaluating a technical solution to ensure that it will have the desired versatility and market endurance.
The acquirer may also want to select interfaces for analysis to help decide which interfaces require acquirer management. (See specific goal 2 of this process area.)
Tip
Criteria for selection can include the impact of the technical solution on cost, schedule, performance, and risk. How these criteria are defined in detail, however, depends on the requirements.
Example Work Products
1. Criteria used to select supplier technical solutions for analysis
2. Lists of supplier technical solutions selected for analysis
3. Analysis methods for each selected supplier solution
Example Supplier Deliverables
1. List of supplier deliverables
Tip
Screening criteria may involve setting thresholds for selected quality attributes (e.g., response time) that must be met by technical solutions.
Subpractices
1. Develop criteria for determining which supplier technical solutions to analyze.
Refer to the Decision Analysis and Resolution process area for more information about establishing evaluation criteria.
2. Identify supplier technical solutions for analysis.
Hint
Explore the use of COTS products (or open source or new technology) by the supplier early in technical evaluations. To use COTS solutions effectively, you may need to consider changes to requirements. Fully understand the tradeoffs entailed in such requirements and designs early, before committing to (and putting under contract) a particular development approach.
3. Identify the functional and quality attribute requirements to be satisfied by each selected technical solution.
A traceability matrix is a useful tool for identifying requirements for each selected technical solution, as it typically includes information that relates requirements to work products. When identifying requirements for each selected technical solution, consult the appropriate traceability matrix.
Refer to the Maintain Bidirectional Traceability of Requirements specific practice in the Requirements Management process area for more information about tracing requirements to work products.
X-Ref
For more information regarding principles, methods, and techniques for creating systems from COTS products, see the SEI website at www.sei.cmu.edu/acquisition/tools/methods/cotsintro.cfm.
4. Identify the analysis methods to be used for each selected technical solution.
5. Include analysis methods and review activities in the project plan.
Refer to the Project Planning process area for more information about establishing and maintaining plans that define project activities.
SP 1.2 Analyze Selected Technical Solutions
Analyze selected supplier technical solutions.
Depending on the type of technical solution being analyzed (i.e., preliminary design, detailed design, design implementation of components or their interfaces), the results of the analysis are provided to the technical review described in the next specific practice.
Hint
To gain the insight needed to fully evaluate supplier solutions, you may need to refine operational concepts and scenarios so that you understand the implications of each alternative solution. Also, you may need to develop scenarios for other phases of the product lifecycle.
The acquirer should select a supplier’s design to analyze by exploring the adequacy and completeness of that design by reviewing product representations (e.g., prototypes, simulations, models, scenarios, storyboards) and by obtaining feedback about them from relevant stakeholders.
The acquirer should confirm the following:
• The selected design adheres to applicable design standards and criteria.
• The design adheres to allocated functional and quality attribute requirements.
• The resulting product will perform appropriately in its intended-use environment.
Hint
A design is a document used by stakeholders over the life of the product; thus it must communicate clearly and accommodate change. Consider these issues when selecting criteria to be used in evaluating the value of a design.
During design implementation, the supplier implements the design reviewed and analyzed by the acquirer by developing product components, integrating those components, conducting unit and integration testing of the product, and developing end-user documentation.
A successful analysis of the supplier’s implementation is predicated on the acquirer’s determination that the requirements are fully met in the final production configuration, that the production capability forms a satisfactory basis for proceeding into pilots or full-rate production, and that the product is ready to be brought into the acquirer environment for further integration and acceptance testing.
The acquirer can require delivery of verification results from the supplier of the technical solution, as applicable. The suppliers can conduct verifications in an iterative fashion, concurrently with the acquirer’s technical analyses, or the supplier can be required to conduct follow-on verifications of technical solutions.
The acquirer should also confirm that sufficient end-user documentation has been developed and is in alignment with the tested implementation. The supplier can develop preliminary versions of the installation, operations, and maintenance documentation in early phases of the project lifecycle for review by acquirer and relevant stakeholders.
Tip
Installers, operators, end users, and maintainers may have different documentation needs that may be addressed in different documents.
Example Work Products
1. Record of analysis
2. Results of analysis
Tip
Documentation assists product maintenance and support later in the life of the product.
Example Supplier Deliverables
1. Alternative solutions
2. Product architecture
3. Product component designs
4. Architecture evaluations
5. Unit and integration test results
6. Verification results
Tip
Documentation can be treated as a type of product component for which a solution may be selected, designed, and implemented. Design and implementation methods and standards have been developed for documentation.
Subpractices
1. Confirm that the selected technical solution adheres to applicable standards and criteria.
2. Confirm that the selected technical solution adheres to allocated functional and quality attribute requirements.
3. Use analysis results to compare actual performance measurements to specified thresholds of technical performance measures.
Refer to the Measurement and Analysis process area for more information about analyzing measurement data.
Hint
It is not enough to consider product functionality and behavior in the intended operational environment when evaluating a solution. Ask questions about other phases in the life of the product, including whether the solution can be manufactured; whether it is easy to test, install, repair, migrate to new versions or platforms, and support; and what the costs and legal implications will be.
4. Conduct technical interchange meetings as necessary.
Technical interchange meetings are scheduled meetings between the supplier and acquirer to discuss technical progress. These meetings are less formal than the event-driven technical reviews in the next specific practice. One of the purposes of these meetings is to discuss and understand issues encountered during acquirer analyses or provided results of supplier verification activities.
5. Confirm that the selected technical solution is sufficiently analyzed and meets entrance criteria to begin technical review.
Hint
No solution will rank best on every criterion used. Instead, analyze a solution’s ability to provide a balanced approach to cost, schedule, performance, and risks across the product lifecycle.
6. Review critical verification results and data from verifications conducted by the supplier.
Hint
Following the evaluation, you may conclude that the selection criteria were not complete or detailed enough to adequately assess the ability of the solution to meet the entrance criteria for the technical review. If so, you may need to iterate through SP 1.1 and 1.2.
SP 1.3 Conduct Technical Reviews
Conduct technical reviews with the supplier as defined in the supplier agreement.
Technical reviews are used by the acquirer to confirm that products and services being developed or produced by suppliers meet end user needs and requirements.
Technical reviews should have the following characteristics:
• Are conducted when the technical solution under development satisfies review entry criteria (i.e., event driven, not schedule driven)
• At a minimum, are conducted at the transition from one acquisition phase to the next and at major transition points of technical effort
• Have their processes and requirements addressed in and required by the supplier agreement
Tip
The purpose of a technical review is to review the supplier’s technical progress and identify and resolve issues.
Refer to the Project Planning process area for more information about establishing and maintaining plans that define project activities.
Example Work Products
1. Review schedule
2. Entry and exit criteria
3. Review results
4. Documented issues (e.g., issues with customer requirements, product and product component requirements, product architecture, product design)
Example Supplier Deliverables
1. Progress reports and process, product, and service level measurements
2. Technical performance measurements
3. Review materials and reports
4. Action items tracked to closure
5. Documentation of product and document deliveries
Tip
Involving relevant stakeholders (e.g., engineers, technical writers, QA personnel) in reviews of a supplier’s technical solution can reduce the number of serious issues that might be discovered late in the acquisition process. In these reviews, issues affecting installation, operation, and so on can be identified and resolved.
Hint
You may want to examine the rationale for a particular technical review result if you later learn that a promising technology or COTS component has become available. It may be unnecessary to interrupt product development to explore the implications if they were already explored earlier and records were maintained.
Subpractices
1. Identify participants for the technical review.
2. Conduct the technical review.
3. Analyze and record results of the review.
4. Use the results of technical reviews to improve the supplier’s technical solution.
The results of some reviews may require changes to the supplier agreement.
Refer to the Solicitation and Supplier Agreement Development process area for more information about preparing a solicitation package, selecting one or more suppliers to deliver the product or service, and establishing and maintaining the supplier agreement.
SG 2 Perform Interface Management
Selected interfaces are managed.
Many integration and transition problems arise from unknown or uncontrolled aspects of both internal and external interfaces. Effective management of interface requirements, specifications, and designs helps to ensure implemented interfaces are complete and compatible.
The supplier is responsible for managing the interfaces of the product or service it is developing. However, the acquirer identifies interfaces that it will manage as well (particularly external interfaces).
Tip
Overlooked or incompletely described interfaces are risks to be identified and managed (RSKM). The realization of these risks may lead to safety recalls and can prove very costly.
Hint
Manage interfaces early in the project to help prevent inconsistencies from arising.
SP 2.1 Select Interfaces to Manage
Select interfaces to manage.
The interfaces considered for selection include all interfaces with other products and services in the operations and support environment as well as environments for verification and validation and services that support those environments. The acquirer should review all supplier interface data for completeness to substantiate the complete coverage of all interfaces when making the selection.
Hint
Where possible, reach early agreement with the supplier on interface design. For some external interfaces, agreement must also be reached by the acquirers or owners of the other interfacing products or services. Lack of agreement leads to wasted time, as assumptions made by the teams on each side of an interface must be validated. Project costs may increase and the schedule may slip.
Example Work Products
1. Criteria to be used in selecting acquirer managed interfaces
2. Categories of interfaces
3. List of interfaces per category
Example Supplier Deliverables
1. Interface description documents
2. Categories of interfaces
3. List of interfaces per category
4. Mapping of interfaces to product components and the product integration environment
5. Interface design specifications
Tip
You can use a formal evaluation (DAR) process to select the interfaces to manage. Criteria for selection can include the impact of the interface on cost, schedule, performance, and risk.
6. Interface control documents
7. Interface specification criteria
Tip
Interface control documents define interfaces in terms of data items passed, protocols used for interaction, and so on. These documents are particularly useful in controlling products and product components being built by different teams.
Subpractices
1. Select the criteria to be used for determining which interfaces the acquirer should manage.
Refer to the Decision Analysis and Resolution process area for more information about establishing evaluation criteria.
2. Identify interfaces that are candidates for acquirer management.
X-Ref
Requirements for the interfaces are developed in ARD. The acquirer should pay special attention to those interfaces that were particularly important to the stakeholders.
3. Review identified interfaces against the selection criteria.
4. Include acquirer managed interfaces in the project plan.
X-Ref
It may be prudent to mitigate risks by establishing teams across related acquisition programs when the final capabilities require effective interfaces between or within programs. IPM SP 1.6 provides guidance for such teams.
SP 2.2 Manage Selected Interfaces
Manage selected interfaces.
Managing interfaces includes the maintenance of the consistency of the interfaces throughout the life of the product, compliance with architectural decisions and constraints, and the resolution of conflict, noncompliance, and change issues. In a system-of-systems environment, the management of interfaces between products or services acquired from suppliers and other systems within the system of systems is critical for success of the project.
Tip
Interface management may sometimes follow a formal evaluation (DAR) process: Establish criteria for correctness of an interface (based in part on interface requirements), evaluate alternatives, and so on.
Interface changes are documented, maintained, and readily accessible.
Refer to the Acquisition Requirements Development process area for more information about validating requirements.
Refer to the Configuration Management process area for more information about tracking and controlling changes.
Refer to the Requirements Management process area for more information about managing requirements changes.
1. Table of relationships among the supplier’s product or service and the external environment
2. Updated interface description or agreement
Example Supplier Deliverables
1. Table of relationships among the product components and the external environment (e.g., main power supply, fastening product, computer bus system)
2. Reports from interface control working group meetings
3. Action items for updating interfaces
4. Application program interface (API)
Subpractices
1. Review and analyze selected interface definitions and designs.
2. Confirm that interface descriptions adhere to allocated requirements.
3. Confirm the compatibility of selected interfaces throughout the life of the product or service.
Confirm that interface descriptions adhere to applicable standards, criteria, and interface requirements between the supplier’s product and acquirer’s intended environment.
4. Verify that interfaces have been sufficiently tested by the supplier.
5. Verify that issues identified during testing have been resolved appropriately, with product revisions, if necessary.
6. Resolve conflict, noncompliance, and change issues for the selected interfaces.
7. Periodically review the adequacy of interface descriptions.
Once established, interface descriptions should be periodically reviewed to ensure there is no deviation between existing descriptions and the products being developed, processed, produced, or bought.
The interface descriptions should be reviewed with relevant stakeholders to avoid misinterpretations, reduce delays, and prevent the development of interfaces that do not work properly.