When Matt briefed the hidden truths to the board of directors and other senior managers, he was surprised that it created such a stir. “They all want us to get moving fast, delivering solutions like clockwork,” he says to Paula. He pauses and then continues.

Acquirers of technology solutions must keep in mind the overall picture and then continuously reinforce and improve the integration of acquirer, supplier, and stakeholder activities. Paula describes what it’s like when a team hasn’t achieved the desired level of integration.

Steve is also sitting in on the meeting.

Achieving integrated project management requires an approach that is new to most organizations. You must make certain that your preferred technologies and your choices about your overall technology environment are aligned with the supplier’s system processes and delivery capabilities, and you must do this early enough in the overall process that the two parties can influence and shape one another in a timely and effective way. You can examine suppliers’ qualifications early in the process as requirements and technical constraints emerge. The suppliers can begin to participate in early discussions aimed at obtaining firsthand information about potential problems and opportunities inherent in the design trade-offs you present.

However, people who work on technology projects don’t have to reinvent the wheel. After you have documented some processes, procedures, templates, or guidelines, projects can leverage your organizational process and tailor it to their specific needs. A defined process not only covers the technical aspects of system delivery but also addresses organizational acquisition guidance, regulations, instructions, and practices that are established for use across projects. The project’s processes logically sequence acquirer activities and supplier deliverables (as identified in the supplier agreement) to deliver a product that meets the requirements. Examples of integrated project activities include proposal evaluation, supplier selection, development and management of supplier agreements, and acceptance of supplier deliverables.

Although all processes must be aligned with the acquisition strategy, this alignment is particularly important if you plan to make processes available for projects so that they can tailor them. Markets and technologies are dynamic, and time is a critical element of competition. Therefore, deeper, more intensive acquirer-supplier integration is crucial for effectively delivering solutions.

Steve picks up the dialog about integrated project management.

An integrated work environment fosters careful and practical planning.

Developing an integrated project plan requires an in-depth understanding of your and the supplier’s capabilities. For instance, projects are often asked to accelerate the delivery schedule. If you can’t accelerate work on your side, the relationship with the supplier comes into play. In a long-term acquirer-supplier relationship, the supplier might provide additional short-term capability without adding cost to help the acquirer out of a difficult situation.

Kristin confirms Steve’s observation.

Many acquirers (especially their purchasing departments) are tempted to dip into the project’s management reserve or buffer and suggest that reducing the buffer is a painless way of reducing the planned project cost and delivery time. But this is a myth. Project buffers protect critical tasks from the project’s inherent risk. Project buffers can consist of, for instance, additional resources that become available when you need to protect a critical task, or they may provide additional monies to buy capacity or capability to keep the project on track. Reducing the project buffer has no impact on project execution time; all it reduces are the chances that the promised delivery date will be missed, and potentially it causes excessive replanning.

Therefore, you should not cut the project buffer. Instead, clearly identify the risk associated with every project activity, and then size the project buffer according to the aggregate project risk. This practice not only makes the project risk visible but also provides a single convenient indicator for the impact of the risk on the project.

In an integrated project environment, acquirers communicate with suppliers on a variety of issues common to both organizations throughout the project. This communication requires that you establish forums and create a context in which two-way communication is effective: You must comprehend issues of “manufacturability” or system delivery processes, foster skills in developing robust design constraints that exploit manufacturing capability, and be prepared to trim designs accordingly. Similarly, suppliers must be oriented toward customer satisfaction: They must comprehend issues of product performance and total cost of ownership and must be prepared to deliver and develop a solution that allows the design to achieve its objectives in the intended environment.

Matt’s incisive statement brings home the point. Successful acquirers attempt to maximize cooperation among stakeholders. In other words, the acquirer involves and integrates all relevant stakeholders, and the supplier agreement provides the basis for managing the supplier’s involvement. Kristin agrees.

Involving all stakeholders throughout the project is critical.

Steve laughs.

Depending on the scope and risk of the project, your coordination effort among stakeholders and with the supplier can be significant. To integrate the effort across functions and suppliers, you need to add specific activities that support cross-functional work. For example, a supplier might build prototypes to support your desire to develop richer customer understanding early in the process. To complete the loop, the supplier participates with your personnel when interacting with customers to strengthen and deepen their understanding of the experience that the product will create.

The supplier aligns its system delivery processes with your acquisition processes early in the project and, if necessary, collaborates with you to tailor processes. Moreover, prototype testing and evaluation are conducted jointly by all the functions involved in development.

All these interactions happen only when an integrated plan is in place that establishes clear accountability. George offers a comment.

The CMMI-ACQ provides specific processes for handling difficulties you might have with integrated project work. It attacks problems at their root and starts you on a path to improving your project execution. The Integrated Project Management process area focuses on establishing and managing projects and involving the relevant stakeholders according to an integrated and defined process that is tailored from your organization’s set of standard processes (see Table 4-1).

According to the CMMI-ACQ, a project’s defined process addresses all activities, tasks, templates, procedures, and guidelines that a project needs to execute to acquire and maintain the technology solution (Table 4-1, practice 1.1). The product-related life-cycle processes—for instance, support processes—are developed concurrently with the product. The project’s defined process must satisfy the project’s contractual and operational needs, opportunities, and constraints. A project’s defined process is designed to provide a best fit for the project’s needs and is based on the following factors:

Project managers usually select a life-cycle model from those available in the organization’s library of process assets. One approach is to contact another project manager who has executed a similar acquisition and who can help you select a subset of standard processes that best fit the needs of the project.

Some activities, templates, checklists, and guidelines might apply to a project only to some degree or in a modified form. Modifying the standard processes to fit a specific project is what we mean by tailoring. Typically, you provide tailoring guidelines to support the project when defining its processes. Some acquirers even provide “pretailored” versions to simplify and expedite tailoring.

To develop a realistic plan, you must understand the relationships among the various tasks and work products of the defined process and the roles fulfilled by the stakeholders (Table 4-1, practice 1.2). When the results of previous planning and execution are available, use them as predictors of the relative scope and risk of the effort you’re estimating. To arrive at accurate estimates during project planning, successful acquirers typically do the following:

To support the desired level of performance and reliability, it is critical to establish an appropriate work environment for the project (Table 4-1, practice 1.3). An appropriate work environment comprises the facilities, tools, and equipment people need to perform their jobs effectively. As with any other product, the critical aspects of the work environment are driven by requirements. You should explore the work environment’s functionality and operations with the same rigor as you do for any other product development.

Depending on the scope of the outsourcing endeavor and the type of the technology solution, you might maintain your own environments for integrating, verifying, and validating the delivered technology solutions. Acquirers that productize and sell technology to their customers, or those that develop technology solutions in-house in addition to acquiring them, often prefer to establish their own integration and test environments. In contrast, organizations that rely solely on suppliers to provide (and maintain) technology solutions tend to hold their suppliers responsible for creating and maintaining integration and testing environments. This situation more closely resembles the purchase of commodities, where buyers pay to use the product but don’t expend their own resources to ensure the product’s quality.

Your integrated plan addresses additional planning activities such as incorporating the project’s defined process, coordinating with relevant stakeholders, using organizational process assets, incorporating plans for peer reviews, and establishing objective entry and exit criteria for tasks (Table 4-1, practices 1.4–1.6). For you to continuously improve, projects must contribute to your process asset library, which documents lessons learned on projects and programs. Your process asset library is also a repository for storing process and product measures.

You manage and coordinate stakeholder involvement according to the project’s integrated and defined process (see Table 4-2, practice 2.1). You manage the supplier’s involvement according to the supplier agreement.

Acquirers conduct frequent reviews with stakeholders (Table 4-2, practice 2.2) and typically establish milestones for each critical dependency in the project schedule. Tracking the critical dependencies typically includes the following activities:

Inevitably, communication issues arise that require you to work even more closely with the stakeholders (Table 4-2, practice 2.3). In some cases, it might be necessary to escalate to the appropriate stakeholder managers any issues that cannot be resolved. You should anticipate this scenario and define clear, concise escalation procedures that are well understood by all stakeholders.

Most people take configuration management (CM) for granted, but you will suffer great misery if your acquisition artifacts, products, and services aren’t version controlled, stored, and easily accessible so that you can get to them when you need them. Only too quickly you will find yourself looking for the famous needle in the haystack.

Steve, the project manager, wishes he hadn’t picked up the phone. Now he’s scrambling to answer Paula’s piercing questions.

After Steve snaps his cell phone shut, his face brightens. He thinks, “I’ll call George Taylor.” As the supplier’s project manager, George delivered the first release of the financial system. Steve places the call.

Silence greets Steve on the other end of the line. “Did I lose the connection?” he wonders. Finally George speaks.

Steve’s heart sinks. He thinks, “I should’ve told Paula weeks, not hours. We’ll have to conduct an archeological dig or, worse, re-create almost everything.” He’s determined to never let this happen again. He pulls together his team along with representatives from supplier organizations to discuss configuration management.

Your approach to configuration management depends on acquisition-specific factors such as the acquisition approach, the number of suppliers, the design responsibility, the support concept, and associated costs and risks. Configuration items can vary widely in complexity, size, and type, from COTS software to a test metric or project plan.

Typically, any item required for product support and designated for separate procurement is a configuration item, along with acquirer work products provided to suppliers (for example, solicitation packages and technical standards). Paula, the internal customer for the current acquisition project, speaks up.

To avoid unexpected costs to procure, reformat, and receive configuration items from the supplier, you need to identify configuration items early and incorporate them into the integrated project plan. For acquirers, this includes considering how configuration items are shared between the acquirer and supplier and among relevant stakeholders. If you allow a supplier to use its CM system, you are responsible for implementing security and access control procedures. In many cases, an alternative solution is for the acquirer to grant physical possession of acquirer-created configuration items. In turn, the supplier gives the acquirer access to supplier deliverables.

If a CM system is shared between acquirer and supplier, then the supplier agreement must specify the clear responsibility for managing the configuration items, baselines, security, access restrictions, and backup and restore processes. You must ensure that the supplier agreement specifies the CM requirements for the supplier (e.g., status reporting and providing configuration audit results).

The supplier agreement should also specify appropriate acquirer rights to the supplier deliverables, in addition to the requirements for delivery or access. For instance, you typically maintain the right to access configuration data at any level to implement planned or potential design changes and support options. Configuration management of legacy systems should be addressed on a case-by-case basis as design changes are contemplated.

When do you create a baseline for the configuration items? Steve ponders this question for a while.

With regard to the technical solution, you maintain configuration control of your requirements, and the supplier performs configuration management of the technical solution (e.g., establishing and maintaining the product baseline). In this way, you retain the authority and responsibility for approving any design changes that affect the product’s ability to meet contractual requirements. The supplier manages other design changes.

The level of control is typically selected based on project objectives, risk, and the value of the configuration item to the project. Levels of control can range from informal control (simply tracking changes made when you are developing the configuration items or when supplier work products are delivered or made accessible to you) to formal configuration control using baselines that can be changed only as part of a formal CM process. Although the supplier may manage configuration items on your behalf, you are responsible for approval and control of changes to such configuration items.

Change requests can be initiated either by you or by the supplier. Changes that affect your work products and supplier deliverables as defined in the supplier agreement are typically handled through your CM process. You analyze the impact of submitted change requests on the supplier agreements.

The CMMI-ACQ provides specific recommendations on how you can optimize configuration management as an acquirer of technology solutions. Configuration management specifically focuses on establishing and maintaining the integrity of work products using configuration identification, control, status accounting, and audits (see Table 4-3).

An acquirer typically designates work products or supplier deliverables as configuration items when they’re critical to the project, are used by two or more groups, or are expected to change over time either because of errors or changes in requirements (Table 4-3, practice 1.1). Each configuration item has a responsible owner. In addition, each item has a unique identifier. Frequently, projects also specify important characteristics of each item for easier classification and retrieval.

To avoid unexpected costs to procure, reformat, and deliver configuration items from the supplier, you conduct your planning for managing configuration items—including during transition to operations and support—as part of project planning. Include plans and the infrastructure for managing these items within the project teams and between the acquirer, supplier, operational users, and other relevant stakeholders. You need to clearly define when each configuration item is placed under CM. Sample criteria include the following:

To store configuration items with the responsible party and as close as possible to their primary use, you need to designate a CM system (Table 4-3, practice 1.2). Such a system describes the storage media, the procedures, and the tools for accessing the system. Sometimes this system also includes the tools for recording and accessing change requests. In addition to storing and retrieving configuration items, you include contractual provisions to ensure the necessary sharing and transfer of configuration items between you and the supplier.

You review and approve the release of the product baselines created by the supplier (Table 4-3, practice 1.3). A baseline is a set of specifications or work products that has been formally reviewed and agreed on. Thereafter it serves as the basis for further development or delivery and can be changed only through change control procedures. You also create baselines for acquirer work products to mark the agreement on the description of the project, requirements, funding, schedule, and performance parameters and to make a commitment to manage the program to those baselines. To create or release baselines of configuration items, a project must obtain authorization from the change control board.

Consistently tracking changes to baselined configuration items is critical for acquirer and supplier (see Table 4-4, practice 2.1). Change requests address not only new or changed requirements but also failures and defects in the work products. You analyze the impact that submitted change requests will have on the supplier agreements. You also determine the impact that the change will have on the work product, related work products, and schedule and cost.

Frequently, stakeholders request more changes than can be accommodated within a project. In this case, the change control board consults the relevant stakeholders and decides whether such change requests can be addressed in the next release of the solution; the stakeholders then agree to defer some the proposed changes. This negotiation includes keeping records of the disposition of each change request and the rationale for the decision, including success criteria, a brief action plan if appropriate, and needs met or not met by the change.

You maintain overall control of the configuration of the work product baseline (Table 4-4, practice 2.2). It’s important to control changes to configuration items throughout the life of the product. This control includes tracking the version of each item, approving a new configuration of the solution if necessary, and updating the baseline. For example, authorization to access configuration items may come from the change control board, the project manager, or the customer. Only with the proper authorization are configuration items checked in or checked out of the CM system for review or modification. Changes must be incorporated in a manner that maintains the correctness and integrity of the items. In many cases, this requires reviews to ensure that changes do not cause unintended effects on the baselines (e.g., ensure that the changes have not compromised the safety or security of the system).

To establish and maintain the integrity of baselines, you record CM actions in sufficient detail that the content and status of each item is known and previous versions can be recovered (see Table 4-5, practice 3.1). In addition, projects can ensure the integrity of baselines by following these practices:

Periodically, you perform configuration audits to confirm that the resulting baselines and documentation conform to a specified standard or requirement (Table 4-5, practice 3.2). The completeness and correctness of the content are based on the requirements as stated in the plan and the disposition of approved change requests.

Considering the track record of organizations that implement technology solutions, it is risky business. A variety of sources contributes to the risks that every acquirer must manage.

Acquirers can’t afford to possess this kind of attitude toward risk. To survive, you must be able to take rational risks. This means accepting the downside of a decision if the payoff is high enough.

By employing a disciplined process, acquirers can uncover risks before they occur so that risk-handling activities can be planned and invoked to mitigate adverse impacts on program objectives. The need for early and aggressive risk detection is compounded by the complexity of projects due to the acquirer-supplier relationship. For example, the supplier’s experience of working with the acquirer, the supplier’s financial stability, or the availability of well-defined dispute resolution processes all influence the risk of a project.

Because of the inherent risks of projects attempting to produce a technology solution, especially in an outsourcing environment, it is paramount for any acquirer to document the risks and make them visible. It’s much harder to ignore a risk if it’s documented. Unfortunately, this task is fraught with peril. Your mind can play tricks on you when it comes to admitting risks. Steve explains.

A simple risk assessment works well when a project possesses little data. Typically this is the case when an organization initially establishes an acquisition strategy. As a project or program progresses, however, greater knowledge justifies more detailed risk analysis.

As you gain experience with the simple risk assessment, you can develop a risk scorecard based on the ten-question risk analysis presented in Chapter 2. This extended question set (see Figure 4-1) has weights or level of impact for each parameter.

Instead of intuitively assigning a score for each risk parameter, the project team can formulate and answer a series of detailed questions that produce a score. Additionally, the project team may know that some parameters affect overall risk more than others.

The questions, their ratings, and the level of impact reflect an organization’s current capabilities and areas of emphasis. These data will change as the acquisition strategy and core capabilities change. George agrees.

The CMMI-ACQ provides specific recommendations on how to identify and analyze risks. Risk management identifies potential problems before they occur so that you can plan and invoke risk-handling activities as needed throughout the life of the product to mitigate adverse impacts on achieving objectives (see Table 4-6).

Analyzing risks entails identifying risks from internal and external sources and then evaluating each risk to determine its likelihood and consequences. Identifying potential issues, hazards, threats, and vulnerabilities that could negatively affect work efforts or plans is the basis for sound and successful risk management (Table 4-6, practice 2.1). Risks must be identified and described in an understandable, consistent way before they can be analyzed and managed properly. Risks are documented in a concise statement that includes the context, conditions, and consequences if the risk occurs.

You should follow an organized, thorough approach to seek out probable or realistic risks that might affect your achieving your objectives. To be effective, risk identification should not be an attempt to address every possible event no matter how improbable it may be. Instead, you should follow a disciplined, streamlined approach that uses the categories and parameters developed in your risk management strategy, along with the identified sources of risk. The identified risks form a baseline on which to initiate risk management activities. You should periodically review the list of risks to reexamine possible sources of risk and changing conditions and thereby uncover sources and risks previously overlooked or nonexistent when the risk management strategy was last updated.

Typical methods for identifying risks include the following:

You can identify some risks by using the categories and parameters developed in your risk management strategy to examine the supplier’s work breakdown structure, product, and processes. You can identify risks in many areas (e.g., requirements, technology, design, testing, vulnerability to threats, life-cycle costs). Examining the project or program in each of these areas can help you develop or refine the acquisition strategy and the risk-sharing structure between you and the supplier.

You should also consider the risks associated with a supplier’s capability (e.g., meeting schedule and cost requirements), including the potential risks to your intellectual capital or security vulnerabilities introduced by using a supplier.

Certain kinds of risks are often missed. These include risks thought to be outside the scope of the project; in fact, even though the project does not control whether they occur, you can mitigate their impact. Examples include the weather, natural or manmade disasters that affect the continuity of operations, political changes, and telecommunications failures.

To the extent that they affect project objectives, cost, schedule, and performance, risks should be examined during all phases of the product life cycle in the intended environment. You may discover potential risks that are outside the scope of the project’s objectives but are vital to customer interests. For example, the risks in development costs, product acquisition costs, cost of spare (or replacement) products, and product disposition (or disposal) costs have design implications. The customer may not have provided requirements for the cost of supporting the fielded product. The customer should be informed of such risks, but it may not be necessary to actively manage those risks. The mechanisms for making such decisions should be examined at project and organization levels and put in place if deemed appropriate, especially for risks that affect the ability to verify and validate the product.

Risk statements are typically documented in a standard format that contains the risk context, conditions, and consequences. The risk context provides additional information so that the intent of the risk can be easily understood. In documenting the context of the risk, you should consider the relative time frame, the circumstances or conditions, and any doubt or uncertainty.

You need to evaluate risks so that you can assign relative importance to each identified risk and determine when appropriate management attention is required (Table 4-6, practice 2.2). Each risk is evaluated and assigned values in accordance with the defined risk parameters, which may include likelihood, consequence (severity of impact), and thresholds. You can integrate the assigned risk parameter values to produce additional measures, such as risk exposure, which can be used to prioritize risks. Often it is useful to aggregate risks based on their interrelationships and then develop options at an aggregate level. When you form an aggregate risk by rolling up lower-level risks, take care that important lower-level risks are not ignored.

You categorize risks into defined risk categories, providing a means to view risks according to their source, taxonomy, or project component. Related or equivalent risks can be grouped for efficient handling. You should document the cause-and-effect relationships between related risks. Your risk categories might include sourcing, contract management, and supplier execution in addition to project management, technology, and requirements.

Acquirers prioritize risks for mitigation. You use clear criteria to determine a relative priority for each risk based on the assigned risk parameters. Your goal is to determine the areas to which risk mitigation resources can be applied with the greatest positive impact to the project or program.

Risks are handled and mitigated, when appropriate, to reduce adverse effects on achieving your objectives.

You first identify the most important risks and then decide how many of them you have the resources to mitigate.

You must identify which risks can be accepted (that is, you can live with them if they do become problematic) and which risks can be assigned to someone who is better able to manage them.

Paula adds an important point.

The ultimate responsibility for risk management rests with you, but a close partnership between you and your suppliers is crucial if you are to effectively identify and mitigate risks. Both you and the supplier need to share risk information, understand each other’s risk assessments, and develop and execute risk mitigation efforts. You must involve the supplier as early as possible so that you can establish effective risk assessment and reduction. For instance, the supplier has often much deeper insight into the technical risks than you have.

You need to include specific risk assessment and risk-handling tasks in the RFP and subsequent supplier agreement.

The progress of project activities should lower the project’s perceived risk. In fact, when an organization pursues any new technical solution, it must deliberately acquire competencies to reduce risk and ensure success. If successive reviews demonstrate steady or rising project risk, it sends a strong signal that you need to reexamine the project’s viability. In contrast, steadily decreasing risk increases your confidence of a successful outcome. A properly designed and executed risk management process lets you make decisions about the fate of a project, thereby enabling you to make intelligent bets. In general, acquirers are better off receiving risk information early, because it gives them an opportunity to decrease the downside and raise the upside.

Moving fast is an additional risk mitigation technique commonly used by successful acquirers.

Controlling the technical risk of integration should be at the forefront of any acquirer’s mind: How do you ensure that the suppliers integrate the various components into a value-added solution and then integrate the technical solution into your environment? Steve has a couple of ideas.

When you close a risk, you document the rationale for closing it, along with successful and unsuccessful actions, assumptions that were proven wrong, mitigation costs, and project savings or return on investment.

The CMMI-ACQ provides specific recommendations on how to handle risks, including developing risk-handling options, monitoring risks, and performing risk-handling activities when defined thresholds are exceeded (see Table 4-7).

You develop and implement risk mitigation plans for selected risks to proactively reduce their potential impact. A critical component of a risk mitigation plan is to develop alternative courses of action, workarounds, and fallback positions, with a recommended course of action for each critical risk (Table 4-7, practice 3.1). The risk mitigation plan for a given risk includes techniques and methods to avoid, reduce, and control the probability of occurrence, the extent of damage incurred should the risk occur (sometimes called a contingency plan), or both. You monitor risks, and when they exceed the established thresholds, you deploy the risk mitigation plans to return the affected effort to an acceptable risk level. If the risk cannot be mitigated, a contingency plan can be invoked. Both risk mitigation and contingency plans are often generated only for selected risks whose consequences are determined to be high or unacceptable; other risks may be accepted and simply monitored.

Options for handling risks typically include alternatives such as the following:

Often, especially for high risks, you should generate more than one approach.

In many cases, risks are accepted or watched. Usually you accept a risk when you judge it too unimportant for formal mitigation, or when there appears to be no viable way to reduce it. If a risk is accepted, the rationale for this decision should be documented. Risks are watched when there is an objectively defined, verifiable, and documented threshold of performance, time, or risk exposure (the combination of likelihood and consequence) that will trigger risk mitigation planning or invoke a contingency plan if it is needed. Thresholds for supplier risks that affect the project (e.g., schedule, quality, or risk exposure due to supplier risks) are specified in the supplier agreement, along with escalation actions if the thresholds are exceeded.

Examine risk mitigation activities and assess the benefits they provide versus the resources they will expend. As with any other project activity, you may need to develop alternative plans and assess the costs and benefits of each alternative. You then select the most appropriate plan for implementation. At times the risk activity may be significant and the benefit small, but the risk must be mitigated to reduce the probability of incurring unacceptable consequences.

To effectively control and manage risks during the work effort, follow a proactive program to regularly monitor risks and the status and results of risk-handling actions (Table 4-7, practice 3.2). For that purpose, you establish a schedule or period of performance for each risk-handling activity that includes the start date and anticipated completion date. This requires continued commitment of resources for each plan.

The risk management strategy defines the intervals at which the risk status should be revisited. This activity may result in the discovery of new risks or new risk-handling options that can require replanning and reassessment. In either event, you should compare the acceptability thresholds associated with the risk against the status to determine the need for implementing a risk mitigation plan.

You share selected risks with the supplier. Risks associated specifically with the acquisition process are tracked and then resolved or controlled until mitigated. This monitoring includes risks that may be escalated by the supplier.

To effectively manage projects, you must have the necessary data or information at your fingertips when you need it. For many organizations, reaching this goal proves elusive. Matt, however, no longer worries about the usual disconnect between the data an acquirer needs to run its business and the data an acquirer has available to make decisions. Matt has what he calls the “House of Measures.” Like an instrument panel in a car, the House of Measures provides critical information in easy-to-read graphics, assembled from data extracted in real time from the acquirer’s measurement databases and, in some instances, directly from the suppliers’ measurement repositories. Let’s see how the House of Measures is built.

Before you can decide what the right measures are, you need to decide what to measure.

Without the right measures, you can start pulling the wrong levers—to use Matt’s analogy—and you go out of business even though all indicators show green, full steam ahead into the abyss.

Creating a good, precise set of measures to predict your success is difficult. It is often difficult to get consensus on how to measure the achievement of acquisition goals and the authoritative source for the measure. Yet getting buy in of key stakeholders and constituents is critical in obtaining a meaningful measurement of project execution. It is equally important to manage this process so that you end up with the few critical measures that truly matter.

So how do you arrive at the few critical measures? Acquiring software, systems, and IT, like any other business activity, works best with timely feedback. Such feedback indicates how a process is functioning and signals the need for intervention. Short-duration, repetitive processes, which are common in manufacturing environments, provide ample opportunities for measuring their efficacy through demonstrated results. But acquisition projects have comparatively long lead times and unrepetitive tasks, and that poses difficulties for acquirers that manage with results data only. You cannot afford to receive status reports only at the completion of a project or significant milestone; you need interim feedback. You must manage the acquisition process itself through results or output measures and progress measures.

Outcome metrics characterize activities’ results by describing how well objectives are accomplished. Progress metrics, in contrast, characterize the current status of activities that are not yet complete. Outcome metrics allow you to look in the rearview mirror. They describe after-the-fact results, whereas progress metrics enable intervention that can affect results.

To get a balanced mix of outcome and progress measures, you need to get a handle on how to use supplier measures. To manage projects, you use supplier-reported measures in addition to your own measures of progress and output. The addition of supplier measures allows you to comprehensively address the measurement objectives and determine the progress and output of the project. Supplier measures are the foundation of the House of Measures (see Figure 4-2).

Figure 4-2. The House of Measures

To build a solid foundation for the House of Measures, you should use accepted standards within the industry when defining measures that the supplier reports to you.

Let’s hope that enough acquirers and suppliers adopt a common set of measurement definitions to increase the efficiency of acquiring technology solutions. However, standard or not, supplier measures must be defined in the supplier agreement. The agreement specifies which measurement data the supplier must provide, in what format, how the data will be collected and stored by the supplier (e.g., a retention period for data), how and how often it will be transferred to the acquirer, and who will have access to the data. Some suppliers may consider some of their data proprietary, so you may need to protect it. Also consider that some of your measurement data (e.g., total project cost data) may be proprietary and should not be shared with suppliers. You need to plan for the collection, storage, and access control of these types of sensitive data.

Several measures indicated in Figure 4-2 offer a good starting point while the search for better measures continues. The regularly used measures can typically be traced to one or more of these measurement information categories: schedule and progress, effort and cost, size and stability, and quality.

It’s frustrating when there is no accepted measure of size.

Measures can be either base or derived. Data for base measures are obtained by direct measurement. Data for derived measures come from combining the results of two or more base measures. Derived measures are typically expressed as ratios, composite indices, or other aggregate summaries. Often, summary measures are more quantitatively reliable and meaningful than the base measures from which they’re derived. A direct relationship exists between measurement objectives, measurement categories, base measures, and derived measures, as depicted in Table 4-8.

In some cases, you augment your measures with supplier measures. For instance, your return on investment metric can incorporate the supplier’s cost performance index. In most cases, however, the suppliers’ measures are the primary source of data, especially with regard to the development of the acquired product or service. For instance, for effective management of the project’s quality, size, cost, and schedule, it’s essential to measure and analyze the supplier-provided product or components by using technical performance measures (e.g., amount of new, modified, reused code; percentage of function points completed; defect removal efficiency).

Because you receive a significant amount of measurement data from the supplier, it is important that you periodically audit how the supplier collects and analyzes this data. The supplier agreement defines the required data analysis and the definition and examples of the measures the supplier must provide.

Having pristine measurement data and project measures is a prerequisite, but it isn’t sufficient for achieving long-term success. You must also consolidate project measures into a portfolio of project measures—program and organizational measures—that help illuminate to what degree you’re accomplishing your purpose through your collection of projects.

The CMMI-ACQ provides specific recommendations on how to develop and sustain a measurement capability to support management information needs (see Table 4-9).

To tightly integrate measurement and analysis into your processes, you specify the objectives of measurement and analysis so that they are aligned with identified information needs and project objectives (Table 4-9, practice 1.1).

Measurement objectives document the purposes of measurements and analyses and specify the kinds of actions that may be taken based on the results. You establish measurement objectives for your activities and work products as well as for supplier deliverables and milestones. Measurement objectives can range from “reduce time to delivery” to “reduce total life-cycle cost of the acquired technology solution” to “improve customer satisfaction.” They must be documented, reviewed by management and other relevant stakeholders, and updated as necessary.

The measurement objectives are refined into specific measures. Candidate measures are categorized and specified by name and unit of measure. To manage projects, you use supplier data (base measures) and supplier-reported derived measures, in addition to measures of your own progress and output (Table 4-9, practice 1.2). Supplier measures must be defined in the supplier agreement, including requirements for the supplier’s measurement collection and the measurement reports to be provided to you. When you specify supplier measures, it is particularly important to specify acceptance criteria for the data to ensure that the intended use of the supplier measures, such as aggregation and analysis by the acquirer, is feasible. To that end, you state measurement specifications in precise and unambiguous terms. They address the following two important criteria:

It is equally important to clearly specify how measurement data will be obtained and stored (Table 4-9, practice 1.3). In other words, explicit specifications are made for how, where, and when the data will be collected and retained. You must ensure that appropriate mechanisms are in place to obtain measurement data from the supplier in a consistent way—for instance, how data is to be transferred from the supplier to you. It is critical for you to insist in the supplier agreement on accurate data collection by the supplier. This includes requiring the use of applicable standard report formats and tools for reporting by the supplier.

It’s also critical to clearly define in advance how the measures will be used (Table 4-9, practice 1.4). The analyses must explicitly address the documented measurement objectives, and the results of the measurement analyses must be clearly understandable by its intended audiences. Just as the need for measurement drives data analysis, clarification of analysis criteria can affect the measures in use. Specifications for some measures may be refined further based on the data analysis procedures.

The CMMI-ACQ provides specific recommendations on how to develop and sustain a measurement capability that is tied to information needs and the objectives of the project or program (see Table 4.10).

The proof is in the pudding. You must be able to obtain specified measurement data (Table 4.10, practice 2.1). This requires the use of checks and balances to verify the completeness and integrity of acquirer and supplier measurement data. Checks can include scans for missing data, out-of-bounds data values, and unusual patterns and correlation across measures. Follow up with suppliers if data is not available or data checks indicate potential errors.

Rarely are the results of analyzing measurement data self-evident. For this reason, you use explicitly stated criteria for interpreting the results and drawing conclusions (Table 4.10, practice 2.2). Because suppliers provide important measurement data, it is paramount to discuss the results and preliminary conclusions with them. The results of planned analyses may suggest (or require) additional, unanticipated analyses. In addition, they may identify the need to refine existing measures, to calculate additional derived measures, or even to collect data for additional base measures to properly complete the planned analyses. Similarly, preparing the initial results for presentation may identify the need for additional analyses.

Storing measurement-related data enables the timely and cost-effective future use of historical data and results (Table 4.10, practice 2.3). You protect measurement data provided by the supplier according to the terms outlined in the supplier agreement. The agreement might specify that you must restrict access of a supplier’s measurement data to your employees only. Typically, you store the following measurement data:

Data sets for derived measures can typically be recalculated and need not be stored. However, it may be appropriate to store summaries based on derived measures (e.g., charts, tables of results, or report prose). Results of interim analyses need not be stored separately if they can be efficiently reconstructed.

Results are reported in a clear and concise manner so that they are understandable, easily interpretable, and clearly tied to identified information needs and objectives. For that purpose, you establish and maintain a standard format for communicating measurement data to stakeholders, including suppliers (Table 4.10, practice 2.4).

The CMMI-ACQ provides specific recommendations on how to quantitatively manage the project’s defined process to achieve the project’s established objectives for quality and process performance (see Table 4-11).

You establish the project’s quantitative objectives for quality and process performance based on the objectives of the organization and the project’s objectives (Table 4-11, practice 1.1). The quantitative quality and process performance objectives for the supplier are documented in the supplier agreement. You typically expect the supplier to execute its processes and apply its performance models to meet your quality performance objectives.

It’s critical for you to statistically manage those processes that are most critical to the overall process (Table 4-11, practices 1.2 and 1.3). You should pay particular attention to the subprocesses for interacting with a supplier—for example, negotiating a supplier agreement and conducting supplier reviews.

You monitor the performance of selected subprocesses, including those that involve interaction with a supplier, as well as the quality and performance of the supplier deliverables, for adherence to the quality and performance objectives (Table 4-11, practices 1.2 and 1.4). This selective monitoring provides you with insight into project and supplier performance so that you can predict the likelihood of achieving the project’s objectives for quality and process performance. You use this information to manage the risks of the project and to initiate corrective actions in time to preserve the ability to meet the project’s objectives.

The CMMI-ACQ provides specific recommendations on how to quantitatively manage the project’s defined subprocesses (see Table 4-12).

To support quantitative project management, you need to identify common measures that support statistical management (Table 4-12, practice 2.1). For that purpose, the measure must be an adequate performance indicator of how well the subprocess performs relative to its objectives. The measure must be controllable in that you can change its values by changing how the subprocess is implemented.

To detect the cause of a particular variation, such as an unexpected change in process performance, you typically need to understand variations in process performance and have insight into potential sources of anomalous patterns (Table 4-12, practice 2.2). You monitor process performance by comparing quality and process performance objectives to the established target limits of the measure (Table 4-12, practices 2.3 and 2.4). This comparison provides an appraisal of the process capability for each measured attribute of a subprocess.

You are responsible for monitoring the progress and output of the project. Monitoring and control functions are instituted early in the project during project planning and definition of the acquisition strategy. As the acquisition unfolds, monitoring and controlling are essential to ensure that appropriate resources are being applied and that activities are progressing according to plan.

After a supplier is selected and a supplier agreement is established, monitoring and control assume a twofold role. You are concerned with (1) continuing to monitor and control acquirer activities and work products while also (2) monitoring and controlling the progress and performance of the supplier’s execution under the supplier agreement and the supplier’s project plans.

You use the supplier’s measurement data, reflecting its progress and output, to monitor the supplier’s progress and ensure that it achieves service levels established in the supplier agreement. This task also includes monitoring the availability of resources and the skills and knowledge possessed by the personnel provided by the supplier. In addition to monitoring the status of your own work products, you monitor the fitness of supplier deliverables by establishing and exercising stringent acceptance criteria. You can obtain additional information by analyzing the supplier’s industry research results.

Kristin agrees.

Steve confirms Kristin’s ideas.

It’s critical for any acquirer to stay focused and yet consider all relevant measures when monitoring and controlling a program or a portfolio of projects. Based on George’s experience, this isn’t always the case.

George prompts Kristin for more information.

In any event, the effective use of measures to monitor and control a project is strongly influenced by the acquirer-supplier relationship.

If a corrective action is required to resolve a deviation from project plans, this action should be defined and tracked to closure. You take corrective action for acquirer deviations and also when supplier execution does not satisfy the supplier agreement or align with project. You may assign some corrective actions to a supplier. When you identify, for example, through your monitoring of measurement data, that supplier progress does not appear to be sufficient to meet a service level defined in the supplier agreement, then you initiate and manage corrective action for the supplier.

The CMMI-ACQ provides specific recommendations about how to monitor the actual performance and progress of the project against the project plan (see Table 4-13). This monitoring begins as soon as a project plan is established. You are also responsible for monitoring the progress and output of the project as a whole, that is, your activities as well as the supplier’s project execution. You monitor the progress of the supplier, including achievement of service levels established in the supplier agreement, by using the supplier’s measurement data about its progress and output.

A project’s documented plan is the basis for monitoring activities, communicating project status, and taking corrective action. You determine progress primarily by comparing project planning parameters to the estimates in the plan at prescribed milestones or control levels within the project schedule or WBS, and identifying significant deviations (Table 4-13, practice 1.1). Project planning parameters include work products and tasks; cost, effort, and schedule; and the knowledge, skills, and experience of project personnel. Attributes of the work products and tasks include size, complexity, weight, form, fit, or function. You document significant deviations that apply either to your project execution or to the supplier’s deviations from the project plan.

You also monitor overall project risk and the commitments made by relevant stakeholders to the plan (Table 4-13, practices 1.2 and 1.3). Many risks are your sole responsibility and may include sensitive information that should not be shared with the supplier (e.g., source selection, competition, internal staffing, or other risks). There can also be risks that require careful coordination with suppliers and appropriate escalation of risks and risk status (e.g., the feasibility of the technology to meet end user performance requirements). Shared risks may affect the mitigation approaches and result in jointly planned mitigations. Moreover, every acquirer needs to ensure that project data is managed against the project plan and that the level of stakeholder involvement aligns with the project plan (Table 4-13, practices 1.4 and 1.5).

It is crucial to monitor the technology solution during the operations and support phases (Table 4-13, practice 1.8). You make adequate provisions through the supplier agreement or in-house organizations to support the acquired technology solution. Typically, you use verification best practices to confirm that the organization, the physical environment, and the operations and support personnel are equipped to execute the operations and support activities.

You review operations and support organizations that have been designated to take responsibility for operation of the technology solution. You need to ensure that the workers have been identified and budgeted and are available when needed. This needs to happen very early and monitored throughout the project. The designated operations and support organizations demonstrate their readiness (capability and capacity) to accept responsibility for the product and to ensure uninterrupted support. In a typical readiness demonstration, the candidate organization executes all the activities of operations. Acquirers tend to use transition readiness criteria and verification and validation practices to determine whether the support organization meets the specified requirements. The criteria also address the readiness of the product for maintenance during the planned product life cycle.

A smooth transition to maintenance also requires monitoring to ensure that those who receive, store, use, and maintain the technology solution are trained. Typically, the supplier develops training materials for the solution it has developed. The training materials and resources to be provided by the supplier are specified in the supplier agreement and should meet the needs of various audiences (e.g., operations staff, support staff, end users). You verify that the training is provided at the appropriate times to the appropriate audiences and determine whether the training is adequate.

To monitor project performance, it is invaluable to conduct periodic progress and milestone reviews to keep stakeholders informed (Table 4-13, practices 1.6 and 1.7). These project reviews can be informal and need not be specified explicitly in the project plans. Reviews cover the results of collected and analyzed measures used for controlling the project.

You and the supplier may coordinate and conduct project reviews jointly. These reviews tend to fall into two categories: management and technical. Management reviews typically include acquirer and supplier measures, critical dependencies, and project risks and issues. Technical reviews are an important oversight tool that you can use to review and evaluate the state of the product and of the project, redirecting activity after the review if necessary.

Technical reviews typically include the following activities:

As a result of project reviews, you identify and document significant issues and deviations from the plan. This includes identifying and documenting both acquirer and supplier issues and deviations. In addition to seeking ways of improving your own performance, use the results of reviews to improve the supplier’s performance and to establish and nurture long-term relationships with preferred suppliers.

Appropriate visibility enables you to take timely corrective action when performance deviates significantly from the plan (see Table 4-14, practice 2.1). A deviation is significant if, when left unresolved, it prevents the project from meeting its objectives. Managing issues and corrective actions is your sole responsibility. The documentation of this effort can include sensitive information that should not be shared with the supplier (e.g., source selection, competition, and internal staffing).

You take decisive corrective action by determining and documenting the appropriate actions needed to address the identified issues (Table 4-14, practice 2.2). If a corrective action is required to resolve variances from project plans, this action should be defined and tracked to closure (Table 4-14, practice 2.3). You also analyze results of corrective actions to determine their effectiveness. In addition, lessons learned as a result of taking corrective action can be used as inputs to planning and risk management processes.

In addition to the project team’s monitoring and control activities, your quality assurance processes provide staff and management with objective insight into processes and associated work products (see Table 4-15). The tips for project monitoring and control apply equally to quality assurance.

You evaluate how the project team executes your processes, including interactions with suppliers. In addition, you review evaluation reports provided by suppliers to determine whether the supplier follows its processes. A popular means of doing this is to review recent CMMI maturity or capability ratings. Sufficient process quality assurance will help detect noncompliance issues, which may affect your or the supplier’s ability to successfully deliver the products to the customer as early as possible.

In the supplier agreement, you should retain the right to audit supplier processes if there is an indication that the suppliers are not following acceptable processes. You can review the results of the supplier’s quality assurance activities for selected supplier processes to ensure that the supplier is following its own processes. Typically, some supplier processes are considered critical, such as engineering or verification processes, where the supplier is required (through the supplier agreement) to follow project- or program-specified standards. In exceptional cases, you may directly perform product and process quality assurance for selected supplier processes. You and the supplier periodically share quality assurance issues and findings of mutual interest.

The practices in the Process and Product Quality Assurance process area ensure that planned processes are implemented, and the practices in the Acquisition Verification process area ensure that the specified requirements are satisfied. These two process areas may sometimes address the same work product, but from different perspectives. Projects should take advantage of this overlap to minimize duplicate efforts while taking care to maintain separate perspectives.

When it comes down to it, every acquisition project must be able to answer one question: How can you insert these changes—this technology solution—into its intended environment?

A key ingredient for the successful deployment of technology solutions is its technical readiness. You want to make sure—to verify—that the supplier’s product meets the contractual requirements and is ready to be installed in the intended environment for further integration and user acceptance testing (UAT). Verification is inherently an incremental process, because it occurs throughout the acquisition. You begin by verifying the requirements and plans, and then you verify the evolving work products such as design and test results, and in the end you verify the completed product.

It is critical that you insist contractually that your suppliers apply proven best practices for verification (e.g., structured coding techniques, peer code reviews) to increase the likelihood that the product will meet the contractual requirements.

It is equally important to find a cost-effective way to consistently and comprehensively verify your own work products and the supplier’s deliverables. Select your work products to verify based on the products’ relative importance to meeting project objectives and requirements and to addressing project risks.

At the same time, Steve was part of an effort to select supplier deliverables for which the supplier must provide verification records such as peer review results.

As with other supplier deliverables, peer review results must meet predefined acceptance criteria. Supplier deliverables are considered delivered only when the acceptance criteria are satisfied. There is no partial credit.

Note that a product can be released in increments. In that case, it’s a good idea to timebox each release. A timebox is essentially a fixed unit of development capacity. The capability to be implemented for a release of the technology solution must fit (be delivered) within a given time frame (e.g., within six weeks) and often within a given cost to the acquirer. The work to be performed in each timebox consists of both a functionality and a quality component. Verification falls into the quality component. By predefining (fixing) the duration of a release, you can better plan for adequate verification of the delivered functionality.

Testing the supplier’s product is a big part of verification.

Does an acquirer need to test every permutation or combination of every application in the system?

Steve smiles.

Because the supplier deliverables go through different stages of testing, it is critical that you clearly designate which testing environment is expected for each type of testing.

Matt pauses before describing how this works.

Steve pauses to consult his notes.

The CMMI-ACQ provides specific recommendations about how to ensure that an acquired product, intermediate acquirer work products, and supplier deliverables meet their contractual requirements (see Table 4-16). The Acquisition Verification process area includes selection, inspection, testing, analysis, and demonstration of acquirer work products and supplier deliverables.

Up-front preparation ensures that verification provisions are embedded in the contractual requirements, constraints, plans, budgets, and schedules. Preparation also entails defining support tools, test equipment and software, simulations, prototypes, and facilities.

You select work products to verify based on the products’ relative importance to meeting project objectives and requirements, and to addressing project risks (Table 4-16, practice 1.1). You also select supplier deliverables for which the supplier must provide verification records. The required methods and criteria are described in the supplier agreement. Typical acquirer verification activities include reviewing the solicitation package, supplier agreements, plans, requirements, documents, and design constraints.

In addition to selecting work products for verification, you select the verification methods. This typically begins with the definition of contractual requirements to ensure that the requirements are verifiable. The methods should address reverification to ensure that rework does not cause defects. Suppliers should be involved in your selection process to ensure that the project’s methods are appropriate for the supplier’s environment. To set clear expectations, you establish in the supplier agreement requirements for verification of supplier deliverables. The supplier agreement typically includes the following:

You must establish an environment to enable verification (Table 4-16, practice 1.2). This environment can be acquired, developed, reused, modified, or a combination of these, depending on the needs of the project. The type of environment depends on the work products selected for verification and the methods applied. The requirements for the environment can vary greatly. For instance, a peer review might require little more than a package of materials, some reviewers, and a room. In contrast, a product verification test might require simulators, emulators, scenario generators, data reduction tools, environmental controls, and interfaces with other systems.

Verification criteria are defined in the supplier agreement to ensure that the work products meet their requirements (Table 4-16, practice 1.3). You generate a set of comprehensive, integrated verification procedures for work products and any COTS products as necessary.

You use the verification methods, procedures, and criteria in the appropriate verification environment to verify the selected work products and any associated maintenance, training, and support services. It’s good to verify acquired products and work products incrementally, because it promotes early detection of problems and can result in the early removal of defects (see Table 4-17, practice 2.1). Incorporating the results of verification into a repair process can save considerable cost for fault isolation and rework.

You must compare actual verification results to established verification criteria to determine whether the results are acceptable (Table 4-17, practice 2.2). For each work product, all available verification results are incrementally analyzed and corrective actions are initiated to ensure that the documented requirements are met for both acquirer and supplier. Corrective actions are typically integrated into project monitoring activities. Because a peer review is one of several verification methods, you should include peer review data in this analysis to ensure that the verification results are analyzed sufficiently. Analysis reports or “as-run” method documentation can indicate that bad results are being caused by problems with the verification methods, verification criteria, or verification environment.

The CMMI-ACQ includes the process area Causal Analysis and Resolution. This process area provides recommendations for identifying the causes of defects and other problems and taking action to prevent them from occurring in the future. When determining which defects to analyze further, you should consider the impact of the defects, the frequency of occurrence, the similarity between defects, the cost of analysis, the time and resources needed to correct the defects, the safety considerations, and so on. Causal analysis is performed by people who have the best understanding of the selected defect or problem being studied.

Before the technology solution goes live and is available to the users, the project team needs to make sure that all involved parties are on the same page. For this, communication throughout the project is critical.

Consistently delivering a successful premiere—seamlessly inserting new or enhanced technology solutions into your environment—is not without its perils. For instance, you don’t want your project to be the one in which UAT came to mean “user angry at technology” instead of “user acceptance testing.”

Unfortunately, these UAT surprises are common. Steve adds his favorite one.

A key ingredient for successful validation of an acquired product is to have the true customer and users involved throughout the project life cycle. This is especially challenging on projects that have long life cycles.

George pauses for a moment and then continues.

What can you do to avoid UAT surprises and increase your chances for a smooth go-live of the technology solution? It’s vital to have a time-tested, proactive set of verification procedures and criteria to ensure that the product or component will fulfill its intended use when placed in its intended environment. You need to clearly identify the applicable verification procedures and criteria and then reference these procedures in the solicitation package and supplier agreement.

Steve strongly supports George’s ideas.

The closer you get to going live with the new or enhanced product, the more important it is to train the potential users and support personnel to use it. Effective training requires assessment of needs, planning, instructional design, and appropriate training media (e.g., workbooks, software) as well as a repository of training process data. As an organizational process, training has these main components: a managed training development program, documented plans, personnel with appropriate mastery of specific disciplines and other areas of knowledge, and mechanisms for measuring the effectiveness of the training.

When the big go-live day finally arrives, acquirers can only hope that the process plays out like the Vienna Philharmonic performing Strauss at New Year’s—breathtaking, flawless, and smiles on everyone’s faces when it’s finished.

With the technology solution ready for use, you transfer responsibility for operations and support from the development supplier to the appropriate group. This could be the same supplier or an in-house team on your side. The development supplier, however, is still very much responsible for successfully completing the warranty period for the technology solution. Steve stresses the importance of the warranty period.

After the warranty period is complete and the responsibility for the technology solution is transferred to the operational and support organizations, you review and analyze the results of the transition activities and determine whether any corrective actions must be completed before closing the project.

The CMMI-ACQ provides specific recommendations on how to demonstrate that an acquired product or service fulfills its intended use when placed in its intended environment (see Table 4-18). Validation demonstrates that the acquired product, as provided, will fulfill its intended use. In other words, validation ensures that the product meets the stakeholders’ intentions and the customer requirements.

Preparation activities include selecting products and components for validation and establishing and maintaining the validation environment, procedures, and criteria. Products or services are selected for validation on the basis of their importance to stakeholder intentions and customer requirements (Table 4-18, practice 1.1). The items selected for validation may include only the acquired product or may include appropriate levels of the product components that are used by the supplier to build the product. Any product or component—including replacement, maintenance, and training products, to name a few—may be subject to validation.

You should select validation methods early in the project life cycle so that they are clearly understood and agreed to by the relevant stakeholders. Validation methods include the following:

The supplier agreement should capture your expectations of suppliers for participation in validation of the product and components. The requirements for the validation environment are driven by the product or service selected, by the kind of work products being tested (e.g., design, prototype, final version), and by the methods of validation being used (Table 4-18, practice 1.2). The environment may be purchased or may be specified, designed, and built. The environments used for verification and for validation may be considered concurrently to reduce cost and improve efficiency or productivity.

You define validation procedures and criteria to ensure that the product or component fulfills its intended use when used in its intended environment (Table 4-18, practice 1.3). The validation procedures and criteria include validation of maintenance, training, and support services. They also address validation of requirements and the acquired product or service throughout the project life cycle. Typically, you establish formal user acceptance testing procedures and criteria to ensure that the delivered product meets stakeholder intentions before it is deployed in the intended environment. The validation procedures and criteria applicable to the supplier are typically referenced in the solicitation package and supplier agreement.

Validation activities are performed early and incrementally throughout the project life cycle (see Table 4-19, practice 2.1). They can be applied to all aspects of the product in any of its intended environments, such as operation, training, manufacturing, maintenance, and support.

The data resulting from validation tests, inspections, demonstrations, or evaluations are analyzed against the defined validation criteria (Table 4-19, practice 2.2). Analysis reports should indicate whether the stakeholders’ intentions were met; if there are deficiencies, these reports document the degree of success or failure and categorize probable causes of failures. You compare the collected test, inspection, or review results with established acceptance criteria to determine whether to proceed or to address requirements or design issues in the requirements development or technical solution processes. Analysis reports or “as-run” method documentation can indicate that bad validation results are being caused by validation method problems, validation criteria problems, or validation environment problems.

Often, acceptance of supplier deliverables is tied to supplier payments. You must ensure that payment terms defined in the supplier agreement are met and that supplier compensation is linked to supplier progress, as defined in the supplier agreement (see Table 4-20, practice 2.1). You should not make final payment to the supplier until you have certified that all the supplier deliverables meet the contractual requirements and that all acceptance criteria are satisfied. When you encounter nonperformance, exercise the contract provisions for withholding or reducing payments to the supplier to the appropriate degree.

You ensure that all acceptance criteria are satisfied and that all discrepancies are corrected before final acceptance of the acquired technology solution (Table 4-20, practice 2.2).

Your authorized representative assumes ownership of existing identified supplier products or deliverables tendered, or approves specific services rendered, as partial or complete performance of the supplier agreement by the supplier. You, usually through your authorized supplier agreement or contract administrator, provide the supplier with formal written notice that the supplier deliverables have been accepted or rejected.

Typically, the supplier integrates and packages the products and prepares for the transition to operations and support, including support for business user acceptance, and you oversee the supplier’s activities (Table 4-20, practice 2.3). These expectations and the acceptance criteria for transition to operations and support are included in the solicitation package and the supplier agreement.

After appropriate reviews of the transition activities, you make the product available for use according to the plans for pilot and transition. During this pilot, or initial period of production, you validate that the product is capable and ready for full operational use. During this defined transition or warranty period—for example, 30 days or 90 days for certain information systems—you oversee activities to make sure that the product is operating as planned and identify any corrective actions required. Although the product is in the operational environment, full responsibility for operations and support is not transitioned until this pilot period is complete and any identified corrective actions have been successfully completed. During this defined transition period, you ensure support of the product (for example, you may assign the supplier the responsibility to maintain support during transition).

Responsibility for operations and support of the technology solution is transferred by you to the operations and support organizations, which may be suppliers, only after these organizations demonstrate their capability and capacity to support the product and accept the responsibilities to perform their assigned operations and support processes. You ensure that these organizations understand post-transition service requirements from the supplier.

You maintain oversight responsibilities until the transition activities are complete and the transfer of responsibility for operations and support of the product has been accepted. This includes oversight of any supplier activities, based on the supplier agreement, for the execution of the transition of the product to operations and support.

After the transition is complete and the responsibility is transferred to the operational and support organizations (e.g., at the end of the warranty period for a software product), you review and analyze the results of the transition activities and determine whether any corrective actions (such as process improvement) must be completed before close.