Establish and maintain measurement objectives derived from identified information needs and objectives.

Measurement objectives document the purposes for which measurement and analysis are done and specify the kinds of actions that may be taken based on results of data analyses.

Measurement objectives focus on acquirer performance, supplier performance, and understanding the effects of their performance on customer operational and financial performance. Measurement objectives for the supplier enable defining and tracking service level expectations documented in the supplier agreement.

Measurement objectives identify what information is needed to do the following.

Sources of measurement objectives include management, technical, project, product, and process implementation needs.

Measurement objectives may be constrained by existing processes, available resources, or other measurement considerations. Judgments may need to be made about whether the value of the result is commensurate with resources devoted to doing the work.

Modifications to identified information needs and objectives may, in turn, be indicated as a consequence of the process and results of measurement and analysis.

Refer to the Project Planning process area for more information about estimating project attributes and other planning information needs.

Refer to the Project Monitoring and Control process area for more information about project performance information needs.

Refer to the Requirements Management process area for more information about maintaining requirements traceability and related information needs.

Typical Work Products

1. Measurement objectives

Subpractices

1. Document information needs and objectives.
2. Prioritize information needs and objectives.
3. Document, review, and update measurement objectives.

   Carefully consider the purposes and intended uses of measurement and analysis.

   Measurement objectives are documented, reviewed by management and other relevant stakeholders, and updated as necessary. Completing these activities enables traceability to subsequent measurement and analysis activities and helps ensure that analyses will properly address identified information needs and objectives.

4. Provide feedback for refining and clarifying information needs and objectives as necessary.

   Identified information needs and objectives may be refined and clarified as a result of setting measurement objectives. Initial descriptions of information needs may be unclear or ambiguous. Conflicts may arise between existing needs and objectives. Precise targets on an already existing measure may be unrealistic.

5. Review appropriate measurement objectives with potential suppliers throughout the solicitation, obtaining their feedback and commitment.

   Refer to the Solicitation and Supplier Agreement Development process area for more information about solicitations and interacting with potential suppliers.

6. Maintain traceability of measurement objectives to identified information needs and objectives.

   Of course, measurement objectives may also change to reflect evolving information needs and objectives.

Specify measures to address measurement objectives.

Measurement objectives are refined into precise, quantifiable measures.

Measurement of project and organizational work can typically be traced to one or more measurement information categories. These categories include the following: schedule and progress, effort and cost, size and stability, and quality.

Measures may be either base or derived. Data for base measures are obtained by direct measurement. Data for derived measures come from other data, typically by combining two or more base measures.

Derived measures typically are expressed as ratios, composite indices, or other aggregate summary measures. They are often more quantitatively reliable and meaningfully interpretable than the base measures used to generate them.

Base measures enable the creation of many derived measures or indicators from the same standard data sources. In addition, there is a direct relationship between measurement objectives, measurement categories, base measures, and derived measures. This direct relationship is depicted using some common examples in Table 9.1:

As a part of their measurement and analysis activities, projects may also consider the use of Earned Value Management (EVM) for measures related to cost and schedule [GEIA 748 2002]. EVM is a method for objectively measuring cost and schedule progress and for predicting estimated total costs and target completion dates based on past and current performance trends.

Typical EVM data include the planned cost of accomplishing specific and measurable tasks, the actual cost of completing tasks, and earned value, which is the planned cost of the work actually completed for each task. Using these or similar base measures, the project can calculate derived measures such as schedule and cost variance and more complex measures. These include schedule and cost performance indices. EVM derived measures can assist with estimating the cost for completion and additional resources that may be required.

To manage projects, an acquirer uses supplier data (i.e., base measures) and supplier-reported derived measures in addition to measures of acquirer progress and output. Supplier measures required by the acquirer allow the acquirer to comprehensively address measurement objectives and to comprehensively determine the progress of the project. In some cases, these supplier measures will augment acquirer measures (e.g., supplier’s schedule performance index and size estimation accuracy).

In most cases, supplier measures are the primary source of data, especially with regard to the development of the acquired product or service. For instance, measurement and analysis of the product or product components provided by a supplier through technical performance measures is essential for effective management. Technical performance measures are precisely defined measures based on a product requirement, product capability, or some combination of requirements and capabilities.

It is important to use measures to track high-risk items to closure and to help determine risk mitigation and corrective actions. These supplier measures must be defined in the supplier agreement, including a supplier’s measurement collection requirements and measurement reports to be provided to the acquirer.

Typical Work Products

1. Specifications of base and derived measures
2. Acceptance criteria for supplier measures

Subpractices

1. Identify candidate measures based on documented measurement objectives.

   Measurement objectives are refined into measures. Identified candidate measures are categorized and specified by name and unit of measure.

2. Identify existing measures that already address measurement objectives.

   Specifications for measures may already exist, perhaps established for other purposes earlier or elsewhere in the organization.

3. Specify operational definitions for measures.

   Operational definitions are stated in precise and unambiguous terms. They address two important criteria:

   X-Ref

   Operational definitions are a key to the effective specification of measures. Activities in ARD may be of assistance in determining many of these.

   • Communication: What has been measured, how was it measured, what are the units of measure, and what has been included or excluded?

   • Repeatability: Can the measurement be repeated, given the same definition, to get the same results?

   X-Ref

   For additional information on specifying measures, see the SEI’s Software Engineering Measurement and Analysis Web site (www.sei.cmu.edu/sema), the Practical Software & Systems Measurement Web site (www.psmsc.com), and the iSixSigma Web site (www.isixsigma.com).

4. Specify acceptance criteria based on operational definitions for measures that come from suppliers to the acquirer in a way that enables their intended use.

   Measures may be provided by the supplier as detailed measurement data or measurement reports. Measures that come from suppliers must be associated with the acquirer’s acceptance criteria for supplier measures. Acceptance criteria may be captured in measurement specifications or by checklists.

   Acceptance criteria should be defined in a way that enables the intended use of supplier measures, such as potential aggregation and analysis. These criteria must include criteria associated with the collection and transfer mechanisms and procedures that must be performed by the supplier. Consider all characteristics about supplier measures that may impact their use, such as differences in financial calendars used by different suppliers.

5. Prioritize, review, and update measures.

   Proposed specifications of measures are reviewed for their appropriateness with potential end users and other relevant stakeholders. Priorities are set or changed, and specifications of measures are updated as necessary.

Specify how measurement data is obtained and stored.

Explicit specification of collection methods helps ensure that the right data are collected properly. This specification may also help further clarify information needs and measurement objectives.

Proper attention to storage and retrieval procedures helps ensure that data are available and accessible for future use.

The supplier agreement specifies the measurement data the supplier must provide to the acquirer, in what format they have to be provided to the acquirer, how the measurement data will be collected and stored by the supplier (e.g., retention period of data), how and how often they will be transferred to the acquirer, and who has access to data. Some supplier data may be considered proprietary by the supplier and may need to be protected as such by the acquirer. Also consider that some acquirer measurement data (e.g., total project cost data) may be proprietary and should not be shared with suppliers. An acquirer must plan for the collection, storage, and access control of sensitive data.

The acquirer must ensure that appropriate mechanisms are in place to obtain measurement data from the supplier in a consistent way. It is critical for the acquirer to insist in the supplier agreement on accurate data collection by the supplier for the acquirer’s measurement and analysis.

Typical Work Products

1. Data collection and storage procedures
2. Data collection tools

Typical Supplier Deliverables

1. Recommendations for data collection and storage procedures

Subpractices

1. Identify existing sources of data that are generated from current work products, processes, or transactions.
2. Identify measures for which data are needed but are not currently available.
3. Specify how to collect and store the data for each required measure.

   Explicit specifications are made of how, where, and when data will be collected. Procedures for collecting valid data are specified. Data are stored in an accessible manner for analysis. This analysis helps determine whether data will be saved for possible reanalysis or documentation purposes.

   Questions to be considered typically include the following.

   • Have the frequency of collection and points in the process where measurements will be made been determined?

   • Has the timeline that is required to move measurement results from points of collection to repositories, other databases, or end users been established?

   • Who is responsible for obtaining data?

   • Who is responsible for data storage, retrieval, and security?

   • Have necessary supporting tools been developed or acquired?

   • Have required data collection requirements and applicable procedures been specified in supplier agreement standards and related documents?

4. Create data collection mechanisms and process guidance.

   Data collection and storage mechanisms are well integrated with other normal work processes. Data collection mechanisms may include manual or automated forms and templates. Clear, concise guidance on correct procedures is available to those responsible for doing the work. Training is provided as needed to clarify processes required for the collection of complete and accurate data and to minimize the burden on those who must provide and record data.

   Create mechanisms to transfer data and process guidance from the supplier to the acquirer, as appropriate. Data collection from a supplier may be integrated with periodic monitoring and review of supplier activities. Applicable standard report formats and tools to be used for reporting by the supplier must be specified in the supplier agreement.

5. Support automatic collection of data as appropriate and feasible.

   Tip

   In today’s environment, automation is often used. Further, some organizations use multiple tools and databases to address their measurement needs. If this is the case, you must carefully manage the compatibility among these tools and databases.

6. Prioritize, review, and update data collection and storage procedures.

   Proposed procedures are reviewed for their appropriateness and feasibility with those who are responsible for providing, collecting, and storing data. They also may have useful insights about how to improve existing processes or may be able to suggest other useful measures or analyses.

   Review data collection and storage procedures with potential suppliers throughout the solicitation. Update data collection and storage procedures, as appropriate, and obtain supplier commitment to collect and store measurement data and reference procedures in the supplier agreement.

7. Update measures and measurement objectives as necessary.

Specify how measurement data are analyzed and communicated.

Specifying analysis procedures in advance ensures that appropriate analyses will be conducted and reported to address documented measurement objectives (and thereby the information needs and objectives on which they are based). This approach also provides a check that necessary data will, in fact, be collected.

The supplier agreement defines the required data analysis and the definition and examples of measures the supplier must provide to the acquirer.

Typical Work Products

1. Analysis specifications and procedures
2. Data analysis tools

Typical Supplier Deliverables

1. Recommendations for analysis specification and procedures

Subpractices

1. Specify and prioritize the analyses to be conducted and the reports to be prepared.

   Early on, pay attention to the analyses to be conducted and to the manner in which results will be reported. These should meet the following criteria.

   • The analyses explicitly address the documented measurement objectives.

   • Presentation of results is clearly understandable by the audiences to whom the results are addressed.

   Priorities may have to be set within available resources.

   Establish and maintain a description of the analysis approach for data elements, a description of reports that must be provided by the supplier, and a reference to analysis specifications and procedures in the supplier agreement.

2. Select appropriate data analysis methods and tools.

   Refer to the Select Measures and Analytic Techniques specific practice and the Apply Statistical Methods to Understand Variation specific practice of the Quantitative Project Management process area for more information about the appropriate use of statistical analysis techniques and understanding variation.

   Descriptive statistics are typically used in data analysis to do the following.

   • Examine distributions on specified measures (e.g., central tendency, extent of variation, or data points exhibiting unusual variation).

   • Examine interrelationships among specified measures (e.g., comparisons of defects by phase of the product’s lifecycle or by product component).

   • Display changes over time.

3. Specify administrative procedures for analyzing data and communicating results.

   Data collected from a supplier are subject to validity checks that can be achieved by periodic audits of the supplier’s execution of data collection and analysis procedures for acquirer-required measures. The acquirer’s option to perform validity checks of measurement data collected by the supplier and the supplier’s execution of required analysis procedures must be defined in the supplier agreement.

   Tip

   Those responsible for analyzing the data and presenting the results should include those whose activities generated the measurement data or their management whenever possible. There should also be support provided by a process group, QA group, or measurement experts for these data-related activities.

4. Review and update the proposed content and format of specified analyses and reports.

   All of the proposed content and format are subject to review and revision, including analytic methods and tools, administrative procedures, and priorities. Relevant stakeholders consulted should include end users, sponsors, data analysts, and data providers.

   Review specified analyses and reports with suppliers and identify their commitment to support the analysis, and review recommendations they may provide related to the analysis of measurement data.

5. Update measures and measurement objectives as necessary.

   Just as measurement needs drive data analysis, clarification of analysis criteria can affect measurement. Specifications for some measures may be refined further based on specifications established for data analysis procedures. Other measures may prove unnecessary, or a need for additional measures may be recognized.

   Specifying how measures will be analyzed and reported may also suggest the need for refining measurement objectives themselves.

   X-Ref

   Refer to specific practice 1.1 when refining your measurement objectives.

6. Specify criteria for evaluating the utility of analysis results and for evaluating the conduct of measurement and analysis activities.

   Criteria for evaluating the utility of the analysis might address the extent to which the following apply.

   • The results are (1) provided in a timely manner, (2) understandable, and (3) used for decision making.

   • The work does not cost more to perform than is justified by the benefits it provides.

   Tip

   The criteria are divided into two lists. The first comprises criteria that any organization can use. The second is a bit more sophisticated and might be used by organizations once they establish their measurement program.

   Criteria for evaluating the conduct of the measurement and analysis might include the extent to which the following apply.

   • The amount of missing data or the number of flagged inconsistencies is beyond specified thresholds.

   • There is selection bias in sampling (e.g., only satisfied end users are surveyed to evaluate end-user satisfaction, or only unsuccessful projects are evaluated to determine overall productivity).

   • Measurement data are repeatable (e.g., statistically reliable).

   • Statistical assumptions have been satisfied (e.g., about the distribution of data or about appropriate measurement scales).

Obtain specified measurement data.

Data necessary for analysis are obtained and checked for completeness and integrity.

Supplier measurement data are collected according to data collection and storage procedures as defined in the supplier agreement. Data necessary for analysis are obtained and checked for completeness and integrity.

Typical Work Products

1. Base and derived measurement data sets
2. Results of data integrity tests

Typical Supplier Deliverables

1. Base and derived supplier measurement data sets
2. Results of data integrity tests of supplier measurement data

Subpractices

1. Obtain data for base measures.

   Data are collected as necessary for previously used and newly specified base measures. Existing data are gathered from project records or elsewhere in the organization.

   Data are obtained from the supplier for base measures as defined in the supplier agreement.

2. Generate data for derived measures.

   Values are newly calculated for all derived measures.

   Derived measures are obtained from the supplier as defined in the supplier agreement.

3. Perform data integrity checks as close to the source of data as possible.

   Tip

   If too much time has passed, it may be inefficient or even impossible to verify the integrity of a measure or to identify the source of missing data.

   All measurements are subject to error in specifying or recording data. It is always better to identify these errors and sources of missing data early in the measurement and analysis cycle.

   Checks can include scans for missing data, out-of-bounds data values, and unusual patterns and correlation across measures. It is particularly important to do the following.

   • Test and correct for inconsistency of classifications made by human judgment (i.e., to determine how frequently people make differing classification decisions based on the same information, otherwise known as intercoder reliability).

   • Empirically examine the relationships among measures that are used to calculate additional derived measures. Doing so can ensure that important distinctions are not overlooked and that derived measures convey their intended meanings (otherwise known as criterion validity).

   Use acceptance criteria to verify the results of data integrity tests conducted by the supplier and to verify the integrity of supplier data. Follow up with suppliers if data are not available or data integrity checks indicate potential errors in data.

   Refer to the Agreement Management process area for more information about resolving supplier agreement issues.

Analyze and interpret measurement data.

Measurement data are analyzed as planned, additional analyses are conducted as necessary, results are reviewed with relevant stakeholders, and necessary revisions for future analyses are noted.

Typical Work Products

1. Analysis results and draft reports

Typical Supplier Deliverables

1. Responses to analysis results and draft reports

Subpractices

1. Conduct initial analyses, interpret results, and draw preliminary conclusions.

   The results of data analyses are rarely self-evident. Criteria for interpreting results and drawing conclusions should be stated explicitly.

   Discuss results and preliminary conclusions with suppliers, as appropriate.

   Tip

   Often, someone can misinterpret analyses and draw incorrect conclusions. By specifying criteria for interpreting results in advance, you can reduce the risk of drawing incorrect conclusions.

2. Conduct additional measurement and analysis as necessary, and prepare results for presentation.

   Results of planned analyses may suggest (or require) additional, unanticipated analyses. In addition, these analyses may identify needs to refine existing measures, to calculate additional derived measures, or even to collect data for additional base measures to properly complete the planned analysis. Similarly, preparing initial results for presentation may identify the need for additional, unanticipated analyses. Coordinate additional analyses with suppliers, as appropriate.

3. Review initial results with relevant stakeholders.

   It may be appropriate to review initial interpretations of results and the way in which these results are presented before disseminating and communicating them widely.

   Relevant stakeholders with whom reviews may be conducted include intended end users and sponsors, as well as data analysts and data providers.

   Review initial results related to supplier progress or output with suppliers and determine if revisions are appropriate based on their response.

4. Refine criteria for future analyses.

   Tip

   Measurement and analysis is a learning process. You will typically go through many cycles before the measures and analyses are fine-tuned. Recognize that this learning may also be occurring on the supplier side.

   Lessons that can improve future efforts are often learned from conducting data analyses and preparing results. Similarly, ways to improve measurement specifications and data collection procedures may become apparent, as may ideas for refining identified information needs and objectives.

   Update data acceptance criteria for supplier measures, as appropriate.

Manage and store measurement data, measurement specifications, and analysis results.

Storing measurement-related information enables its timely and cost-effective use as historical data and results. The information also is needed to provide sufficient context for interpretation of data, measurement criteria, and analysis results.

Stored information contains or references other information needed to understand and interpret the measures and to assess them for reasonableness and applicability (e.g., measurement specifications used on different projects when comparing across projects).

Typically, data sets for derived measures can be recalculated and need not be stored. However, it may be appropriate to store summaries based on derived measures (e.g., charts, tables of results, or report prose).

Interim analysis results need not be stored separately if they can be efficiently reconstructed.

Projects may choose to store project-specific data and results in a project-specific repository. When data are shared across projects, they may reside in the organization’s measurement repository.

Refer to the Establish the Organization’s Measurement Repository specific practice of the Organizational Process Definition process area for more information about establishing the organization’s measurement repository.

Refer to the Configuration Management process area for more information about managing measurement work products.

Typical Work Products

1. Stored data inventory

Subpractices

1. Review data to ensure their completeness, integrity, accuracy, and currency.
2. Store data according to data storage procedures.
3. Make stored contents available for use only to appropriate groups and personnel.

   The acquirer protects measurement data provided by the supplier according to the supplier agreement. The supplier agreement might specify that the acquirer must restrict access to a supplier’s measurement data to acquirer employees only.

4. Prevent stored information from being used inappropriately.

Communicate results of measurement and analysis activities to all relevant stakeholders.

The results of the measurement and analysis process are communicated to relevant stakeholders in a timely and usable fashion to support decision making and assist in taking corrective action.

Relevant stakeholders include intended users, sponsors, data analysts, and data providers.

Relevant stakeholders also include suppliers.

Typical Work Products

1. Delivered reports and related analysis results
2. Contextual information or guidance to help interpret analysis results

Subpractices

1. Keep relevant stakeholders apprised of measurement results in a timely manner.

   To the extent possible and as part of the normal way they do business, users of measurement results are kept personally involved in setting objectives and deciding on plans of action for measurement and analysis. Users are regularly kept apprised of progress and interim results.

   Refer to the Project Monitoring and Control process area for more information about the use of measurement results.

2. Assist relevant stakeholders in understanding results.

   Results are communicated in a clear and concise manner appropriate to the methodological sophistication of relevant stakeholders. Results are understandable, easily interpretable, and clearly tied to identified information needs and objectives.

   Tip

   As organizations mature, management and staff members should become more comfortable with measurement, be more likely to interpret the analyses correctly, and be able to ask the right questions to help them draw the right conclusions.

   The acquirer establishes and maintains a standard format for communicating measurement data to relevant stakeholders.

   Data are often not self-evident to practitioners who are not measurement experts. The following measurement choices should be explicitly clarified:

   • How and why base and derived measures were specified

   • How data were obtained

   • How to interpret results based on the data analysis methods used

   • How results address information needs