The purpose of Measurement and Analysis (MA) is to develop and sustain a measurement capability used to support management information needs.
Use this process area when you need to measure project progress, product size or quality, or process performance in support of making decisions and taking corrective action. In addition, use this process area when providing acquisition office perspectives on measurement data provided by suppliers.
The Measurement and Analysis process area involves the following activities:
• Specifying objectives of measurement and analysis so they are aligned with identified information needs and objectives
• Specifying measures, analysis techniques, and mechanisms for data collection, data storage, reporting, and feedback
This process area uses the term objective both as a noun meaning “a goal to be attained” (e.g., first bullet) and as an adjective meaning “unbiased” (e.g., fourth bullet).
• Implementing the collection, storage, analysis, and reporting of data
• Providing objective results that can be used in making informed decisions, and taking appropriate corrective action
The integration of measurement and analysis activities into the processes of the project supports the following:
• Objective planning and estimating
• Tracking actual performance against established plans and objectives
• Identifying and resolving process-related issues
Measurement involves everyone. A centralized group, such as a process group or a measurement group, may provide help in defining the measures, the analyses to perform, and the reporting content and charts used.
• Providing a basis for incorporating measurement into additional processes in the future
The staff required to implement a measurement capability may or may not be employed in a separate organization-wide program. Measurement capability may be integrated into individual projects or other organizational functions (e.g., quality assurance).
The initial focus for measurement activities is at the project level. However, a measurement capability may prove useful for addressing organization- and enterprise-wide information needs. To support this capability, measurement activities should support information needs at multiple levels, including the business, organizational unit, and project to minimize rework as the organization matures.
Projects may choose to store project-specific data and results in a project-specific repository. When data are shared widely across projects, data may reside in the organization’s measurement repository.
Measurement and analysis of product components provided by suppliers is essential for effective management of the quality and costs of the project. It is possible, with careful management of supplier agreements, to provide insight into data that support supplier-performance analysis.
The acquirer specifies measures that enable it to gauge its own progress and output, supplier progress and output per contractual requirements, and the status of the evolving products acquired. An acquirer establishes measurement objectives for its activities and work products and supplier activities and deliverables. Measurement objectives are derived from information needs that come from project objectives, organizational objectives, and business needs.
It is important to include in the supplier agreement any measurement and analysis activities that you want the supplier to perform. Refer to SSAD, AM, and ATM for more information about supplier activities.
Measurement objectives are used to define measures as well as collection, analysis, storage, and usage procedures for measures. These measures are specified in the project plan. Measures for the supplier, data collection processes and timing, expected analysis, and required storage should be specified in the supplier agreement.
In projects where multiple products are acquired to deliver a capability to the end user or where there are relationships with other projects to acquire joint capabilities, additional measures may be identified to track and achieve interoperability for programmatic, technical, and operational interfaces.
Refer to the Project Planning process area for more information about estimating project attributes and other planning information needs.
Refer to the Project Monitoring and Control process area for more information about monitoring project performance information needs.
Refer to the Configuration Management process area for more information about managing measurement work products.
Refer to the Requirements Management process area for more information about maintaining requirements traceability and related information needs.
Refer to the Solicitation and Supplier Agreement Development process area for more information about including supplier measures in the solicitation package and in the supplier agreement.
Refer to the Organizational Process Definition process area for more information about establishing the organization’s measurement repository.
Refer to the Quantitative Project Management process area for more information about understanding variation and the appropriate use of statistical analysis techniques.
Specific Goal and Practice Summary
SG 1 Align Measurement and Analysis Activities
SP 1.1 Establish Measurement Objectives
SP 1.2 Specify Measures
SP 1.3 Specify Data Collection and Storage Procedures
SP 1.4 Specify Analysis Procedures
SG 2 Provide Measurement Results
SP 2.1 Obtain Measurement Data
SP 2.2 Analyze Measurement Data
SP 2.3 Store Data and Results
SP 2.4 Communicate Results
Measurement objectives and activities are aligned with identified information needs and objectives.
The specific practices under this specific goal may be addressed concurrently or in any order:
• When establishing measurement objectives, experts often think ahead about necessary criteria for specifying measures and analysis procedures. They also think concurrently about the constraints imposed by data collection and storage procedures.
When starting a measurement program, an iterative process is usually helpful since you often do not know all of your objectives. On long acquisition projects, needed measurement activities may change significantly over the life of the project.
• Often, it is important to specify the essential analyses to be conducted before attending to details of measurement specification, data collection, or storage.
Establish and maintain measurement objectives derived from identified information needs and objectives.
When establishing measurement objectives, ask yourself what question you are answering with the data, why you are measuring something, and how these measurements will affect project behavior.
Measurement objectives document the purposes for which measurement and analysis are done and specify the kinds of actions that may be taken based on results of data analyses.
Measurement objectives focus on acquirer performance, supplier performance, and understanding the effects of their performance on customer operational and financial performance. Measurement objectives for the supplier enable defining and tracking service level expectations documented in the supplier agreement.
Measurement can be an expensive activity. Think about why you are collecting the measures and try to ensure that the measures are used.
Measurement objectives identify what information is needed to do the following.
• Maintain alignment to project objectives and provide results that keep a project on track to its successful conclusion.
• Support the organization’s ability to establish an infrastructure that reinforces and grows acquirer capabilities, including processes, people, and technologies, as appropriate.
• Support the enterprise’s ability to monitor and manage its financial results and customer expectations, as appropriate.
Sources of measurement objectives include management, technical, project, product, and process implementation needs.
Measurement objectives may be constrained by existing processes, available resources, or other measurement considerations. Judgments may need to be made about whether the value of the result is commensurate with resources devoted to doing the work.
Modifications to identified information needs and objectives may, in turn, be indicated as a consequence of the process and results of measurement and analysis.
Refer to the Project Planning process area for more information about estimating project attributes and other planning information needs.
Refer to the Project Monitoring and Control process area for more information about project performance information needs.
Refer to the Requirements Management process area for more information about maintaining requirements traceability and related information needs.
Typical Work Products
Subpractices
Carefully consider the purposes and intended uses of measurement and analysis.
Measurement objectives are documented, reviewed by management and other relevant stakeholders, and updated as necessary. Completing these activities enables traceability to subsequent measurement and analysis activities and helps ensure that analyses will properly address identified information needs and objectives.
Identified information needs and objectives may be refined and clarified as a result of setting measurement objectives. Initial descriptions of information needs may be unclear or ambiguous. Conflicts may arise between existing needs and objectives. Precise targets on an already existing measure may be unrealistic.
Refer to the Solicitation and Supplier Agreement Development process area for more information about solicitations and interacting with potential suppliers.
Of course, measurement objectives may also change to reflect evolving information needs and objectives.
Specify measures to address measurement objectives.
Measurement objectives are refined into precise, quantifiable measures.
Measurement of project and organizational work can typically be traced to one or more measurement information categories. These categories include the following: schedule and progress, effort and cost, size and stability, and quality.
Measures may be either base or derived. Data for base measures are obtained by direct measurement. Data for derived measures come from other data, typically by combining two or more base measures.
Derived measures typically are expressed as ratios, composite indices, or other aggregate summary measures. They are often more quantitatively reliable and meaningfully interpretable than the base measures used to generate them.
Base measures enable the creation of many derived measures or indicators from the same standard data sources. In addition, there is a direct relationship between measurement objectives, measurement categories, base measures, and derived measures. This direct relationship is depicted using some common examples in Table 9.1:
As a part of their measurement and analysis activities, projects may also consider the use of Earned Value Management (EVM) for measures related to cost and schedule [GEIA 748 2002]. EVM is a method for objectively measuring cost and schedule progress and for predicting estimated total costs and target completion dates based on past and current performance trends.
Typical EVM data include the planned cost of accomplishing specific and measurable tasks, the actual cost of completing tasks, and earned value, which is the planned cost of the work actually completed for each task. Using these or similar base measures, the project can calculate derived measures such as schedule and cost variance and more complex measures. These include schedule and cost performance indices. EVM derived measures can assist with estimating the cost for completion and additional resources that may be required.
Table 9.1 and the discussions of EVM are not currently captured in CMMI-DEV. Acquirers may wish to share these ideas with their suppliers to facilitate integrated approaches to this measurement challenge.
To manage projects, an acquirer uses supplier data (i.e., base measures) and supplier-reported derived measures in addition to measures of acquirer progress and output. Supplier measures required by the acquirer allow the acquirer to comprehensively address measurement objectives and to comprehensively determine the progress of the project. In some cases, these supplier measures will augment acquirer measures (e.g., supplier’s schedule performance index and size estimation accuracy).
The project must determine when an integrated answer is needed with the supplier, and when it must deliver an independent analysis as part of its acquisition responsibilities.
In most cases, supplier measures are the primary source of data, especially with regard to the development of the acquired product or service. For instance, measurement and analysis of the product or product components provided by a supplier through technical performance measures is essential for effective management. Technical performance measures are precisely defined measures based on a product requirement, product capability, or some combination of requirements and capabilities.
It is important to use measures to track high-risk items to closure and to help determine risk mitigation and corrective actions. These supplier measures must be defined in the supplier agreement, including a supplier’s measurement collection requirements and measurement reports to be provided to the acquirer.
An example of a derived measure that is typically expressed as a ratio is a productivity measure (e.g., proposal pages read per hour). An example of a derived measure that is typically expressed as a composite index is a process capability index (e.g., Cpk, which indicates how well centered and tightly distributed a stable process is relative to selected specification limits).
Typical Work Products
Subpractices
Measurement objectives are refined into measures. Identified candidate measures are categorized and specified by name and unit of measure.
Specifications for measures may already exist, perhaps established for other purposes earlier or elsewhere in the organization.
Operational definitions are stated in precise and unambiguous terms. They address two important criteria:
• Communication: What has been measured, how was it measured, what are the units of measure, and what has been included or excluded?
• Repeatability: Can the measurement be repeated, given the same definition, to get the same results?
For additional information on specifying measures, see the SEI’s Software Engineering Measurement and Analysis Web site (www.sei.cmu.edu/sema), the Practical Software & Systems Measurement Web site (www.psmsc.com), and the iSixSigma Web site (www.isixsigma.com).
Measures may be provided by the supplier as detailed measurement data or measurement reports. Measures that come from suppliers must be associated with the acquirer’s acceptance criteria for supplier measures. Acceptance criteria may be captured in measurement specifications or by checklists.
Acceptance criteria should be defined in a way that enables the intended use of supplier measures, such as potential aggregation and analysis. These criteria must include criteria associated with the collection and transfer mechanisms and procedures that must be performed by the supplier. Consider all characteristics about supplier measures that may impact their use, such as differences in financial calendars used by different suppliers.
Proposed specifications of measures are reviewed for their appropriateness with potential end users and other relevant stakeholders. Priorities are set or changed, and specifications of measures are updated as necessary.
Specify how measurement data is obtained and stored.
Explicit specification of collection methods helps ensure that the right data are collected properly. This specification may also help further clarify information needs and measurement objectives.
Proper attention to storage and retrieval procedures helps ensure that data are available and accessible for future use.
Ensuring appropriate accessibility of data and maintenance of data integrity are two key concerns related to data storage and retrieval. Both concerns are magnified when there are multiple suppliers or complex interactions among customers, acquirers, and suppliers.
The supplier agreement specifies the measurement data the supplier must provide to the acquirer, in what format they have to be provided to the acquirer, how the measurement data will be collected and stored by the supplier (e.g., retention period of data), how and how often they will be transferred to the acquirer, and who has access to data. Some supplier data may be considered proprietary by the supplier and may need to be protected as such by the acquirer. Also consider that some acquirer measurement data (e.g., total project cost data) may be proprietary and should not be shared with suppliers. An acquirer must plan for the collection, storage, and access control of sensitive data.
The acquirer must ensure that appropriate mechanisms are in place to obtain measurement data from the supplier in a consistent way. It is critical for the acquirer to insist in the supplier agreement on accurate data collection by the supplier for the acquirer’s measurement and analysis.
Typical Work Products
Typical Supplier Deliverables
Subpractices
Explicit specifications are made of how, where, and when data will be collected. Procedures for collecting valid data are specified. Data are stored in an accessible manner for analysis. This analysis helps determine whether data will be saved for possible reanalysis or documentation purposes.
Data collection and storage mechanisms are well integrated with other normal work processes. Data collection mechanisms may include manual or automated forms and templates. Clear, concise guidance on correct procedures is available to those responsible for doing the work. Training is provided as needed to clarify processes required for the collection of complete and accurate data and to minimize the burden on those who must provide and record data.
Create mechanisms to transfer data and process guidance from the supplier to the acquirer, as appropriate. Data collection from a supplier may be integrated with periodic monitoring and review of supplier activities. Applicable standard report formats and tools to be used for reporting by the supplier must be specified in the supplier agreement.
Proposed procedures are reviewed for their appropriateness and feasibility with those who are responsible for providing, collecting, and storing data. They also may have useful insights about how to improve existing processes or may be able to suggest other useful measures or analyses.
Review data collection and storage procedures with potential suppliers throughout the solicitation. Update data collection and storage procedures, as appropriate, and obtain supplier commitment to collect and store measurement data and reference procedures in the supplier agreement.
Specify how measurement data are analyzed and communicated.
Specifying analysis procedures in advance ensures that appropriate analyses will be conducted and reported to address documented measurement objectives (and thereby the information needs and objectives on which they are based). This approach also provides a check that necessary data will, in fact, be collected.
The supplier agreement defines the required data analysis and the definition and examples of measures the supplier must provide to the acquirer.
Often, someone can manipulate data to provide the picture he or she wants to convey. By specifying the analysis procedures in advance, you can minimize this type of abuse.
Typical Work Products
Typical Supplier Deliverables
Early on, pay attention to the analyses to be conducted and to the manner in which results will be reported. These should meet the following criteria.
• The analyses explicitly address the documented measurement objectives.
• Presentation of results is clearly understandable by the audiences to whom the results are addressed.
Priorities may have to be set within available resources.
Establish and maintain a description of the analysis approach for data elements, a description of reports that must be provided by the supplier, and a reference to analysis specifications and procedures in the supplier agreement.
Refer to the Select Measures and Analytic Techniques specific practice and the Apply Statistical Methods to Understand Variation specific practice of the Quantitative Project Management process area for more information about the appropriate use of statistical analysis techniques and understanding variation.
Data collected from a supplier are subject to validity checks that can be achieved by periodic audits of the supplier’s execution of data collection and analysis procedures for acquirer-required measures. The acquirer’s option to perform validity checks of measurement data collected by the supplier and the supplier’s execution of required analysis procedures must be defined in the supplier agreement.
Those responsible for analyzing the data and presenting the results should include those whose activities generated the measurement data or their management whenever possible. There should also be support provided by a process group, QA group, or measurement experts for these data-related activities.
All of the proposed content and format are subject to review and revision, including analytic methods and tools, administrative procedures, and priorities. Relevant stakeholders consulted should include end users, sponsors, data analysts, and data providers.
Review specified analyses and reports with suppliers and identify their commitment to support the analysis, and review recommendations they may provide related to the analysis of measurement data.
Just as measurement needs drive data analysis, clarification of analysis criteria can affect measurement. Specifications for some measures may be refined further based on specifications established for data analysis procedures. Other measures may prove unnecessary, or a need for additional measures may be recognized.
Specifying how measures will be analyzed and reported may also suggest the need for refining measurement objectives themselves.
Measurement results, which address identified information needs and objectives, are provided.
The primary reason for conducting measurement and analysis is to address identified information needs and objectives. Measurement results based on objective evidence can help to monitor performance, fulfill obligations documented in a supplier agreement, make informed management and technical decisions, and enable corrective actions to be taken.
MA provides the foundation for the behavior required in high-maturity organizations. By the time organizations reach maturity level 4, management and staff members will use measurement results as part of their daily work.
Obtain specified measurement data.
Data necessary for analysis are obtained and checked for completeness and integrity.
Supplier measurement data are collected according to data collection and storage procedures as defined in the supplier agreement. Data necessary for analysis are obtained and checked for completeness and integrity.
Typical Work Products
Typical Supplier Deliverables
Subpractices
Data are collected as necessary for previously used and newly specified base measures. Existing data are gathered from project records or elsewhere in the organization.
Data are obtained from the supplier for base measures as defined in the supplier agreement.
Values are newly calculated for all derived measures.
Derived measures are obtained from the supplier as defined in the supplier agreement.
If too much time has passed, it may be inefficient or even impossible to verify the integrity of a measure or to identify the source of missing data.
All measurements are subject to error in specifying or recording data. It is always better to identify these errors and sources of missing data early in the measurement and analysis cycle.
Checks can include scans for missing data, out-of-bounds data values, and unusual patterns and correlation across measures. It is particularly important to do the following.
• Test and correct for inconsistency of classifications made by human judgment (i.e., to determine how frequently people make differing classification decisions based on the same information, otherwise known as intercoder reliability).
• Empirically examine the relationships among measures that are used to calculate additional derived measures. Doing so can ensure that important distinctions are not overlooked and that derived measures convey their intended meanings (otherwise known as criterion validity).
Use acceptance criteria to verify the results of data integrity tests conducted by the supplier and to verify the integrity of supplier data. Follow up with suppliers if data are not available or data integrity checks indicate potential errors in data.
Refer to the Agreement Management process area for more information about resolving supplier agreement issues.
Analyze and interpret measurement data.
Measurement data are analyzed as planned, additional analyses are conducted as necessary, results are reviewed with relevant stakeholders, and necessary revisions for future analyses are noted.
Typical Work Products
Typical Supplier Deliverables
Subpractices
The results of data analyses are rarely self-evident. Criteria for interpreting results and drawing conclusions should be stated explicitly.
Discuss results and preliminary conclusions with suppliers, as appropriate.
Results of planned analyses may suggest (or require) additional, unanticipated analyses. In addition, these analyses may identify needs to refine existing measures, to calculate additional derived measures, or even to collect data for additional base measures to properly complete the planned analysis. Similarly, preparing initial results for presentation may identify the need for additional, unanticipated analyses. Coordinate additional analyses with suppliers, as appropriate.
It may be appropriate to review initial interpretations of results and the way in which these results are presented before disseminating and communicating them widely.
Relevant stakeholders with whom reviews may be conducted include intended end users and sponsors, as well as data analysts and data providers.
Review initial results related to supplier progress or output with suppliers and determine if revisions are appropriate based on their response.
Measurement and analysis is a learning process. You will typically go through many cycles before the measures and analyses are fine-tuned. Recognize that this learning may also be occurring on the supplier side.
Lessons that can improve future efforts are often learned from conducting data analyses and preparing results. Similarly, ways to improve measurement specifications and data collection procedures may become apparent, as may ideas for refining identified information needs and objectives.
Update data acceptance criteria for supplier measures, as appropriate.
Manage and store measurement data, measurement specifications, and analysis results.
Storing measurement-related information enables its timely and cost-effective use as historical data and results. The information also is needed to provide sufficient context for interpretation of data, measurement criteria, and analysis results.
Stored information contains or references other information needed to understand and interpret the measures and to assess them for reasonableness and applicability (e.g., measurement specifications used on different projects when comparing across projects).
Typically, data sets for derived measures can be recalculated and need not be stored. However, it may be appropriate to store summaries based on derived measures (e.g., charts, tables of results, or report prose).
Interim analysis results need not be stored separately if they can be efficiently reconstructed.
Projects may choose to store project-specific data and results in a project-specific repository. When data are shared across projects, they may reside in the organization’s measurement repository.
Refer to the Establish the Organization’s Measurement Repository specific practice of the Organizational Process Definition process area for more information about establishing the organization’s measurement repository.
Refer to the Configuration Management process area for more information about managing measurement work products.
Typical Work Products
Subpractices
The acquirer protects measurement data provided by the supplier according to the supplier agreement. The supplier agreement might specify that the acquirer must restrict access to a supplier’s measurement data to acquirer employees only.
Communicate results of measurement and analysis activities to all relevant stakeholders.
An indicator of a mature organization is the daily use of measurement data by both staff members and management to guide their activities. Such use of measurement data requires effective communication of measurement data and the results of analyses.
The results of the measurement and analysis process are communicated to relevant stakeholders in a timely and usable fashion to support decision making and assist in taking corrective action.
Relevant stakeholders include intended users, sponsors, data analysts, and data providers.
Relevant stakeholders also include suppliers.
Typical Work Products
Subpractices
To the extent possible and as part of the normal way they do business, users of measurement results are kept personally involved in setting objectives and deciding on plans of action for measurement and analysis. Users are regularly kept apprised of progress and interim results.
Refer to the Project Monitoring and Control process area for more information about the use of measurement results.
Results are communicated in a clear and concise manner appropriate to the methodological sophistication of relevant stakeholders. Results are understandable, easily interpretable, and clearly tied to identified information needs and objectives.
As organizations mature, management and staff members should become more comfortable with measurement, be more likely to interpret the analyses correctly, and be able to ask the right questions to help them draw the right conclusions.
The acquirer establishes and maintains a standard format for communicating measurement data to relevant stakeholders.
Data are often not self-evident to practitioners who are not measurement experts. The following measurement choices should be explicitly clarified:
• How and why base and derived measures were specified
• How data were obtained
• How to interpret results based on the data analysis methods used
• How results address information needs
3.17.183.186