Measurement and Analysis: A Support Process Area at Maturity Level 2
Purpose
The purpose of Measurement and Analysis (MA) is to develop and sustain a measurement capability used to support management information needs.
Introductory Notes
The Measurement and Analysis process area involves the following activities:
• Specifying objectives of measurement and analysis so that they are aligned with identified information needs and work, organizational, or business objectives
• Specifying measures, analysis techniques, and mechanisms for data collection, data storage, reporting, and feedback
• Implementing the analysis techniques and mechanisms for data collection, data reporting, and feedback
• Providing objective results that can be used in making informed decisions and taking appropriate corrective action
The integration of measurement and analysis activities into the processes of the work supports the following:
• Objective planning and estimating
• Tracking actual progress and performance against established plans and objectives
• Identifying and resolving process related issues
• Providing a basis for incorporating measurement into additional processes in the future
The staff required to implement a measurement capability may or may not be employed in a separate organization-wide program. Measurement capability may be integrated into individual work groups or other organizational functions (e.g., quality assurance).
The initial focus for measurement activities is at the work group level. However, a measurement capability can prove useful for addressing organization- and enterprise-wide information needs. To support this capability, measurement activities should support information needs at multiple levels, including the business, organizational unit, and work group to minimize re-work as the organization matures.
Work groups can store work-specific data and results in a work-specific repository, but when data are to be used widely or are to be analyzed in support of determining data trends or benchmarks, data may reside in the organization’s measurement repository.
Measurement and analysis of product components provided by suppliers is essential for effective management of the quality and costs of the work. It is possible, with careful management of supplier agreements, to provide insight into data that support supplier performance analysis.
Measurement objectives are derived from information needs that come from work, organizational, or business objectives. In this process area, when the term “objectives” is used without the “measurement” qualifier, it indicates either work, organizational, or business objectives.
Related Process Areas
Refer to the Configuration Management process area for more information about establishing and maintaining the integrity of work products using configuration identification, configuration control, configuration status accounting, and configuration audits.
Refer to the Organizational Process Definition process area for more information about establishing the organization’s measurement repository.
Refer to the Quantitative Work Management process area for more information about quantitatively managing the work to achieve the established quality and process performance objectives for the work.
Refer to the Work Monitoring and Control process area for more information about monitoring work planning parameters.
Refer to the Work Planning process area for more information about establishing estimates.
Specific Practices by Goal
SG 1 Align Measurement and Analysis Activities
Measurement objectives and activities are aligned with identified information needs and objectives.
The specific practices under this specific goal can be addressed concurrently or in any order.
When establishing measurement objectives, experts often think ahead about necessary criteria for specifying measures and analysis procedures. They also think concurrently about the constraints imposed by data collection and storage procedures.
Often it is important to specify the essential analyses to be conducted before attending to details of measurement specification, data collection, or storage.
SP 1.1 Establish Measurement Objectives
Establish and maintain measurement objectives derived from identified information needs and objectives.
Measurement objectives document the purposes for which measurement and analysis are done and specify the kinds of actions that can be taken based on results of data analyses. Measurement objectives can also identify the change in behavior desired as a result of implementing a measurement and analysis activity.
Measurement objectives may be constrained by existing processes, available resources, or other measurement considerations. Judgments may need to be made about whether the value of the result is commensurate with resources devoted to doing the work.
Modifications to identified information needs and objectives can, in turn, be indicated as a consequence of the process and results of measurement and analysis.
Refer to the Requirements Management process area for more information about maintaining bidirectional traceability of requirements.
Refer to the Work Monitoring and Control process area for more information about monitoring work planning parameters.
Refer to the Work Planning process area for more information about establishing estimates.
Example Work Products
1. Measurement objectives
Subpractices
1. Document information needs and objectives.
2. Prioritize information needs and objectives.
It can be neither possible nor desirable to subject all initially identified information needs to measurement and analysis. Priorities may also need to be set within the limits of available resources.
3. Document, review, and update measurement objectives.
Carefully consider the purposes and intended uses of measurement and analysis.
The measurement objectives are documented, reviewed by management and other relevant stakeholders, and updated as necessary. Doing so enables traceability to subsequent measurement and analysis activities, and helps to ensure that analyses will properly address identified information needs and objectives.
It is important that users of measurement and analysis results be involved in setting measurement objectives and deciding on plans of action. It may also be appropriate to involve those who provide the measurement data.
4. Provide feedback for refining and clarifying information needs and objectives as necessary.
Identified information needs and objectives can be refined and clarified as a result of setting measurement objectives. Initial descriptions of information needs may be ambiguous. Conflicts can arise between existing needs and objectives. Precise targets on an already existing measure may be unrealistic.
5. Maintain traceability of measurement objectives to identified information needs and objectives.
There should always be a good answer to the question, “Why are we measuring this?”
Of course, measurement objectives can also change to reflect evolving information needs and objectives.
SP 1.2 Specify Measures
Specify measures to address measurement objectives.
Measurement objectives are refined into precise, quantifiable measures.
Measurement of work can typically be traced to one or more measurement information categories. These categories include the following: service continuity, capacity, availability, service performance, and service quality.
Measures can be either base or derived. Data for base measures are obtained by direct measurement. Data for derived measures come from other data, typically by combining two or more base measures.
Derived measures typically are expressed as ratios, composite indices, or other aggregate summary measures. They are often more quantitatively reliable and meaningfully interpretable than the base measures used to generate them.
There are direct relationships among information needs, measurement objectives, measurement categories, base measures, and derived measures. This direct relationship is depicted for service work using some common examples in Table 8.1.
Table 8.1 Example Measurement Relationships
Example Work Products
1. Specifications of base and derived measures
Subpractices
1. Identify candidate measures based on documented measurement objectives.
Measurement objectives are refined into measures. Identified candidate measures are categorized and specified by name and unit of measure.
2. Maintain traceability of measures to measurement objectives.
Interdependencies among candidate measures are identified to enable later data validation and candidate analyses in support of measurement objectives.
3. Identify existing measures that already address measurement objectives.
Specifications for measures may already exist, perhaps established for other purposes earlier or elsewhere in the organization.
4. Specify operational definitions for measures.
Operational definitions are stated in precise and unambiguous terms. They address two important criteria:
• Communication: What has been measured, how was it measured, what are the units of measure, and what has been included or excluded?
• Repeatability: Can the measurement be repeated, given the same definition, to get the same results?
5. Prioritize, review, and update measures.
Proposed specifications of measures are reviewed for their appropriateness with potential end users and other relevant stakeholders. Priorities are set or changed, and specifications of measures are updated as necessary.
SP 1.3 Specify Data Collection and Storage Procedures
Specify how measurement data are obtained and stored.
Explicit specification of collection methods helps to ensure that the right data are collected properly. This specification can also help further clarify information needs and measurement objectives.
Proper attention to storage and retrieval procedures helps to ensure that data are available and accessible for future use.
Example Work Products
1. Data collection and storage procedures
2. Data collection tools
Subpractices
1. Identify existing sources of data that are generated from current work products, processes, or transactions.
Existing sources of data may have been identified when specifying the measures. Appropriate collection mechanisms may exist whether or not pertinent data have already been collected.
2. Identify measures for which data are needed but are not currently available.
3. Specify how to collect and store the data for each required measure.
Explicit specifications are made of what, how, where, and when data will be collected and stored to ensure its validity and to support later use for analysis and documentation purposes.
4. Create data collection mechanisms and process guidance.
Data collection and storage mechanisms are well integrated with other normal work processes. Data collection mechanisms can include manual or automated forms and templates. Clear, concise guidance on correct procedures is available to those who are responsible for doing the work. Training is provided as needed to clarify processes required for the collection of complete and accurate data and to minimize the burden on those who provide and record data.
5. Support automatic collection of data as appropriate and feasible.
6. Prioritize, review, and update data collection and storage procedures.
Proposed procedures are reviewed for their appropriateness and feasibility with those who are responsible for providing, collecting, and storing data. They also may have useful insights about how to improve existing processes or may be able to suggest other useful measures or analyses.
7. Update measures and measurement objectives as necessary.
SP 1.4 Specify Analysis Procedures
Specify how measurement data are analyzed and communicated.
Specifying analysis procedures in advance ensures that appropriate analyses will be conducted and reported to address documented measurement objectives (and thereby the information needs and objectives on which they are based). This approach also provides a check that necessary data will, in fact, be collected. Analysis procedures should account for the quality (e.g., age, reliability) of all data that enter into an analysis (whether from the work group, organization’s measurement repository, or other source). The quality of data should be considered to help select the appropriate analysis procedure and evaluate the results of the analysis.
Example Work Products
1. Analysis specifications and procedures
2. Data analysis tools
Subpractices
1. Specify and prioritize the analyses to be conducted and the reports to be prepared.
Early on, pay attention to the analyses to be conducted and to the manner in which results will be reported. These analyses and reports should meet the following criteria:
• The analyses explicitly address the documented measurement objectives.
• Presentation of results is clearly understandable by the audiences to whom the results are addressed.
Priorities may have to be set for available resources.
2. Select appropriate data analysis methods and tools.
Refer to the Select Measures and Analytic Techniques specific practice and Monitor the Performance of Selected Subprocesses specific practice in the Quantitative Work Management process area for more information about the appropriate use of statistical techniques and understanding variation.
3. Specify administrative procedures for analyzing data and communicating results.
4. Review and update the proposed content and format of specified analyses and reports.
All of the proposed content and format are subject to review and revision, including analytic methods and tools, administrative procedures, and priorities. Relevant stakeholders consulted should include end users, sponsors, data analysts, and data providers.
5. Update measures and measurement objectives as necessary.
Just as measurement needs drive data analysis, clarification of analysis criteria can affect measurement. Specifications for some measures may be refined further based on specifications established for data analysis procedures. Other measures may prove unnecessary or a need for additional measures may be recognized.
Specifying how measures will be analyzed and reported can also suggest the need for refining measurement objectives themselves.
6. Specify criteria for evaluating the utility of analysis results and for evaluating the conduct of measurement and analysis activities.
SG 2 Provide Measurement Results
Measurement results, which address identified information needs and objectives, are provided.
The primary reason for conducting measurement and analysis is to address identified information needs derived from work, organizational, and business objectives. Measurement results based on objective evidence can help to monitor progress and performance, fulfill obligations documented in a supplier agreement, make informed management and technical decisions, and enable corrective actions to be taken.
SP 2.1 Obtain Measurement Data
Obtain specified measurement data.
The data necessary for analysis are obtained and checked for completeness and integrity.
Example Work Products
1. Base and derived measurement data sets
2. Results of data integrity tests
Subpractices
1. Obtain data for base measures.
Data are collected as necessary for previously used and newly specified base measures. Existing data are gathered from work records or elsewhere in the organization.
2. Generate data for derived measures.
Values are newly calculated for all derived measures.
3. Perform data integrity checks as close to the source of data as possible.
All measurements are subject to error in specifying or recording data. It is always better to identify these errors and sources of missing data early in the measurement and analysis cycle.
Checks can include scans for missing data, out-of-bounds data values, and unusual patterns and correlation across measures. It is particularly important to do the following:
• Test and correct for inconsistency of classifications made by human judgment (i.e., to determine how frequently people make differing classification decisions based on the same information, otherwise known as “inter-coder reliability”).
• Empirically examine the relationships among measures that are used to calculate additional derived measures. Doing so can ensure that important distinctions are not overlooked and that derived measures convey their intended meanings (otherwise known as “criterion validity”).
SP 2.2 Analyze Measurement Data
Analyze and interpret measurement data.
Measurement data are analyzed as planned, additional analyses are conducted as necessary, results are reviewed with relevant stakeholders, and necessary revisions for future analyses are noted.
Example Work Products
1. Analysis results and draft reports
Subpractices
1. Conduct initial analyses, interpret results, and draw preliminary conclusions.
The results of data analyses are rarely self evident. Criteria for interpreting results and drawing conclusions should be stated explicitly.
2. Conduct additional measurement and analysis as necessary and prepare results for presentation.
Results of planned analyses can suggest (or require) additional, unanticipated analyses. In addition, these analyses can identify needs to refine existing measures, to calculate additional derived measures, or even to collect data for additional base measures to properly complete the planned analysis. Similarly, preparing initial results for presentation can identify the need for additional, unanticipated analyses.
3. Review initial results with relevant stakeholders.
It may be appropriate to review initial interpretations of results and the way in which these results are presented before disseminating and communicating them widely.
Reviewing the initial results before their release can prevent needless misunderstandings and lead to improvements in the data analysis and presentation.
Relevant stakeholders with whom reviews may be conducted include intended end users, sponsors, data analysts, and data providers.
4. Refine criteria for future analyses.
Lessons that can improve future efforts are often learned from conducting data analyses and preparing results. Similarly, ways to improve measurement specifications and data collection procedures can become apparent as can ideas for refining identified information needs and objectives.
SP 2.3 Store Data and Results
Manage and store measurement data, measurement specifications, and analysis results.
Storing measurement related information enables its timely and cost effective use as historical data and results. The information also is needed to provide sufficient context for interpretation of data, measurement criteria, and analysis results.
Stored information contains or refers to other information needed to understand and interpret the measures and to assess them for reasonableness and applicability (e.g., measurement specifications used on different work activities when comparing across work groups).
Typically, data sets for derived measures can be recalculated and need not be stored. However, it may be appropriate to store summaries based on derived measures (e.g., charts, tables of results, report text).
Interim analysis results need not be stored separately if they can be efficiently reconstructed.
Refer to the Configuration Management process area for more information about establishing a configuration management system.
Refer to the Establish the Organization’s Measurement Repository specific practice in the Organizational Process Definition process area for more information about establishing the organization’s measurement repository.
Example Work Products
1. Stored data inventory
Subpractices
1. Review data to ensure their completeness, integrity, accuracy, and currency.
2. Store data according to data storage procedures.
3. Make stored contents available for use only to appropriate groups and staff members.
4. Prevent stored information from being used inappropriately.
SP 2.4 Communicate Results
Communicate results of measurement and analysis activities to all relevant stakeholders.
The results of the measurement and analysis process are communicated to relevant stakeholders in a timely and usable fashion to support decision making and assist in taking corrective action.
Relevant stakeholders include intended end users, sponsors, data analysts, and data providers.
Example Work Products
1. Delivered reports and related analysis results
2. Contextual information or guidance to help interpret analysis results
Subpractices
1. Keep relevant stakeholders informed of measurement results in a timely manner.
To the extent possible and as part of the normal way they do business, users of measurement results are kept personally involved in setting objectives and deciding on plans of action for measurement and analysis. Users are regularly kept informed of progress and interim results.
Refer to the Work Monitoring and Control process area for more information about conducting progress reviews.
2. Assist relevant stakeholders in understanding results.
Results are communicated in a clear and concise manner appropriate to relevant stakeholders. Results are understandable, easily interpretable, and clearly tied to identified information needs and objectives.
The data analyzed are often not self evident to practitioners who are not measurement experts. The communication of results should be clear about the following:
• How and why base and derived measures were specified
• How data were obtained
• How to interpret results based on the data analysis methods used
• How results address information needs