Organizational Process Performance: A Process Management Process Area at Maturity Level 4
Purpose
The purpose of Organizational Process Performance (OPP) is to establish and maintain a quantitative understanding of the performance of selected processes in the organization’s set of standard processes in support of achieving quality and process performance objectives, and to provide process performance data, baselines, and models to quantitatively manage the organization’s projects.
Hint
OPP establishes assets for use by projects (QPM and CAR) and the organization (OPM). Consult these PAs to better understand which roles these assets play. This understanding, as well as your business objectives and project needs, will help you design appropriate assets as you implement OPP.
Introductory Notes
X-Ref
OPP and QPM are tightly coupled process areas. Refer to QPM extensively when considering how you should implement OPP. In particular, note the correspondence between OPP SP 1.1 and 1.2 and QPM SP 1.1, 1.3, and 1.4.
The Organizational Process Performance process area involves the following activities:
• Establishing organizational quantitative quality and process performance objectives based on business objectives (see the definition of “quality and process performance objectives” in the glossary)
• Selecting processes or subprocesses for process performance analyses
• Establishing definitions of the measures to be used in process performance analyses (see the definition of “process performance” in the glossary)
• Establishing process performance baselines and process performance models (see the definitions of “process performance baselines” and “process performance models” in the glossary)
X-Ref
The CMMI principle that specific practices are agnostic as to who performs them applies here as well. The OPP specific practices, including those establishing Process Performance Baselines (PPBs) and Process Performance Models (PPMs) (SP 1.4 and SP 1.5, respectively), may in some cases provide more benefit when implemented for an individual project.
The collection and analysis of the data and creation of the process performance baselines and models can be performed at different levels of the organization, including individual projects or groups of related projects as appropriate based on the needs of the projects and organization.
The common measures for the organization consist of process and product measures that can be used to characterize the actual performance of processes in the organization’s individual projects. By analyzing the resulting measurements, a distribution or range of results can be established that characterize the expected performance of the process when used on an individual project.
Tip
A subprocess measure’s “central tendency and spread,” normalized appropriately (e.g., for work product size), and determined for a set of similar projects, for multiple teams, or perhaps just for a single team, together with sufficient contextual data to interpret the individual measurements (to allow creation of data subsets from which these statistics are derived), may compose a PPB.
Measuring quality and process performance can involve combining existing measures into additional derived measures to provide more insight into overall efficiency and effectiveness at a project or organization level. The analysis at the organization level can be used to study productivity, improve efficiencies, and increase throughput across projects in the organization.
The expected process performance can be used in establishing the project’s quality and process performance objectives and can be used as a baseline against which actual project performance can be compared. This information is used to quantitatively manage the project. Each quantitatively managed project, in turn, provides actual performance results that become a part of organizational process assets that are made available to all projects.
The acquirer can use quality and process performance objectives to define performance and service level expectations for suppliers.
Hint
PPBs can be displayed or summarized in different ways, depending on the nature of the data and the contexts from which the data arose. These techniques might include control charts, box plots, or histograms.
Process performance models are used to represent past and current process performance and to predict future results of the process. For example, the latent defects in the delivered product can be predicted using measurements of work product attributes such as complexity and process attributes such as preparation time for peer reviews.
The same measures of latent defects, analyzed using a supplier’s past projects data, can be used to predict the quality of products delivered by that supplier. The acquirer can use supplier process performance models to predict the overall capability of the acquirer to deliver the product.
Tip
This paragraph describes in a simplified way how the organization’s PPBs (referred to here as “expected process performance”) support the activities described in QPM, which in turn provide data used to update these PPBs.
When the organization has sufficient measures, data, and analytical techniques for critical process, product, and service characteristics, it is able to do the following:
• Identify aspects of processes that could be improved across acquirer-supplier interfaces
• Determine whether processes are behaving consistently or have stable trends (i.e., are predictable)
• Identify processes in which performance is within natural bounds that are consistent across projects and could potentially be aggregated
• Identify processes that show unusual (e.g., sporadic, unpredictable) behavior
• Identify aspects of processes that can be improved in the organization’s set of standard processes
• Identify the implementation of a process that performs best
This process area interfaces with and supports the implementation of other high maturity process areas. The assets established and maintained as part of implementing this process area (e.g., the measures to be used to characterize subprocess behavior, process performance baselines, process performance models) are inputs to the quantitative project management, causal analysis and resolution, and organizational performance management processes in support of the analyses described there. Quantitative project management processes provide the quality and process performance data needed to maintain the assets described in this process area.
Related Process Areas
Refer to the Measurement and Analysis process area for more information about specifying measures, obtaining measurement data, and analyzing measurement data.
Tip
Mastering the practices in MA is a prerequisite to effectively implementing OPP and QPM. In particular, it is important to evaluate the effectiveness of your measurement system. Are subprocess measures sufficiently accurate and precise to be useful for your intended purposes? Techniques such as ANOVA Gauge Repeatability and Reproducibility (GR&R) and Fleiss’s kappa (see Wikipedia) may help answer these questions.
Refer to the Organizational Performance Management process area for more information about proactively managing the organization’s performance to meet its business objectives.
Refer to the Quantitative Project Management process area for more information about quantitatively managing the project to achieve the project’s established quality and process performance objectives.
Tip
Another suitable reference is “Measurement and Analysis Infrastructure Diagnostic: Version 1.0: Method Definition Document,” by Mark Kasunic. See www.sei.cmu.edu/library/abstracts/reports/10tr035.cfm.
Specific Practices by Goal
SG 1 Establish Performance Baselines and Models
Baselines and models, which characterize the expected process performance of the organization’s set of standard processes, are established and maintained.
Prior to establishing process performance baselines and models, it is necessary to determine the quality and process performance objectives for those processes (the Establish Quality and Process Performance Objectives specific practice), which processes are suitable to be measured (the Select Processes specific practice), and which measures are useful for determining process performance (the Establish Process Performance Measures specific practice).
The first three practices of this goal are interrelated and often need to be performed concurrently and iteratively to select quality and process performance objectives, processes, and measures. Often, the selection of one quality and process performance objective, process, or measure will constrain the selection of the others. For example, selecting a quality and process performance objective relating to defects delivered to the customer will almost certainly require selecting the verification processes and defect related measures.
The intent of this goal is to provide projects with the process performance baselines and models they need to perform quantitative project management. Many times these baselines and models are collected or created by the organization, but there are circumstances in which a project may need to create the baselines and models for themselves. These circumstances include projects that are not covered by the organization’s baselines and models. For these cases the project follows the practices in this goal to create its baselines and models.
SP 1.1 Establish Quality and Process Performance Objectives
Establish and maintain the organization’s quantitative objectives for quality and process performance, which are traceable to business objectives.
The organization’s quality and process performance objectives can be established for different levels in the organizational structure (e.g., business area, product line, function, project) as well as at different levels in the process hierarchy. When establishing quality and process performance objectives, consider the following:
• Traceability to the organization’s business objectives
• Past performance of the selected processes or subprocesses in context (e.g., on projects)
• Multiple attributes of process performance (e.g., product quality, productivity, cycle time, response time)
• Inherent variability or natural bounds of the selected processes or subprocesses
Tip
By ensuring traceability to business objectives, the organization’s quality and process performance objectives (QPPOs) become the mechanism for aligning the activities described in OPP and QPM with the organization’s business objectives.
The organization’s quality and process performance objectives provide focus and direction to the process performance analysis and quantitative project management activities. However, it should be noted that achieving quality and process performance objectives that are significantly different from current process capability requires use of techniques found in Causal Analysis and Resolution and Organizational Performance Management.
Tip
Ideally, the QPPOs should be attainable, but perhaps a “stretch” beyond the current PPBs. This comparison can help establish feasible objectives. CAR or OPM can also be invoked to improve process performance.
Example Work Products
1. Organization’s quality and process performance objectives
Subpractices
1. Review the organization’s business objectives related to quality and process performance.
2. Define the organization’s quantitative objectives for quality and process performance.
QPPOs can be established at multiple levels of an organization. One approach that aligns the whole organization toward achieving the organization’s QPPOs and links QPPOs across all levels is Hoshin Kanri or Hoshin Planning (see Wikipedia). Quality and process performance objectives can be established for process or subprocess measurements (e.g., effort, cycle time, defect removal effectiveness) as well as for product measurements (e.g., reliability, defect density) and service measurements (e.g., capacity, response times) as appropriate.
3. Define the priorities of the organization’s objectives for quality and process performance.
Tip
Commitment to the organization’s QPPOs means senior management periodically reviews how well projects are performing relative to them. Project management and senior technical staff members incorporate them into their projects and strive to achieve them.
4. Review, negotiate, and obtain commitment to the organization’s quality and process performance objectives and their priorities from relevant stakeholders.
5. Revise the organization’s quantitative objectives for quality and process performance as necessary.
Hint
Because not all processes contribute equally to a business objective, yet all such analyses consume time, effort, and money, you should start with a small selection. As you learn more, you can modify and expand the selection accordingly.
SP 1.2 Select Processes
Select processes or subprocesses in the organization’s set of standard processes to be included in the organization’s process performance analyses and maintain traceability to business objectives.
Refer to the Organizational Process Definition process area for more information about establishing organizational process assets.
Hint
Select process elements, not composite processes (processes composed of multiple elements). Multiple sources of variation are often hidden within a composite process and may not be easily analyzed. What might appear to be a stable process may, in fact, be a collection of unstable process elements. Selection of such a process would leave you unable to correctly predict future behavior or accurately identify opportunities for improvement.
The organization’s set of standard processes consists of a set of standard processes that, in turn, are composed of subprocesses.
Typically, it is not possible, useful, or economically justifiable to apply statistical management techniques to all processes or subprocesses of the organization’s set of standard processes. Selection of processes or subprocesses is based on the quality and process performance objectives of the organization, which are derived from business objectives as described in the previous specific practice.
Example Work Products
1. List of processes or subprocesses identified for process performance analyses with rationale for their selection including traceability to business objectives
1. Establish the criteria to use when selecting subprocesses.
2. Select the subprocesses and document the rationale for their selection.
Refer to the Decision Analysis and Resolution process area for more information about analyzing possible decisions using a formal evaluation process that evaluates identified alternatives against established criteria.
3. Establish and maintain traceability between the selected subprocesses, quality and process performance objectives, and business objectives.
4. Revise the selection as necessary.
• It may be necessary to revise the selection in the following situations:
• The predictions made by process performance models result in too much variation to make them useful.
• The objectives for quality and process performance change.
• The organization’s set of standard processes change.
• The underlying quality and process performance changes.
SP 1.3 Establish Process Performance Measures
Establish and maintain definitions of measures to be included in the organization’s process performance analyses.
Refer to the Measurement and Analysis process area for more information about specifying measures.
Example Work Products
1. Definitions of selected measures of process performance with rationale for their selection including traceability to selected processes or subprocesses
Subpractices
1. Select measures that reflect appropriate attributes of the selected processes or subprocesses to provide insight into the organization’s quality and process performance.
Tip
A commonly heard objection is that it is not possible to measure certain attributes. In his book How to Measure Anything: Finding the Value of “Intangibles” in Business, Doug Hubbard provides strategies for how to proceed in these circumstances.
It is often helpful to define multiple measures for a process or subprocess to understand the impact of changes to the process and avoid sub-optimization. Also, it is often helpful to establish measures for both product and process attributes for the selected process and subprocess, as well as its inputs, outputs, and resources (including people and the skill they bring) consumed.
The Goal Question Metric paradigm is an approach that can be used to select measures that provide insight into the organization’s quality and process performance objectives. It is often useful to analyze how these quality and process performance objectives can be achieved based on an understanding of process performance provided by the selected measures.
X-Ref
The Goal Question Metric (GQM; www.cs.umd.edu/~mvz/handouts/gqm.pdf) is a well-known approach to deriving measures that provide insight into issues of interest. The SEI’s variant of GQM is called the Goal Question Indicator Metric (GQIM; www.sei.cmu.edu/training/p06.cfm).
For quality and process performance objectives addressed through acquisition, select process, product, and service level measures that provide insight into the process performance of suppliers and into the quality of their deliverables.
2. Establish operational definitions for the selected measures.
Refer to the Measurement and Analysis process area for more information about specifying measures.
3. Incorporate selected measures into the organization’s set of common measures.
Measures expected to be collected and reported by suppliers are incorporated into standard supplier agreement templates and standard service level agreements as appropriate.
Refer to the Organizational Process Definition process area for more information about establishing organizational process assets.
Hint
To begin systematic collection of these measures from new projects, incorporate them into the organization’s set of common measures (OPD SP 1.4).
4. Revise the set of measures as necessary.
Measures are periodically evaluated for their continued usefulness and ability to indicate process effectiveness.
SP 1.4 Analyze Process Performance and Establish Process Performance Baselines
Analyze the performance of the selected processes, and establish and maintain the process performance baselines.
The selected measures are analyzed to characterize the performance of the selected processes or subprocesses achieved on projects. This characterization is used to establish and maintain process performance baselines. (See the definition of “process performance baseline” in the glossary.) These baselines are used to determine the expected results of the process or subprocess when used on a project under a given set of circumstances.
Process performance baselines are compared to the organization’s quality and process performance objectives to determine if the quality and process performance objectives are being achieved.
Tip
QPPOs should motivate superior performance. QPPOs that set the bar too high may demoralize more than they motivate. What do the performance data say about how well a project can do relative to the QPPOs? There is a need for balance between “desires” and “reality.”
The process performance baselines are a measurement of performance for the organization’s set of standard processes at various levels of detail. The processes that the process performance baselines can address include the following:
• Sequence of connected processes
• Processes that cover the entire life of the project
• Processes for developing individual work products
There can be several process performance baselines to characterize performance for subgroups of the organization.
Tailoring the organization’s set of standard processes can significantly affect the comparability of data for inclusion in process performance baselines. Effects of tailoring should be considered in establishing baselines. Depending on the tailoring allowed, separate performance baselines may exist for each type of tailoring.
Refer to the Quantitative Project Management process area for more information about quantitatively managing the project to achieve the project’s established quality and process performance objectives.
Example Work Products
1. Analysis of process performance data
2. Baseline data on the organization’s process performance
1. Collect the selected measurements for the selected processes and subprocesses.
The process or subprocess in use when the measurement was taken is recorded to enable its use later.
Refer to the Measurement and Analysis process area for more information about specifying measurement data collection and storage procedures.
Hint
Record sufficient contextual information with a measurement to enable identification of when that information was generated, by whom, and in which PPB it should be included (or which PPB it should be used to regenerate, for a different class of process instances).
2. Analyze the collected measures to establish a distribution or range of results that characterize the expected performance of selected processes or subprocesses when used on a project.
This analysis should include the stability of the related process or subprocess, and the impacts of associated factors and context. Related factors include inputs to the process and other attributes that can affect the results obtained. The context includes the business context (e.g., domain) and significant tailoring of the organization’s set of standard processes.
Tip
Unless the process is stable, the data from the process may actually comprise a mixture of measurements taken from different processes. PPBs developed from such data are limited in their usefulness to projects (e.g., the estimated natural bounds are likely to be far apart).
The measurements from stable subprocesses in projects should be used when possible; other data may not be reliable.
3. Establish and maintain the process performance baselines from collected measurements and analyses.
Refer to the Measurement and Analysis process area for more information about aligning measurement and analysis activities and providing measurement results.
Hint
Investigate subgrouping when data from multiple projects (and teams) will be incorporated into the same PPB. Even if the process is stable within individual projects, its performance across projects may vary so much that the resultant single PPB will be too “wide” to be useful.
Process performance baselines are derived by analyzing collected measures to establish a distribution or range of results that characterize the expected performance for selected processes or subprocesses when used on a project in the organization.
4. Review and get agreement with relevant stakeholders about the process performance baselines.
5. Make the process performance information available across the organization in the measurement repository.
The organization’s process performance baselines are used by projects to estimate the natural bounds for process performance.
Hint
Even when organizational PPBs are established, individual projects may still benefit from establishing their own individual PPBs when they have accumulated sufficient data.
6. Compare the process performance baselines to associated quality and process performance objectives to determine if those quality and process performance objectives are being achieved.
These comparisons should use statistical techniques beyond a simple comparison of the mean to gauge the extent of quality and process performance objective achievement. If the quality and process performance objectives are not being achieved, corrective actions should be considered.
Refer to the Causal Analysis and Resolution process area for more information about determining causes of selected outcomes.
Refer to the Organizational Process Focus process area for more information about planning and implementing process actions.
Refer to the Organizational Performance Management process area for more information about analyzing process performance data and identifying potential areas for improvement.
7. Revise the process performance baselines as necessary.
SP 1.5 Establish Process Performance Models
Establish and maintain process performance models for the organization’s set of standard processes.
Tip
Part of mastering any discipline is developing “nuance” for factors that matter in how a situation will unfold, yet realizing every situation is somewhat different. Thus, while PPMs can never be perfectly accurate, they offer a systematic approach to using the data available in a situation and similar situations toward making, scientifically, the best possible judgment or prediction.
High maturity organizations generally establish and maintain a set of process performance models at various levels of detail that cover a range of activities that are common across the organization and address the organization’s quality and process performance objectives. (See the definition of “process performance model” in the glossary.) Under some circumstances, projects may need to create their own process performance models.
Process performance models are used to estimate or predict the value of a process performance measure from the values of other process, product, and service measurements. These process performance models typically use process and product measurements collected throughout the life of the project to estimate progress toward achieving quality and process performance objectives that cannot be measured until later in the project’s life.
Hint
Establish PPMs that provide insight at different points in a project that will help the project assess progress toward achieving its QPPOs (QPM SP 2.2).
Process performance models are used to estimate or predict when to fund, hold, cancel, migrate, re-engineer, or retire a project. Process performance models allow the acquirer to synchronize processes with customer needs. The organization’s process performance baselines provide quantitative data on those aspects of the projects and organization that can approximate the throughput potential of its processes. Focusing on these critical constraints, process performance models allow the acquirer to predict how to best maximize the flow of work through projects and the organization.
X-Ref
Experiences from using PPMs are described in “Performance Effects of Measurement and Analysis: Perspectives from CMMI High Maturity Organizations and Appraisers” found at www.sei.cmu.edu/library/abstracts/reports/10tr022.cfm.
Process performance models are used as follows:
• The organization uses them for estimating, analyzing, and predicting the process performance associated with processes in and changes to the organization’s set of standard processes.
• The organization uses them to assess the (potential) return on investment for process improvement activities.
• Projects use them for estimating, analyzing, and predicting the process performance of their defined processes.
• Projects use them for selecting processes or subprocesses for use.
• Projects use them for estimating progress toward achieving the project’s quality and process performance objectives.
Process performance models are also used to set quality and process performance objectives for suppliers and to provide data that can help suppliers achieve these objectives.
Tip
In general, data-driven approaches to judgment and prediction should, over the long term, perform better than human judgment alone. Perhaps the performance of both should be analyzed and contrasted, and then the team should determine how to best use both. Be aware, however, that in situations unlike those experienced before, neither approach is likely to perform well.
These measures and models are defined to provide insight into and to provide the ability to predict critical process and product characteristics that are relevant to the organization’s quality and process performance objectives.
Results of the acquirer’s process performance models are shared with suppliers to help ensure the synchronized delivery of products and services.
Tip
Many books on the limitations of human judgment are available, including an update to a classic text, Rational Choice in an Uncertain World: The Psychology of Judgment and Decision Making by Hastie and Dawes. A more recent book is The Invisible Gorilla: And Other Ways Our Intuitions Deceive Us by Chabris and Simons.
Refer to the Quantitative Project Management process area for more information about quantitatively managing the project to achieve the project’s established quality and process performance objectives.
Example Work Products
1. Process performance models
1. Supplier process performance models
Hint
Subject-matter experts from different relevant disciplines and functions can help identify process and product characteristics important to prediction. A variety of analytic techniques such as ANOVA and sensitivity analysis can also help screen factors for potential inclusion in PPMs. PPBs provide a primary source for the information needed for this analysis.
Subpractices
1. Establish process performance models based on the organization’s set of standard processes and process performance baselines.
2. Calibrate process performance models based on the past results and current needs.
3. Review process performance models and get agreement with relevant stakeholders.
4. Support the projects’ use of process performance models.
5. Revise process performance models as necessary.
Hint
Meet with relevant stakeholders to discuss PPMs (e.g., their usefulness and limitations) and the support required to make effective use of such models on projects. Acquirers can often benefit by teaming with mature suppliers, as the PPMs they use may help predict acquisition project outcomes.
Tip
To use PPMs effectively, project staff members and management may need significant support. There may be an initial tendency to dismiss or misuse either PPMs or their results.
X-Ref
Example PPMs are described in “Approaches to Process Performance Modeling: A Summary from the SEI Series of Workshops on CMMI High Maturity Measurement and Analysis” found at www.sei.cmu.edu/library/abstracts/reports/09tr021.cfm.