Not everything that can be counted counts, and not everything that counts can be counted. | ||
| --Albert Einstein (1879–1955) | ||
Good, fast, cheap: Pick any two. Sign in print shop If you don’t like something, change it. If you can’t change it, change your attitude. Don’t complain. | ||
| --Maya Angelou (1928– ) | ||
A comprehensive discussion of process appraisals, their history, the way in which they are addressed in CMMI, rules for ratings, and so forth would require more than the “distilled” presentation offered by this book. The objective of the current chapter is more modest: to briefly introduce the CMMI appraisal process, explain basic appraisal concepts, and discuss some of the issues that an organization may confront when deciding to conduct an appraisal. The appraisal method that is included in the CMMI Product Suite is called the Standard CMMI Appraisal Method for Process Improvement (SCAMPISM).[1] To help understand why the SCAMPI approach has the features that it does, we begin by reviewing the set of requirements that apply to any appraisal method used with the CMMI model.[2]
One key part of the CMMI Product Suite is the Appraisal Requirements for CMMI, Version 1.2 (ARC, V1.2). This technical report comprises 40 “requirements” that provide a mixed set of appraisal method requirements and design constraints. To be “ARC-compliant,” an appraisal method must satisfy all (or an appropriate subset) of the 40 ARC requirements. In the jargon of CMMI, an ARC-compliant appraisal method that meets all 40 requirements is called a Class A method. The ARC also defines Class B and Class C method classes, which address only a subset of the 40 ARC requirements. Table 9-1, which was taken from the ARC technical report, outlines some of the salient differences between the method classes.[3]
Table 9-1. Requirements of CMMI Appraisal Method Classes
Requirements | Class A | Class B | Class C |
|---|---|---|---|
Types of objective evidence gathered | Documents and interviews | Documents and interviews | Documents or interviews |
Ratings generated | Goal ratings required | Not allowed | Not allowed |
Organizational unit coverage | Required | Not required | Not required |
Minimum team size | 4 | 2 | 1 |
Appraisal team leader requirements | Lead appraiser | Person trained and experienced | Person trained and experienced |
SCAMPI A is the appraisal method included in the CMMI Product Suite that is ARC-compliant as a Class A method. Just from Table 9-1 alone you already know several features of the SCMAPI A method: Documents and interviews provide evidence to the appraisal team during an appraisal; ratings are generated for the goals in the model, on whether or not they are satisfied; the scope of the appraisal must adequately represent the organization being appraised; an appraisal team of at least four people must be used; and the effort must be led by an authorized lead appraiser.
Taking a broader look at the ARC, we see that it comprises seven categories of requirements:
Responsibilities. Lists the responsibilities of the appraisal sponsor and the leader of the appraisal team.
Appraisal Method Documentation. Provides items of guidance that should be included in the documentation of a method. As examples, guidance should be provided for identifying the appraisal’s purpose, model scope, organizational scope, team member and team leader qualifications, team size, team member and team leader roles, resource estimation, logistics, data collection, findings, data protection, non-attribution, and appraisal records.
Planning and Preparation for the Appraisal. Identifies a minimum set of appraisal preparation activities and the items needed in an appraisal plan.
Appraisal Data Collection. Requires the use of two types of objective evidence (conducting interviews and reviewing documentation) for Class A and Class B appraisals, and requires just one of the two types of objective evidence for Class C appraisals.[4]
Data Consolidation and Validation. Specifies constraints on the way in which an appraisal method provides for the consolidation and validation of the data collected. The topics required include the following:
Team consensus leading to findings and ratings
A method for consolidating the data into accurate findings
A method for validating findings
A set of criteria for objective evidence to be considered “corroborated” (which entails the use of two different sources and having at least one source reflect work actually being done)[5]
Sufficiency of data to cover the scope of the appraisal (which entails corroborated objective evidence for each specific or generic practice, such that the extent of the implementation is understood and the data is representative of the organizational unit and the relevant life-cycle phases)
A mechanism to determine strengths and weaknesses relative to the CMMI model being used
Feedback of preliminary appraisal findings to the appraisal participants
Rating. Provides more constraints on what is needed for a capability-level or maturity-level rating. The requirements include the pre-requisite of corroborated objective evidence for each practice prior to the rating of goals, a judgment that goals are “satisfied” only when the associated practices are implemented and the aggregate of any weaknesses does not have a significant negative impact on goal achievement, and the basis for a capability-level or maturity-level rating to be the satisfaction of all relevant goals.
Reporting Results. Requires documentation of the findings and ratings for the appraisal sponsor, and requires a report made to the CMMI Steward (at Carnegie Mellon University).
In three of these seven areas (appraisal data collection, data consolidation and validation, and rating), some of the ARC “requirements” are what might be called “appraisal method design constraints,” rather than strict requirements. Performance requirements in this area would focus on issues such as accuracy, repeatability, stability, usefulness, and efficiency. The ARC authors included design constraints for one primary reason: to try to ensure the accuracy and repeatability of the appraisal results and ratings. This concern is a carry-over from the Software CMM and the belief that you have to tightly control how appraisals are conducted if you want to get a single maturity rating number—one that could serve as a benchmark and the basis for comparisons across different organizations.[6]
Perhaps we should deepen our understanding of which performance requirements are appropriate for any appraisal method and how to measure against those requirements the results from the application of a method. Then perhaps the ARC might be recast to allow for greater flexibility in developing methods that make different trade-offs between the various quality factors and efficiency (cost).[7]
As noted earlier, SCAMPI A (version 1.2) is a Class A method that meets all of the ARC requirements. It was designed to produce benchmark-quality ratings, with multiple uses in mind—namely, internal process improvement and external capability determinations (supplier selection and process monitoring). Its creators wanted to address not only ARC compliance, but also the essential characteristics of accuracy, repeatability, cost- and resource-effectiveness, and meaningfulness of the appraisal results. A central goal of any Class A appraisal is to determine the extent to which the model is implemented.
Of course, if an organization is focused primarily on discovering opportunities for improvement, and so is less interested in a rating, a less-costly Class B or Class C appraisal may be the prudent path. Often such less-costly appraisals are excellent preparation for embarking on the path to a SCAMPI A appraisal.
SCAMPI A can provide a comprehensive look at the process strengths and weaknesses in an organization, which typically leads to planning for process improvement. The ratings that are produced during a SCAMPI A appraisal should reflect the extent to which practices (which support the goals in the CMMI models) are planned and implemented in the organization. To make this determination, the appraisal team collects objective evidence, both at the organization level and within projects, seeking to identify compliance with the CMMI model as well as gaps in compliance.
When planning for an appraisal, the projects (and support groups) that the appraisal team is to review must be identified. The SCAMPI A method provides sampling rules for project (and support group) selection that are intended to lead to a representational look at the organization during the appraisal. For example, both “focus” projects and “non-focus” projects must be identified: Focus projects provide enough evidence to rate every process area in the model, whereas non-focus projects need only provide applicable evidence for a small group of process areas.[8]
During a SCAMPI A appraisal, the organization is encouraged to provide to the appraisal team evidence of its implemented processes, with clear mapping to the goals and practices of the model. Having this information permits the appraisal team to operate in a “verification” mode rather than one of “discovery,” making the team’s appraisal activity much more efficient and less prone to error.
The SCAMPI A method requires the use of three types of practice implementation indicators. These “footprints” of an activity or practice provide the basis for verification by the appraisal team:
Direct artifacts—tangible outputs that result from performing a practice (e.g., the plan that results from the practice of establishing a plan)
Indirect artifacts—the consequence of performing a practice, but not its purpose (e.g., the minutes from a planning meeting)
Affirmations—oral or written statements confirming that a practice has been implemented (e.g., an interview with the persons who created or maintain the plan)
A combination of these three indicator types is required during a SCAMPI A appraisal: direct artifacts for each practice in the model, and either indirect artifacts or affirmations for each practice.
A consensus across the entire appraisal team is needed whenever a SCAMPI A appraisal team reaches conclusions about whether practices are implemented across an organizational unit, whether the model goals have been satisfied, and which capability level or maturity level rating is appropriate. Work that leads up to such conclusions may be assigned to smaller subteams for review. Individual strengths and weaknesses that a SCAMPI A appraisal team has identified are aggregated into preliminary findings, which are then presented to the organization for verification, prior to drafting the final findings (and ratings).
Because the CMMI-DEV model is integrated across all development disciplines (systems, hardware, and software), an organization that wishes to obtain ratings (capability or maturity levels) has several options to consider. Specifically, does it want a single rating across the disciplines, or does it want individual ratings for each discipline? While the former permits a stronger discipline to achieve higher ratings than might be warranted through a collective look, a single rating (especially a higher one) may be desired as it would suggest the existence of a well-integrated process structure across the disciplines.
A SCAMPI A appraisal is divided into three phases: (1) plan and prepare for the appraisal; (2) conduct the appraisal; and (3) report the results. Each of these phases includes multiple steps. Phase 1 involves analyzing the organization’s business needs and objectives, developing and documenting an appraisal plan, selecting and preparing the appraisal team, obtaining and analyzing the initial practice implementation data, conducting a readiness review, and preparing for the collection on-site of additional objective evidence. Phase 2 involves preparing those who will participate in the on-site appraisal, collecting and examining information obtained both from interviews and from documents,[9] creating records that document practice implementation as well as strengths and weaknesses, verifying the detailed information collected (i.e., looking for gaps and comparing to the CMMI model the practices as implemented in the organization[10]), validating the preliminary findings based on feedback from the organization, and generating the appraisal results (i.e., rating whether the goals in the model are satisfied and determining which capability or maturity levels would be appropriate). Phase 3 involves reporting the appraisal results to the organization and appraisal sponsor, and appropriately packaging and archiving the appraisal assets, including appraisal feedback to the CMMI Steward at Carnegie Mellon.[11]
Any ARC Class A appraisal must be directed by a lead appraiser who has received recognition from an authorizing body to perform that role for the particular appraisal method.[12] In the case of the SCAMPI A method, the lead appraiser must be trained and authorized by the CMMI Steward. To be a lead appraiser, an individual must have relevant discipline experience, knowledge of the CMMI models, and a good understanding of appraisal techniques.[13] The appraisal team leader has the following responsibilities: ensure that the appraisal method is followed; confirm the sponsor’s commitment to the appraisal; ensure that the participants are adequately informed about the appraisal plan; ensure that appraisal team members have appropriate experience, knowledge, and skills with regard both to the CMMI model and the appraisal method; verify that all appraisal method requirements have been met; and confirm delivery of the appraisal results to the sponsor.
A significant investment of resources is needed to conduct a cross-disciplinary SCAMPI A appraisal that covers all (or most) of the CMMI model. Data on actual resources used is not always easy to obtain. One set of real data shows that a ML 5 appraisal covering the disciplines of systems, hardware, and software involved about 7000 hours of effort. This number includes time spent by the lead appraiser, the appraisal team, the sponsor, the appraisal coordinator, and the appraisal participants. Of course, the number of hours would vary based on the model scope, the number of disciplines covered, the number of projects interviewed, the size of the appraisal team, the extent to which objective evidence was compiled in advance of the appraisal, and so on.
For organizations that seek to conform to ISO/IEC 15504 (the international standard on process appraisals), SCAMPI A is designed for conformance with that standard.
In summary, here are some key principles to keep in mind when planning any SCAMPI A appraisal:
Get strong sponsorship from senior management.
Focus on the business goals for the organization.
Use the SCAMPI A documented appraisal method.
Use an appropriate CMMI process reference model.
Assure confidentiality for interviewees.
Select and engage a lead appraiser early in the process.
Take care when selecting appraisal team members, including taking the following considerations into account: balancing experience in CMMI with the desire to train and expose to CMMI; assuring discipline coverage; matching the organizational scope coverage; and balancing “insiders,” who understand the current process well and where to look for evidence, with “outsiders,” who bring a fresh set of eyes to the analysis.
Run an appraisal effort the same way you would run a project.
Use experienced staff to compile the process implementation indicators.
Present a process briefing to the appraisal team to (among other things) provide affirmations across the generic practices.
For high-maturity appraisals, provide to the appraisal team a matrix that maps high-maturity activities both to process areas from the model and to disciplines from the organization.
Whenever possible, present to the appraisal team process improvement initiatives that are cross-disciplinary in nature.
Just as a lean engineering value-stream map can facilitate the identification of waste and highlight opportunities to make gains in process efficiency by reducing that waste, so a CMMI appraisal can provide insight into needed change. The findings from an appraisal might include shortcomings such as these:
A standard process for something is on the books, but the understanding and use of that process are currently spotty.
Some programs make important decisions but often fail to capture the assumptions and justifications for those decisions.
Cost estimates are based on information that in some cases is outdated.
Efforts in statistical management and control produce potentially useful data, but follow-through is lacking.
The transfer of information between systems and software is not always well handled.
Engineering design changes are being used to make architectural changes that have major implications for both cost and schedule.
Information on employee training and skills is not kept current, and programs are finding it difficult to obtain the skills that they need.
Some employees believe that making risks known would prove detrimental to their careers.
The quality group lacks adequate personnel to perform evaluations across all the programs that need them.
While process improvement efforts have been made in the past, there is little effort to quantify or evaluate the impact of those changes.
When an appraisal is conducted well, each finding can present an opportunity to delve into the root causes of a problem and to make improvements in how the business is conducted. For example, after an appraisal an organization might:
Conduct a Six Sigma project focusing on how to make better use of statistical control techniques.
Perform a value-stream mapping exercise focused on how one discipline hands off information to another in an effort to improve that interface.
Develop a training module to encourage the more widespread use of some successful practice.
Given that the maturity level produced by a CMMI SCAMPI A appraisal can attract large amounts of attention both inside and outside the organization, upper management has a special obligation to keep the focus on improving the business. While getting a maturity level number can be beneficial and help in new business-development efforts, management needs to champion the improvement aspect of a CMMI appraisal as well. After all, the resources expended in conducting an appraisal are such that it would be short-sighted to only care about the number produced by that effort.
CMMI appraisals are an important tool in the continuous improvement toolkit, sometimes highlighting otherwise unnoticed weaknesses and vulnerabilities. A subsequent evaluation of those weaknesses could generate some pointed questions: Can the organization reach its objective of being an employer of choice if current employees are afraid to make risks known? Can financial performance remain good if information used for estimation purposes is outdated? Will customer satisfaction remain high if quality evaluations are skipped?
When a SCAMPI lead appraiser is selected and the team is formed, management needs to make sure everyone knows that it is not just the final number that matters. Whatever maturity-level target may exist, insights into needed improvements to enhance the business are at least as important as a number, if not more so.
[1] SCAMPI is a service mark of Carnegie Mellon University.
[2] Of course, in English the word “scampi” denotes large shrimp, or by extension an Italian dish of shrimp served over pasta. While the CMMI appraisal method called SCAMPI has nothing to do with shrimp (certainly not the fishy smell), one may feel the irresistible and perhaps unprofessional urge to develop colorful characterizations of appraisal types. For example, what is shrimp SCAMPI? Answer: an appraisal focused on just one process area. What is chicken SCAMPI? Answer: an appraisal where the lead appraiser comes from inside the organization being appraised. What is pork SCAMPI? Answer: an appraisal conducted by the government. What is squash SCAMPI? Answer: an appraisal conducted by upper management. What is prune SCAMPI? Answer: you tell us! We leave the continuation of this list as an exercise for the reader.
[3] CMU/SEI-2006-TR-011, Chapter 3, page 5, Table 1.
[4] Version 1.1 of the ARC also required “administering instruments” such as questionnaires or surveys. That requirement has been dropped in ARC version 1.2.
[5] Version 1.1 of the ARC also required that there be data from at least two data-gathering sessions. That requirement has been dropped in ARC version 1.2.
[6] After years of experience in this area, your authors continue to be highly skeptical of the validity of using the CMMI model to make maturity-level comparisons across organizations that inherently have many differences. Benchmarking within a single organization and monitoring change over time make sense. Beyond that activity, maturity levels have little value, we believe.
[7] This might result also in a shift in the way in which “lead appraisers” compete for business. They could utilize different appraisal methods that offer to an organization increased improvement benefits with decreased expense.
[8] Rules in the SCAMPI A method specify how many focus and non-focus projects are required, based on variables such as the number of projects in the organization and the stages of those projects in the development life cycle.
[9] A document is defined as “a collection of data, regardless of the medium on which it is recorded, that generally has permanence and can be read by humans or machines.”
[10] The team’s characterization regarding how each practice from the model is implemented uses these categories: fully implemented (FI), largely implemented (LI), partially implemented (PI), not implemented (NI), and not yet (NY). The NY characterization is appropriate when the practice in the model applies to a stage in the development life cycle that has not yet been reached.
[11] With version 1.2 of the CMMI Product Suite, there is a three-year (maximum) period of validity for SCAMPI Class A appraisals; after that period of time, the appraisal results “expire.”
[12] Starting with version 1.2 of the CMMI Product Suite, any SCAMPI A appraisal whose results will become part of the public record (such as being announced in a press release) or will be used in a proposal response to U.S. Department of Defense requirements must be headed by a lead appraiser from an external, third-party organization. In other words, an internal lead appraiser is not permitted. Also, before any public-record announcement is issued, the appraisal must first be reviewed and “accepted” (not “confirmed” or “certified”) by the CMMI Steward (the SEI at Carnegie Mellon University). Acceptance by the CMMI Steward means that the SCAMPI A method definition was followed and that the Appraisal Disclosure Statement (ADS) meets certain criteria. The goal for turnaround by the SEI Steward is 30 calendar days to provide a response that the appraisal results are either accepted, not accepted, or rejected.
[13] Recently, there has been an effort by the CMMI Team to better define what the higher maturity levels (ML 4 and ML 5) really mean. To promote greater consistency, there is a new requirement that any lead appraiser conducting a “high maturity” appraisal must have supplemental training and certification related to high-maturity practices.