Chapter 10
Managing an Appraisal Life Cycle

When you start working with almost any model, the topic of appraisal—or assessment, or evaluation, or gap analysis; there are many similar terms—comes up. That’s because any model that can be used for a guide to improve practices can, just about automatically, be used to evaluate practice conformance. The attractiveness of this is that you can use the same (usually public) model for both guiding your improvement and understanding achievements in relation to it.

In this chapter, we’ll talk in general about decisions related to appraisal and a small amount on CMMI-specific appraisal. You will not get an in-depth understanding of how to prepare for and conduct an appraisal from this section. If you want or need that—at least for SCAMPI, the appraisal method that accompanies CMMI—we recommend the book SCAMPI Distilled, by Dennis Ahern et al.¹ It will give you a good understanding of the mechanics of these types of appraisals and how to prepare for them. Another good source on appraisal philosophy is Tim Kasse’s Action-Focused Assessment book.²

1. Ahern, Dennis, et al. CMMI SCAMPI Distilled: Appraisals for Process Improvement. (Boston: Addison-Wesley, 2004).

2. Kasse, Tim. Action-Focused Assessment. (Boston: Artech House, 2001).

10.1 To Appraise or not to Appraise: Is that Really the Question?

Some people think that you can have a successful improvement effort without ever doing an appraisal. If what they mean is that you never do a formal, externally validated appraisal, we can agree with that statement, as long as you don’t need to publicize some achievement against your model of choice. As soon as publicity becomes a requirement, you really need to do some kind of external, validated appraisal before you publicize your achievements.

Aside from external appraisals such as SCAMPIs, there are lots of appraisal activities beneficial to an improvement effort that, in our minds, should not be skipped. For example, the Process Change Management method that is taught in the SEI’s Mastering Process Improvement course emphasizes an “Organizational Scan.” This is a type of appraisal that explicitly seeks projects with best practices to leverage in the improvement effort. So, from our viewpoint, “to appraise or not to appraise” is not the question; the question is, “when and how should you appraise?” Appraisals are one of the best ways to get the information you need to understand whether you have achieved your improvement goals.

The flip side of the organization that “never” appraises is the organization that never stops appraising. This type of organization sees the entire improvement effort as one big “getting ready for the next appraisal” activity. In this type of organization, continuous appraisal is seen as equivalent to continuous improvement. We don’t agree with that perspective either. The reason we consider this approach problematic is that focusing everyone’s attention on appraisal and conformance to the practices is likely to cause what we term “foolish compliance”—that is, compliance without thinking about the contribution of the practices to meeting your business goals. Any time that link is lost, you stand the risk of your improvement effort’s becoming a model-compliance effort, and our experience, as well as what’s seen in the improvement literature, argues that a model-compliance effort does not result in long-term (or even short term) improvement in the business.

10.2 Different Appraisal Philosophies

So what are your choices in terms of using appraisals effectively? There are several philosophies related to when and how you should appraise. Table 10-1 summarizes our perspective on the ones we’ve had experience with. Note that this is a general summary, more or less neutral as to technique or model. The promoters of individual models usually have more specific guidance on their Web sites as to how and when to appraise, and how to select qualified appraisers for their model. Go to www.sei.cmu.edu/cmmi/ appraisals for the home page for SCAMPI (Standard CMMI Appraisal Method for Process Improvement), the SEI’s appraisal method for CMMI. There, you can find links to authorized lead appraisers, and look at appraisal results posted for companies that have undergone a SCAMPI Class A appraisal and have permitted publication of their results.

Table 10-1: Appraisal Approaches

10.3 Managing the Resources Needed to Plan and
Conduct Appraisal Activities

You will need to manage several classes of resources when you decide to plan and conduct an appraisal. How much of what you will need depends on the type of appraisal you are planning to conduct. The resource classes are

•   External labor. Consultants and/or appraisal team staff to help you prepare for and conduct the appraisal planning and execution.

•   Internal labor. Appraisal team members, process improvement team members, interview candidates, and logistics support staff to help prepare for and conduct the appraisal planning and execution.

•   Appraisal support materials. Depending on the type of appraisal, special forms to record evidence (for CMMI, these are called PIIDS, or Process Implementation Indicator Documents), electronic documents that point appraisal team members to process assets, appraisal recording tools (both commercial and shareware tools are available), measurement reports, and so on.

•   Appraisal team support materials. Flip charts, markers, sticky notes, meeting rooms, laptops, catering, and other materials needed to keep the appraisal running smoothly.

Rather than duplicate the very useful information in appraisal method descriptions and books like SCAMPI Distilled, in this section we’ll point out a couple of the ways that organizations often cause themselves more pain in conducting appraisals than is needed.

10.3.1 Preparing Evidence for Appraisals

One of the potentially time-consuming and seemingly wasteful activities that is necessary for many types of appraisals, particularly those in which some kind of compliance rating will be given, is preparing evidence. Evidence is usually collected in the form of artifacts that verify certain types of activities have taken place, or artifacts that document some aspect of guidance for a process. And to minimize the time for the appraisal team to process that evidence, some sort of mapping between the evidence and the reference model you’re appraising against (for example, CMMI) is usually recommended. That is generally the time-consuming part. Doing this kind of mapping requires a decent knowledge of your process implementation and your reference model.

A job aid produced by the SEI to support recording evidence is a document called a PIID. This document is organized by Process Area and Specific Practice, and provides fields for recording different types of evidence. Microsoft Word and Microsoft Excel versions of the PIIDs are available from various Web sites (do a Google search for PIID templates, and you’ll get plenty of hits, some of which contain free downloadable templates), and processable versions of them are the basis for several of the commercial appraisal tools that are available.

If you know you will be conducting an appraisal as one of your improvement activities, you will want to make the process of evidence recording as painless as possible. This generally means collecting and recording evidence as you actually perform a process. The trick is that to make this efficient, whoever is recording the evidence should know enough about the model to map the artifacts to the reference model of interest. This is one of the reasons that some organizations go overboard (in our opinion) in training everyone in the organization in the model; they think that by getting everyone trained in the model-ese, they’ll be better at helping with the evidence collection/recording. And some of them will be, but unless you’re in a very small organization where the staff working in improvement are also the staff doing most of the management and engineering, you probably will have many people who, despite model training, won’t be very good at recognizing how their activities and artifacts map onto the model of interest.

This is one of the places where having some skilled model support is helpful; whether it’s a person from inside or outside your organization, having access to someone who knows both the processes you’re implementing and the model you’re referencing can speed this task along tremendously. In smaller organizations, having one person who takes the artifacts from projects and your process descriptions and maps them into the evidence library, whatever approach you’re using, will improve consistency of the mappings and provide a focal point for appraisal team questions. You can use a device as simple as a project notebook (electronic or paper) with a list of the artifacts that should be collected as a way of minimizing impact on project staff while making it feasible for the evidence mapper to collect and record efficiently.

The biggest mistake we see people make in their evidence collection, especially when they’re getting started, is to wait until just before the appraisal to gather all the evidence. You’d be amazed by how difficult and time-consuming it can be to find something (“I know I have it here somewhere”) that is just the right piece of evidence to support an aspect of the appraisal. Of course, there is also the disruption of people’s work to find something that could easily have been provided at the time it was generated without interrupting the project.

So how do you know what things to collect? Actually, that’s one of the benefits of using the PIIDs mentioned earlier; they have taken the “typical work products” section from CMMI and transcribed it into this form. That’s not to say that you must have the work product mentioned; the work product suggestions that are provided, however, will probably give you ideas for work products that might serve the same function on your own project. Some of the downloadable PIIDs contain additional instructional example material to help you.

10.3.2 Composing Appraisal Teams

Another problematic area, especially for small organizations, is composing a useful but efficient appraisal team. For certain types of appraisals, such as a SCAMPI A appraisal that results in Capability Level or Maturity Level ratings, there is a minimum number of appraisal team members (four) who have all been through “official” Introduction to CMMI training, and one of those team members must be an authorized SCAMPI A Lead Appraiser.³ Chances are that if you’re just starting out, you won’t have a lead appraiser in your organization, and you may not even want to dedicate three of your own staff members to an appraisal activity for several days.

Here are a few things to consider when putting together an appraisal team:

•   Having one team member from inside the organization who knows both the process implementation and something about the model you’re using as your basis is almost always a good idea. The money you spend training that person and the time he or she spends preparing for and participating in the appraisal will generally be made up for easily in terms of the time you save for the external members of the team, for whom you’re probably paying a daily rate.

3. Assessment Methods Team. Standard CMMI Appraisal Method for Process Improvement (SCAMPI), Version1.1: Method Definition Document. CMU/SEI-2001-HB-001. (Pittsburgh: Carnegie Mellon University, 2001).

•   The lead appraiser is probably going to be your highest-cost external labor item. Choose one who knows your type of business and has some experience with your context. Hiring a lead appraiser who has done appraisals only in a large-organization context could cause some interpretation problems if your company is very small.

•   If you decide not to staff the rest of the team with internal team members, you may want to look around for people in your area who are “candidate” lead appraisers needing appraisal team experience. One of the requirements to become a lead appraiser is that you must participate in two SCAMPI A appraisals as a team member. There are often candidates looking for opportunities to participate in a team to complete their prerequisites for taking the next step toward authorization. Often, they are willing to discount their labor/travel fees as a way to make it attractive for you to allow them to participate in your appraisal. Make sure that you coordinate all your team-member candidates with your lead appraiser.

There are many other things to consider in preparing for an appraisal, and as we’ve said, several books on the subject cover the necessary elements to consider in planning and conduct. We recommend that you obtain one of them before getting involved with selecting a lead appraiser and your appraisal team.