I once needed to know what cities were adjacent to a town I planned to visit. I went to one of the travel Web sites to look up a map. At first, I looked at the largest map of the area, which gave me a relatively wide radius. I centered the map on the specific town, and then clicked down another level to zoom in closer. That narrowed the options considerably. I continued to work my way down the available levels until I was at the lowest level.

That experience of starting at a high level and gradually working down to a quite detailed level is similar to developing measures. At the highest level, you have your goals and the audience for your goals. At the next level, you zero in on defining your measures. You then move to specifying what data will be collected and how. At the lowest level, you determine how your measures will be displayed. In this chapter, we’ll look at a generic process for developing measures.

The generic process for developing measures has the following steps:

Developing measures is a reality check for your goals. If they’re not concrete and clear enough, this will show where the cracks are. For measures, you need goals that define the following:

Expect to spend some time discussing and refining the goals you developed previously. It’s one thing to put together goals for a program or approach. It’s quite a different kettle of fish to put together goals concrete and clear enough to drive measures.

Also, expect to revisit your goals periodically throughout the process of developing measures. You may discover that a goal needs a bit more work, or even that a goal isn’t realistic to measure.

At the same time, don’t let yourself jump ahead to a much later step in the process, like what data will be needed. If you skip ahead to the tail end of the process, you’ll find yourself slinking back to the goals. They are your foundation for the rest of the process.

So who decides what success is? The second step in the process answers the question of what success means to certain groups. Those groups are the audiences for your measures. Since defining success is part of determining your goals, these two steps in the process are intertwined.

The level and scope of your effort helps name your audience for your measures. If you’re involved in a global knowledge management effort that costs big bucks and spans your global organization, I’d suggest strongly that you consider your CEO and corporate board to be a potential audience, not to mention your external stakeholders. If, on the other hand, your effort is located within a specific business unit, your corporate board may be an option but not a necessity. At the same time, the senior managers in your business unit are an important audience.

Regardless of your level, some critical audiences you should consider are

For example, in Chapter 8, “Communities of Practice—The Killer Application,” we learned that you have three potential audiences for communities of practice measures:

There’s no set answer for who the audience should be. However, there is one ironclad rule involving the audience: They are the ultimate judges of how good your measures are. No matter how good you may think they are, it must make sense to them.

As the proud parent of the measure, you have the right to give it a name. This doesn’t give you free rein for your imagination; the name needs to be in simple, everyday language understandable by the audience for the measure. Resist any urge to give an overly long, detailed, and precise name. A good name is brief and gives a general idea of what is being measured.

You’ll build on the name to develop an operational definition. An operational definition is your blueprint for the rest of your work developing the measure. It gives you the specifications in enough detail to guide the work. It also tells you what is and isn’t included in the measure.

It’s another link in the long chain from your goals to your measures to the data collected and displayed to how the measures are used. While all links in the chain must be solid and tightly connected, the operational definition has a particular impact on the validity of the measure. Your operational definition is a critical step in moving from a more abstract goal to a concrete, measurable something.

It is also an important step for developing a good measure, one that actually measures what it is supposed to measure, not something else. This is called validity, a key concept in measurement. A valid measure is one that gives you the correct information you need for your intended actionable understanding.

Validity for measures is contextual. It depends upon what question you are asking and what inference you will make from the measure. For example, if a goal defines success for a community of practice, a valid measure actually gives you the information you need to decide if the community of practice is successful. An invalid measure could lead you to assume that a community is successful when, in fact, it is not, or that a community has failed when it is, in fact, successful.

You have to take your best shot at validity. A measurement tenet is that validity is impossible to prove conclusively. You can collect proof of validity, but you can never prove it. Over time you may get unfortunate proof that your reasoning was faulty and that the measure doesn’t give you the right information.

Or, you may get proof that your thinking was spot on. That’s why, after you implement your measures, you need to evaluate and monitor how they’re working, which we’ll discuss later, in evaluating your team of measures.

However, one thing you can do immediately is to go back and look at your reasoning for the measure. Some of your conclusions may have been unvoiced, and thus unexamined. Force your reasoning out into the light of day. Think through your reasoning aloud, step-by-step. Ask yourself again:

Measurement is an invasive procedure, like surgery. It changes the patient—in your case, your organization. It’s also a good idea to think through the possible consequences of the invasion, the measure itself. What sort of actions could result from this measure? For example, if you only measure individual sales, what does that say to your salespeople about the importance of team sales? Will the measure encourage them to concentrate only on sales they can make alone, even though team sales might be more effective? What will happen if you push the amount of individual sales to the max?

Unintended and unwanted consequences also hurt the validity of any measure. They’re certainly not what you wanted to measure.

However, validity is useless unless you will take action, if necessary, based on the results of the measures. Answering the following questions is the acid test for the ability to take action:

If you can’t answer any of those questions, your measure is useless. Stop. Think about what you can do to make the measure actionable. If you can’t, drop this one and develop other measures.

If you’re a detail person, now’s your time to shine. Specifying what data will be collected and how it will be collected is a process of spelling out necessarily excruciating detail. Expect to spend a lot of time on this. It’s also hard to shift your mental gears from defining a measure, which is still fairly general, to a level of extreme detail.

To determine what data will be collected and how, you need to answer these questions:

You need not only to be specific, but also be able to describe and communicate the specifications well enough so that the data collected is consistent. This brings us to another key concept for measurement—reliability. Reliability means consistency. For example, if every time I step on a scale I weigh 98 pounds, the results are reliable (and I will be heartbroken when I wake up).

The reliability of your measures will depend upon the consistency of the data collected. No matter how carefully you described the intended data and its collection, in practice, the actual data and its collection could vary widely from what you intended. For one thing, it may not be collected at the exact times you specify. While you can’t control all of the variation that will happen (variation is inevitable), you do need to take special care when instituting new measures to give detailed descriptions for the data and to use effective communications. You’ll also need to monitor how the data is collected over time.

Reliability and validity are related. To be valid, a measure must be reliable. A scale that gives different results every time isn’t valid. However, a reliable measure isn’t necessarily a valid one. A measure can give you a consistent but dead-wrong result.

The difficulty of collecting new data might tempt you to develop measures using existing data. However, just because reliable data already exists doesn’t justify developing a measure using it. Every measure must provide important information, not nice-to-know stuff. Even though it may not cost anything additional to collect the data for the measure, you’ll expend resources reporting and monitoring the measure in your effort. It still ain’t free. It’ll cost even more if it stunts your thinking and diverts you from developing more important measures.

As mentioned in Chapter 19, “You Get What You Measure,” you are what you present. Think through how the measures will be presented graphically, preferably on your intranet. If you’re able to present using computerized graphics, you’ll have many more options both for presenting and for dissemination.

Don’t try to cram in too much information into a single display. The more things you try to present at one time, the harder it is to understand. Try to limit each graphic presentation to a key facet of the measure.

Determine how often the graphic displays will be updated. This may or may not dovetail with how often the measures are reported.

Imagine what could happen if you went out and recruited star ballplayers for a baseball team without thinking what positions they played and what positions are needed to complete a team. You could come up with three pitchers, one shortstop, and five first basemen. That’s not a winning team. Also, no matter how much money you had to play with, you’d be unable to get your dream team. You’d have to do the best you could with what money you had and what players were available.

Assembling a team of measures is similar to putting together a strong baseball team. You need to fill each position. For you, this means looking to see if you’ve developed measures for each of your goals. Unless you keep track, you can leave out some of your goals.

You also have to make sure that you don’t have more than one player for the same position. Look to see if any measures are close enough to pass for duplicates. Also check to see if the measures are loaded up for a particular goal or a few goals, and thin for others.

Another important consideration for putting together a team is how the individual players complement each other. Each player has strengths and weaknesses. In the case of measures, you need to determine how the measures relate to each other. Using your chart, compare each measure to all of the others, one by one. Ask yourself these questions:

Give enough context so that people understand the measures. Consider a glossary for your labels and various links, like links to define labels and measures. Also consider including a link to give more overall context. Overall, give people a point of contact for more information if they need it.

Don’t drive yourself crazy. There may be no relationship at all, or the relationship may become clear over time. However, you do need to think about it up front.

Finally, ask yourself if this set of measures gives you a good team of measures. Will they do a good job of measuring your goals? Or are there still a few positions open on the team?

Consider your measures your employees. I invite you to hold them to the goals you have for them and to periodically have performance reviews.

Be prepared to send some measures off to the minor leagues. Having a set of measures requires adjustments to the roster just like a major league baseball team. As the season wears on, some measures will fail to perform and need to be replaced.

Developing measures is a difficult process of trial and error. You won’t get it right the first time. Even if you do reach the point that you think you have a champion team, events can surprise you.

Even someone who has been a most valuable player and a member of the all-star team reaches a time when his or her playing days are over. You need to periodically evaluate your measures, looking to see if it’s time for a well-deserved retirement. You can’t be sentimental about old favorites. Nor can you keep on a measure because it is familiar and easy to keep using. Your answers to the following questions will tell you if it’s time for a retirement dinner:

The same questions apply for every measure in your team roster.