What Makes a Good Metric?

Perhaps the most obvious metric is money. If we want the best of something, say the best wine or the best television or the best school, then one of the ways we go about finding this best thing is to find out which is the most expensive wine, the most expensive television, and the most expensive school. If we were to use this method in finding the best doctor we would find the most expensive doctor and conclude that he was the best doctor. That is, we would conclude that there was a correlation between the expense of X and X’s quality (and this is after all what we want from any metric, a well established correlation between the metric and performance). However, there are many problems involved with correlating the expense of X and its quality. There are market failures and other distortions (such as government regulations and other perverse incentives). What these all come down to, however, is that using money as a metric doesn’t work a lot of the time. A performance metric should measure an outcome associated with an objective. Money does not meet this standard. Therefore, other metrics must be generated and used, but in order to generate such metrics a more comprehensive framework is needed.

One such framework was created by Professor William Clark, of Harvard’s Kennedy School of Government. His framework for assessing metrics is one in which we could say that all good metrics are “credible,” “relevant,” and “legitimate.” He writes, “‘Relevancy’ as we use it, is meant to capture the perceived relevance or value of the assessment to particular groups who might employ it to promote any of the effects noted above. ‘Credibility,’ as we use it, is meant to capture the perceived authoritativeness or believability of the technical dimensions of the assessment process to particular constituencies, largely in the scientific community. ‘Legitimacy,’ as we use it, is meant to capture the perceived fairness and openness of the assessment process to particular constituencies, largely in the political community.” (Clark 1999) I hope that all the metrics I discuss here will be credible, relevant, and legitimate. Unfortunately, they will not all turn out to have these important characteristics.

It is extremely important to choose metrics that actually measure the thing that you want them to. An example of not doing this would be to measure random things that are correlated with good performance but which are not actually associated with that performance. This is one of Nassim Nicholas Taleb’s main points in his book Fooled by Randomness, where he writes, “[M]y income started to increase after I discovered my slight nearsightedness and started wearing glasses. Although glasses were not quite necessary, nor even useful, except for night driving, I kept them on my nose as I unconsciously acted as if I believed in the association between performance and glasses. To my brain such statistical association was as spurious as it can get.” (Taleb 2005) If Taleb were to generate a metric that correlated wearing glasses with performance, it would not be an efficacious metric. One of Taleb’s points in his recent book is that we are often fooled by randomness into believing that a metric is measuring performance when in reality it is only randomly correlated with performance.

Of course, many people in the business world have thought about how to generate metrics of performance and they have learned a great deal about how to generate metrics that actually measure performance. Business strategist Geri Stengel generated a list of Ten Tips for Measuring and Improving Performance. Most important on his list are that you should:

In the examples that follow I will discuss the metrics of many fields that use many of these suggestions when generating their performance metrics.