Standards and benchmarks, in the realm of metrics, are strongly interconnected. Standards, from the Industrial Age through today, are invaluable for providing a means for interoperability. Standards in the industrial world allow you to use a light bulb that you bought at Walmart in a lamp that you bought at a high-end designer furniture store. Standards allow you to get gas for your car from any station in the United States, without worrying if the gas pump nozzle will fit into your gas tank. From the ingredients label on a can of soup to the technology that allows you to tune your radio, standards give consistency and interoperability for manufacturers, distributors, builders, and customers alike.

Unlike the manufacturing industry, performance measures are more akin to an art than a science. The use of standards for how we measure things for improvement is arguable. What need is there? If our questions are unique, and thus our answers are unique, why do we need standards for our measures?

Since I advocate creating measures to answer your specific questions, I have trouble taking up the other side of the debate. Why indeed?

I would love to have standards for how you develop metrics; as in the use of expectations over targets, for example. Or for the definition of the data owner. But, standards for performance measures as a whole? Why?

Before I answer this question, let’s look at benchmarks and why I think the two are interrelated.

Benchmarks: Best Used to Provide Meaningful Comparisons

Benchmarks are best used to provide meaningful comparisons for your ­metrics. Outside of defining expectations, you usually want to know how well you perform against your peers. If you’re ambitious, you’ll want to know how well you perform compared to the best—the best in your industry, the best in your country and, perhaps if you’re really ambitious, the best in the world.

Benchmarks are also useful for drawing a “line in the sand.” You can establish a baseline from your own measures so that you can compare your present performance to your past performance. This is critical when your goal is to improve.

Establish Baselines

One of my joys when working with clients on metrics is helping them establish a baseline; mostly because it forces them to put the metrics upfront in their improvement-process thinking. I almost always run into goals to improve effectiveness, improve efficiency, improve productivity, or improve customer satisfaction.

“Improve” is a lousy verb to use in a goal statement. You have to qualify it with more information—as in “how much” of an improvement? By a certain percentage? By a certain number?

My favorite recollection of the poor use of an “improve” goal was in my ­parish council. I was hoping to bring organizational development expertise to the council. I was teamed with a retired police officer, a successful business owner, a nurse, our priest, and a retired grandmother. The goal? Improve membership in the church. I wasn’t perturbed because I had seen this type of goal (increase, decrease, etc.) many times before.

We eventually worked out a reasonable and measurable increase over the expected growth without making any changes. The purpose of the goal was to build up the parish. The purpose of the metric was to see if our efforts were successful. We had plans, ideas, and activities scheduled for the purpose of bringing in new members and bringing back parishioners who had fallen away. We needed to (1) set a goal to focus our efforts and ideas; and (2) set measures to tell us what worked and what didn’t.

We also needed a benchmark. We could not determine if any of our efforts were producing the desired result if we didn’t know the norm. Consider the benchmark in this case a “control group” or value. You have to know what you get if you do nothing different. Then, when you do new things in new ways, you can at least assume that any changes that you made caused the change to the outcome. Even if you implement so many changes that you can’t determine what exactly worked or what exactly didn’t work, you at least know whether the overall effort(s) worked.

So, benchmarks basically allow you to know where you are and, therefore, where you end up.

Even when a benchmark is used to compare you against your peers, it is essentially a starting line—a baseline to measure your progress against. The purest form of the benchmark is when you set it as an internal baseline that allows you to measure progress.

Set Benchmarks Responsibly

Benchmarks falter from time to time when leaders want to use the comparison benchmark as the baseline. Most times, it starts with something like, “Can you get our competitors’ average availability, response time, or customer satisfaction ratings so that we can compare ourselves to it?”

This requires that the performance of your competitors is a good starting point.

My simple and first argument against chasing this data is: “What if you are already better than your competitors? Does that mean you’re doing well enough?”

So, while gathering information on another organization’s performance can be enlightening, if your goal is to improve, it is not overly useful. If your goal is to be better than your peers, then, of course, this benchmark is essential. But even if your goal is to be better than your competitors, you’ll need to know (1) whether your efforts are helping you improve; and (2) how far you are from the performance of your peers.

If you choose to look only internally at your performance, standards are not necessary. But, if (and when—because eventually you’ll want to see how you compare to others) you decide to compare your performance to your peers or competitors, standards will be critical. You can’t compare yourself to others when the methods of measurement are different.

Let’s say you define the availability of the network as the amount of time without an outage divided by the total amount of time in a given period.

• Availability = 1,440 minutes (number of minutes in one day (or 24 hours)) – 20 minutes (of outage) divided by 1,440 minutes
• Availability = (1,440 – 20) ÷1,440 = 98.6%

So far, so good. But, let’s say your closest competitor (or peer) has a 100 percent availability rate for the same period. Are you going to step up your game a bit? Are you going to work harder? Is your competitor doing better than you?

Well, without standards, you can’t tell if your competitor is doing a better job than you. What if you define an outage as any time span that your customers cannot use the network, but your competitors consider an outage as only those times when the network is unavailable due to unscheduled or unplanned downtime? In other words, let’s say that during the same 24-hour period, your competitor had scheduled maintenance for four hours. If you define an outage as any time the service is unavailable (which is likely the way customers will interpret it), then the competitor’s availability should be reported as follows:

• Competitor Availability: (1,440 – 240) ÷ 1,440 = 83.3%

If you had all of the raw data for your competitor’s reports, you could use your personal “standard” and determine how well you perform against your competitor. But this is highly unlikely to happen. What you will get, if you are extremely lucky, is the “score”.

So, how you define an outage compared to the way your competitor defines an outage is critical to your use of their measures as a benchmark.

Standards Allow Comparison to Others

Standards in performance measurement come down to providing the ability to compare measures between different organizations. Just as manufacturing standards allow you to use your products seamlessly with another organization’s products, standards in performance measurement allow you to “use your measures” seamlessly with another organization’s measures. If there were standardization of performance measures, you could “borrow” another unit’s measures for your own purposes. If you had the same questions, you could trust that that metrics used by Company A could be used to measure the same things in your organization.

Without standards, it is hard to imagine how you could use the metrics of a different organization—even if you had the same exact root questions. And using measures produced by another organization as a benchmark is even more improbable.

Getting Good Data

This problem with benchmarks and standards for performance measurement also creates problems for well-meaning organizations that seek to provide data warehouses of information. This information invariably is intended for your comparison. To make the data warehouse effective, it has to have a clear set of standards for the information provided by the different organizations.

HDI, a third-party survey company, offers benchmarking on customer satisfaction data by controlling data definitions. HDI administers the survey to your customers, collects the responses, and provides you with reports, analysis, and raw data. Since HDI standardizes its questions (the same questions are asked along with the same set of possible replies on a 5-point Likert scale), it is able to offer you comparisons against other organizations, including the following:

• Comparison of your scores (average, percentage satisfied, or other) against all other organizations who have used the HDI service
• Comparison of your scores with others in your industry (self-selected from a list)
• Comparison of your scores to the top nth percentile of others’ customer base

This provides you with a higher confidence in the comparability of the information. Of course, there are some drawbacks. Only organizations that use HDI’s service are included in the comparison, and your main competitors or peers may not be among these organizations. Even if you compare your results against the entire customer base, this still may not reflect the pool you want to compare to. Another minor drawback is that you are forced to use one set of questions. No deviations. If you want to use the 10-point scale suggested by Reichheld in The Ultimate Question, you could not use HDI’s service. And even if you could use the 10-point scale, you could not compare the data to those who used a 5-point scale.

The Goal: Reliable Industry Standards

Industry standards for performance measures would make it possible to truly benchmark, rank, and compare peer organizations. It should be feasible to convince an industry (like higher education IT) to standardize performance measures before the chance of adopting universal standards. The tighter you can make the pool for standardization, the easier it should be to come to agreement. Higher education information technology is a pretty specific pool. If you started with information technology performance measures—your pool for coming to consensus on the standards is too large. If you narrow it to education information technology, you’re doing better. Higher education tightens it a little more.

Consensus is required for success. Publishing a standard does not make it effective. You must have the majority of organizations (in your industry) using the standard to make it useful. Since you need high participation in the use of the standards, it logically follows that you should involve as many of the target audience in the creation of the standard as possible

I offer that the consortium structure is the best bet for creating standards for performance measures. The consortium creates, evaluates, reviews, and manages standards for an industry. The problem may be that the “industry” in this case is hard to define. Of course, if you do as I suggested and find a tighter definition of the target audience, you can make it happen. But, looking at performance measurement as an industry is obviously too large. So, as a performance measurement expert, you’ll have to define your “industry” to build a consortium. If you have some standards for performance measurement to reference, you are ahead of the game.

Recap

Standards are tools that allow for interoperability. In the case of performance measures, standards allow for comparison between organizations.

Benchmarks are either the starting line (baseline) for your improvement efforts or a goal for you to achieve. As a baseline, it helps you determine how far you have to progress to achieve your goals, how well you’re getting there, and how far you have come. As a goal, it represents how good you want to become—“as good as Company A” or “better than the average.”

To have real external benchmarks, you must have standards that are in agreement among the organizations you choose to compare to.

If you can find or develop standards for your performance measures, and your peers agree to them, you can compare measures.