Chapter Eight

Attainment: Measurement of Customer Satisfaction

One of the new measurement areas we specify in the Learning Impact Measurement Framework is “attainment,” which refers to the measurement of how well the organization has met specific customer requirements. This could be considered the “customer satisfaction” measure. This measure, not considered in the Kirkpatrick model, may be the most important one of all.

Remember that Learning & Development is a service organization and, as such, customer satisfaction should be your number-one goal. In the service business, customer satisfaction measures are defined by the customer—not by you. Customers establish the criteria for success. Some program sponsors may measure success by being delivered on-time, for example. Others may measure success by being delivered on-budget. Others may measure success by meeting their limited time requirements, technology requirements, or learner satisfaction requirements.

This measure, called “voice of the customer” in Six Sigma terms, is one of the most valuable measurement processes used in manufacturing, sales, and other business processes. It should be incorporated into the measurement program. Let me briefly introduce you to some Six Sigma concepts.

The Six Sigma Approach

Six Sigma¹ is an approach for measuring, monitoring, and improving manufacturing and business processes to generate the highest possible quality results. It was developed with the goal of establishing a clear process to measure and improve the quality of any business process. It has been refined through implementation at hundreds of companies, and elements of it can easily be applied to training. The Depository Trust Clearing Corporation (DTCC) uses Six Sigma to measure training development, delivery, and customer satisfaction.

Simply explained, Six Sigma lays out a set of processes to define:

• Customer needs and measures of success

• Customer “critical-to-quality” issues (the things that “must be done right” in order for the customer to be successful)

• Processes that result in this quality (processes in your training organization, in the rollout, delivery, or follow-up)

• Measures of the performance of these processes (indicators of each process, such as those in the Impact Measurement Framework^®)

• A way to collect and analyze this data

• A commitment to continually use this data to improve these processes through solutions (process improvement, training, etc.)

• Ongoing improvements over time

Six Sigma is a process that goes on forever. You establish these measures and you monitor them at all times, continually making process improvements and increasing your measures to get better and better over time. Six Sigma enables Japanese auto manufacturers to develop higher and higher quality automobiles every year. When a new car is designed (as when a new learning program is designed), the measures and processes must be applied to this new car. Processes that were highly successful in a prior car (or learning program) may need tweaking or improvement in a new car (or program).

For example, suppose one of your Six Sigma measures is the ability to deliver a program precisely on-time. This may be customer-critical for a product launch. If you are ten days late on an original schedule of one hundred days of development, this would represent a 10 percent schedule slip. For a subsequent program, you may be five days behind on a development schedule of twenty weeks, or 2.5 percent schedule slip. To meet the “Six Sigma” goals, you would strive to be 100 percent on-time more than 99.9 percent of the time (a very lofty goal).

In Six Sigma terminology, the success of an improvement is measured by how well the training organization meets the customers’ critical-to-quality issues—as defined by the customer.

In Six Sigma, these criteria are specific and unique to each customer. For example, for a compliance program, the customer may have a requirement that 100 percent of employees successfully complete the program by a certain date. There may not be any learning or business objectives beyond this. Here, success may be measured by completion rate. In a food-safety program, the customer’s criteria of success may be a 90 percent or higher score on the exit exam. This is a “learning” objective.

A training organization should capture these requirements on the signoff form (or during the performance consulting process) and establish a process for capturing this data to meet the customer’s needs. Can success be gauged against these customer-driven requirements? Yes.

How Close Is the Training Organization to Perfection?

Six Sigma addresses this question. The term “sigma” refers to the statistical standard deviation (a measure of how much variance there is in the results). In a sense, the sigma level is “how close we are to perfection.”

In Six Sigma terms, perfection is defined by these customer requirements. A Six Sigma defect is defined as anything outside of these customer specifications.

Suppose, again, that one customer’s issue is “time to market”—the customer needs the course to be out in three weeks. If 90 percent of the time the course is delivered within three weeks, then the training organization is between 4 Sigma and 5 Sigma. A second customer requirement may be cost. The client demands that the program come in on-budget. If the ability to meet this need is measured and it is achieved 66 percent of the time, only 2 Sigma has been attained (far below 6).

How Do You Operationalize the Measurement of Customer Satisfaction?

Once you come to grips with the “attainment” of “customer satisfaction” concept, it is not difficult to measure. At DTCC Learning, the Six Sigma approach is used to measure every single learning program. The voice-of-the-customer requirements are identified up-front, turned into specifications, and then used to measure the sigma of each measure and, ultimately, the program itself. Programs can then be ranked by their “sigma” results.

For example, if one of the Six Sigma measures was on-time delivery, DTCC measures the “days behind schedule” for each program. If another measure was meeting budget compliance, it is easy to measure the “percent over budget” for each program. If a third measure is “percent of target audience completed within 180 days,” then again this can easily be measured.

Note that “How satisfied was the customer?” is not a Kirkpatrick measure, so if you follow the traditional Kirkpatrick model you may miss it completely.

We encourage every training organization to capture these voice-of-the-customer needs through a business unit signoff form or performance consulting process. These processes can then track progress against goals and use these as a way of “scoring” delivery success. You can establish a set of eight to ten of these measures yourself and easily develop your own benchmarks.

These benchmark measures serve two important values: first, they will give you excellent operational targets to improve your learning operations; second, they will give your business unit customers tremendous confidence in your organization.

Notes

1. For more information on Six Sigma, please visit www.sixsigma.com, which offers articles and easy-to-read examples of how to apply Six Sigma to any business process.