Appendix III

Research: The State of Training Measurement Today

As part of this research effort, in 2006 we surveyed research members to understand current trends, issues, and benchmarks in training measurement. This survey was a follow-up to a similar survey we conducted in the summer of 2004.

The survey was sent to CLOs, directors, and senior managers of training throughout North America, and we received 136 detailed responses.

In general, this survey illustrates the relative immaturity of measurement in many organizations and the need for easier, more business-focused measurement solutions in corporate training. It illustrates the challenge that organizations have in general planning and alignment, which, in turn, leads to challenges in measurement.

What Do Training Organizations Consider Important?

One of the first questions asked was what measures organizations consider important, as shown in Figure A3.1.

As the figure indicates, organizations are very interested in understanding impact: job impact, business impact, and specific business impacts. These measures are considered important because they help the organization demonstrate value and refine programs to better drive results.

Figure A3.1 Research Results: What Measures Are Considered Important

A significant number of organizations (47 percent) consider cost a critical measure, but interestingly 53 percent do NOT consider cost a valuable measure. This is somewhat surprising, considering that training is, in fact, a cost center and, without understanding the nature of these costs, it is impossible to measure ROI or improve efficiency.

Another interesting finding is that only 44 percent believe that learning results (Level 2) are extremely valuable and only 43 percent believe that satisfaction is extremely valuable. These two foundational measures of the Kirkpatrick model are clearly limited in value. This is not surprising, given the fact that corporate training is a business performance improvement function (not an education function) and that measures should focus more on performance than learning. It further justifies the motivation for Bersin & Associates’ Impact Measurement Framework.

What Do Training Organizations Actually Measure?

Understanding the measurement desires, we then wanted to understand the current state of measurement. Unfortunately, the current state of measurement is almost inverse to the perceptions of value. Organizations are frequently measuring things that they do not see as highly valuable.

As Figure A3.2 shows, most organizations focus on measuring standard course operations: enrollments, and completion. Level 1 surveys are now very widely used (81 percent of organizations use these regularly).

A very small number of organizations routinely measure ROI, business impact, or job impact. This further supports our hypothesis (explained in detail in this book) that training organizations are looking for help creating easy-to-implement processes for measuring impact. The Bersin Impact Measurement Framework is designed to help in this area.

Figure A3.2 Research: What Do Organizations Measure?

Figure A3.3 What Is Frequently Measured: 2006 Versus 2004

The other interesting finding (which we consider disappointing) is that only one-third of organizations or fewer measure cost. Thirty-seven percent measure the total cost of a training program, and only 31 percent measure development cost or cost per student. This is an indicator that many training organizations are not able to focus their time on operational excellence and efficiency.

When comparing this data with the results from our 2004 study (Figure A3.3), little major progress was found. In almost every high-value area, the percent of organizations that routinely measure these areas is lower in 2006 than it was in 2004. While the samples were not identical, we believe this indicates that, in general, there is a similar level of frustration in learning measurement today as there was in 2004.

Budgets for Training Measurement

Respondents were asked to estimate what percent of their companies’ total training budget is spent on measurement, as seen in Figure A3.4. Our research shows that, without a dedicated resource, training measurement is difficult to implement in a consistent way.

Figure A3.4 Training Spending on Measurement

The result was an overall average of 2.6 percent of budget spent on training measurement.

However, this reflects a high standard deviation of results. In fact, one-third of companies surveyed spend far more than the average, and 42 percent spend far below the average. There are a large number of companies (nearly half) who spend almost nothing and there are a small number of companies (16 percent) who invest heavily—spending 5 percent to 10 percent or more.

Looking at the spending by organization size, as seen in Figure A3.5, spending is, in fact, fairly evenly distributed. Very large organizations can leverage their dedicated resources across a large audience, so their percentages are low. But small-and mid-sized organizations all spend to some degree, so we believe the findings indicate that companies of any size can afford to spend 3 to 5 percent of their budgets on measurement if they choose to do so.

Figure A3.5 Spending on Measurement as a Function of Organization Size

The Importance of a Business Plan

An important factor in an organization’s ability to measure training is how well they manage the overall planning and budgeting process. The business planning process is instrumental in measuring alignment. In this survey, only half (52 percent) of respondents stated that they have a written business plan for budgets, volumes, and program deliverables. (See Figure A3.6.) Of course, without such a plan it is impossible to measure “results versus plan.”

This somewhat surprisingly low number of organizations that have a business plan led us to hypothesize that the organizations without a business plan are likely to be doing less measurement. In fact, this is true.

Figure A3.6 Is There a Written Plan for the Training Organization?

• Organizations with a written business plan spend 60 percent more on training measurement than those without a business plan;

• Organizations with a written business plan are twice as likely to measure alignment through executive reviews and steering committees than those without a business plan;

• Organizations without a business plan are four times more likely to see a need for significantly higher spending on measurement than they are spending today; and

• Organizations without a business plan are three times more likely to rate themselves “poor” or “marginal” in their overall success at measuring the effectiveness and efficiency of training.

How Do Organizations Measure Alignment?

We specifically asked organizations to explain more about how they measure and manage business alignment. Here the answers were again somewhat disappointing:

• Sixty percent of organizations measure alignment by their ability to “meet their budget”;

• Fifty-four percent of organizations measure alignment through an annual executive review of their operations (this seems very low);

• Forty-one percent of organizations measure alignment by their ability to “deliver programs on time”;

• Thirty-nine percent of organizations measure alignment by their ability to “meet learner satisfaction targets”;

• Twenty-four percent of organizations measure alignment by their ability to “maintain job satisfaction in the training department”; and

• Fifteen percent of organizations measure alignment by their ability to “meet target volumes delivered.”

These results indicate that training organizations do not have good measures in place to make sure they are consistently aligned with corporate priorities.

Training managers and executives were then asked to relate what techniques they use to make sure they are staying close to ever-changing business requirements (an alignment issue). The results were similarly disappointing, as seen by Figure A3.7.

In our High-Impact Learning Organization¹ research, it was found very conclusively that organizations with high levels of effectiveness and efficiency focus heavily on a three-tier governance process. This includes an executive steering committee, a series of learning “councils” (which often include line of business representation) and a series of operational teams (which work together on internal training processes). We also found a significant correlation between organizations that implement effective measurement programs and those rated high in effectiveness and efficiency.

Figure A3.7 How Do Organizations Stay Aligned with Changing Business Needs?

Training organizations with high levels of effectiveness and efficiency focus heavily on a three-tier governance process.

The research from this study further reinforces these findings by illustrating how often training departments do not have a detailed operational plan (nearly half) and how even those with a plan do not measure themselves against the plan on a regular basis.

Why, in fact, are training organizations having such a difficult time with such operational measurements? One problem may be the lack of performance consulting processes in place.

The Role of Performance Consulting

Because we know that performance consulting is such an important factor in effectiveness, efficiency, and measurement, respondents were asked in this survey whether or not they had a performance consulting role. See Figure A3.8 for the results.

Performance consulting is such an important factor in effectiveness, efficiency, and measurement.

Figure A3.8 Does the Organization Have a Performance Consulting Role?

Again the findings were somewhat disappointing—almost half of the organizations in this survey do not have such a role (45 percent). These organizations without a performance consulting role, on average, spend 20 percent less on measurement. Organizations with written business plans are three times as likely to have a performance consulting role, illustrating that, in general, more sophisticated and business-focused learning organizations have all three processes in place. These organizations:

• Have a written business and operational plan;

• Have a performance consulting role that works closely with lines of business; and

• Spend above average on training measurement and are more satisfied with their measurement processes.

Satisfaction with LMS Systems

In this survey, we also asked organizations how satisfied they were with their LMS systems. As seen in Figure A3.9, only about two-thirds of the organizations responding had a full-functioning LMS system. We specifically asked those with LMS systems how well the systems were helping to facilitate the measurement process. As with other research we have performed in this area, the results were relatively low.

Figure A3.9 How Well Does the LMS Support Training Measurement Needs?

Only 29 percent of respondents stated that their LMS was excellent or good at providing the desired measurements and reporting capabilities, and 39 percent rated their systems “poor.” As described in this study, this is due to several factors:

• Lack of data standards;

• Lack of measurement process;

• Poor content integration; and

• The difficulty of using the LMS reporting system itself.

As discussed throughout this book, many of these problems can be solved through the implementation of a sound measurement program—some can only be solved by an investment in data standards, integration, and more advanced LMS features.

Is Enough Being Spent on Training Measurement?

Finally, we asked respondents whether they were spending enough or should they be spending more. As expected, and shown in Figure A3.10, almost all respondents felt they should be spending more.

A full 82 percent of organizations feel they should be spending “more” or “much more,” and almost one-third (31 percent) feel they should be spending “much more.” In general, when talking with training managers individually, we found that the biggest challenges they face are not lack of budget but, rather, a difficulty understanding precisely how to implement a measurement program and, from there, build the business case for the budget.

Figure A3.10 Are You Spending Enough on Training Measurement?

Demographics from the Survey

This survey came from a broad number of organizations in a wide range of sizes and industries, as seen in Figures A3.11 and A3.12.

Figure A3.11 2006 Survey Demographics: Organization Size

Figure A3.12 2006 Survey: Industry

We did not find any significant trends in spending, maturity, or measurement sophistication across these different organization sizes or industries. Interestingly, the challenge of developing and implementing a consistent and powerful measurement program haunts small, medium, and large organizations equally.

Note

1. For more information, see The High-Impact Learning Organization: WhatWorks^® in The Management, Organization, and Governance of Corporate Training. Bersin & Associates, June 2005. Available at www.bersin.com.