5

Transferring Learning and Evaluating Results: How Do You Demonstrate Success?

 

In This Chapter

• How your organization determines success

• Available evaluation methods

• Impact on learners

• Impact on the organization

At first blush, it appears that this chapter is the place to read about evaluation and demonstrating success. That’s only partially true. Think back to what you read in the first four chapters:

• Chapter 1: “Although we won’t discuss the evaluation plan until chapter 5, you need to begin thinking about how you will measure success. What tools are available to you? What do you anticipate your leaders will want to measure?”

• Chapter 2: “This means that you need to begin the discussion about how you will evaluate the results. It is far easier to sustain a measured successful effort than one that is successful but has no data to support the success.”

• Chapter 3: “End your strategy by identifying your indicators of success. How will you measure success and how will you demonstrate efficiency, effectiveness, quality and the way the design meets the organization’s needs? Present your evaluation plan: what and how you expect to evaluate the results.”

• Chapter 4: “If you’ve followed the plan, by now you know what your leaders expect and how they will evaluate the program.”

That’s a pretty good argument for stating that you need to think about evaluation first, last, and all the times between.

Learning is a strategic enabler in your organization. That means that you and your talent development team are responsible for showing that talent development efforts are not only aligned to the organization’s strategy, but actually produce a return on the investment. That return on investment could be substantial. Data compiled by the Korn/Ferry Institute show that when companies align their talent strategy with their business strategy, the results may include:

• a boost in morale

• increased productivity

• lower turnover

• higher customer satisfaction

• better company financial performance.

You can demonstrate that talent development efforts contribute to these results with a good evaluation process.

With that in mind, what could I possibly add to what Jim, Don, and Wendy Kirkpatrick and Jack and Patti Phillips have already said in their many books? Of the organizations that have an evaluation process in place, almost all use a measurement method from one of them. Their techniques are well researched and proven. If you need an evaluation refresher, be sure to check out the books listed in the resources at the end of this chapter.

Determine and Demonstrate Success

No matter how perfect your talent development program is, most people agree that it is difficult to demonstrate a return on investment. So, what can you do to show business impact? Your measurement plan will begin with understanding what effect talent development will have on the organization. But it’s not just the C-suite who wants proof. Think about who else needs to know. Then talk to your stakeholders about how you plan to measure success, and ask them for feedback on whether you’re measuring the most meaningful items. In addition, consider your strategy for making recommended changes to any training or development events. Remember, you can’t wait until the end to think about what and how you will measure.

Measuring the impact of implementing a talent development program may seem to be difficult—even impossible. However, if you identify your objectives—the reasons you are starting a talent development program—it is achievable. In addition to measuring performance, you’ll want to monitor indicators that measure the development of the learning culture you are trying to create. For example, you could measure things like:

• how many employees have IDPs and what percentage of items were completed

• employee turnover or changes to engagement

• how many employees embrace self-directed learning

• sales growth in various regions

• decreases in accidents on the shop floor.

Depending upon the situation, some of these may simply be part of your strategy, such as increase sales or reduce accidents; but in other cases, they may be used as indicators of what you ultimately want to measure.

Early in this book you learned about the importance of communicating with senior leaders to identify organizational strategy, values, and goals. Those discussions become even more critical when we address evaluation because they provide the basis for what should be evaluated—establishing how the talent development program determines and demonstrates success.

Talent development professionals need to show how the work they do contributes to organizational success. Multiple reasons exist for evaluation. A robust evaluation system provides valuable information for the organization, managers, talent development professionals, and of course the learner.

The organization benefits because the data show whether the learning strategy helped achieve the organization’s goals. This benefit may be shown by return on investment or return on expectations. Evaluation can also provide a cost-benefit ratio and identify the bottom-line value of the learning strategy to the organization.

Managers benefit because evaluation can help them understand how to continue coaching their employees. It shows them the benefits of investing in staff, where the shortfalls may be, and what skills and knowledge have been gained that can be used as springboards to additional learning. Probably most important, the evaluation data demonstrate what has been transferred to the workplace.

Talent development professionals benefit because the data provide ways to improve the design of the learning experience. It can determine whether the objectives were met and to what extent. Facilitators can use evaluation data to determine whether the delivery and content were adequate and practical. Ultimately, evaluation helps to determine if any additions, changes, or deletions are required to improve the talent development program.

Available Evaluation Methods

Many evaluation methods are available, so you will want to establish a framework for defining the value of learning for your organization. Many frameworks target the four levels of evaluation introduced by Don Kirkpatrick to guide their process of collecting, analyzing, and reporting quantitative and qualitative data on key performance measures.

As a reminder, the Kirkpatrick four levels are:

• Level 1, Reaction: Learner attitudes toward the training opportunity, such as a satisfaction with involvement or what was learned.

• Level 2, Learning: Knowledge and skills learned, such as being able to state the best practices or for having new skills for tasks on the job.

• Level 3, Behavior: Changes in execution and implementation of skills learned and practiced on the job.

• Level 4, Results: Quantifiable results that demonstrate the impact that training has on the organization.

We all know that most organizations are good at measuring Levels 1 and 2. Unfortunately, most talent development programs seem to shy away from measuring Level 4. However, you may be surprised to learn that Level 4 results are actually the simplest and least resource-intensive to evaluate!

According to the Kirkpatrick model, if something is a true Level 4 result, it is important enough that someone is probably already measuring it. That means it is simply a matter of obtaining the data. A true Level 4 evaluation measures a combination of the organization’s purpose and mission, and it must be an organization-wide measure. The difficulty arises when trying to link specific learning events or training to the ultimate result (Kirkpatrick and Kirkpatrick 2015).

How to Get Started With Evaluation

You will be able to attribute your success to an excellent talent development program if you have a solid evaluation framework in place. The previous chapters mentioned the importance of having senior-level buy-in right from the start. They may not think about evaluating the talent development program immediately, but that doesn’t mean that you can’t open the discussion. As a conscientious employee, you have an obligation to do so.

Perhaps not immediately, but eventually, leadership will want a regular accounting that identifies the details and costs to develop employees, cost of various offerings, payroll, facilities, external suppliers, and so forth. To balance the cost, you will need evaluation to identify the benefits.

You have a chance to act as the conscience of the organization by ensuring that there is a maximum result for the investment in talent development. Senior leaders will increase their confidence in you and the talent development program when you commit to using metrics. You can get started by following a few guidelines:

• Start small. Don’t think that you need to measure everything. Go back to your discussions with senior leaders and figure out what the program’s purpose and strategic imperatives are. Then, identify a few leading indicators that you can use to measure. Start small and grow slowly to ensure that employees and processes are ready.

• Use the data. Once you have gathered the data, use it to drive the talent development program and business results. Success is dependent on demonstrating and quantifying the value in evaluation.

• Use qualitative data, too. The evaluations will also produce learner stories, examples, and feedback, which you can use to bring the data alive and to tell the talent development story.

• Use data positively. Ensure that everyone understands that evaluation is a positive part of the program. It should never be used in a punitive way. A proactive approach where you collaborate with leaders, supervisors, employees, mentors, trainers, external suppliers, and others demonstrates that you want to use the evaluation data to constantly improve the talent development program.

Getting started with evaluation is a big task, so don’t put it off too long! One of the easiest evaluation processes is a comparison. For example, think about how you can incorporate a pre- and post-evaluation strategy. The benefit is in interpreting results and showing the financial impact of the talent development program. It will help you to build a better business case for more learning.

Cost-Benefit Analysis

Eventually you will want to present a cost-benefit analysis to your senior leaders. This tool was created in the 1840s by a French economist and engineer, Jules Dupuit. The tool affords you the ability to weigh the pros and cons or benefits and costs of your talent development program. The three steps are:

1. Identify the costs. This should include monetary costs such as payroll, equipment, suppliers, materials, participants’ time away from the job, and so forth. It also includes nonmonetary costs, such as risks or productivity, that could affect the organization. You will need to assign a monetary value to each nonmonetary cost.

2. Identify benefits. It may be difficult to determine exact amounts because other environmental actions may also have helped to create the benefit. Benefit values include profits, increased production, enhanced services, improved employee engagement, and others. You also assign a monetary value to each.

3. Compare cost to benefit. As you compare costs and benefits, consider how long it will take for the benefits to repay the costs.

Use your calculations to make recommendations for adding to, reducing, or changing the talent development program. The tool can also be used to make improvements or to rally for a larger budget. My best advice? Don’t exaggerate.

Impact of Talent Development

When implemented thoughtfully, there is little doubt that talent development programs will benefit employees and the organization. Demonstrating impact isn’t easy, but it is important. It is difficult to segregate what occurred due to the talent development program from other effects, such as changes in the economy, changed regulations, or new competition. But this should not stop you from establishing measures that demonstrate the impact of talent development on employees and the organization.

Impact on Employees

We know that employees change jobs more frequently now than ever before and that the reasons are often lack of development and a poor supervisor. An effective talent development program can address that for sure. But what else? Additional benefits that you might not expect can result from a good talent development program. These unexpected, yet beneficial spinoffs indicate that employees are not only more satisfied with their jobs and the organization, but more motivated and committed. Gaining new skills and knowledge increases their self-esteem and morale. If they are more informed and feel like they are part of the organization, they’ll have more confidence in the organization’s future. This also leads them to see more options for growth and development within the organization.

A talent development program can have a positive effect. Employee satisfaction and engagement are important to both the employee and the organization.

Impact on the Organization

When organizations support the talent development department, all employees and all departments can do their part to help the organization reach its business goals and objectives. Like any other aspect of business—research, marketing, sales, manufacturing—talent development requires an investment. Organizations also expect a return on their investment. In this case, that return will be expected in terms of improved customer satisfaction, higher sales, improved productivity, an overall increase in the bottom line, or better regulation compliance.

To increase the positive impact on the organization, you’ll need to look beyond competence, beyond training, and at the entire system.

Competence, Commitment, Confidence

As talent development professionals, we tend to focus on skills and knowledge. While they are important, to have real impact for our organization, we need to go beyond them. Learners must leave every development opportunity with three accomplishments. We need to use the 3Cs—competence, commitment, and confidence—when evaluating learning and development opportunities to ensure we are delivering what our learners require.

The first one is easy and logical. We are supposed to improve our learners’ knowledge and skills. That’s competence. But if we do not also increase learners’ commitment to change and their confidence that they can change, they will not implement the new skills, performance will not change, and everything will remain the same. Developmental events must help participants return to the job site and put into practice the skills and knowledge that were delivered. When evaluating learning opportunities, ask:

• Is content delivered that ensures learners’ competence?

• Do learners have the commitment to implement what they learned?

• Do learners have confidence to be successful?

Is Training the Answer?

A lack of competence may not be the only reason performance is suffering. You also have to keep the six areas of human performance improvement (HPI) in mind. As a reminder, the areas that you can analyze include:

• Physical resources: The tools, technology, materials, and equipment performers need to do the job.

• Structure or process: The methods and steps that describe how work gets done in an organization, including the reporting relationships, incentives, and consequences.

• Information: How data are exchanged between people and machines to ensure information is consistent, accessible, timely, and in a useful format for performers—especially when used to make decisions.

• Knowledge or skill: Well-defined skills that are learned and practiced to perform a job; performers know how and when to use the skills.

• Motivation: Proper incentives to support the desired performance; the performer’s values aligned with the tasks.

• Ability or wellness: Ensuring the performer is free of emotional and physical problems.

Each area can have limitations, be absent, or be misused, which will ultimately affect employee performance—even with the required skills and knowledge present.

Look at the Entire System

We also need to look beyond talent development essentials and HPI. Organizations are systems—the individuals and the processes within the organization are dependent upon one another. Talent development—improving individuals’ capabilities and ensuring they have the right attitude—is important, of course. But as you continue to add elements to the talent development program, remember all the other factors that contribute to an employee’s ability to contribute to and affect the organization’s success.

From an evaluative perspective, begin to think more about what the organization needs in addition to what the individual needs to learn. What learning activities will build organizational capabilities, not just individual capability? How can coaching and mentoring broaden employees’ focus and help them consider how to develop to integrate with other functions in the organization? Is there a way to assess the performance of the team, not just individual contributors? When identifying high-potential talent, is it possible to assign them to the teams that are more critical to contribute to the organization? These ideas may seem far-fetched right now, but treating talent development from a systems perspective is not.

The story in the sidebar about Credicorp in Peru is a good example of how the employee development and organizational systems came together to make a major improvement.

Measuring Informal Learning

Having a learning culture is where it all starts. When employees recognize that they are responsible for their own development, they take learning more seriously and they are more likely to transfer what they’ve learned. When they learn the skills on the job, the results increase even more because there is no transfer, such as from a class. They needed help and they received the knowledge or skills immediately.

If transfer of learning is critical and informal learning is the way that most employees learn, can you evaluate informal learning? As a matter of fact, you can.

Informal learning activities, either learning from others or on the job, can be measured using tools similar to what you would use when measuring formal learning, such as questionnaires, appraisals, and opinion surveys. Platform-based social learning could be evaluated using the platform’s analytical capabilities. If coaching sessions are included as part of the program, coaches might lead participants through self-reflection exercises to determine what they’ve learned and how they are using their new skills and knowledge. You might also be able to track learners’ progress as they move around an organization. What do the patterns in the succession pipeline tell you?

Because evaluating informal learning usually requires different questions than formal learning, you’ll need to modify Kirkpatrick’s four-level framework. Here are some questions that provide a way to collect the data:

• What did employees learn informally?

• How did employees learn the informal content?

• How can employees receive recognition for their informal learning?

• What is the extent of participation in various informal learning activities?

• What is the extent of satisfaction with the resources used for informal learning?

• In what way does the organization benefit from the informal learning?

• Which informal learning efforts provide tangible benefits to the organization?

• How can the organization better support informal learning efforts? (Carliner 2012)

Best Practices

Your organization is coping with dizzying rates of change, shifting markets, unexpected competition, a young, inexperienced workforce, and probably a shortage of leaders. A talent development program can and does resolve many of these concerns. However, to be effective, it must be targeted and aligned with the organization’s strategy, and driven by the organization’s business requirements. Keeping that in mind is the “best practice” you can achieve.

That being said, there are a few other best practices you can keep in mind when evaluating your talent development program. Let’s look at some of them:

• Have a plan and work the plan. What you decide to evaluate in the beginning will not always work. Make sure you re-evaluate the plan at your annual review meeting. This is a learning experience for you, too.

• Keep evaluations short. When possible, make evaluations a part of the task if you want more input. If you ask for everything from learners, you will get nothing. Keep it short and easy.

• Think long term. Changes may not occur overnight, so be patient—learning is a long-term investment! While the benefits are not immediately obvious, they will be observable as employees become more knowledgeable and stick around. Commitment will build the organization’s reputation as an employer of choice.

• Focus your resources. Resist the temptation to measure everything, everywhere. Too many directions can be overwhelming. Although evaluation is important, it is not the only thing you need to do.

• Look for value everywhere. Talent development can be costly. However, its effect can’t always be translated into bottom-line dollars and cents. The result may be fewer customer complaints or an increase in sales. In these cases, tout good news stories or happy employees. After all, there are data that show the value in increased engagement.

• Look for small things. Evaluating boulders can be exciting, but it can also be risky. Focusing on pebbles will eventually add up to the kind of results you want to see.

• Evaluation is good. Do not use evaluation sheets as punishment—use them as a chance to improve. A facilitator who gets a poor evaluation needs development. An employee who fails a test needs development. A talent development program that is not living up to leadership’s expectations needs development.

Evaluation Now and What’s Next?

Evaluation gives you some insight into how effective talent development is today. The real value, however, is to use what you learn through evaluation today to create tomorrow. The data from evaluations—both quantitative and qualitative—provide a wealth of ideas for what you need to do next.

Questions to Explore

• How do you plan to transfer what you learned from senior leaders during your early interviews and meetings to the evaluation plan?

• What evaluation philosophy do you adhere to?

• What evaluation methodology might you experiment with?

• What specifically will demonstrate success the most?

• At this point, what do you think should be measured at the Level 4 category?

• What are the primary benefits of evaluation to employees? To managers? To the organization? To the talent development program?

• Several suggestions were given about how to get started with evaluation. Which one is most important to your organization?

• What would be included in your first cost-benefit analysis? Why?

• What are your thoughts about evaluating informal learning?

• How do you plan to approach the 3C conundrum: competence, commitment, and confidence?

• Looking at the entire organizational system as opposed to looking at improving individual competencies requires a holistic perspective. What are your thoughts, and how do you think they would align with your organization?

• What best practices would you add to the list?

 

Tools for Support

Questions That Turn Evaluation Into Action

This worksheet will help you determine how to make your evaluation more beneficial for your organization, managers, and talent development professionals and their department.

Organization

  What organizational strategies need talent development’s support? How can we measure improvement?

  What business functions have the most impact on the organization’s success? What can we measure to demonstrate impact?

  What roles are the most important to achieve the organization’s strategy? What measurements would demonstrate this?

  What general capabilities do all employees need to support the organization’s mission, vision, and values? How can we measure this?

Managers

  What core competencies are most critical to the organization’s success? How can we measure improvement?

  What competencies will employees need in the future to address key strategies and business goals? What should we measure?

  How can the talent development department help you be more successful as you develop your employees? What should be measured?

Talent Development Department and Professionals

  What needs to be measured to ensure your skills are enhanced?

  What does the talent development department need to do better? How can we measure improvement?

  How are the talent development department’s measures of success aligned with the organization’s measures of success? What could be improved?

Used with permission from ebb associates inc © 2017.

Comparisons of Learning Evaluation Methods

This summary helps you see the differences between the evaluation tools used most often by talent development professionals. Select an approach that will fit best with your program.

Used with permission from Downs (2015).

Evaluation Considerations for Formal Learning Events

Results are best when managers and supervisors are involved and aware of what formal learning events should produce. Use these questions as you consider how involved managers and supervisors are. Hopefully the discussion leads to additional valuable and informed conversations.

During the Formal Learning Event

• Did the pre-work and conversations ensure that participants were prepared as well as they needed to be?

• Did participants know why they were attending the training session?

• How well were participants able to connect what they do to appropriate business goals?

• Did participants know how they contribute to achieving organizational results?

• Are participants confused by mixed messages?

After the Formal Learning Event

• Are managers and supervisors involved in the follow-up as planned?

• Do participants know where they can receive support?

• Was coaching available as necessary?

• Are participants held accountable?

• Are managers and supervisors held accountable?

• How accurate was the measure of value?

Used with permission from Biech (2009).

Sample Evaluation Plan

Use this as a sample data collection design plan to present each program’s objectives, how they will be measured, where the data are found, when measurement will occur, and who is responsible.

These questions will help you focus on what is important to include in your plan:

• What is your organization’s philosophy about evaluation and measurement regarding employee development?

• What are the goals of the program?

• What expectations or leading indicators did you identify?

• What questions can you ask to clarify what your organization’s leaders want to accomplish?

 

Data Collection Approaches

The tool you select to collect data should reflect the way the data will be used, budget, people available, and how much data you anticipate collecting. Be sure to pilot test your chosen collection tool. Here are a few advantages and disadvantages of each, as well as things you need to think about.

Cost-Benefit Analysis Template

Use this tool to help you make a case for change or improvement of your talent development program, or just one aspect of it.

Informal Evaluation Worksheet

Measuring learning informally does not use traditional methods. To capture data, use a method such as those suggested by Saul Carliner.

What You Want To Evaluate	Methods Available To Evaluate It	Which Methods Suit Your Evaluation Needs And Why?
Informal learning at the individual level
Identify what workers learned	• Self-assessments • Process portfolios • Coaching interviews
Identify how workers learned it	• Process portfolios • Coaching
Recognize acquired competencies	• Employee education records • Skills assessments • Certifications • Badges
Informal learning across groups of workers
Determine the extent to which individual resources for informal learning were used	• Analytics • Compiling data from evaluations of individual learning efforts
Assess satisfaction with individual resources	• Surveys • Focus groups
Identify the impact of individual resources	• Rater systems • Specialized reports • Long-term studies

Used with permission from Carliner (2012).

References and Additional Resources

Anand, P. 2017. “Executive Dashboards to Win Over the C-Suite.” TD at Work. Alexandria, VA: ATD Press.

Biech, E. 2009. 10 Steps to Successful Training. Alexandria, VA: ASTD Press

Brinkerhoff, R. 2014. “The Growing Importance of Measuring Results and Impact.” Luminary Perspective From Section V in ASTD Handbook: The Definitive Reference for Training & Development, 2nd edition, edited by E. Biech. Alexandria, VA: ASTD Press.

Bucy, M., T. Fagan, B. Maraite, and C. Piaia. 2017. “Keeping Transformations on Target.” Our Insights, March. McKinsey & Company.

Carliner, S. 2012. Informal Learning Basics. Alexandria, VA: ASTD Press.

Downs, L. 2015. “Managing Learning Programs Step by Step.” TD at Work. Alexandria, VA: ATD Press.

Kirkpatrick, J., and W. Kirkpatrick. 2015. “The Four Levels of Evaluation—an Update.” TD at Work. Alexandria, VA: ATD Press.

———. 2016. Kirkpatrick’s Four Levels of Training Evaluation. Alexandria, VA: ATD Press.

Levenson, A. 2016. “Measuring and Maximizing the Impact of Talent Development.” TD at Work. Alexandria, VA: ATD Press.

Orr, J.E., I. Gochman, and M. McGowan. 2014. “Talent Strategy That Drives Business Strategy.” Los Angeles: Korn/Ferry Institute.

Patterson, T., S. Stawiski, K. Hannum, H. Champion, and H. Downs. 2017. Evaluating the Impact of Leadership Development, 2nd edition. Greensboro, NC: Center for Creative Leadership.

Phillips, P., and J. Phillips. 2016. Real World Training Evaluation: Navigating Common Constraints for Exceptional Results. Alexandria, VA: ATD Press.

———. 2017. The Business Case for Learning. Alexandria, VA: ATD Press; West Chester, PA: HRDQ Press.

Reinhold, D., T. Patterson, and P. Hegel. 2015. Make Learning Stick: Best Practices to Get the Most Out of Leadership Development. Center for Creative Leadership Whitepaper. Greensboro, NC.

Rothwell, W., A. Stopper, and A. Zaballero. 2015. “Measuring and Addressing Talent Gaps Globally.” TD at Work. Alexandria, VA: ATD Press.