7

Measuring Transformation, Not Just Participation

In an age where concepts like big data are front and center, L&D measures evolve by moving from measuring participation to measuring transformation. In the Track Action, select measures that indicate progress on the desired on-the-job behavior and KPI changes, as defined by the Change Action. This is how you will confirm that the learning cluster is working for everyone—learners, the business, and even L&D. With these data, L&D shifts from providing the standard backward-looking report card, to providing leading data that the business and L&D can use for continuing transformation.

With the Track Transformation of Everyone’s Results Action, C-level leaders stop asking the backward-looking question:

and start asking the forward-thinking question:

Today, whether we are talking about online or offline learning, we in L&D most often report learner usage and reaction. We measure how many people use a particular platform or have completed a particular course or class. We measure satisfaction through end-of-class “smile sheets” or the “Like” button next to online course offerings. Yet, as emphasized in the Change Action (in chapter 3), where you put your focus is where you are going to go. When it comes to measures, is focusing on learner reaction and usage where we want to go? If we tell our leaders and employees that this is what L&D contributes to the enterprise, will they appreciate and value our work?

By putting focus on usage and reaction, we are saying that what’s important is the volume of people who use our products and their feelings about the in-the-moment experience. Of course, after many years of seeing such data, this is exactly the data that organizational leaders now expect. But what do we learn about business impact from such metrics? In this Action, L&D shifts the focus to on-the-job transformation by reporting changes both in KPIs and in employee behavior and attitudes. By doing so, we share a fuller story about what’s important: sustained performance improvement and collecting forward-looking data that will help shape future learning initiatives.

In this chapter, we will first look at a different approach to L&D measurement, an approach that tracks transformation for everyone—learners, the business, and L&D. Then we’ll discuss the Track Tool, which helps break down the measurement selection process, step by step. Finally, we’ll describe changes in our measurement approach that can shape the modern learning approach.

The Action Explained

Recall from the Change Action in chapter 3 how we in L&D have been constricted in defining our goals because we’ve limited ourselves to what can be accomplished by the end of a course. While L&D has long attempted to measure business impact and ROI, it has been difficult to measure. After all, how would anyone know if one particular learning asset had sustained impact on performance on the job? Trying to isolate the impact of a single program is one reason many L&D teams give up on tracking transformation. With the OK-LCD model, it’s time to make another go of it. The Track Action is about new measurement thinking, made possible with learning clusters; using measures to report L&D’s impact; measuring the combined effect of multiple learning assets; and choosing measures that are relevant for the three learning touchpoints.

The New Possibility With Learning Clusters

By delivering learning clusters with learning assets that meet the many moments of learning need, we have shortened the distance between L&D’s work and employee application on the job. This reduces the unpredictability of what happens after learners leave the classroom (Figure 7-1). Now, with the broader scope of learning clusters, we can have more confidence that L&D products are affecting performance on the job. However, by increasing the number of learning assets in our product line, the number of possible measures also increases. We need to be intentional about what measures we track so that we can report an accurate, meaningful picture of the impact of a learning cluster without being buried in data.

Figure 7-1. Reducing the Unpredictability of Measuring Impact

The Track Action is about selecting measures for a learning cluster that track the transformation we set out to achieve, as captured in the strategic performance objective. To track transformation in an impactful way, consider the main reasons measures are important. Powerful measures can:

• Articulate meaningful targets so learners, L&D, and the business know the “finish line” that they are currently shooting for (KPIs, on-the-job behaviors).

• Demonstrate value that invites further support through funding and resources.

• Spread awareness of the learning cluster and its availability to learners.

• Provide internal feedback so L&D knows what is working for learners and what needs improvement.

There’s no doubt metrics are important. Now we need to figure out how to make them work for us.

Emphasizing Metrics That Show and Tell Transformation

One of the key principles behind the OK-LCD model is to change on-the-job behavior. When starting on the Track Action, review the Change Action, which describes the desired behaviors and business metrics for the learning cluster. For those of you familiar with Kirkpatrick’s Levels of Evaluation (see the sidebar for a quick refresh), you might notice the consistency between OK-LCD philosophy on measures and the Kirkpatrick Levels. In our experience, L&D struggles for resources because the value isn’t clear. That’s why our emphasis in the Track Action is to measure and report Kirkpatrick’s Level 3 and 4. These are the data that the business cares about the most!

But our measures may fall short if we don’t build awareness. First, learners must be aware that the learning cluster exists. Be sure to create a marketing plan or system so that employees will use your products. Confirm that your plan or system works by measuring awareness of the learning cluster. Next, ensure that users of your products know that the product comes to them through L&D. Every learning cluster, every learning asset, and every L&D system should have L&D’s logo or brandmark on it. L&D can then include metrics like Net Promoter Score (NPS) to gain the customer perspective on our L&D products.

NPS is based on research by Frederick Reichheld and Bain, which found that a single question is most effective in understanding customer loyalty: “How likely are you to recommend xyz to another?” (Reichheld 2003). While NPS doesn’t tell you why or what to do about unsatisfactory scores, high NPS scores indicate viral enthusiasm about a product, so much so that many would be willing to stake their personal reputation on the value of a product. This is an idea worth reapplying for our learning clusters. Just by reporting good NPS, we gain traction and awareness that can help spread transformation.

Finally, to meet the need of feedback for improvement to L&D, and to meet base expectations of measurement, Level 1 and 2 should still be tracked. However, the primary focus is on metrics that reflect performance change across the target learner groups, and on the resulting business impact.

Measuring the Combined Effect of Multiple Assets

When it comes to measurement, a key difference between the OK-LCD model and past models is that the Track Action considers the impact of multiple learning assets that are organized into a learning cluster. This is something very new for L&D. Existing methodologies are strictly focused on evaluating one learning asset, and, in practice, only formal learning assets. OK-LCD practitioners measure Level 3 and 4—Behavior and Results—at the overall learning cluster level, instead of the individual learning asset level. This is absolutely critical.

In the past, L&D’s success or failure was often reliant on usage of a single learning asset. If people were not using that asset, it might be removed, updated, or left collecting dust. In the learning cluster world, however, L&D designs assets knowing that everyone will not use each asset equally! That was intentional, as part of our work in the Learn Action. Now, instead of seeing low usage as a failure on the part of L&D, we celebrate the ability to meet nuanced learner needs by reporting the success of the full learning cluster instead. But, we can still remove assets if they aren’t working. Metrics like NPS and Kirkpatrick Levels 1 and 2 tell us if an asset is meeting a persona need or not, and if its inclusion in the learning cluster needs to be reviewed.

Choosing the Right Metric for Each Learning Touchpoint

This Action highlights how Reaction, Learning, and Behavior measures may manifest differently depending on if the learning asset is a social, formal, or immediate learning touchpoint. For example, we know from the neuroscience of learning and the AGES model that emotion has a lot to do with learning. Consider how the emotions of a learner might be different during a coaching call with a peer versus covering the same topic in an e-learning course. When it comes to learning assets that belong to the three different learning touchpoints, what you measure and how you measure it could look very different. As you measure these differences, you will discover what works well for your learner personas to drive the desired behavior.

Here’s an example. Consider a YouTube-like video as an immediate learning asset. How could we evaluate its effectiveness at the Application level? The metric could be as simple as asking if the learner was able to complete the task at hand after watching the YouTube video. Alternatively, you could measure whether the learner had to seek out additional materials to achieve success. In this case, one metric could be to directly ask the learner: “Was this video all you needed to complete your task?” Another method would be to use background analytics to indirectly measure and track how many people stop the video early, or move on to another asset, instead of completing the video and getting back to the work task at hand.

This same how-to situation could look very different in a social or formal learning situation. For example, at the end of a class, you might only be able to measure the Learning level. You might use a knowledge check or a practice simulation to see if learners can complete the task on their own. The metric is simply reporting how many people pass the knowledge check or complete the simulation. An alternative for in-class activities is to ask the trainer to complete an after-class evaluation. Ask such questions as, “In your opinion, did at least 80 percent of participants demonstrate competency during the in-class activity?”

Finally, look for measures that track L&D efficiency and effectiveness. For example, a simple number-of-hits measure on a reading asset can provide clues about user preferences, as well as needs for L&D to increase awareness that a learning asset exists. If no one is clicking a learning asset, and you have confirmed awareness through other means, maybe it’s time for L&D to reduce their own workload by eliminating assets like this. A caution: Don’t drop a learning asset type just because only 10 percent of learners use an asset. It’s highly possible that those 10 percent of learners really value and need this asset. Keep this in mind as you design your measures so you can get the full picture of what your learners want and need to enable them to learn and behave in a way that drives results.

When you complete this fifth and final Action, you will have selected the measures and systems you will be tracking, so that you can begin to Track the Transformation of Everyone’s Results: L&D, your learners, and your business!

The Action’s Impact

The Track Action is at the very right side of the model because it is the last Action you take before rolling out your learning cluster to the organization despite the fact that you’ve been considering these measures during the Change and Surround Actions. It’s also an ongoing Action that provides feedback and clues to improving the previous four Actions over time, as indicated by the feedback arrow in the model graphic.

Many organizations do not track results, relying instead on intuition, gut feelings, and historical procedures as they design their training programs. Other organizations track Level 1 to determine if learners like a training, based on the idea that if they like it, they are more likely to be learning from it. Others use Level 2 evaluations to prove that participants learned something. But what is the value to the company if participants like it, learn from it, and yet don’t do it? L&D can do better. We can prove to ourselves and others that our training drives behavior that drives results. And what if it doesn’t? Well, that’s learning, too. And who would want to spend their career building things that don’t work? Let’s dig into the measures, find out what drives behavior and results, and go build those things (Figure 7-2).

Figure 7-2. Summary of Track Action Mindset Shifts

The Action Implemented

The Track Tool can help you collect your learning from other Actions and convert it into strong measures that demonstrate the impact of L&D’s product—the learning cluster. In this section we will walk you through the nuances of the three-step tool (Figure 7-3).

Figure 7-3. Track Tool

Learning Cluster Name: ______________________________________________________

Learning Cluster Strategic Performance Objective:

Step 1: Select Measures for the Overall Learning Cluster

The purpose of this set of measures is to determine if the approach is working. Can you identify a trend as you build a critical mass of employees who are changing their behavior such that the KPIs improve?

Step 2: Measures for Key Learning Assets

The purpose of this set of measures is to determine if L&D is meeting learner needs and to identify assets whose metrics share a story that gets attention.

Step 3: Select a Few Measures to Share

1.

2.

3.

4.

5.

There are three main steps for the Track Tool. The first is to consider how to measure the learning cluster by reviewing the strategic performance objective developed in the Change Action. The second is to identify different methods and metrics for key learning assets, in part by thinking through differences in learning touchpoints. Finally, make strategic choices to narrow down the measures and metrics, and from this reduced list of measures, select a handful that the L&D team would like to report to leadership for this learning cluster.

Step1: Review the Strategic Performance Objective

Identify what is most important to measure for the learning cluster by reviewing your strategic performance objective. If you did a thorough job on the Change Action, you will see that the KPI and behaviors portions outline “what to measure” for the performance of the overall learning cluster. Identify measures of the behavior change that stakeholders expected. Then determine how to get these data. You will likely need to partner with other organizations to get them. Such behavior change measures may be a valuable early predictor that the KPI is about to improve.

Step 2: Review Each Learning Asset

What learning touchpoints do the learning assets fall under? List ways you can get Level 1 Reaction evaluations for each learning asset, with a focus on gathering feedback for asset improvement. Here are some ideas:

• Videos: Include a thumbs-up/thumbs-down feature with a comment box. Precede a comment box with a question, such as “Was this helpful?”

• Job Aids: On interactive PDF links, include a “Was this helpful? yes/no” question with an open box for inputting “Why or why not?”

• Peer Coaching or Mentoring: Have a quick two- or three-question evaluation sheet each party fills out at the end of an interaction. For virtual meetings, this could be a part of a mobile app or online social community tool such as Microsoft Teams.

• Any Asset: Measure NPS by simply asking, “Would you recommend this learning asset to others?”

Then go beyond Level 1 by thinking about how learners can demonstrate growth in performance for the learning assets. For Social methods, one way is to let others—perhaps it’s the SME or a peer of the learner, or the manager—remark on the growth in performance. Think of ways to capture such remarks within your context. For assets in the immediate learning touchpoint, consider such things as self-evaluation or self-reflection on performance using an L&D-provided rubric. For formal learning touchpoints, try using face-to-face demonstrations of competence.

Consistent qualitative data are important in demonstrating Level 3 Behavior and Level 4 Results. For example, if a learning cluster is on a culture change, and a learning asset is a discussion forum, you could create a weekly word cloud of the forum responses to see the nature of the language changes over time. Imagine how much more powerful these data would be to stakeholders in comparison with simply reporting the number of posts in the forum. This is where emerging technology like big data can really help. (See the Future Technology sidebar to learn more.)

Step 3: Make a Selection

Finally, look through all the methods and metrics you’ve planned to track. Which ones do you think are most likely to communicate with stakeholders and demonstrate the impact of L&D’s work toward the goals of the strategic performance objective? Remember, people pay attention to what you share. Here, you really want to create a separation between metrics that are important for you in L&D, and those that are important for sharing with stakeholders and employees in order to gain sponsorship and awareness of your contributions to their success.

Level 1 evaluation metrics typically become a part of L&D’s ongoing improvement strategy, but this type of measure does not necessarily add value for stakeholders and employees. Be careful that you do not expend all your resources responding to Level 1 suggestions for improvements. Instead, step back and look at the bigger picture, selecting Level 3 and 4 evaluation metrics that demonstrate to stakeholders and employees the impact and value of L&D. In your reports and scorecards, always include a measure on the impact of the learning cluster.

The Track Tool represents our current approach to measuring the impact of learning clusters. It’s a framework to get you thinking about how different learning assets could require different measures, and how to look at the combined impact of multiple assets on performance change. We encourage you to dig deeper into the topic of measurement, possibly by exploring other proven processes to identify metrics such as:

• Phillips’ ROI Methodology, which is used by more than 5,000 organizations around the world, systemizes and expands on Kirkpatrick’s Levels of Evaluation work. The basis of the methodology is to collect data for a single learning asset at each level of evaluation: efficiency/cost, reaction, learning, application, impact, and ROI (where ROI is strictly a monetary value).

• Basarab’s Predictive Evaluation model, which centers on identifying the intended goals of a learning asset and the behaviors that need to be adopted by learners. Then, the L&D team strives to predict the percent of learners who will buy into the intended goal and behaviors, thereby creating the ability to calculate the impact on the business.

As the L&D industry moves forward, we need to identify ways to modify these rigorous methods to measure the combined effects of multiple learning assets. If you know of a measurement method, or have ideas to share, please reach out and let us know! Sharing among our own L&D networks is a great way for all of us to continue to build respect for our profession. Then, check back on LearningClusterDesign.com to see which ideas are gaining traction in our industry.

Final Note

In this chapter, we shared the new possibilities for evaluating your work based on the new job of taking responsibility for behavior change in the workplace, not just meeting the end-of-class objectives. As the L&D industry shifts into developing and delivering learning clusters, we can be confident of our impact back on the job and on business KPIs and ROI. We talked about how to address the unique challenges of measuring the impact of multiple learning assets and how each learning touchpoint calls for different methods and metrics of measurement. We walked through the tool to help guide your work in this Track Action. To support you further, consider using some of the learning assets listed here to supplement or reinforce what you have gained from this chapter:

• The Track Tool. We’ve included the most recent version for you in the appendix. For future evolutions, head to LearningClusterDesign.com/Book-Bonus.

• The OK-LCD Learning Cluster (see chapter 9) that we designed to help you grow your capability to use the model in your workplace.

• The chapter 8 example of the OK-LCD model with the five Actions and associated Tools in practice.

• The chapters on the other Actions in the OK-LCD model, so that you can see how each Action supports the others.

Ultimately, this fifth and final Action puts your focus on owning the story stakeholders and employees will hear about successful learning, performance improvement, and your contribution to everyone’s results. Because of the core principles of multiple assets and targeting behavior change on the job, the OK-LCD model opens new doors for L&D to gain sponsorship, meaningful insights, and perception as an essential culture-builder in the organization. Rather than focusing only on attendance, usage, and satisfaction, the tool in this final Action encourages you to tell a fuller, more accurate picture of the power of your learning clusters.