Organizations in the United States spent approximately $134 billion on employee learning and development in 2007, according to ASTD's 2008 State of the Industry Report. Increasing the effectiveness of employee learning and development programs was cited as a number one or number two priority by 44 percent of organizations surveyed for the 2008 report. This means that companies may be spending billions on programs that do not achieve business results. Given this dismaying statistic, it is no surprise that training budgets are among the first to be cut during economic downturns.
If you are tasked with implementing training, you may find yourself in the position of evaluating instructional materials to ensure training program effectiveness. In this Infoline, you will learn what questions you should ask—and answer—to make certain the instructional materials and thus the training are effective and a worthwhile investment.
High-quality instructional materials
• add value by supporting the organization's strategy implementation
• are delivered in a way that maximizes the potential to achieve business results
• are structured to foster learners' ability to immediately apply what they learn to their work
• explain why it is in learners' best interest to learn and set expectations about what will happen during and after training
• contain content and activities that mirror the job as closely as possible
• are instructionally sound
• achieve results on the job.
Assess whether your instructional materials are effective by asking these 20 questions arranged by seven categories:
Value
1. Does the training program support strategy implementation?
2. Is training the right solution?
Delivery
3. Is the training program part of a process?
4. Does the delivery method promote training effectiveness?
Structure
5. Do the instructional materials present a single model for performance?
6. Does the course objective describe what learners will be able to do after training?
7. Do the lesson objectives describe what learners will be able to do by the end of each lesson?
8. Do the learning objectives describe what learners will be able to do by the end of each topic within a lesson?
9. Is there a clear path from course objective to lesson objectives to learning objectives?
Expectations
10. Do the instructional materials answer the question, “What's in it for me?”
11. Do learners know what to expect before, during, and after training?
Content
12. Can you identify the learning objectives by looking at the course content?
13. Does the content explain not only what and how, but also why?
14. Does the content include examples, analogies, and anecdotes?
Activities
15. Do learners have the opportunity to actively participate at least 75 percent of the time?
16. Do learning activities support learners' ability to achieve one or more lesson objectives?
17. Do learning activities mirror the job as closely as possible?
18. Are learning activities debriefed so that learners understand their purpose?
19. Are there opportunities for learners to receive feedback?
Results
20. Does the training achieve results?
VALUE
1. Does the training program support strategy implementation?
Every training program should directly support the implementation of the organization's strategy. In Benjamin B. Tregoe and John W. Zimmerman's book, Top Management Strategy, strategy is defined as “the framework which guides those choices that determine the nature and direction of an organization.” For example, a company's strategy may be to be the market leader in technological innovation.
Once an organization has determined its strategy, it identifies the tactics or steps it will take to achieve that strategy. Depending on these tactics, employees may need to develop new knowledge and skills to
• solve current problems
• take advantage of future opportunities.
Problems can include issues such as an increase in customer complaints or employee turnover. Opportunities may include the introduction of a new product, service, or system; a reorganization of roles and responsibilities; or an improvement to a key process.
Case in Point: A coffee shop, Lava Java, had hand-roasted its gourmet coffee since its inception. With a surge in sales, though, it was unable to keep up with customer demand. Because its strategy was to be the market leader in sales of coffee drinks, the organization's leadership decided to automate the roasting process to increase production capacity. In response, the company rolled out a training program to teach the roasting artisans how to use the new systems and equipment. The training program made a direct contribution to the company's ability to implement its strategy.
2. Is training the right solution?
Another factor to evaluate is whether training is an appropriate solution. Just because employees do not have the knowledge or skills to demonstrate the desired performance does not mean that a full-blown training course is required. A checklist, information sheet, or job aid may be a more cost-effective way to get employees up-to-speed. This is especially true with a small number of learners. In this case, the cost per participant for a full course may be prohibitively expensive.
Along with deciding if training is necessary, you must decide if and to what extent you will evaluate the results of training. For more information, see the sidebar When to Evaluate.
DELIVERY
3. Is the training program part of a process?
People learn through repeated exposure, practice, and feedback. It is not possible to use a single training program to inject new knowledge and skills into employees. Thus, it is rarely appropriate to think of a training program as a stand-alone event.
Rather, a training program might consist of
• pre-work to identify strengths, areas for growth, and opportunities to use new knowledge and skills immediately after the course
• a training course delivered in the classroom, through online meeting software (such as WebEx), or as an e-learning module
• shoulder-to-shoulder coaching on the job that allows the employee to immediately apply new knowledge, practice new skills, and receive feedback targeted to improve performance.
Case in Point: A major retailer decided to implement a computer system that would help its distribution analysts (DAs) do a better job of ensuring the right items were in the right stores at the right time to meet customer demand. Before the training, DAs and their managers decided to use the system to plan the distribution for the next season's merchandise. Next, the DAs spent four days in the classroom learning the system. Afterward, they attended mandatory lab sessions where they were able to bring their work and get answers to any problems encountered back on the job. This process led to better use of the new system and a smarter distribution of products.
4. Does the delivery method promote training effectiveness?
There are many possible ways to deliver training: Classroom, online meeting software, and e-learning are just a few. Unfortunately, the delivery method often is chosen based more on expediency than effectiveness.
WHEN TO EVALUATE
In the oft-quoted words of management guru Tom Peters, “What gets measured, gets done.” This alone makes the case for why you should perform training evaluation beyond smile sheets, which gauge learner satisfaction. Evaluation can foster transfer of what was learned in the classroom to the job.
But, when should you measure? To find out if learners actually achieved learning objectives, evaluate their application of newly acquired knowledge and skills during the training. Ideally this measurement involves putting what they learned to work to solve a problem or perform a task based on several likely job-based scenarios. Tests that involve learners regurgitating information don't tell you whether learners know what to do and how to do it once they get back to their desks.
About three to six weeks after the training, it's time to see if learners have been successfully able to apply what they learned to their jobs. Make sure to ask how the instructional materials could be improved to better support the transition from the classroom to the job.
For example, it may be more expedient to use an asynchronous e-learning module to train a global customer service team on how to handle difficult calls. However, this module is not as effective as role-playing difficult call situations with another human being, which gives learners an opportunity to practice managing emotions and responding in ways that are ultimately helpful to the customer.
Another consideration is your organization's culture. In some organizations, employees find it difficult (or impossible) to make time to complete self-paced training because of the demands of their work. In this situation, even the most effective self-paced training program (e-learning or printed workbooks) may end up gathering dust because employees never have time to use it.
Case in Point: All American Insurance needed to train new employees on basic insurance terminology and concepts. The instructional design team decided that an asynchronous e-learning module would be the best delivery method because the information was straightforward and stable.
The company spent close to $100,000 on the design and development of the program. The corporate culture, though, was collegial. The employees enjoyed collaborating with one another in both learning and work. The e-learning program was seen as stale and dull; as a result, it had a disappointing completion rate.
In addition to delivery, another important factor is the look of the learning materials. See the sidebar, Looks Count.
STRUCTURE
5. Do the instructional materials present a single model for performance?
A model for performance consists of the steps employees would take to perform the task or solve the problem described in the course objective. For example, the model might include the six steps to delegate work or forecast a budget, or the four components to check when troubleshooting equipment problems.
If the instructional materials contain multiple models—such as the six steps to delegate work, along with the four steps to provide feedback, and the three steps for successful coaching—learners will most likely leave the training more confused than when they arrived. Effective training programs give learners an opportunity to learn and master one skill set—delegating, for example—before they are introduced to a second.
Case in Point: ABC Construction wanted to teach its project managers the ins and outs of good project management. Over the last year, the company had experienced an unacceptable number of delays and costly scope changes.
Rather than create a single course that included everything but the kitchen sink, the company hired an instructional design firm to create a project management curriculum. Each course within the curriculum focused on a single topic such as project scope, budget, schedule, and quality within a single phase of a typical construction project.
As a result, employees had the opportunity to learn the content for each phase before moving on to the next phase in the next course. In addition, the curriculum included repetition of key concepts. This helped employees remember what they learned so they could successfully apply it to their jobs.
Another pitfall is training that doesn't present a model at all. In this case, information is presented topically. The drawback to a topical presentation is that it doesn't mirror the job. This makes it very difficult for employees to apply what they learned.
Case in Point: Worldwide Cellular was getting ready to roll out a new cellular service—World Power. World Power was superior in every way to older services and to that offered by the competition.
Instructional designers at Worldwide Cellular were tasked with developing instructional materials for the sales and customer service teams. The materials covered the following topics:
• history of World Power
• World Power technology
• features
• benefits
• World Power vs. the competition.
Unfortunately, the information was not presented in a way that made it easy for salespeople and customer support representatives to answer early adopters' questions. The training provided no process for the employees to follow. So, for the first few months after the World Power rollout, employees struggled to answer even the simplest technical questions. Ultimately, they were able to learn much of what they needed to know on the job, but, unfortunately, not before Worldwide Cellular posted disappointing results for the first quarter.
Finally, this single model for performance should be based on real—versus anticipated—best practices. For example, process improvements should be tested and proven to work before the entire organization is trained on them. Often this step is skipped in the rush to fix a problem. This course of action can lead to unanticipated problems that often are as costly as the original problem. When there isn't enough time to do it right, companies are frequently left having to do it again.
6. Does the course objective describe what learners will be able to do after training?
Each training program should have a course objective that describes what learners will be able to do after the training. In other words, the course objective describes the goal of the course. It also serves as the framework to guide decisions about what content to include and what learning activities to use to teach that content.
Ideally, the course objective should describe a task employees will be able to perform or a problem they will be able to solve as a result of the training. For example, employees might be expected to forecast a budget, enter a customer order, delegate work, or troubleshoot equipment problems as a result of the training.
LOOKS COUNT
You were probably told not to judge a book by its cover. While we can all appreciate the sentiment, in the world of training and development, the look of instructional materials really does matter.
High production values that include high-resolution images, crisp charts, lots of white space, and a lack of typographical errors can help you gain credibility. And, with cheap images plentifully available on sites such as istockphoto.com plus the ability of standard programs such as Microsoft PowerPoint to handle sophisticated designs, there really is no excuse for poor-looking instructional materials.
Case in Point: In a book on business math published by a top publisher, the author's ideas were innovative, interesting, and well supported and explained. Yet there were spelling mistakes and missing or duplicate words in various places throughout the text. It left the reader with a nagging feeling that the material lacked professionalism and possibly value. This is not a reason to dismiss the author's ideas. However, many people will not be generous in their assessment and acceptance of new information if this type of lack of professionalism is present in instructional materials.
Make sure the course objective is clear and concrete. It should describe a specific, observable, measurable performance. For example, if the course objective is to provide excellent customer service, what exactly does that mean? How will you know excellent customer service when you see it?
7. Do the lesson objectives describe what learners will be able to do by the end of each lesson?
A training course is made up of lessons. Each lesson should have a lesson objective that describes what learners will be able to do upon its completion. In other words, the lesson objective describes the goal of the lesson. As with course objectives, lesson objectives should guide your decisions about what content to include and what activities to use.
Ideally, each lesson will map to a step in the performance model that is presented in the training. So, if there are six steps to delegating, there should be six lessons, one for each step, plus an introduction and a summary lesson for a total of eight lessons.
Sample lesson objectives that map to the steps for using a new system might include
• Lesson 1: Log in to the system
• Lesson 2: Open the discrepancy database
• Lesson 3: Navigate within the database
• Lesson 4: Add incidents.
As you look at the lesson objectives, make sure that if learners are able to achieve them, they will be able to achieve your course objective. For example, if the course objective is to add incidents into a database, are all steps included in the lessons? Perhaps learners should search for similar incidents before they add a new incident to make sure there are no double entries. If this is true, there should be another lesson on searching the database before the current Lesson 4. This means that the steps in your performance model will need revising as well.
Like the course objective, lesson objectives should describe specific, observable, measurable job performance. This means objectives such as “understand the supply chain process” won't work because you cannot tell by observation what someone understands about the supply chain.
In addition, objectives should not describe learning activities, such as, “work through a simulation of the banking system”; or the instructor's performance, such as, “assist learners in the development of interview questions.” Neither describes specific, observable, measurable job performance by learners.
8. Do learning objectives describe what learners will be able to do by the end of each topic within a lesson?
Each lesson should have multiple learning objectives that directly link to the topics within the lesson. In other words, each learning objective should describe the goal of a particular topic contained in the lesson.
For example, the learning objectives for the lesson on logging in might include:
• Identify the type of incidents to add to the database.
• Explain the requirements for adding incidents to the database.
• Set up a password.
• Log in to the system.
Like lesson objectives, learning objectives should describe specific, observable, measurable job performance.
9. Is there a clear path from the course objective to lesson objectives to learning objectives?
There should be a clear path, like stepping stones, from the course objective to the lesson objectives to the learning objectives. In fact, simply by looking at the learning objectives you should be able to identify the lesson objective. And, simply by looking at the lesson objectives you should be able to identify the course objective.
EXPECTATIONS
10. Do the instructional materials answer the question, “What's in it for me?”
Answering the question, “What's in it for me?” provides participants with the motivation to learn and then apply what they learn to their jobs. If learners do not have a rational self-interest to learn, you can be sure that, while they may click through the e-learning module or sit through the two days of classroom training, they are actually absorbing very little.
In fact, this is the reason any training program should start with an explanation of how learners will benefit. Possible examples include:
• Your job will be easier.
• You'll be able to focus on the interesting, strategic work because boring, routine tasks will be automated.
• You'll be set up for success in your career.
• You'll be able to make a visible, meaningful contribution to the goals of the organization.
• You'll be able to stay safe or out of trouble.
• You'll be able to keep up with your peers.
Case in Point: The market intelligence team at ABC, Inc., suffered from a perpetual staffing shortage. This made it very difficult for them to respond to the requests of their internal customers—the sales team—who needed market intelligence data in a timely manner. It also made for a high-pressure environment and very long hours that contributed to the staffing shortage.
The department manager decided to offer training to the sales force on how to search the company's database to get answers to simple market intelligence questions. She figured that if her department could limit its focus to complicated requests, she would be able to serve her internal customers more quickly plus reduce the stress of the department's staff.
Unfortunately, the training did not answer the question, “What's in it for me?” for the sales force. As a result, the members of the sales force saw the training as a way to shift the work of the market intelligence department to them. The training was poorly attended, and the market intelligence department was left where it had started, only without the goodwill of the sales force.
11. Do learners know what to expect before, during, and after training?
The training should include an explanation of what learners can expect before, during, and after training. This helps learners prepare for learning and apply what they learn to their work.
Possible questions to answer include the following:
Pre-work
• What is it?
• How long will it take?
• Is it mandatory?
• What should I do with it? Do I need to bring it to class?
About the Training
• When is it?
• Where is it?
• How do I get there?
• How long will it take?
• Why do I need to go?
• What will I learn?
• How does this relate to what I am currently doing?
• What's in it for me? How will I benefit?
• Is it mandatory?
• What will we be doing?
• When are breaks and lunch?
• How will my workload be covered while I am in training?
After the Training
• What do I need to do when I get back to my desk?
• How will I know I am doing it right?
• Who can I turn to with questions or problems?
CONTENT
12. Can you identify the learning objectives by looking at the course content?
It can be tempting to include in a training program everything that learners might ever need to know. Unfortunately, this can result in information overload where learners leave the training feeling overwhelmed, confused, and frustrated. This makes it very hard, if not impossible, for them to apply what they learned to their jobs.
In fact, the number one question learners have when they finish training is, “What do I do when I get back to my desk?” That question should be answered very specifically by the lesson objectives and the supporting learning objectives. In fact, if you study the content of each lesson, you should be able to arrive at the learning objectives. If content is included that does not map to a specific learning objective, you either have nice-to-know information or you are missing a learning objective. Conversely, if you have a learning objective that is not supported by related content, you either have a nice-to-know objective or you are missing content.
13. Does the content explain not only what and how, but also why?
Content should cover not only what and how, but also why whenever possible. People are more likely to remember what and how to do something if they know why it is important. Knowing the importance of something makes it a priority. In school, teachers accomplish this by pointing out when something will be on the test. In training, trainers accomplish this by pointing out why something should be done a particular way to avoid a problem or optimize results.
14. Does the content include examples, analogies, and anecdotes?
Stories in the form of examples, analogies, and anecdotes help learners retain information and form compelling mental pictures that promote understanding, as well as explain complicated concepts. In fact, they act like Velcro by giving learners something interesting to which they can stick dry or complicated concepts, as well as bland facts, policies, and rules.
Case in Point: Jade Summers teaches a beginner ceramics class at the local community college. During the first class, she always tells the story of how, as a freshman in college, she inadvertently fired a piece of greenware (an unfired clay piece) that had not completely dried. Her greenware exploded, destroying the other students' pieces in the kiln and damaging the kiln itself. She has never forgotten her embarrassment from this incident, and her students never forget to check whether their greenware is completely dry before firing.
When evaluating materials, you may be faced with comparing courses against one another. For some helpful tips, see the sidebar Comparing Multiple Courses.
ACTIVITIES
15. Do learners have the opportunity to actively participate at least 75 percent of the time?
The more time learners spend actively engaged, the higher their level of mastery. This is because learning is about creating knowledge, not absorbing it. Are you more likely to bake the perfect soufflé after listening to a lecture on baking techniques or after you've had a chance to practice making and baking soufflés? Lectures and demonstrations can lay conceptual groundwork, but they do not substitute for hands-on experience.
Case in Point: You Can Learn It offers a range of computer classes to help its customers learn to use programs such as Microsoft Word, Excel, PowerPoint, and Project.
Jan Hoffman, a new project manager, took the introductory Microsoft Project course. The course was jam-packed with information, and the instructor clearly demonstrated each feature and function.
The problem for Jan was that there was very little time for practice. As a result, she ended up spending as much time reviewing the course book as she had spent in the class. In hindsight, Jan wished that she had just bought the book and skipped the time she had wasted in the class watching instructor demonstrations. From Jan's perspective, the class gave the instructor plenty of time to practice, while the students were left sitting on the sidelines.
In addition, it is critical that learners have multiple opportunities to practice applying new knowledge and skills in different ways. Unless the task is quite simple, it is rare that someone can actually learn how to do something by watching a demonstration and practicing a single time.
COMPARING MULTIPLE COURSES
Comparing multiple courses on the same topic can feel like comparing apples to oranges to grapes. They are all fruit, but distinctly different.
To make sure that you are evaluating the same categories of information for each course, set up a grid. List the 20 questions down the left side—one per row. Then, write the name of each course at the top of each column in the matrix.
Before you get started, star the questions that represent the most important criteria given your situation. This will give you a basis for prioritizing once you start comparing the courses.
Then, evaluate each course, one at a time, against the questions. Take notes as you discuss each question so you can remember why you answered the question the way that you did.
Once you have examined the courses on their own merits, compare them one to another. Refer to your notes and the priorities you set at the start of your evaluation.
Cost may be a part of your evaluation if you are planning to buy the instructional materials from a vendor. Remember, no matter how cheap the materials are, though, they are prohibitively expensive if they are not effective.
As a result, it is better to cover less content and spend more time practicing than to jam too much content into the session.
16. Do learning activities support learners' ability to achieve one or more lesson objectives?
The purpose of every single learning activity is to give learners the hands-on practice they need to achieve a specific learning objective. For example, if an objective is for learners to be able to clarify prospect objectives during a sales call, an appropriate learning activity might be a role-play that gives learners the chance to practice asking questions and restating answers to clarify prospect objectives. Note the explicit, direct tie between the two.
While this logic may seem obvious, some courses include activities strictly as a way to raise the fun factor for learners. There is nothing wrong with having fun as a part of learning. However, the learners' time is in short supply. In most organizations, work does not stop accumulating because someone is in training. As a result, it is critical that time in training be focused on achieving learning objectives, rather than having a good time.
17. Do learning activities mirror the job as closely as possible?
The purpose of training is to improve job performance. As a result, learning activities should look and feel as much like real work as is feasible. If possible, learners may even bring real work to training to give them a chance to apply new skills immediately. This can provide learners with the support they need to get over the rough spots of applying what they have learned in the neat world of training to the messy world of real life.
Case in Point: We Sell Everything Cheap wanted its merchants to be able to use weekly sales reports to make better decisions about which items to mark down and when to sell inventory in a more timely and profitable manner. The discount retail chain found too many items were being migrated off the retail floor into the backroom to make space for new inventory. Once in the backroom, these items were more likely to be damaged or disappear. Remaining inventory had to be sold at a deep discount through the chain's outlet stores or given to charity. This situation resulted in an unacceptable decrease in We Sell Everything Cheap's profit margin.
The training department developed a two-hour class that included several case studies to allow merchants to practice interpreting the numbers on reports and making decisions about markdowns. The class also included a hindsight activity where merchants could examine the results of previous decisions to see what happened to inventory in retrospect. These realistic activities allowed learners to immediately feel more comfortable and make better decisions on the job.
Learning activities also should mirror the job by pointing learners to resources that will be available on the job. Such resources could include online help, policies and procedures, reference guides, marketing collateral, and job aids, for example. Well-designed learning activities give learners an opportunity to practice using these resources to accomplish a task. Poorly designed learning activities allow learners to turn to the instructor as the source of all knowledge. Unfortunately, when they get back to their desks, the instructor won't be available to answer questions.
18. Are learning activities debriefed so that learners understand their purpose?
Some learning activities use a familiar analogy to introduce a new concept or topic, or to point out existing knowledge and skills that transfer to a new situation.
For example, learners may
• solve a murder mystery to uncover behaviors that contribute to a high-performing team
• construct a block tower to see what happens when a new member is introduced to the group
• assemble the components of a recipe to learn the importance of including all elements in a plan
• participate in a simulation to draw conclusions about how a system works.
These types of activities can be tricky. The analogy must be similar enough to the situation the training addresses that the instructor can get the point of the activity across. It is also extremely important that they be debriefed thoroughly and correctly, or learners will miss the point of the activity.
A good debrief includes the following questions:
• What happened?
• Why did we do this activity? How does it relate to the real world?
• What did you learn?
• How does what you learned relate to the real world?
These questions help learners see how what they experienced in the activity can be translated into lessons learned that apply on the job.
19. Are there opportunities for learners to receive feedback?
Bottom line: The point of learning activities is to give learners an opportunity to practice using new knowledge and skills. As a result, it is critical that they receive feedback. Without feedback, they will not have any way to know how they are doing and what, if any, adjustments to make.
For information on reviewing off-the-shelf courses, see the sidebar Evaluating Vendor Materials.
EVALUATING VENDOR MATERIALS
Sometimes it makes sense to purchase generic materials from a vendor. In this situation, you'll primarily use the questions on structure, content, and learning activities to evaluate the materials.
No doubt, some customization will be required so the materials fit your specific situation. For example, you may need to add or tweak learning activities to more realistically mirror the job employees in your organization perform. Or, you may need to think about how to answer the question, “What's in it for me?” for learners.
In addition, you may need to supplement the vendor's materials to make the entire experience effective. This may involve developing a post-implementation plan to provide employees support on the job. You also may also develop pre-work to help them prepare for the training and arrive ready to learn.
RESULTS
20. Does the training achieve results?
At a minimum, learners should know what to do and how to do it when they get back to their desks. With the support of their managers and supervisors, they should have the opportunity to apply what they learned in training to their jobs. Ideally, their application of these new skills and knowledge will translate into business results.
While it is not necessary for instructional materials to meet all of the criteria described, the more criteria they do meet, the more effective the training is likely to be.