How many times have you heard someone say, “It was a great training program but…”? Unfortunately, that statement (or a variation) is spoken far too often and reflects a growing concern by both line managers and senior management that training is costly and not always worth the investment of time and money. This attitude is also reflected in the actions of senior managers who cut training budgets first when times are tough.
Following are some of the main reasons to evaluate:
Clearly, needs assessment is critical to the success of any training initiative. It provides the basis for program development and establishes the criteria for measuring the success of the program after its completion. For evaluation to have any meaning, it must be tied to the needs-assessment process. Always remember that the evaluation process should reflect specific business-related or performance-related outcomes.
Evaluation is an ongoing process, not just something that happens at the end of a session or program. Consider evaluation during the session, at the end, and after the participants return to the job.
As people participate in skill practice, case studies, exercises, simulations, and other activities, observe the degree to which they have mastered the content.
Participants' evaluation questionnaires will indicate their personal reactions to the training session.
A few weeks to several months after the session, observe the participants' job performance to determine whether they are applying what they learned to their work situations.
Any category or number of people can be involved in the evaluation process, depending on what you want to know and what level of evaluation you conduct. The participants are directly involved because they are the customers whose level of satisfaction you are trying to determine. The participants are the major source of information on how much they have actually learned in the training. In addition to the participants, involve others in the evaluation process. Survey or interview the participants' managers, their co-workers, peers, subordinates, or even customers or vendors. At some point, you may need to involve senior management.
The methods used for evaluation are pretty much the same as those used to gather data during the needs assessment. Trainers will often ask, “Which method(s) should I use?” Unfortunately, there is no clear-cut answer. Use whatever method that gives you the information you need.
Before developing an evaluation process, be clear about what you want to evaluate. This is not as easy as it might seem. Do you want to evaluate how much the participants have learned? Or do you want to know what the participants thought about the program and you? Do you want to find out whether the participants are applying what they learned on the job? If they are using the learned skills and information on the job, is their enhanced performance making a difference to the organization?
The most widely known model for evaluating training programs was introduced by Donald Kirkpatrick in 1959. It is regarded as a classic by training practitioners. Although all four levels of the model (reaction, learning, behavior, results) are important, you may choose not to evaluate at all four levels. Studies show that a vast majority of organizations evaluate reaction. A significantly high percentage measure learning as well. The evaluation or measurement of behavior lags behind the first two levels; evaluation of results finishes last.
Today's organizations are much more cost-conscious, and the need to measure the effectiveness of training will continue to grow. Should you need to undertake a comprehensive approach to evaluation, you will be able to make appropriate recommendations or respond confidently when someone asks you to prove that training gets results.
Table 12.1 provides an overview of the four levels of evaluation.
Table 12.1 Measuring Training Results
What | Who | When | How | Why | |
Level 1 | Reaction: Did they like it? | Participants | End of program | “Smile sheet” | Determine level of customer satisfaction; may indicate need for revision |
Level 2 | Learning: What knowledge or skills did they retain? | Participants; trainer | During, before/after program | Pretest/posttest; skills application through role plays, case studies, exercises | Identify whether trainer has been successful in delivery of course content and achieving program objectives |
Level 3 | Behavior: How are they performing differently? | Participants; bosses; subordinates; peers | 3 to 6 months after program | Surveys; interviews; observation; performance appraisal | Determine extent to which participants have transferred what they learned in the session to the actual work situation |
Level 4 | Results: What is the impact on the bottom line? | Participants; control group | After completion of Level 3 follow-up | Cost/benefit analysis; tracking; operational data | Determine whether benefits outweigh costs; ascertain degree of contribution of program to organizational goals |
The Trainer's Handbook, 4th Ed. Copyright © 2016 by Karen Lawson. Reproduced by permission of Pfeiffer, an imprint of John Wiley & Sons, Inc.
Level 1 deals with participants' reactions, that is, “customer” satisfaction. Level 1 evaluations are often referred to as “smile sheets,” implying that participants' reactions are based on how much “fun” they had in the training session. For that reason, trainers frequently dismiss Level 1 evaluations as a waste of time.
On the contrary, Level 1 is an important first step in determining the success of a training program. Participants' reactions can help you determine the effectiveness of a program and how it can be improved. Kirkpatrick believes that you cannot bypass this first level because, as he puts it, “If they [participants] do not react favorably, they will not be motivated to learn” (Kirkpatrick, 1994).
One of the problems with and the main cause of criticism of Level 1 evaluation is that it is too subjective and often becomes nothing more than a popularity contest. Before constructing a participant end-of-session evaluation form, understand what it cannot and is not intended to do: (1) it does not measure learning or the ability to apply learning on the job; (2) it also cannot measure changes in attitudes or beliefs; (3) because it deals only with participants' perceptions and reactions, a Level 1 instrument can in no way measure organizational impact; (4) also, although frequently asked, participants cannot measure the trainer's knowledge. Think about it. How could the participants have any way of knowing what the trainer does and does not know about the subject? Your ability to communicate or demonstrate your knowledge is an entirely different story.
Before you design a Level 1 instrument, you need to be clear about what you want to know, why you want to know it, and what you are going to do with the information. Do not ask for information about something you cannot change or have no intention of analyzing or reporting.
Categories. First decide what you want to measure and create questions or response items that address or fall into certain categories, including many, if not all, of the following:
It is also a good idea to provide an opportunity for respondents to make recommendations as to how the program can be improved and also to express their overall reactions to the session.
Format. To counteract people's tendency to respond the same way to every item on a questionnaire or survey, use a variety of response formats. Choose at least four from the following options:
A question could also be added focusing on the impact of the session on the participant, designed to obtain a deeper and more personal response, for example: “Imagine that a co-worker(or friend) of yours is thinking about attending this program. He or she asks you: ‘What was this program like for you?’ How would you respond?”
Evaluation forms are more difficult to construct than you might imagine. Use the following guidelines:
In addition to the end-of-session questionnaires, you can use interviews to increase the reliability of the data collected from the questionnaires. This method of data collection is quite flexible, allowing the interviewer to probe for more specific answers and to clarify questions as needed. The method also allows the interviewer to record spontaneous answers and, therefore, get a more complete picture of the participants' reactions. The interviewer can explore in more detail the reactions gleaned from the questionnaires.
Plan on spending about 30 minutes per interview. You will not be able to interview every participant, so select a random sample. It is important to hold the interviews within one week of the session so that the experience is fresh in their minds. Through one-on-one interviews, you can further explore the reasons for participants' reactions and solicit suggestions for improvement. Either record the interviews and have them transcribed, allowing you to analyze or interpret the responses more thoroughly, or simply take notes during the interviews.
When developing the interview questions, do not duplicate the questions on the written form. Instead, ask specific questions about the methods used or the content covered. For example, below are several questions about the methods used in a training session on leadership:
Level 2 evaluation deals with what the participants actually learned during the training session. Kirkpatrick defines learning as the “extent to which participants change attitudes, improve knowledge, and/or increase skill as a result of attending the program” (Kirkpatrick, 1994). It is far easier to determine what new knowledge or skills the participants acquired than to find the ways in which the training changed their opinions, values, and beliefs.
The three most appropriate methods used to evaluate learning are tests, observation, and interviews, with tests being the most frequent. See Table 12.2 for a summary of the advantages and disadvantages of each method.
Table 12.2 Level 2 Evaluation Methods
Method | Advantages | Disadvantages |
Tests: Objectives tests Multiple choice Matching True-false Fill-in Subjective tests Essay Short answers |
Easy to score Inexpensive to use Easy to write Inexpensive to create |
Difficult to write Time-consuming to write Grading is time-consuming Expensive to grade |
Observation: Observe behavior in class Skills demonstrated in skills practices and learning activity |
Immediate application Lends itself to on-the-spot coaching and feedback |
Subjective, open to interpretation Unable to spend adequate time observing behavior of all participants |
Interviews: Individual interviews conducted shortly after the training Random sampling of participants |
Can gather more information Instant feedback |
Time-consuming; expensive Must be tightly structured to obtain quantifiable responses |
The Trainer's Handbook, 4th Ed. Copyright © 2016 by Karen Lawson. Reproduced by permission of Pfeiffer, an imprint of John Wiley & Sons, Inc.
Testing should be kept fairly simple. Many trainers give both a pretest and a posttest to get an even more accurate picture of what the participants have learned.
Types of tests. First determine whether you want to construct subjective (short-answer or essay) or objective (multiple-choice or true-false) items or even a combination of the two types. When constructing test items, consider the time needed to grade the test as well as the validity and reliability of each item. Make sure the test assesses the learning as specified in the learning objectives. When an item measures what it is supposed to measure, it has validity. Each test item must also be reliable, that is, give consistent results from one application to another. For a broader discussion of test construction, refer to “Testing for Learning Outcomes” by Deborah Grafinger Hacker, one of the INFO-LINE booklets published by the American Society for Training and Development (revised in 1998). Hacker describes various types of tests, provides directions on writing test items, discusses measuring the validity and reliability of tests, and includes a helpful and extensive resource list.
Make sure the test is meaningful. Instead of asking for simple information or factual recall, ask questions that require the participants to apply or interpret what they learned in the session.
Question formats. All multiple-choice questions consist of a stem and a response. The stem presents a problem, asks a question, or takes the form of an incomplete statement. Responses include possible answers, all of which must be plausible. The greater the number of items, the better the test's reliability. Following are some formats to consider:
Test-writing guidelines. In most cases, you will probably choose to develop multiple-choice questions. They are easy to grade, but not necessarily easy to write. To help you construct a multiple-choice test that will provide valuable information about the participants' content mastery, consider the following guidelines:
To gauge retention of the information learned in the program, administer another test several months following the training.
Trainers can watch participants practicing and applying skills, tools, and techniques during the session. As the trainers observe participant behavior in skill practices, role plays, simulations, case studies, and other activities, they can get a good idea of what the trainees have really learned.
Shortly after the training, interview the participants and ask them what they learned in the session. Conduct the interviews within the week following the session.
The critical question answered by Level 3 is, “How has the training affected the way participants perform on the job?”
Although both managers and training professionals agree that the success of a training program is determined by what the participants do with the information or skills back on the job, these results are often ignored. Level 3 evaluation is both time-consuming and costly. It also requires good organizational and follow-up skills and processes.
Use follow-up evaluation to do the following:
Use these guidelines for follow-up evaluation:
The trainer or another designated observer can actually observe employees back on the job. Carefully watch employees as they perform their routine job tasks and responsibilities. To facilitate the process and ensure consistency in the data gathering, create a checklist of desired behaviors and then observe whether or not the employee is demonstrating these behaviors.
For example, if you are observing someone who has recently attended a customer service program, your list of customer service behaviors might look like this:
Not only should you interview those who went through the training, but you should interview those who are affected by or closely associated with the program participants. Possible interviewees include the participants' managers, co-workers, customers, or subordinates. The interview questions would have to be carefully constructed and designed to focus on specific applications and behavior changes.
Surveys are more efficient and less expensive than interviews to find out whether the participants are actually applying what they learned. Once again, do not limit your sources of information. Others who interact with those who participated in the training are often a more reliable source of feedback. You will want to know not only if trainees are using the training on the job but also how they are using it to perform better.
A control group is also helpful for a Level 3 evaluation. To validate your results, choose a control group of employees from the same function as the participants who received the training. Ask the members of the control group to complete the same surveys, questionnaires, and tests as those completed by the participants. Monitor their job performance and compare it with that of employees who received training.
Regardless of which evaluation method(s) you use, make sure you allow enough time for the behavior change to take place. The length of time depends on the program, but three to six months should give the participants ample opportunity to apply what they learned and develop new behaviors.
Level 4 evaluation determines the impact of the training on the organization. Ideally, it shows how the training has contributed to accomplishing organizational goals and objectives—business results. If an organization chooses to conduct a Level 4 evaluation, the area of measurement must be the same as those identified in the needs assessment.
To measure training's impact on the bottom line, return to the data gathered during the needs assessment. Determine your critical success factors up-front. Results measured could include any of the following or any measurable item:
Level 4 evaluation is both difficult and time-consuming. It can also be quite costly. It is difficult to measure because of the many variables that can come into play once the participants leave the session. For these reasons, a Level 4 evaluation is not appropriate for all training. From a practical standpoint, consider Level 4 for those programs that are near and dear to senior management and have been identified as a top priority.
Because of the complexity of Level 4 evaluation and its infrequency of use, it is addressed here only in very basic terms—just enough to give you an idea of what it involves and the purpose it serves. To explore Level 4 in more detail, refer to the listings in Appendix B, including Jack Phillips, Donald Kirkpatrick, and Dana Gaines Robinson and James Robinson.
As a starting point, test your current knowledge of measurement terms by matching the terms in the left-hand column below with the correct definitions in the right-hand column (see Appendix A for the correct answers):
Term | Definition |
_______ 1. Return on investment (ROI) | a. Survey data of employee attitudes |
_______ 2. Direct costs | b. Salaries and benefits paid to participants for the time they are attending the program |
_______ 3. Indirect costs | c. Expenses for creating the training program |
_______ 4. Overhead | d. Involves quantity, quality, cost, time; easily converted to monetary value |
_______ 5. Development costs | e. Costs of shared resources such as heating and building maintenance |
_______ 6. Hard data | f. Expenses of operating the training department that are not directly related to any training program |
_______ 7. Soft data | g. Mathematical formula for calculating the difference between cost and value |
_______ 8. Compensation for participants | h. Costs related to the delivery of the training program, such as materials and trainer's salary |
Unlike traditional evaluations, e-learning evaluations are somewhat difficult because of delivery options and individual learning solutions that often prevent consistency and uniformity in the evaluation process. For example, not all learners will complete self-study courses from start to finish. Some will choose only the modules they think they need. Others may have to go back and repeat a module to gain a greater understanding of its content.
For the most part, however, you can apply the same principles and strategies of traditional evaluation methods and levels discussed earlier in this chapter to the e-learning process. The basic difference, of course, is that the various evaluation methods will be done electronically. The following are types of evaluations that can be adapted to an e-learning environment.
As with any evaluation process, it is important to collaborate with the organization's leaders to determine the success criteria. What specifically do the key people want to measure and how will they use the information? In addition to the basic questions addressed in the section on the four levels of evaluation, you may also be interested in evaluating the e-learning process by establishing ways to measure the following:
Throughout the business world, accountability for all functions is increasing. Staff functions such as training are expected to prove their contribution and value to the organization. Top executives are demanding that training departments offer proof of their worth or take budget cuts. Two approaches to measuring the value of training are covered below: (1) cost/benefit analysis and (2) return on investment.
The cost/benefit analysis looks at the total cost to produce a training program and attempts to quantify the benefits. Cost includes everything from the needs assessment, through design, development, delivery, and finally to follow-up. Both direct and indirect costs are used to determine the total cost of the program. As Table 12.3 shows, determining training costs is complex.
Table 12.3 Determining Training Costs
Direct Costs | |||
People | Facilities | Materials | |
Design and Development | Salaries, benefits, travel for: course development, clerical support or consultant fee and expenses or costs of certifying in-house trainer for purchased programs | Marketing brochures Participant materials Instructor manual Purchased resource materials Purchased program |
|
Delivery | Salaries, benefits, travel for: trainer (s), participants (average salary), clerical support, consultant fees and expenses | Room rental Equipment rental Refreshments |
Note books Folders Tent cards Paper Pencils/pens Flip charts Handouts Film rental/purchase Transparencies/slides Stationery items Certificates Books Articles reprints |
Direct Costs | |||
Evaluation | Salaries, benefits, travel for: training personnel, participants, clerical support, bosses, subordinates, peers |
Surveys Questionnaires |
|
Indirect Costs | |||
Training space Custodial services Utilities Postage Telephone |
Computer time Equipment depreciation Equipment Maintenance/repair Support services |
The Trainer's Handbook, 4th Ed. Copyright © 2016 by Karen Lawson. Reproduced by permission of Pfeiffer, an imprint of John Wiley & Sons, Inc.
Total benefits of the program may be reduced costs or increased revenues directly attributable to the training. In many cases, however, benefits can only be estimated.
Subtract the total costs from the total benefits to find the net benefit of the training program. The program is considered a financial success if the costs are lower than the benefits.
The return on investment simply shows what the payback is for the training program. To determine return on investment, you must wait three to six months and even longer for operational results. The formula for determining return on investment is: ROI = (net program benefits/program costs) × 100. Stated simply, if training programs fail to show a reasonable return on the company's investment, future (or even current) training initiatives are at risk. For more information on determining ROI, refer to Jack Phillips's books listed in Appendix B.
Evaluation is a complex issue. For one reason, many variables enter into the equation. No matter how hard you try to fine-tune the evaluation process, the reality is that effects can only be estimated, and economic benefits cannot be calculated precisely. As the role of training continues to change and trainers reposition themselves as performance consultants, there will be more pressure to measure the effectiveness of training. The good news, however, is that the field of training and development continues to grow at a rapid pace. This trend will continue to provide many opportunities for training professionals, both internal and external, to make an impact on the growth and development of individuals and organizations throughout the world.
The training process is a never-ending cycle: analysis, design, development, implementation, and evaluation. Part I provided you with the basic information for you to create new training programs and revamp existing ones. As mentioned several times throughout this book, the role of the workplace learning and development professional continues to evolve. Part II provides information on specific considerations and applications that you may need to address now or in the future.
18.116.14.245