Chapter 12
Evaluating Training
Evaluation Basics
How many times have you heard someone say, “It was a great training program but…”? Unfortunately, that statement (or a variation) is spoken far too often and reflects a growing concern by both line managers and senior management that training is costly and not always worth the investment of time and money. This attitude is also reflected in the actions of senior managers who cut training budgets first when times are tough.
Purposes of Evaluation
Following are some of the main reasons to evaluate:
- To determine whether the training achieves its objectives
- To assess the value of training programs
- To identify areas of the program that need improvement
- To identify the appropriate audience for future programs
- To review and reinforce key program parts for participants
- To sell a program to management and participants
Linking Evaluation to the Needs Assessment
Clearly, needs assessment is critical to the success of any training initiative. It provides the basis for program development and establishes the criteria for measuring the success of the program after its completion. For evaluation to have any meaning, it must be tied to the needs-assessment process. Always remember that the evaluation process should reflect specific business-related or performance-related outcomes.
When to Evaluate
Evaluation is an ongoing process, not just something that happens at the end of a session or program. Consider evaluation during the session, at the end, and after the participants return to the job.
During the Session
As people participate in skill practice, case studies, exercises, simulations, and other activities, observe the degree to which they have mastered the content.
At the End of the Session
Participants' evaluation questionnaires will indicate their personal reactions to the training session.
After the Training
A few weeks to several months after the session, observe the participants' job performance to determine whether they are applying what they learned to their work situations.
Whom to Involve
Any category or number of people can be involved in the evaluation process, depending on what you want to know and what level of evaluation you conduct. The participants are directly involved because they are the customers whose level of satisfaction you are trying to determine. The participants are the major source of information on how much they have actually learned in the training. In addition to the participants, involve others in the evaluation process. Survey or interview the participants' managers, their co-workers, peers, subordinates, or even customers or vendors. At some point, you may need to involve senior management.
How to Evaluate
The methods used for evaluation are pretty much the same as those used to gather data during the needs assessment. Trainers will often ask, “Which method(s) should I use?” Unfortunately, there is no clear-cut answer. Use whatever method that gives you the information you need.
What to Evaluate
Before developing an evaluation process, be clear about what you want to evaluate. This is not as easy as it might seem. Do you want to evaluate how much the participants have learned? Or do you want to know what the participants thought about the program and you? Do you want to find out whether the participants are applying what they learned on the job? If they are using the learned skills and information on the job, is their enhanced performance making a difference to the organization?
Four-Level Model for Training Evaluation
The most widely known model for evaluating training programs was introduced by Donald Kirkpatrick in 1959. It is regarded as a classic by training practitioners. Although all four levels of the model (reaction, learning, behavior, results) are important, you may choose not to evaluate at all four levels. Studies show that a vast majority of organizations evaluate reaction. A significantly high percentage measure learning as well. The evaluation or measurement of behavior lags behind the first two levels; evaluation of results finishes last.
Today's organizations are much more cost-conscious, and the need to measure the effectiveness of training will continue to grow. Should you need to undertake a comprehensive approach to evaluation, you will be able to make appropriate recommendations or respond confidently when someone asks you to prove that training gets results.
Table 12.1 provides an overview of the four levels of evaluation.
Table 12.1 Measuring Training Results
| What | Who | When | How | Why | |
| Level 1 | Reaction: Did they like it? | Participants | End of program | “Smile sheet” | Determine level of customer satisfaction; may indicate need for revision |
| Level 2 | Learning: What knowledge or skills did they retain? | Participants; trainer | During, before/after program | Pretest/posttest; skills application through role plays, case studies, exercises | Identify whether trainer has been successful in delivery of course content and achieving program objectives |
| Level 3 | Behavior: How are they performing differently? | Participants; bosses; subordinates; peers | 3 to 6 months after program | Surveys; interviews; observation; performance appraisal | Determine extent to which participants have transferred what they learned in the session to the actual work situation |
| Level 4 | Results: What is the impact on the bottom line? | Participants; control group | After completion of Level 3 follow-up | Cost/benefit analysis; tracking; operational data | Determine whether benefits outweigh costs; ascertain degree of contribution of program to organizational goals |
The Trainer's Handbook, 4th Ed. Copyright © 2016 by Karen Lawson. Reproduced by permission of Pfeiffer, an imprint of John Wiley & Sons, Inc.
Level 1: Reaction
Level 1 deals with participants' reactions, that is, “customer” satisfaction. Level 1 evaluations are often referred to as “smile sheets,” implying that participants' reactions are based on how much “fun” they had in the training session. For that reason, trainers frequently dismiss Level 1 evaluations as a waste of time.
On the contrary, Level 1 is an important first step in determining the success of a training program. Participants' reactions can help you determine the effectiveness of a program and how it can be improved. Kirkpatrick believes that you cannot bypass this first level because, as he puts it, “If they [participants] do not react favorably, they will not be motivated to learn” (Kirkpatrick, 1994).
What Level 1 Cannot Measure
One of the problems with and the main cause of criticism of Level 1 evaluation is that it is too subjective and often becomes nothing more than a popularity contest. Before constructing a participant end-of-session evaluation form, understand what it cannot and is not intended to do: (1) it does not measure learning or the ability to apply learning on the job; (2) it also cannot measure changes in attitudes or beliefs; (3) because it deals only with participants' perceptions and reactions, a Level 1 instrument can in no way measure organizational impact; (4) also, although frequently asked, participants cannot measure the trainer's knowledge. Think about it. How could the participants have any way of knowing what the trainer does and does not know about the subject? Your ability to communicate or demonstrate your knowledge is an entirely different story.
Deciding What to Measure
Before you design a Level 1 instrument, you need to be clear about what you want to know, why you want to know it, and what you are going to do with the information. Do not ask for information about something you cannot change or have no intention of analyzing or reporting.
Designing an End-of-Session Evaluation Form
Categories. First decide what you want to measure and create questions or response items that address or fall into certain categories, including many, if not all, of the following:
- Content
- Materials
- Instructional methods
- Trainer
- Environment
- Logistics
It is also a good idea to provide an opportunity for respondents to make recommendations as to how the program can be improved and also to express their overall reactions to the session.
Format. To counteract people's tendency to respond the same way to every item on a questionnaire or survey, use a variety of response formats. Choose at least four from the following options:
- Two-choice questions with room for explanation or comments. These would include responses such as “yes” or “no” and “agree” or “disagree.” Example: Did the course meet the stated objectives? Yes, No, Why, or Why not?
- Short answers. These items are written as open-ended questions and require the respondent to write down a brief response instead of just checking a box. Example: What parts of the workshop were most valuable/beneficial to you? Why?
- Complete the sentence. With this item, the respondent is asked to complete a sentence. Example: What I want/need to know more about is…
- Ratings. Participants respond to a question or statement using some type of scale or rating such as a Likert scale. The Likert scale measures both the direction (positive to negative) and intensity (strongly positive to strongly negative) of an individual's opinion or attitude. Example: Today's session was an enjoyable and satisfying learning experience for me.
- 1 2 3 4 5 6 7 8
- Strongly Disagree Neutral Strongly Agree
- Rankings. This item asks respondents to indicate priorities or preferences. Example: Please rank each topic in order of its importance or relevance to your job: 1 = most important to 5 = least important.
- Checklist. A checklist provides a “laundry list” from which participants can choose words that express their reactions. Example: Check the words that describe your reaction to today's session:
Exceeded my expectations- Met my expectations
- Fell short of my expectations
A question could also be added focusing on the impact of the session on the participant, designed to obtain a deeper and more personal response, for example: “Imagine that a co-worker(or friend) of yours is thinking about attending this program. He or she asks you: ‘What was this program like for you?’ How would you respond?”
Evaluation Form Guidelines
Evaluation forms are more difficult to construct than you might imagine. Use the following guidelines:
- Keep the form brief. Participants should be able to complete it quickly.
- Create a balance among the various types of information you are collecting. For example, do not ask five questions about the instructor and only two about content.
- Obtain participants' immediate reactions. Have participants complete the evaluation before they leave the room. This will ensure that you receive a 100 percent response rate. It will also prevent “mob mentality” response, the possibility of several people getting together to discuss the class either before or while they are completing the evaluation.
Interviews
In addition to the end-of-session questionnaires, you can use interviews to increase the reliability of the data collected from the questionnaires. This method of data collection is quite flexible, allowing the interviewer to probe for more specific answers and to clarify questions as needed. The method also allows the interviewer to record spontaneous answers and, therefore, get a more complete picture of the participants' reactions. The interviewer can explore in more detail the reactions gleaned from the questionnaires.
Plan on spending about 30 minutes per interview. You will not be able to interview every participant, so select a random sample. It is important to hold the interviews within one week of the session so that the experience is fresh in their minds. Through one-on-one interviews, you can further explore the reasons for participants' reactions and solicit suggestions for improvement. Either record the interviews and have them transcribed, allowing you to analyze or interpret the responses more thoroughly, or simply take notes during the interviews.
When developing the interview questions, do not duplicate the questions on the written form. Instead, ask specific questions about the methods used or the content covered. For example, below are several questions about the methods used in a training session on leadership:
- What feelings did you have about the methods used in the program?
- What did you like about the jigsaw design?
- What did you like about the learning tournament?
- What didn't you like about the jigsaw design?
- What didn't you like about the learning tournament?
Level 2: Learning
Level 2 evaluation deals with what the participants actually learned during the training session. Kirkpatrick defines learning as the “extent to which participants change attitudes, improve knowledge, and/or increase skill as a result of attending the program” (Kirkpatrick, 1994). It is far easier to determine what new knowledge or skills the participants acquired than to find the ways in which the training changed their opinions, values, and beliefs.
The three most appropriate methods used to evaluate learning are tests, observation, and interviews, with tests being the most frequent. See Table 12.2 for a summary of the advantages and disadvantages of each method.
Table 12.2 Level 2 Evaluation Methods
| Method | Advantages | Disadvantages |
| Tests: Objectives tests Multiple choice Matching True-false Fill-in Subjective tests Essay Short answers |
Easy to score Inexpensive to use Easy to write Inexpensive to create |
Difficult to write Time-consuming to write Grading is time-consuming Expensive to grade |
| Observation: Observe behavior in class Skills demonstrated in skills practices and learning activity |
Immediate application Lends itself to on-the-spot coaching and feedback |
Subjective, open to interpretation Unable to spend adequate time observing behavior of all participants |
| Interviews: Individual interviews conducted shortly after the training Random sampling of participants |
Can gather more information Instant feedback |
Time-consuming; expensive Must be tightly structured to obtain quantifiable responses |
The Trainer's Handbook, 4th Ed. Copyright © 2016 by Karen Lawson. Reproduced by permission of Pfeiffer, an imprint of John Wiley & Sons, Inc.
Tests
Testing should be kept fairly simple. Many trainers give both a pretest and a posttest to get an even more accurate picture of what the participants have learned.
Types of tests. First determine whether you want to construct subjective (short-answer or essay) or objective (multiple-choice or true-false) items or even a combination of the two types. When constructing test items, consider the time needed to grade the test as well as the validity and reliability of each item. Make sure the test assesses the learning as specified in the learning objectives. When an item measures what it is supposed to measure, it has validity. Each test item must also be reliable, that is, give consistent results from one application to another. For a broader discussion of test construction, refer to “Testing for Learning Outcomes” by Deborah Grafinger Hacker, one of the INFO-LINE booklets published by the American Society for Training and Development (revised in 1998). Hacker describes various types of tests, provides directions on writing test items, discusses measuring the validity and reliability of tests, and includes a helpful and extensive resource list.
Make sure the test is meaningful. Instead of asking for simple information or factual recall, ask questions that require the participants to apply or interpret what they learned in the session.
Question formats. All multiple-choice questions consist of a stem and a response. The stem presents a problem, asks a question, or takes the form of an incomplete statement. Responses include possible answers, all of which must be plausible. The greater the number of items, the better the test's reliability. Following are some formats to consider:
- Correct answer. The correct-answer format asks a simple question to which there is only one correct answer. It is used primarily to test the recall of facts. This type of question is appropriate to test product knowledge, for example.
- Best answer. With this type of question, there is more than one correct choice. Some or all of the choices may be correct to some degree. Because the best-answer question requires a higher level of thinking, the respondent must evaluate the choices and draw conclusions. This type of question can create many problems. Because the answer is open to interpretation, the test item can be challenged quite easily, and you might find yourself either arguing with the individual or group and most probably having to give credit for other answers.
- Combined response. This question is the most complicated and time-consuming for both the test writer and the test taker. The choices, one or more that may be correct, are numbered. A second set of choices lists combinations of possible correct responses. This type of question assesses complex cognitive skills and the ability to analyze and evaluate. Exercise a great deal of thought when writing the item. Because of its complexity, the respondent will probably have to spend more time thinking about the item.
Test-writing guidelines. In most cases, you will probably choose to develop multiple-choice questions. They are easy to grade, but not necessarily easy to write. To help you construct a multiple-choice test that will provide valuable information about the participants' content mastery, consider the following guidelines:
- Avoid “all of the above” and “none of the above” options.
- Make sure the stem (the main part of the question) contains most of the information and defines the problem; place blanks for fill-ins near the end.
- Maintain grammatical consistency or parallel structure for both the stem and the answer choices.
- Try to create choices of equal length.
- Avoid ambiguity and reading difficulty by stating questions in the positive rather than in the negative.
- Keep the sentence stem simple and limit it to one idea.
- Use conversational language when phrasing the item and its choices.
- Arrange the questions in logical order.
- Do not give clues to the correct answer in the question.
To gauge retention of the information learned in the program, administer another test several months following the training.
Observation
Trainers can watch participants practicing and applying skills, tools, and techniques during the session. As the trainers observe participant behavior in skill practices, role plays, simulations, case studies, and other activities, they can get a good idea of what the trainees have really learned.
Interviews
Shortly after the training, interview the participants and ask them what they learned in the session. Conduct the interviews within the week following the session.
Level 3: Behavior
The critical question answered by Level 3 is, “How has the training affected the way participants perform on the job?”
Although both managers and training professionals agree that the success of a training program is determined by what the participants do with the information or skills back on the job, these results are often ignored. Level 3 evaluation is both time-consuming and costly. It also requires good organizational and follow-up skills and processes.
Purpose of Level 3 Evaluation
Use follow-up evaluation to do the following:
- Measure lasting results of the training
- Identify areas in which trainees show the greatest and least improvement
- Compare follow-up and end-of-program responses
Follow-Up Guidelines
Use these guidelines for follow-up evaluation:
- Prepare participants. At the end of the training session, tell participants that you will be conducting a follow-up evaluation and what type of evaluation it will be.
- If the training wasn't effective, find out why. Encourage participants to identify reasons why they haven't improved and what factors obstruct their progress. Sometimes there are factors that inhibit or prevent the application of the new knowledge and skills on the job. These barriers might include poor environmental conditions, lack of proper equipment, the supervisor, existing policies and procedures, or even the organizational climate.
- Share follow-up evaluations with participants' managers or supervisors. These individuals should know about program results and follow-up information and should be involved with the participants' practice and application of training.
Observations
The trainer or another designated observer can actually observe employees back on the job. Carefully watch employees as they perform their routine job tasks and responsibilities. To facilitate the process and ensure consistency in the data gathering, create a checklist of desired behaviors and then observe whether or not the employee is demonstrating these behaviors.
For example, if you are observing someone who has recently attended a customer service program, your list of customer service behaviors might look like this:
- ______ Smiles.
- ______ Greets customer with “Good morning” or “Good afternoon.”
- ______ Uses the customer's name.
- ______ Asks “How may I help you?”
- ______ Offers additional assistance.
- ______ Gives the customer choices.
- ______ States what we can do, rather than what we can't.
Interviews
Not only should you interview those who went through the training, but you should interview those who are affected by or closely associated with the program participants. Possible interviewees include the participants' managers, co-workers, customers, or subordinates. The interview questions would have to be carefully constructed and designed to focus on specific applications and behavior changes.
Surveys
Surveys are more efficient and less expensive than interviews to find out whether the participants are actually applying what they learned. Once again, do not limit your sources of information. Others who interact with those who participated in the training are often a more reliable source of feedback. You will want to know not only if trainees are using the training on the job but also how they are using it to perform better.
A control group is also helpful for a Level 3 evaluation. To validate your results, choose a control group of employees from the same function as the participants who received the training. Ask the members of the control group to complete the same surveys, questionnaires, and tests as those completed by the participants. Monitor their job performance and compare it with that of employees who received training.
Regardless of which evaluation method(s) you use, make sure you allow enough time for the behavior change to take place. The length of time depends on the program, but three to six months should give the participants ample opportunity to apply what they learned and develop new behaviors.
Level 4: Results
Level 4 evaluation determines the impact of the training on the organization. Ideally, it shows how the training has contributed to accomplishing organizational goals and objectives—business results. If an organization chooses to conduct a Level 4 evaluation, the area of measurement must be the same as those identified in the needs assessment.
To measure training's impact on the bottom line, return to the data gathered during the needs assessment. Determine your critical success factors up-front. Results measured could include any of the following or any measurable item:
- Production output
- Sales
- Operating costs
- Customer satisfaction
- Quality standards
- Safety record
- Turnover rate
- Absenteeism
- Employee grievances
- Employee satisfaction
- Budget variances
- Promotions
Level 4 evaluation is both difficult and time-consuming. It can also be quite costly. It is difficult to measure because of the many variables that can come into play once the participants leave the session. For these reasons, a Level 4 evaluation is not appropriate for all training. From a practical standpoint, consider Level 4 for those programs that are near and dear to senior management and have been identified as a top priority.
Because of the complexity of Level 4 evaluation and its infrequency of use, it is addressed here only in very basic terms—just enough to give you an idea of what it involves and the purpose it serves. To explore Level 4 in more detail, refer to the listings in Appendix B, including Jack Phillips, Donald Kirkpatrick, and Dana Gaines Robinson and James Robinson.
As a starting point, test your current knowledge of measurement terms by matching the terms in the left-hand column below with the correct definitions in the right-hand column (see Appendix A for the correct answers):
| Term | Definition |
| _______ 1. Return on investment (ROI) | a. Survey data of employee attitudes |
| _______ 2. Direct costs | b. Salaries and benefits paid to participants for the time they are attending the program |
| _______ 3. Indirect costs | c. Expenses for creating the training program |
| _______ 4. Overhead | d. Involves quantity, quality, cost, time; easily converted to monetary value |
| _______ 5. Development costs | e. Costs of shared resources such as heating and building maintenance |
| _______ 6. Hard data | f. Expenses of operating the training department that are not directly related to any training program |
| _______ 7. Soft data | g. Mathematical formula for calculating the difference between cost and value |
| _______ 8. Compensation for participants | h. Costs related to the delivery of the training program, such as materials and trainer's salary |
Participant Evaluation and Accountability for e-Learning
Unlike traditional evaluations, e-learning evaluations are somewhat difficult because of delivery options and individual learning solutions that often prevent consistency and uniformity in the evaluation process. For example, not all learners will complete self-study courses from start to finish. Some will choose only the modules they think they need. Others may have to go back and repeat a module to gain a greater understanding of its content.
For the most part, however, you can apply the same principles and strategies of traditional evaluation methods and levels discussed earlier in this chapter to the e-learning process. The basic difference, of course, is that the various evaluation methods will be done electronically. The following are types of evaluations that can be adapted to an e-learning environment.
- Level 1 evaluation measures participants' reactions. Methods that can be used include questionnaires completed on-screen within the course or as email feedback. Participant reactions can also be captured via online focus groups or in chat rooms.
- Level 2, which measures what participants actually learned, uses various tests. Almost all types of tests can be adapted to an electronic format: true-false, multiple-choice, essay, fill-in-blanks, matching. Visit William Horton's website (www.horton.com) for great examples of various testing formats. In addition to testing, you can monitor learners by observing their behavior in learning activities such as simulations and learning games as well as role plays conducted in chat rooms.
- To evaluate how well participants apply what they learned (Level 3), you would turn to traditional methods such as observations of the employee's on-the-job performance, surveys completed by the participant and others who interact with him or her, and job performance records. It might also be appropriate to set up control groups: some employees would experience traditional classroom-based training while others would engage in e-learning activities. Employees' individual action plans could also be monitored.
- Level 4 evaluation for distance learning is much the same as it is for traditional learning and would involve determining return on investment (ROI) as well as determining benefits such as a decrease in the number of accidents, safety violations, tardiness, absenteeism, turnover, customer complaints, or grievances. Business metrics such as profitability (sales, revenues, profit) and financial health (stock price, market share) might also be targeted for Level 4 evaluation.
As with any evaluation process, it is important to collaborate with the organization's leaders to determine the success criteria. What specifically do the key people want to measure and how will they use the information? In addition to the basic questions addressed in the section on the four levels of evaluation, you may also be interested in evaluating the e-learning process by establishing ways to measure the following:
- How often users log in
- How long they stay
- How long before they receive a response to a question
- How efficient the online system is
Accountability for Training
Throughout the business world, accountability for all functions is increasing. Staff functions such as training are expected to prove their contribution and value to the organization. Top executives are demanding that training departments offer proof of their worth or take budget cuts. Two approaches to measuring the value of training are covered below: (1) cost/benefit analysis and (2) return on investment.
Cost/Benefit Analysis
The cost/benefit analysis looks at the total cost to produce a training program and attempts to quantify the benefits. Cost includes everything from the needs assessment, through design, development, delivery, and finally to follow-up. Both direct and indirect costs are used to determine the total cost of the program. As Table 12.3 shows, determining training costs is complex.
Table 12.3 Determining Training Costs
| Direct Costs | |||
| People | Facilities | Materials | |
| Design and Development | Salaries, benefits, travel for: course development, clerical support or consultant fee and expenses or costs of certifying in-house trainer for purchased programs | Marketing brochures Participant materials Instructor manual Purchased resource materials Purchased program |
|
| Delivery | Salaries, benefits, travel for: trainer (s), participants (average salary), clerical support, consultant fees and expenses | Room rental Equipment rental Refreshments |
Note books Folders Tent cards Paper Pencils/pens Flip charts Handouts Film rental/purchase Transparencies/slides Stationery items Certificates Books Articles reprints |
| Direct Costs | |||
| Evaluation | Salaries, benefits, travel for: training personnel, participants, clerical support, bosses, subordinates, peers |
Surveys Questionnaires |
|
| Indirect Costs | |||
| Training space Custodial services Utilities Postage Telephone |
Computer time Equipment depreciation Equipment Maintenance/repair Support services |
||
The Trainer's Handbook, 4th Ed. Copyright © 2016 by Karen Lawson. Reproduced by permission of Pfeiffer, an imprint of John Wiley & Sons, Inc.
Total benefits of the program may be reduced costs or increased revenues directly attributable to the training. In many cases, however, benefits can only be estimated.
Subtract the total costs from the total benefits to find the net benefit of the training program. The program is considered a financial success if the costs are lower than the benefits.
Return on Investment
The return on investment simply shows what the payback is for the training program. To determine return on investment, you must wait three to six months and even longer for operational results. The formula for determining return on investment is: ROI = (net program benefits/program costs) × 100. Stated simply, if training programs fail to show a reasonable return on the company's investment, future (or even current) training initiatives are at risk. For more information on determining ROI, refer to Jack Phillips's books listed in Appendix B.
Significance of the Evaluation Process
Evaluation is a complex issue. For one reason, many variables enter into the equation. No matter how hard you try to fine-tune the evaluation process, the reality is that effects can only be estimated, and economic benefits cannot be calculated precisely. As the role of training continues to change and trainers reposition themselves as performance consultants, there will be more pressure to measure the effectiveness of training. The good news, however, is that the field of training and development continues to grow at a rapid pace. This trend will continue to provide many opportunities for training professionals, both internal and external, to make an impact on the growth and development of individuals and organizations throughout the world.
The training process is a never-ending cycle: analysis, design, development, implementation, and evaluation. Part I provided you with the basic information for you to create new training programs and revamp existing ones. As mentioned several times throughout this book, the role of the workplace learning and development professional continues to evolve. Part II provides information on specific considerations and applications that you may need to address now or in the future.