C h a p t e r 5
Evaluating the Workshop
- Reasons for evaluation
- Explanations of evaluation techniques and instruments and how they are used
Reasons for Evaluation of Training
Business guru Tom Peters tells us, “What gets measured gets done. . . . Even imperfect measures provide a strategic indication of progress or lack thereof.” The evaluation of training is an essential process to ensure that the organization’s resources are being used wisely. Evaluation is important to the organization, as well as to the trainer and the participants. Organizations will want to know: “Is the cost of training justified?” Trainers will want to know, “How successful have I been in doing my job?” Participants will want assurances that they have been successful in their learning efforts and will be able to effectively perform their jobs.
Two different, yet related, meanings are attached to the concept of evaluation as related to training.
- Has the participant learned the content according to the specified training objectives?
- Was the process by which that learning occurred effective and appropriate?
In the training business, the generally accepted model for discussing evaluation of learning was developed in the late 1950s by Donald Kirkpatrick. He uses four levels of evaluation:
- Level 1: Reaction
- Level 2: Learning
- Level 3: Behavior
- Level 4: Results
Each of these measures can be applied only to training objectives that were written to those levels. For example, if the objective is that the learners will “understand how to use a decision matrix,” that’s a Level 2 objective and can be measured by asking them to explain how to use it. A Level 3 objective, for example, “Participants will be able to create a decision matrix,” must be measured by having the participant actually create a decision matrix (not just explain how to create one).
Evaluating at the Reaction Level
Trainers almost always use an end-of-course evaluation form to obtain a rating of how the participants liked the program, what they gained most from it, and other information related to the training. An ASTD/i4cp research report, The Value of Evaluation, found that about 90 percent of organizations evaluate training at this level. These Level 1 evaluations are sometimes ironically referred to in the training industry as “smile sheets.”
The reaction-level evaluations may include questions about both content and process. The fact is, at this lowest level, content and process may be somewhat difficult to separate. Because the only practical and immediate way to determine if an attitude or awareness has changed as a result of training is to simply ask the participants, it is common to ask their opinion on both content and process on the same form. Although such analysis is obviously open to significant bias, it still can serve a valid purpose.
Reaction-level evaluations can be done at any time during training, or months after it is over, not only at the conclusion of the last training session. When it’s possible to change the training program while it is underway or if remedial training is an option, daily feedback may help a trainer adjust the training to better meet the needs of the learners or of the next group to be trained.
A word of caution about the interpretation of results from a Level 1 evaluation: Level 1 evaluations tend to be positively biased. Interestingly, whether participants did well in the training or not turns out to be a non-issue. If the delivery of the training was seen to be “good” or at least “adequate,” then high satisfaction and low dissatisfaction among the participants were assured. Ironically, even if delivery was not adequate, there was still a better than 50 percent chance that participants would report high satisfaction and low dissatisfaction. Evaluations of training via Kirkpatrick’s first (reaction) level, therefore, are of the most value in (1) providing a means of bringing closure to the class, and (2) providing narrative comments that may be useful to the trainer. Overall, however, they may tell us very little about what the participant really learned and frequently tell us nothing about either the content or process of the training.
The materials available with this book include Level 1 evaluation forms. The questions deal with both the content (that is, decision making) and the process by which the training was delivered. Adapt these suggested evaluations to better match the way your training is actually conducted.
Evaluating Learning at the Learning Level
Learning-level (Level 2) assessment is typically done by oral or written testing. Evaluation options include using true-false, matching, multiple choice, fill-inthe-blank, short answer, and essay questions, along with variations in each. Any of these formats could be given orally, in writing, or by using an interactive medium.
Most people have taken hundreds of tests in their lifetime and could probably deductively critique most testing formats. Knowledge testing is a complex topic. Anyone with further interest in the subject should refer to books on the subject (see For Further Reading).
Although much easier to do than Level 3 or Level 4 testing, knowledge testing can be deceptive. Properly done, knowledge testing measures learning and retention of the facts and processes addressed in the training. But it does not measure whether people are actually going to use the knowledge or whether it benefits the organization in any way.
A selection of Level 2 questions for evaluating decision-making training has been included in the materials with this book. Modify the questions to reflect the material actually covered in your training sessions.
Evaluating Learning on a Behavioral Level
Behavioral evaluations (Level 3) are related to objectives that demand practical skills. Evaluation requires actual performance of the skill to demonstrate that it has been learned and that the participant can apply the skill. Behavior level evaluations may occur either during the training, in the form of skill tests, or on-the-job, following training. In some cases, posttesting should be delayed, perhaps as much as several months.
A selection of Level 3 decision-making exercises, which can be done during the training, are included in the course materials. However, evaluation of actual on-the-job behavior cannot be done during the training and is difficult to document (especially for a subject like decision making) at any time, even by supervisors or participants.
This leads to another important point about evaluations, especially regarding “results level” evaluations (Level 4). Sometimes, even when the participant has developed and used the skill on the job, it does not make any difference to the organization’s bottom line. That’s what Kirkpatrick’s fourth level is designed to measure: Has the training made a difference to the organization?
Evaluating Learning on a Results Level
The purpose of training in an organization is to enhance results. Training happens because the organization needs the learners to know or do something better, faster, or smarter than they would have been able to do without training. The reason for evaluating at this level is to show how the training has affected business results. Level 4 evaluations are usually conducted some time—maybe even months—after the training to give the learners time to use the new skills on the job.
Beyond Level 4, some organizations may choose to evaluate return-oninvestment. These return-on-investment (ROI) evaluations require skills (such as accounting and economics techniques) that are not necessarily associated with the training profession. The evaluator may need to deal with financial records, market analyses, industrial engineering techniques, and sometimes even the psychological and cultural aspects of organizational operations. For further information on how to do this kind of evaluation, see work by Jack Phillips or Robert Brinkerhoff. Because of the complex nature of ROI evaluations, no means of doing these are included in this training package.
Evaluation Instruments Provided in this Book
A typical Level 1 evaluation form (Evaluation Instrument 5–1) is included at the end of this chapter and also on the website as a separate file, which can be downloaded and modified. A different evaluation form may be used as the end-of-program evaluation; it is shown at the end of this chapter as Evaluation Instrument 5–2.
Level 2 assessment questions and answers (tied to the various training objectives specified in the modules) are located at the end of each chapter. You are welcome to use them as provided or modify them to meet your needs.
Level 3 assessments are an intrinsic part of the worksheets and application exercises, which are provided in separate files. They can be tied to the skill objectives for each module.
What to Do Next
- Review the evaluation instruments provided with the modules.
- Decide what, if any, questions should be added or removed to meet requirements or needs of your organization.
Evaluation Instrument 5–1
Please indicate your response by circling one number for each of the items below. Place this evaluation face down on the front desk as you leave. Thank you.
| RATING | |||
| ABOUT THE EXPERIENCE | LOW | HIGH | COMMENTS |
| Instructor’s knowledge of subject matter | 1 2 3 4 5 | ||
| Instructor’s skill in training | 1 2 3 4 5 | ||
| Effectiveness of the handout materials | 1 2 3 4 5 | ||
| Effectiveness of the visuals | 1 2 3 4 5 | ||
| Learning value of the exercises | 1 2 3 4 5 | ||
| Comfort of the facilities | 1 2 3 4 5 | ||
| Adequacy of the preliminary arrangements | 1 2 3 4 5 | ||
| Overall evaluation of the training program | 1 2 3 4 5 | ||
| RATING | |||
| ABOUT THE SUBJECT MATTER | LOW | HIGH | COMMENTS |
| Amount of information I learned as a result of this training | 1 2 3 4 5 | ||
| Usefulness of this training to me on my job | 1 2 3 4 5 | ||
| My level of knowledge about this subject prior to today’s training | 1 2 3 4 5 | ||
| Difficulty I had in understanding this material. (Note: 5 = a lot of difficulty.) | 5 4 3 2 1 | ||
| What two things were most beneficial to you from this session? | 1 2 |
||
| What improvements would you suggest in this training program? | |||
© 2010 Decision-Making Training, American Society for Training & Development
Evaluation Instrument 5–2
Please indicate your response by circling one number for each of the items below. Place this evaluation face down on the front desk as you leave. Thank you.
| RATING | |||
| ABOUT THE EXPERIENCE | LOW | HIGH | COMMENTS |
| Instructor’s knowledge of subject matter | 1 2 3 4 5 | ||
| Instructor’s skill in training | 1 2 3 4 5 | ||
| Effectiveness of the worksheet materials | 1 2 3 4 5 | ||
| Effectiveness of the visuals | 1 2 3 4 5 | ||
| Learning value of the exercises | 1 2 3 4 5 | ||
| Comfort of the facilities | 1 2 3 4 5 | ||
| Adequacy of the preliminary rrangements | 1 2 3 4 5 | ||
| Overall evaluation of the training program | 1 2 3 4 5 | ||
| RATING | |||
| ABOUT THE SUBJECT MATTER | LOW | HIGH | COMMENTS |
| Amount of information I learned as a result of this training | 1 2 3 4 5 | ||
| Likelihood that I will be make better decisions as a result of this training | 1 2 3 4 5 | ||
| My level of knowledge about decision making prior to this training program | 1 2 3 4 5 | ||
| What three things were most beneficial to you from this program? | 1 2 3 |
||
| What two things from this training program were least useful to you? | 1 2 |
||
| Other comments can be made on the back of the form. | |||