Measuring Level 1

Level 1, or reaction, can be easily measured during the training event or immediately following it, preferably before participants leave the classroom. You usually use a questionnaire, composed of both questions with a rating scale and open-ended questions. Some trainers allow participants to take the evaluation with them or go online to complete it after the session. The drawback is a lower return rate (for in-person or online training).

How do you begin?

1. Determine what you want to learn about the training.

   You will most likely want to know something about the content, including its relevance to the participants’ jobs and the degree of difficulty. You will also want to gather data about the trainer including effectiveness, communication skills, the ability to answer questions, and approachability.

2. Design a format that will be both easy for participants to complete and presents a way for you to quantify participant responses.

   Many formats exist, and the design will be a factor of the first question: What do you want to learn? You may choose to have statements rated on a one- to seven-point scale representing strongly disagree to strongly agree (or poor to excellent). You may also want to ask open-ended questions, or you may choose to do a combination of both types of questions. Even if you develop a format with questions rated on a scale, adding space at the end for comments is a good idea.

3. Plan to obtain 100-percent response ratings that are complete.

   Obtaining a high level of response is mostly in the timing. First, plan time into the training design to complete the questionnaire. Then, 20 minutes prior to the end of the training session, pass out the evaluations and ask participants to complete them. It should take only ten minutes, allowing you time to facilitate your closing activity so that the learners’ last interactions are memorable and positive. Online participants should know where and how to locate the evaluation.

You want honest responses to the evaluations, so allow for anonymity and don’t require participants to sign their evaluations. Also, ask them to drop the evaluations off on a table on their way out versus leaving them at their places. If you have a valid reason for wanting to know who made specific comments —to provide feedback, for example — you can make signing the evaluation optional. Have a similar process prepared for the end of your virtual sessions.

Try using SurveyMonkey or another similar format that learners complete at the end of the class on their tablets or laptops. This approach is easy, fast, and still anonymous.

Your training department will most likely have determined an acceptable standard against which you will measure results. For example, 5.5 on a 7.0 scale may be considered acceptable by your training department.

If you follow these guidelines, you will be well on your way to finalizing an effective Level 1 evaluation. If you are conducting a virtual session, a virtual trainer strengths assessment at the end of this chapter may provide additional ideas to create your Level 1 evaluation. You and your colleagues may want to use the evaluation to provide feedback to each other on your training skills.

Some training practitioners snub Level 1 evaluation as too insignificant or perfunctory. Perhaps this is because trainers can conduct more sophisticated evaluations, or because it’s prudent to tie training to the bottom line. The “smile sheet,” as Level 1 is known, has become a pejorative term to some. Don’t fall into that way of thinking. Dr. Kirkpatrick’s Four Levels are built upon each other. Level 1 is a prerequisite when other levels are used. Even though you’re “only” measuring reactions, these are the first-hand reactions of the customers of the training. There are very few more effective and efficient ways for trainers to obtain feedback about performance. You may have the best training design, but if it’s delivered poorly, it does affect participants’ ability to learn. Level 1 evaluation also serves as a way to focus questions on the content. The key here is to design an instrument that measures the essential foundational elements of the training content and how well you delivered it.

Use the ideas and suggestions your evaluations contain to improve your future performance and program designs. The evaluations can serve as input to your efforts to continually develop your skills and competencies.

Measuring Level 2

You can measure Level 2, or learning, using self-assessments, facilitator assessments, tests, simulations, case studies, and other exercises. You should conduct this type of evaluation at the end of the training before participants leave so that you can measure the degree to which they learned the content. Use pre- and post-test results to compare improvement.

Measuring before and after the training session gives you the best data because you can compare participants’ knowledge levels before and after they complete the training. You measure the knowledge, skills, and attitudes (KSAs; see Chapter 2) that the training session was designed to improve. Use tests to measure knowledge, and use attitude surveys to measure changing attitudes. To measure skill acquisition, use performance tests, which have the participant actually model the skill. As with Level 1, attempt to obtain a 100-percent response rate.

Create straightforward tests. You’re not trying to trip up the participants; you’re trying to measure the learning that occurred to ensure improved job performance.

Some trainers use a control group for comparison, and Dr. Kirkpatrick recommends that you do so if practical. Although this may be the most scientific way to gather data, it may also be a waste of time. For example, if participants have another way to learn between pre- and post-tests, don’t send them to training. On the other hand, if training is the only way that participants can gain the knowledge, skills, and attitudes, why bother with a control group? You’ll need to be your own judge about whether a control group is beneficial.

You can choose from among many types of testing formats, and each brings with it advantages and disadvantages. True/False tests are easy to develop and cover many questions in a short amount of time. The drawback is that if the test is too easy, having superficial knowledge may lead to an inflated score. Be sure to use a subject matter expert (SME, pronounced “smee”) to assist with the design.

Other available testing options include oral tests, essay tests, multiple-choice tests, and measured simulations such as in-basket exercises, business games, case studies, or role plays. Assessments may also include self-assessments, team assessments, or instructor assessments. You can also create an online evaluation. Although an online assessment can be efficient, it brings a couple problems, including the difficulty of guaranteeing that participants are who they say they are when signing in and the ability to protect the questions in the exam banks. Most organizations that use online assessments have addressed both of these issues. Whatever testing option you use, be sure that the results are quantifiable.

Pilot any test before using it with participants. Return measurement results to employees as quickly as possible. The significance of measurement feedback decreases with time.

Measuring Level 3

Level 3, or behavior, determines the successful transfer of learning to the workplace. Unlike with Levels 1 and 2, you need to allow time for the changed behavior to occur. How much time? The experts differ, and with good reason. The amount of time required for the change to manifest will itself depend on the type of training, how soon participants have an opportunity to practice the skill, how long participants take to develop a new behavioral pattern, and other aspects of the job. All this can take anywhere from one to six months. You’ll probably need to work with the SME and supervisors to determine the length of the delay required to allow participants an opportunity to transfer the learning (behavior) to the job.

By the way, as with Level 2, I recommend a pre- and post-testing method, and you’ll need to decide whether to incorporate a control group for comparison. (See the preceding “Measuring Level 2” section for the pros and cons of having a control group.)

Each item in an evaluation instrument for Level 3 should be related to an objective taught as part of the training program. Based on the training objectives, you create a list of skills that describe the desired on-the-job performance. Use a SME to assist you with the design of the evaluation. The SME will understand the nuances of being able to complete a task. For example, a skill maybe “uses the four-step coaching model with employees.” A SME will know that even though the four steps are essential, what truly makes the model successful is the supervisor’s “willingness to be available to respond to questions at any time.” Knowing this, your checklist of skills expands to include availability.

After you identify the skills to measure, you select an evaluation method. Evaluation tools may include interviews, focus groups, on-site observations, follow-up questionnaires, customer surveys, and colleague or supervisory input.

Don’t take Level 3 evaluations lightly; they require a major resource investment. Know what you will measure; know how you will measure; and most important, know how you will use the data.

All trainers should be skilled in Level 3 evaluation methods. The most important reason to conduct training is to improve performance and transfer knowledge, skills, and attitude changes to the workplace. You can’t expect the desired results unless a positive change in behavior occurs. Therefore, use a Level 3 evaluation to measure the extent to which a change in behavior occurs.

What do you do if the results show that performance has not been changed or skills mastered? You need to go back to the training design. Certainly, the first step is to determine whether the skill is required. If yes, examine the training material to ensure that you’ve used appropriate learning techniques and placed enough emphasis on the skill. Perhaps a job aid is required to improve performance. Sometimes you may discover something that didn’t show up in the needs assessment. For example, you may discover that participants aren’t using the skill because it’s no longer important or isn’t frequently used on the job. In that case, you may want to remove it from the training session. This is a perfect example of how the fifth stage of the Training Cycle, evaluation, feeds back into the first stage.

Getting results that demonstrate that performance hasn’t improved is not what a trainer wants to hear. However, that knowledge helps you make an intelligent decision about whether to maintain the training program, overhaul it, or scrap it entirely. Without a Level 3 evaluation, you probably can’t make a wise decision.

Measuring Level 4

Level 4, or results, may be the most difficult to measure. Dr. Kirkpatrick frequently stated that the question he was asked most often was, “How do you evaluate Level 4?” Even though Level 4 is the most challenging to evaluate, training professionals need to be able to demonstrate that training is valuable and can positively affect the bottom line.

At this level, you can never be sure whether external factors affected what happened. There is always a possibility that something other than the on-the-job application contaminated the results. Can you really isolate the effects that training has on business results? This is one time that using a control group can be helpful to the evaluation results. Yet even with a control group, other factors may impact the business, such as the loss of a good customer, the introduction of a new competitor, a new hiring practice, or the economy.

So what do you do when management asks you to provide tangible evidence that training is having positive results? A before-and-after measurement is relatively easy because records for the kinds of things you measure (turnover, sales, expenses, errors, grievances) are generally available. The trick is to determine which figures are meaningful.

A second difficultly arises in predicting the amount of time to allow the change to take effect. It may take six or more months. Gather the data that you believe provides evidence of the impact of the training. This measurement usually extends beyond the training department and utilizes tools that measure business performance, such as sales, expenses, or rework. Remember, a key issue is to try to isolate training’s impact on results. You may not be able to prove beyond a doubt that training has had a positive business impact, but you will be able to produce enough evidence for management to make the final decision.

This book and other sources say that training should align to the business’s requirements, and Dr. Kirkpatrick suggests that you can use his evaluation levels to achieve that alignment. Start with Level 4. Determine, with the business managers, what needs to occur (results). What are their “expectations”? Next decide what performance (behaviors, or Level 3) is required to achieve the desired results. Then identify the knowledge, skills, and attitudes (KSAs, or Level 2) that employees require to achieve performance expectations. Finally, create a plan to offer the correct training, thereby ensuring that employees will be receptive to the changes. Kirkpatrick Partners calls this process Return on Expectations.

Figuring out which evaluation level to use

You may choose to evaluate training at one or all four levels. How do you decide? Base your decision on answers to the following questions:

• What is the purpose of the training?
• How will you use the results of the evaluation?
• What changes or improvements may be made as a result of the evaluations?
• Who are the stakeholders and what do they want you to measure?
• What are the stakeholders’ expectations?
• What business goals will the training impact?
• What resources will be required for the evaluation?

Just because there are four evaluation levels doesn’t mean that you should use all four every time. After answering the preceding questions, decide how far you will go through the four levels. Will you stop after Level 1 or use all four? This depends on what’s important to measure for the training program. Evaluation experts agree that Level 3 and especially Level 4 should be used sparingly because of the time and cost involved. Always start with Level 1. In addition, a rule of thumb seems to be to use all four levels in situations in which the results are a top organizational priority or for training that is expensive.

Objective test formats

This method measures how well trainees learn program content. A facilitator administers paper-and-pencil or computer tests in class to measure participants’ progress. The test should measure the learning specified in the objective. Tests should be valid and reliable. Valid means that an item measures what it’s supposed to measure; reliable means that the test gives consistent results from one application to another. Here are some objective test formats:

• Multiple-choice questions take time and consideration to design. However, they maximize test-item discrimination, yet minimize the accuracy of guessing. They provide an easy format for the participants and an easy method for scoring.
• True/False tests are more difficult to write than you may imagine. They are easy to score.
• Matching tests are easy to write and to score. They require a minimum amount of writing but still offer a challenge.
• Fill-in-the-blank or short-answer questions require knowledge without any memory aids. A disadvantage is that scoring may not be as objective as you may think. If the questions don’t have one specific answer, the scorer may need to be more flexible than originally planned. Guessing by learners is reduced because no choices are available.
• Essays are the most difficult to score, although they do measure achievement at a higher level than any of the other paper-and-pencil tests. Scoring is the most subjective.

Attitude surveys

These question-and-answer surveys determine what changes in attitude have occurred as a result of training. Practitioners use these surveys to gather information about employees’ perceptions, work habits, motivation, values, beliefs, and working relations. Attitude surveys are more difficult to construct because they measure less tangible items. There is also the potential for participants to respond with what they perceive to be the “right” answer.

Simulation and on-site observation

Instructors’ or managers’ observations of performance on the job or in a job simulation indicate whether a learner is demonstrating the desired skills as a result of the training. You facilitate this process by developing a checklist of the desired behaviors, which is sometimes the only way to determine whether skills have transferred to the workplace. Some people panic or behave differently if they think they are being observed. Observations of actual performance or simulated performance can be time consuming. This method also requires a skilled observer to decrease subjectivity.

Criteria checklists

Also called performance checklists or performance evaluation instruments, these are surveys using a list of performance objectives required to evaluate observable performance. The checklists may be used in conjunction with observations.

Productivity or performance reports

Hard production data such as sales reports and manufacturing totals can help managers and instructors determine actual performance improvement on the job. An advantage of using productivity reports is that you don’t have to develop a new evaluation tool. Also, the data is quantifiable. Disadvantages include a lack of contact with the participant, plus records may be incomplete.

Post-training surveys

Progress and proficiency assessments by both managers and participants indicate perceived performance improvement on the job. Surveys may not be as objective as necessary.

Needs/objectives/content comparison

Training managers, participants, and supervisors compare needs analysis results with course objectives and content to determine whether the program was relevant to participants’ needs. Relevancy ratings at the end of the program also contribute to the comparison.

Class evaluation forms

Sometimes called a response sheet or smile sheet, participants respond on end-of-program evaluation forms to indicate what they liked and disliked about the training delivery, content, logistics, location, and other aspects of the training experience. The form lets participants know that you desire their input. You can gather both quantitative and qualitative data.

Interviews

You can use interviews to determine the extent to which skills and knowledge are being used on the job. Interviews may also uncover constraints to implementation. In a way that no other method can, interviews convey interest, concern, and empathy in addition to collecting data. They are useful when observing behaviors directly isn’t possible. The interviewer becomes the evaluation tool, which can pose both an advantage and a disadvantage. Interviews are more costly than other methods, but they give instant feedback, and the interviewer can probe for more information.

Instructor evaluation

Professional trainers administer assessment sheets and evaluation forms to measure the instructor’s competence, effectiveness, and instructional skills. See an example at the end of this chapter.

It’s easy to grow accustomed to your favorite evaluation method. Much better, though, is to incorporate variety in the evaluation methods you use. Gaining expertise with several makes you more skilled in knowing which method will be best for each situation.

Having discussions with individual employees can give you insight into the effectiveness of the training session. You can ask, for example, whether someone is doing anything differently as a result of participating in the training program, or whether they’ve noticed changes in their attitudes or relationships that may be related to the training program.