Chapter 5

Evaluating and Improving Listening Skills Training

Why Bother?

Evaluating training can be extremely beneficial to both the trainer and the organization. Without an evaluation, you are essentially flying blind; you don't know whether the training is effective, whether participants learn anything during the training, or whether training has a positive effect on the organization.

Here are three motivations for, and benefits of, evaluating the listening skills training that you design and deliver:

1. The training outcomes should be aligned with the learning needs that you assessed earlier. In other words, did the training deliver on the needs that the learners in the organization have at the time?

2. You can justify the continuation of investing in listening skills and other training you provide if you can demonstrate that it is on target:

Did the learners like the training?

Did they learn the content?

Did they use the content?

Did it positively affect the organization's results?

3. By using a scientific approach to improve the design and delivery of the training you provide, you can demonstrate value to the organization by linking data to business decisions. Thoroughly evaluating listening skills training ensures that the continuous improvement of your training is driven by real information and not just by general impressions or anecdotes. It also conveys the message that you are serious about results, and it demonstrates your business acumen.

The Classic Levels of Training Evaluation

Donald Kirkpatrick (2006) developed a well-known model for training evaluation that consists of four levels; it guides much of the practice of measuring training outcomes in the learning and development field. The levels are graduated, from the relatively easy to measure to the more complex.

Level 1 - Reaction: Measuring the reaction of participants to the training. Although positive reactions may not ensure that learning takes place, negative reactions can certainly affect the likelihood of learning and whether the training will be offered again in the future.

Level 2 - Learning: Measuring the extent to which learning objectives have been achieved. Has knowledge increased? Have skills improved or attitudes changed as a result of the training?

Level 3 - Behavior: Measuring the extent to which participants changed their behavior in the organization because they attended the training.

Level 4 - Results: Measuring the organizational results from behavioral changes that were achieved because participants attended the training.

The least powerful, but most common, evaluation method is at the first level; they are brief reaction surveys commonly called Smile Sheets, which usually use Likert scales to measure the effectiveness of the training content and delivery, and usually include space to write comments. The questions are often about value and participant enjoyment of the training. Smile Sheets indicate immediate reactions of participants, but they may have no correlation to actual learning. An entertaining facilitator, comfortable learning environment, and good food can produce positive survey results, but it may not affect application of learning and behavior change. Some participants also follow the old adage, “If you can't say anything nice, don't say anything at all.” They either give high scores that don't reflect their real reactions or they don't complete the evaluation. The participants may be more concerned with leaving the training facility and beating rush hour traffic than providing helpful feedback.

Smile Sheets are commonly used because they are easy to administer and can provide some value. Positive scores may not be indicative of effective training, but negative evaluations are a strong indication that the training is ineffective. Also, open-ended questions that allow participants to provide comments on the training can offer important, useful feedback.

Because the four evaluation levels are ranked according to complexity, they are also, in effect, ranked according to decreasing use. Due to ease of use, more organizations use Smile Sheets to evaluate training rather than attempt to measure learning and impact at the other three levels. Some may track learning by conducting pretests and posttests to assess skill level, but this may be limited to IT training subjects in many organizations. Fewer track behavior changes, and still fewer engage in the difficult task of measuring business results or return-on-investment for learning programs. The author strongly recommends that trainers take whatever steps they need to evaluate their sessions more thoroughly. If training is to be considered a key business activity, trainers need to be accountable for the value they claim to add to the organization and the effect on results.

Instruments for Evaluation in This Workbook

Chapter 11 of this workbook includes four instruments that lend themselves to applications of evaluating training. Some can also be used in training designs.

Assessment 11-2: Listening Self-Assessment. This tool calls for listening skills training participants to analyze their strengths and developmental needs. The instrument can be used as prework for an initial training module, as well as a repeat measure either at the end of the session or sometime afterward.

Assessment 11-4: Facilitator Competencies. This form helps establish learning priorities for your own development as a workshop facilitator. It can be used as a self-assessment or as a follow-up questionnaire to solicit feedback from trainees after a session or at a later time.

Assessment 11-5: Listening Skills Follow-Up Assessment. Distribute this questionnaire some time after the end of the listening skills training. Targeting Level 3 of Kirkpatrick's training evaluation model, it can be used in follow-up reunions of trainees or as a survey. It can also include ratings from colleagues or supervisors in the participants' work environments.

Assessment 11-6: Training Evaluation. Use this form to conduct a Level 1 Smile Sheet evaluation. It allows training participants to provide reaction feedback for the workshop and the facilitator.

Trainers are not limited to using these four instruments, of course. It is important to commit yourself to systematic evaluation and to conduct it routinely. In this way, you build up an understanding of what works best with your trainees and communicate your value to the client organization.

Improving Listening Skills Training

For many organizations, the concept of continuous improvement is of critical importance. As a training professional, it is in your best interest to demonstrate your attention to detail, as well as how you achieve the desired results. Be proactive to set yourself up for success; this will enable you to meet the needs of the organization and its learners in the best way possible.

Applied to listening skills training, commitment to continuous improvement means:

Specify the steps you are taking.

Analyze the logic of the sequence.

Look carefully at the effectiveness of each step.

Make changes that offer chances to make the training better.

This approach requires documentation and careful evaluation of the effects, or outcomes, of each step.

Pay close attention to clients' needs to provide work of high quality to organizations. In training, this means assessing the learning needs and preferences of potential participants, involving them in evaluating the training, and providing other services to them—such as one-on-one coaching—as they apply what they learn to their everyday work.

Trainers should avoid using their favorite learning activities; they should be more flexible and adapt to different organization and learner needs. A better approach to improving training is to experiment with both the content and design of the session. If activities do not produce desired results, either change or discontinue them. Try new ways and new activities to deliver the same learning objectives.

Learn ways to evaluate training on more than one of Kirkpatrick's levels, as well. The data gleaned from using Levels 2 through 4 of the model can provide great insight into how to improve your listening skills training. This is more time-consuming than simply distributing a Smile Sheet, but the pay-off can be substantial.

When time has passed after your training, you can also solicit feedback on your competence as a trainer and facilitator. This information can guide you through the process of developing as a learning professional. The root cause of less-than-optimal listening skills training is often the trainer, not the design. You may be interfering with the effectiveness of your sessions. Asking for feedback on what you can change is a direct way to manage your growth as a trainer, but soliciting honest feedback may not be a natural, comfortable thing for some people to do. You can become a role model for other trainers by actively engaging learners in your own quest for excellence.

What to Do Next

Here is a step-by-step method for maximizing the benefits of your efforts in evaluating your listening skills training sessions:

Decide which steps to follow. Lay out a step-by-step plan for evaluating the outcomes (impacts, pay-offs) of your listening skills training. Specify who will do what, when, how much, and for what purpose. Establish a timeline for these steps.

Gather feedback. Solicit data from trainees and any other relevant people. Use the instruments included in this book to assist you in this process.

Analyze results. Conduct both statistical and content analyses of the responses you receive while gathering data for your evaluation. Be as objective as possible during this step because you may be predisposed to use the data to validate your own opinions and observations.

Modify the design as necessary. Your evaluation program is the beginning of your design improvement process. Use the results to strengthen what works well, and change the selection, content, or sequence of activities to reach your training objectives more effectively.