Evidence-Based e-Learning in a Nutshell

In consolidating the principles summarized throughout the book, we can make a couple of general statements about the best use of media elements and instructional methods for multimedia instruction intended for novice learners, who are most susceptible to mental overload. In situations that support audio, best learning will result from concise informal narration of relevant graphics. (Please refer to Chapter 6 for exceptions.) In situations that preclude audio, best learning will result from concise informal written explanations of relevant graphics in which the corresponding text and graphic are placed near each other on the screen. In all cases, learning of novices is best promoted by dividing content into short segments and allowing learners to control the rate at which they access each segment. In addition, in lessons of any complexity, the pretraining principle recommends sequencing supporting concepts prior to the process or procedure that is the focus of the lesson.

To support engagement, novice learners benefit from instructional methods that promote generative learning, including worked examples accompanied by self-explanation questions, supported drawing assignments, job-relevant questions, peer teaching assignments, and individual or collaborative problem solving on challenging authentic problems. Learning also benefits from task-focused feedback that includes an explanation of why a particular response is correct or incorrect. Although we have much to learn about their design and best application, scenario-based learning and educational games are emerging as effective high-engagement multimedia approaches.

Effect Sizes for Principles

Table 18.1 lists the average effect sizes on tests of learning for most of the major themes in this book. Recall from Chapter 3 that effect sizes tell us the proportion of a standard deviation of test score improvement you gain when you apply that principle. For example, if you apply the multimedia principle, you can expect an overall test score improvement of at least one standard deviation greater than the same lesson without visuals. For our purposes, we suggest that any effect size greater than 0.5 indicates a practical improvement worth applying. Table 18.1 lists the number of comparisons on which the effect size is based, as well as the source of the data. In general, a higher effect size based on a larger number of comparisons suggests a more robust principle. All things being equal, feedback is one of the more powerful methods you can use, with an effect size of nearly 0.8 based on nearly seven thousand comparisons. In contrast, we can see that learner control has a negligible effect size, supporting our recommendation for program control in most situations (as described in Chapter 15).

Keep in mind, however, that you cannot directly compare these effect sizes because they represent average values drawn from many studies using diverse learners, materials, and conditions. Furthermore, as we have seen, most of the principles are most effective under specified situations called boundary conditions. For example, feedback is most effective when it provides an explanation of the task or the process to complete the task.

With these caveats in mind, you can see that most of the principles we have summarized have effect sizes that exceed our 0.5 guideline. Effect sizes lower than 0.5 indicate that the particular method may not be worth implementing. As you review the table, also consider the cost-benefit of the different methods. For example, if two methods have an effect size of .56 and one is easy and inexpensive to implement, you may emphasize that method over the other. Other factors to consider are efficiency of learning and learner motivation. Methods with lower but positive effect sizes that lead to more efficient learning or to greater learner motivation may be worth prioritizing higher than methods with higher learning effect sizes. We anticipate that, in the future, we will have more research data on the motivational and efficiency benefits of these methods to provide you with a broader perspective from which to make application decisions.

Review of Sample 1: Excel for Small Business

Figures 18.1 through 18.6 are screen captures from an asynchronous directive Excel lesson. The course is designed to help small business owners use spreadsheets. The course design assumes that learners are new to spreadsheets and Excel. Some of the learning objectives include:

• To identify and name cells
• To construct formulas for common calculations
• To use Excel functions

Take a look at Figure 18.1 on the topic of functions in Excel, review guidelines 1 through 28 and make a list of which guidelines you feel are violated. Then look at Figures 18.2 and 18.3. Put a check by the violations on your list that are remedied in the revisions shown in Figures 18.2 and 18.3. When you are finished, compare your analysis to ours.

As you can see, the screen in Figure 18.1 includes a lot of text presenting an introduction to and procedure for using functions in Excel. Clearly, it violates the multimedia and coherence principles. The revised screen in Figure 18.2 applies the multimedia principle by incorporating a visual of the relevant tool bar as well as the coherence principle by presenting only a small amount of text on the screen. Applying the segmenting principle, the procedure is broken into a few steps organized with the tabs for Access, Insert, and Select Values. Steps are displayed with callouts to maximize contiguity between text and graphics. The revised screen in Figure 18.3 applies the modality principle by using brief audio rather than text to present a few steps at a time. It also helps direct attention to the relevant portion of the visual through the use of cueing circles and numbers corresponding to the steps. As with any audio, controls allow the learner to replay as desired. Since the audio does not repeat on-screen text, the redundancy principle is not violated.

Next take a look at Figure 18.4 and refer to guidelines 29 through 46. Make a list of ways you think this practice exercise could be improved. Then look at a revision in Figure 18.5 and note which violations in your list have been remedied. Are there any further improvements you would make to the revision in Figure 18.5?

Figure 18.4 shows a practice exercise with feedback. We note the following major problems. First, the practice question is a recall or regurgitate question. While recall is needed on occasion, we recommend that for most workforce learning applications, you rely on higher-level application questions. Second, the practice directions and input boxes are separated from the spreadsheet, requiring the learner to expend mental effort integrating the two. We recommend better layout contiguity. Third, note that the feedback tells the learner that the answer is incorrect but does not give an explanation. Either correct or incorrect answers may be the result of guessing, so providing an explanation for all response options improves learning. Some of these shortcomings are improved in Figure 18.5. The question is at an application level, is more contiguous with the spreadsheet, and explanatory feedback is provided. However, the feedback statement “Great Job” may draw attention to the ego rather than the task. Research on feedback recommends that praise be avoided in lieu of explanations that focus on the task or process.

Our final screen sample from the asynchronous Excel course in Figure 18.6 shows a worked example with a self-explanation question. The lesson has demonstrated inputting an incorrectly formatted formula in Cell E6 to calculate February profit. The self-explanation question requires the learner to evaluate the demonstration by identifying the error in the formula.

Review of Sample 2: Synchronous Excel Lesson

Figures 18.7 through 18.10 are taken from a virtual classroom lesson on How to Use Excel Formulas. Synchronous e-learning continues to gain market share in e-learning solutions since our third edition, and you can apply most of the principles in this book to virtual classroom lessons. The goal of the sample lesson is to teach end-user spreadsheet procedures. The lesson objectives are:

• To construct formulas with valid formatting conventions
• To perform basic calculations using formulas in Excel

Figure 18.7 shows a content outline. In applying guideline 17 based on the pretraining principle, the procedural part of the lesson is preceded by important concepts. Before learning the steps to input a formula in Excel, the lesson teaches the concept of a formula, including its formatting conventions. When teaching the procedures, the lesson applies guidelines for worked examples by starting with a full worked example accompanied by self-explanation questions and fades to a full practice exercise.

Although virtual classroom tools can project a video image of the instructor, in this lesson the instructor uses audio alone. Research we reviewed in Chapter 9 shows that it is the voice of a learning agent—not the image—that is most instrumental in promoting learning. Since the main instructional message is displayed on the whiteboard slides, the instructor decided to minimize the potential for split attention caused by a video image. The introductory slide is shown in Figure 18.8. The instructor places her photo on this slide to implement guideline 53 to promote social presence. The instructor further builds social presence by inviting participants to use their audio as they join the session and by calling them by name during the session. One of the advantages of the virtual classroom is the opportunity to leverage social presence during learning through chat and audio participation of the learners as well as through collaboration in breakout rooms.

Figure 18.9 illustrates guidance in the form of example fading and memory support in the virtual classroom. The spreadsheet window in the center of the virtual classroom interface is projected to the learners through application sharing. The instructor has completed the first step in the procedure by typing the equal sign and the first cell reference into the correct spreadsheet cell. The instructor asks participants to finish the example by typing the rest of the formula in the chat window. Note that in applying guideline 20, the directions are displayed on the screen in text, since participants need to refer to them as they work the exercise. Additional memory support is provided in the left-hand box on the spreadsheet, which displays the valid operator syntax. The amount of guidance in this example should be faded as the lesson progresses.

From this brief look at some virtual classroom samples, you can see that just about all of the principles we describe in the book apply. Because the class proceeds under instructor rather than learner control, it is especially critical to apply all guidelines that reduce extraneous mental load. Lesson designers should create effective visuals to project on the whiteboard that will be described verbally by the instructor, applying the multimedia and modality principles. The instructor should use a conversational tone and language and incorporate participant audio to apply personalization. Skill-building classes can apply all of our guidelines for faded worked examples and effective generative methods to promote relevant engagement. The presence of multiple participants in virtual sessions lends itself to collaborative projects. Most virtual classroom tools offer breakout rooms in which small teams can carry out assignments. Apply guidelines 47 through 53 as you plan collaborative activities. As with asynchronous e-learning, instructors should minimize irrelevant visual effects, stories, themes, or audio in accordance with the coherence principle.

Review of Sample 3: Automotive Troubleshooting Simulation

In Chapter 1 we identified the opportunity to accelerate expertise as one of the unique promises of digital learning environments. Figures 18.10 through 18.13 are from a simulation designed to give experienced automotive technicians compressed opportunities to practice unusual troubleshooting situations. The learner starts with a point of view perspective in the auto shop that includes all common troubleshooting tools. In Figure 18.10 you see the trigger event for the case in the form of a work order. Typical of guided discovery learning environments, the learner is free to use various tools in the shop to diagnose and repair the failure. There are several sources of guidance. First, a telephone offers technical advice. Second, the computer opens to the actual reference system the technician uses on the job. Third, if the learner clicks on a tool that is irrelevant to the current problem, the system responds that the test is not relevant to this problem. This response constrains the environment in order to guide learners to the specific tests relevant to the case.

If the learner selects an incorrect failure and repair action, the high idle indicated in Figure 18.12 shows the learner that the failure has not been resolved. Once the case is correctly resolved, the learners get feedback and an opportunity for reflection by comparing their activities in the right window with those of an expert shown in the left window in Figure 18.13.

This lesson applies most guidelines 58 through 64 applicable to e-learning to build thinking skills. By situating the learner in a typical automotive shop, she has virtual access to the tools and resources she would have on the job. The goal, rules, activities, and feedback of the simulation are all aligned to the desired learning outcome, that is, to promote an efficient troubleshooting process to identify and correct the failure. Learners can see a map of their steps and compare it with an expert approach. Thus, the lesson focuses not only on finding the correct answer but on how the answer is derived. There are several sources of structure and guidance available congruent with guideline 62.

Since the structure of the case study is guided discovery, it emphasizes learning during problem solving. Regarding navigation, there is a high level of learner control. Overall, our assessment is that this course offers a good model for simulation environments designed for workers with relevant background knowledge and experience.

Reflections on Past Predictions

In the following section, we review our predictions from our first edition, followed by our observations fifteen years later. Because developing e-learning material for workforce learning is an expensive commitment, we had predicted more examples of online training that apply guidelines proven to lead to return on investment. Specifically, we made the following predictions:

• Fewer Las Vegas–style courses that depress learning by overuse of glitz and games. Instead, the power of technology will be leveraged more effectively to support acquisition and transfer of job-related skills.
• More problem-centered designs that use job-realistic problems in the start of a lesson or course to establish relevance, in the body of the lesson to help learners build related knowledge and skills, and at the end of the lesson to provide practice and assessment opportunities.
• More creative ways to blend computer technology with other delivery media so that the features of a given media are best used to support ongoing job-relevant skill requirements.

Beyond 2016 in Multimedia Research

In this fourth edition, we have been able to add principles and to qualify previous principles based on the expanding research base that is our primary source of guidance. For some principles, so many experiments have been published that meta-analytic studies offer us opportunities to not only recommend guidelines with greater confidence but also to qualify the situations in which those guidelines will be most applicable. For example, a recent review of the modality principle (use audio to describe complex visuals) included more than sixty research studies comparing learning from lessons that presented words in audio with lessons that presented the same words in written text (Mayer & Pilegard, 2014). A median effect size of 0.76 was reported based on diverse lessons that focused on lightning, fish locomotion, graph reading, science games, geometry, and electric circuits, to name just a few. This substantial effect size based on a large number of experiments gives us greater confidence in the modality principle. In addition, meta-analysis also gives insight into the boundary conditions under which any given principle is most effective. For example, the modality principle applies most when learners are novice to the content and when the audio segments are brief.

What changes do we look for in multimedia research over the next ten years? Our predictions follow.

In Conclusion

We began this book in Chapter 1 with a summary of the unique promises and pitfalls inherent in digital technology for instruction. We hope that the guidelines and evidence that we have described in this fourth edition will be a resource that minimizes the pitfalls and optimizes the promise of multimedia learning in your instructional environments.