How do you know whether your instruction has been effective?
How do you know whether your objectives have been met?
What are the components involved in an effective evaluation?
How can Flash send performance data?
Why are user tests important?
What are the differences between formative and summative evaluation?
The final step in designing online learning is to evaluate and revise the instructional materials that have been constructed. This process occurs, to some extent, throughout the design and development process. However, it is important to focus on evaluation at the end of the process because only then can the application be viewed holistically.
Formative evaluation is undertaken periodically while online learning is being designed and developed. Formative evaluation is evaluation done before it is finished. The advantage of formative evaluation is that problems can be identified early. Problems have a tendency to multiply when decisions are being made based on what has already been designed and developed. Figure 14.1 demonstrates how evaluation and revision relates to the entire Online Learning design and development process.
The client should be consulted in the Planning stage once the needs have been identified and once the objectives have been declared. This can save a substantial amount of time, particularly when a designer is working with an external client. This ensures that the client and the designer are on the same page. Any discrepancies can be worked out before any further time and resources are invested in the project.
It is also important to share the needs assessment and learning objectives with subject-matter experts. They can provide an expert's view of whether the important problems are going to be adequately addressed. Identifying needs and analyzing tasks is the foundation on which the rest of the development will rest. It is important to ensure that this part of the planning phase has been adequately addressed.
One useful strategy is to create a prototype. A prototype is a sample or part of an application that allows you to test and demonstrate a specific part or function. For example, a working prototype could be created in Flash that only has the content and interactions without graphics. Often graphics are tailor-made and dependent on the content. Those graphics might have to be remade if changes are required to the content.
Usability refers to an application's ease of use. As you design and develop your Flash application for online learning, you will make a number of assumptions about what the learner will experience. There are several areas that you can test, for example, making sure the application runs well on different sized screens, operating systems, and different browsers. Unfortunately, you will not be in the best position to judge because designers and developers tend to be too close to the application; many elements will appear to you to be clear and appropriate because you are familiar with the context. Your users will rarely have this level of familiarity and will often find the application confusing.
Designers and developers tend to report that a usability test is one of the most valuable events in the design process. Often, testers will provide a wealth of information on topics that the designer has never thought of addressing. It is important, however, not to engage in this process if you do not plan to listen to and implement suggestions. For example, you it makes little sense to have users test the navigation of an application if the clients have imposed one. Of course, if you believe a decision has been imposed that affects the instructional quality of an online application, you might use the data collected from usability tests to convince them that changes need to be made.
Usability tests do not have to be elaborate. Simply observing users without commenting is often sufficient to identify deficiencies and confusing elements in an application. It is important not to verbally, or through body language, provide information to the testers that they cannot obtain for the application. Remember, your target population will not have you sitting next to them. It is also important to remember that mistakes your testers make are because of the design and not because of any limitations they might have, if you keep that in mind, you will adopt an attitude of openness, which will assist you in collecting information.
It is important to select testers who are similar to your target population. For example, if the target population is college educated, mid-level managers aged thirty-five to fifty, you will need to find testers who meet that same demographic. You will need about five testers from that demographic to give you a comprehensive idea of how well your application works. Further, it is important that you not reuse testers; once they have encountered the application once, they will have a biased experience.
Additionally, online learning applications should be tested over a number of different bandwidth connections. You should decide what is the lowest permissible bandwidth configuration tolerable. You also must consider that the Internet rarely provides consistent throughput. The more nodes that a user connects through, the more likely that the data transmission will be compromised. It is important to identify a number of sites that you will be delivering to and test the connections during different time periods of the day and week.
While you are observing, you should note each area of difficulty in a table so that you can compare different testers' responses. This will help you identify problems that consistently come up. One user having difficulty in a particular area may be an anomaly; however, if multiple testers are reporting challenges, it is a sign that there is something wrong with the application. Problems should be prioritized according to which ones cause the most users difficulty.
After you have silently observed and taken notes of the tester working through the application, you can follow up with some targeted questions. Table 14.1 includes a list of questions that can inform a user test.
Table 14.1. Post-User Test Questions.
What is confusing? | How can it be improved? | |
Were the objectives easily comprehended? | Is the language used clear and easily understood? | Does it load in a timely manner? |
Did you have to wait for content? | Is there a method or responding to the developer? | Does the application have a consistent look? |
Is navigation intuitive and consistent? | Is the placement of graphics consistent? | |
Was it easy to find the information you needed? | Was it easy to navigate? | |
Did things work as you expected? | Were you bored at anytime? | |
Were the color combinations pleasing? | Did the graphics support the text? | |
Were practice opportunities clear? | Were practice sequences staged appropriately? | |
Summative evaluation comes after the application has bee designed, developed and implemented; it is concerned with results and outcomes. The two big questions that summative evaluation answers are (1) Did the learners learn? and (2) Have the needs of the sponsoring organization been met? Whether the learner has learned can be evaluated in the short term and for the long term. The test that was designed to match the learning objectives in the planning phase should provide a good indication of whether the learners have learned the material.
The data describing how the learners perform can be collected and sent back to you (see Saving and Sending Data). This data gives you an important look at what is working or not working in the application. For example, if everyone is missing a particular question, you can be confident that material was not well instructed or that there is a problem with the questions. Either may need to be revised to ensure learning.
Long-term retention can be evaluated simply by extracting the assessment portion of the application and resending it to the learners a few months later; this aspect of online learning is an advantage over a face-to-face situation. With online learning users, can be contacted well after the original instruction to determine whether they not only learned but retained the knowledge and skills.
The final summative evaluation technique is to determine whether or not the problems that existed that prompted the call for an instructional intervention have been resolved. For example, if a new piece of machinery was the event that triggered the necessity for training and if the target employees are successfully using the machine, then the needs have been satisfied. Another example would be if complaints to a customer service center were reduced after the training was implemented.
An onsite visit or an interview with managers and co-workers or colleagues may be helpful in determining whether a need has been met. The important thing is to gather data after the instruction and match it with data gathered during the needs assessment from the planning phase. As with all data collection for evaluation, it is a good idea to provide responders with an opportunity to provide some open-ended responses. If needs have not been met, it is a sign that the instruction needs to be revised.
It is also a good idea evaluate the process of development at this point. If this was a team project, how well did the team work together? Did you have the necessary design and technical skills? What resources are needed for future projects? Additionally, you should consider the efficiency of the process. How many hours did it take to develop the applications? What were the costs involved? How many learners benefited. If the Flash application was a part of a larger instructional event, then you should determine how well it was integrated. Was Flash the best platform for the content delivered? Should more be added?
This chapter discussed formative and summative evaluation techniques used to ensure that your instruction meets the learning objectives and is used by the learners as you have intended. Instructional materials are always in a state of revision. You should be continuing to ask for feedback from your learners on their experience with your applications.
This chapter concludes the text. Please refer to the text's website for further details on Designing Online Learning with Flash.