RESEARCH METHOD

94 Usability Testing

Usability testing focuses on people and their tasks, and seeks empirical evidence about how to improve the usability of an interface.¹

Usability testing is an evaluative method that allows teams to observe an individual’s experience with a digital application as he or she walks through the steps of a given task (or set of tasks). The method is designed to help teams identify the parts of an interface that most regularly frustrate and confuse people so that they can be prioritized, fixed, and retested prior to launch.

Tests are designed around tasks and scenarios that represent typical end-user goals. It is common practice that everyone on the interdisciplinary team works together to identify usability testing tasks and scenarios. Tasks should be specific, concrete, and reflect actual goals of the target audience. Scenarios contextualize the task, and are written to provide extra information necessary to complete the task. Tasks and scenarios should neither influence the participant to solve a problem a certain way, nor seek to justify product requirements (which often reflect system or developer goals).

Usability tests typically follow the format of the Think-aloud Protocol technique. Some of the errors that observers and evaluators should try to detect include any instance where the participant:²

1. understands the task but can’t complete it within a reasonable amount of time;

2. understands the goal, but has to try different approaches to complete the task;

3. gives up or resigns from the process;

4. completes a task, but not the task that was specified;

5. expresses surprise or delight;

6. expresses frustration, confusion, or blames themselves for not being able to complete the task;

7. asserts that something is wrong or doesn’t make sense; or

8. makes a suggestion for the interface or the flow of events.

As usability tests reveal problems, the team will realize that how they evaluate and use the interface is different from how typical end users do.³ Also, just as the number of participants in the test directly impacts the number of problems that are detected,⁴ so do the number of evaluators—the more evaluators, the more problems will be detected.⁵

Aside from experiment validity, empiricism, and avoiding bias, the key to successful usability testing is to require the attendance of developers and project stakeholders at research events. Many teams are making the usability test observation session the only opportunity to see and weigh in on prototypes prior to launch. With this approach, you are guaranteed to have observers participate in the empirical testing process where they can observe and weigh in on usability problems firsthand.

1. Gould, John D., and Clayton Lewis. “Designing for Usability: Key Principles and What Designers Think.” Communications of the ACM 28, no. 3 (1985): 300-311.

2. Jacobsen, Niels Ebbe, and Bonnie E. John. “The Evaluator Effect in Usability Studies: Problem Detection and Severity Judgments.” Proceeding of the Human Factors and Ergonomics Society 42nd Annual Meeting, 1998.

3. Mack, Robert, Clayton H. Lewis, and John M. Carroll. “Learning to Use Word Processors: Problems and Prospects.” ACM Transactions on Information Systems 1, no. 3 (1983): 254-271.

4. Virzi, Robert A. “Refining the Test Phase of Usability Evaluation: How Many Subjects is Enough?” Human Factors 34, no. 4 (1992): 457-468.

5. See note 2 above.

Further Reading

Barnum, Carol. Usability Testing Essentials: Ready, Set...Test! San Francisco, CA: Morgan Kaufmann, 2010.

Krug, Steve. Don’t Make Me Think, 2nd ed. Berkeley, CA: New Riders Press, 2006.

Krug, Steve. Rocket Surgery Made Easy. Berkeley, CA: New Riders Press, 2010.

Unlike attitudinal studies, behavioral experiments such as usability tests can be used with fewer participants to isolate enough problems to help teams confidently decide whether an interface needs to be revised, or is ready for release. When the testing process is included early in the process, over several rounds of testing the team will gain confidence in the usability of the interface as fewer problems are identified.