15
Test! Test! Test!
,I urge you to adopt the mantra: “Test! Test! Test!” Usability testing is that important.
Any time you have someone try to use what you’ve developed you are doing usability testing. If you ask someone to read your web content out loud and tell you what it means, you are doing a usability test. If you have someone try to find information on your web site and have them think out loud while doing it, you are doing a usability test.
Usability testing is an incredibly important part of the toolkit of every user experience (UX) specialist – but it’s not the only one. We’ve already discussed
• predesign user research, personas, scenarios – Chapter 2
• content strategy, including inventory and analysis of content – Interlude 1 after Chapter 2
• card sorting – Chapter 5
• persona-based, conversation-based review – Chapter 14
• style guide – Interlude 5 just before this chapter
Why do usability testing?
Almost all usability testing that web teams do now is “formative” testing – having site visitors (or people you would like to be site visitors) try out the web site or app before you launch it. The purpose is diagnostic. You want to see what is working well and what is not working well, so you can fix what is not working well before you launch.
You must do usability testing because
• you are almost certainly not typical of your site visitors
• what matters is what works for the site visitors – not what different members of the web team want or think is best
• by watching a few people struggle, you save many people from pain, frustration, and failure
• the earlier you find what’s not working, the easier and less expensive it is to fix
Don’t assume; test! Don’t argue; test! Don’t risk failure after launch; test!
A few words about words
The people who come to help you in a usability test are “participants.” Don’t call them “subjects.” You’re not doing a psychology experiment or a medical study.
Be careful how you talk about “test.” You are not doing “user testing” – that would be testing the users. You are doing “usability testing” where the participants test the site or app.
Don’t say, “We want to test how well people can use the site.” That blames problems on the users. Say, “We want to know how the site works for people like you.” That puts the burden on the site.
What you learn in a usability test can help you change the site or app. You cannot change the users.
Although I use the words “usability testing” when I’m talking to colleagues, I never say “testing” or even “evaluation” to participants. I ask people to come “try out the site.” When they are with me, I tell them that they’ll be “working with” a particular site. That’s much better than trying to convince them that you are not testing them after you’ve used the words “usability testing.”
What’s needed for usability testing
Usability testing has many variations, but all usability tests share these six attributes:
• Real issues. You have thought about what you want to learn and planned the test to give you answers to your questions.
• Real people. Participants represent (at least some of) the site visitors or app users you want.
• Real tasks. The stories (scenarios, conversations) you have them try out with the web site or app are ones that they really want to do or that are realistic to them.
• Real data. You watch, listen, ask neutral questions, and take notes as they work. (In remote unmoderated tests, you may get only what they did – clickstream data – without hearing why or being able to ask questions.)
• Real insights. You put away your assumptions and biases as you review the data. You see what is working well and what is not.
• Real changes. You use what you learned. You keep what is working well and improve what could be better.
What’s not needed for usability testing
You don’t need:
• Finished product. Don’t wait until your product is finished. Test early. Test often. Test your current site or app before you revise it. Test with paper prototypes. Be informal. Just do it.
• Special space. If you have a usability lab, great! But it’s not necessary. Test in someone’s cubicle, workers’ break room, participants’ homes, at a conference where the people you want to work with gather, or in a conference room – as in Figure 15-1.
• Special software. It’s great if you have it. Several excellent programs are available to capture what is happening and to allow you to take notes. But they are not necessary. When I’m sitting next to a participant, I still use paper and pencil – even if I have a note-taker using software in another room.
• Videotape. The most common use of videotape is to convince people who don’t yet believe in usability testing or who think that only stupid users have problems. Seeing is believing. If you can’t get people to come observe the test in person (which they should!) – or if you don’t have space and have too many observers – then, yes, videotape the sessions. But don’t assume you’ll do detailed data analysis from the tapes. In most schedules, there’s no time for that.
• A formal test report (maybe). What you need as a report depends on the type of usability testing you are doing and on your organization’s or client’s culture and process. You may only need a quick list of findings and recommendations that everyone agrees on.
• Participants to come to you. You save time and can schedule more sessions if participants all come to the same place, but it’s not necessary. You can test remotely with you in one place, the participant in another, and observers each at their own location. If you go to people’s work places or their homes, you also get to see their web setups as well as their physical and social environments. So, don’t let “people don’t have time to come to do a usability test with us” stop you.
Figure 15-1 An informal usability test of a web site, using a paper prototype. Whitney Quesenbery is taking notes at the far table. Caroline Jarrett facilitated the session and took the photo. Used with permission of The Open University, the photographer, and the people in the photo.
How do we do a usability test?
No matter what variation you use, every usability test – like other user research projects – goes through five phases:
In the following sections, I summarize several ways to do usability testing.
What most people do
Formative testing. Small-scale. Five to 15 people, one at a time. Each session is about one hour. The participant works with the site or app while an experienced facilitator sits with the person (or in an adjacent room).
If you have not yet done usability testing, learn how from these excellent resources: Barnum, 2011; Krug, 2010; Rubin and Chisnell, 2008; www.usability.gov
Here’s what typically happens in a usability test session:
• The facilitator starts with a few questions to learn more about the participant and to make the participant comfortable.
• Then the participant tries to complete tasks (conversations, scenarios) with the site or app while thinking out loud.
• At the end, the facilitator asks a few more questions to get the participant’s reaction to the site and to cover any specific issues (such as understanding of specific words on the site).
• Some groups use a formal rating-scale questionnaire at the end.
You can do this with or without video- and audiotaping. Someone should definitely take notes, but it can be with a specialized software program, word processing, or even on paper. Everyone connected to the project should observe either in person or remotely.
The level of analysis, reporting, and recommendations vary widely.
An interesting look at what different groups do for reports from formative usability testing: Theofanos and Quesenbery, 2005 (Although this paper dates from several years ago, the information in it is still what people do.)
Even quicker: “A morning a month”
Formative testing. Three people. An hour each. One morning during the month. Teams should commit to doing testing this way every month on a regular schedule. Each month, the team selects specific issues to focus on and promises to act right away on the top insights from that month’s testing. Each session follows the same pattern as in the previous description of “what most people do.” Team members should observe the testing because everyone who observed (and only those who did) come together immediately after the three sessions to review insights and agree on changes. Brief write-up on agreement about insights and fixes. No formal report.
“A morning a month” is the method in Krug, 2010.
Include people with special needs in your usability test to be sure that the site or app works for them.
Information on recruiting and working with seniors in usability testing – resources at http://redish.net/articles-slides/articles-slides-older-adults
What variations might we consider?
You can test
Remotely, with a facilitator
You can do remote, facilitated tests with either method that I just described. The only difference is that people are not all in the same physical place. You use collaboration software to pass control of the screen from facilitator to participant and back again. You listen and talk either through the computer program or through a telephone connection. Observers can join from anywhere.
Working remotely greatly expands the reach of your testing. You can recruit the best participants no matter where they are. You can gain diversity among your participants without travel expenses.
Remotely, without a facilitator
With unmoderated testing, you trade the richness of think aloud for quantitative results from many people. Unmoderated testing may work well to know if people can find a specific piece of information. It’s not the best technique for testing content when you want to know if the site has the content people need or if they understand the content (beyond finding a specific fact).
Many vendors now offer tools for unmoderated usability testing. For a good article on the pros and cons of unmoderated testing with a list of vendors – Soucy, 2010
Testing around the globe
Context matters. Culture matters. Language matters. If your web site is global, you should be sensitive to cultural differences. If you can, do usability testing in many contexts, cultures, and languages. You can do that either with local facilitators in different countries or remotely from one country to another.
Barnum, 2010, has a running case study of testing the Chinese version of a hotel web site.
Other resources:
Quesenbery and Szuc, 2011, on how UX practitioners work globally
Dray and Siegel, 2005, on many aspects and cautions for international UX work
Before you do any user-centered activities outside your own culture, educate yourself about working globally.
Testing in a group setting
In a typical usability test, the same facilitator holds individual sessions one after the other. And there’s value in the continuity and consistency of having the same person interact with all the participants. But if you can’t afford the time, you might do simultaneous sessions.
Hal Shubin of Interaction Design, Inc. (www.user.com) shared this story:
The client needed quick feedback. Hal got four laptops, four participants, and four facilitators (himself and three quickly-trained people from the client’s web team). They did four one-on-one sessions at the same time around the dining room table of one of the facilitators. Then, all eight people – participants and facilitators – gathered in the living room to discuss what happened during the test sessions. Hal says the client learned a lot about how their customers use the web site and got good ideas about what needed to be done – all in one evening. It helped, of course, that the participants they selected were comfortable working in the same room and being together in the discussion afterwards.
Fielding alternatives (A/B testing)
Today, a common way to find out what works best with actual users is to launch different versions of content and see which does best in the marketplace. This is called A/B testing; although, of course, you could compare A, B, C, D, or more versions.
Check out http://whichtestwon.com. Each week, you get to see the results of a different real A/B test – and you get to match your guess about what happened in the test against other people’s guesses and the actual results.
In A/B testing, “best” is the version that achieves a measurable goal: more sales, more completed forms, more pages looked at, and so on. Because conversion rates (getting people to click on an ad and then to complete a transaction) is so important to e-commerce companies, most now routinely use A/B testing.
A/B testing gives you real-world data. If you change only very specific aspects of the site, you know what made the difference. However, with A/B testing, you don’t know why. Without listening to people, you don’t know what about the change made it work (or not work). It may be hard to generalize from the results.
Why not just do focus groups?
If your web team is part of a marketing department or you are in an agency that works primarily with marketing departments, you may hear, “we do usability testing; we get people together in a focus group.”
A focus group is not the same as usability testing. They are different techniques that get you different information. Table 15-1 shows several points of difference.
Table 15-1. A focus group is different from a usability test
| Focus Group | Usability Test | |
| People | 8 – 10 at a time | 1 at a time |
| Time | usually 2 hours | 1 hour or less |
| Activity | talk; a group discussion where people react to what they are shown or asked | behavior, people actually use the site; usually also a brief individual interview without the influence of other group members |
| Good for | getting people’s opinions, attitudes, desires, self-report | seeing how people actually do tasks; watching people interact with the site or app |
| Best time to do | only very early in the design process | iteratively, throughout the process from testing old site or app before revising, to paper prototypes, to working prototypes, to final site |
What does a focus group need?
A well-run focus group is
• planned carefully with a prepared script and questions to ask
• facilitated by a trained person who can draw out the shy people and keep the overeager people from dominating the discussion
• allowed to stray from the script along interesting and relevant lines but kept within bounds so you get the information you need
Why isn’t a focus group the best technique?
Focus groups are not the best technique for understanding how a web site or mobile app works for site visitors for two major reasons:
• Group dynamics. One person can trigger others agreeing to something they might not otherwise say. Even with a great facilitator, some people may not express their true feelings in the group setting.
• Difference between talk and action. What people say they do is often not what they actually do. People are not very good at imagining their own behavior. They have to actually put their hands on and “do” to realize when and how something will or won’t work for them.
Can we combine usability testing and focus groups?
If you need to, you can combine aspects of usability testing with a focus group:
• First part of the time: Have people use the product either with a person watching and listening to each participant or letting the participants work on their own and take their own notes on where they succeed and where they have problems.
• Second part of the time: Bring them together to talk about the experience.
Hal Shubin’s story would be an example. I’ve done this, calling it a task-based focus group.
A final point: Test the content!!
Lots of companies now do usability testing, but too many still focus only on the information architecture – on whether people can find what they need. That’s necessary. But it’s not enough. If your site visitors get to the right place, but the content doesn’t work for them, they’ll be frustrated and leave.
Don’t stop when site visitors get to the right web page. Have them actually find the information on the page to satisfy their conversation or your test scenario.
If they are doing a transactional task (buying something, booking a ticket), watch them complete the task.
If they are looking up information, ask them to tell you what the content says. Don’t just assume they have absorbed the message because they looked at the page.
Watch what they do on content pages. Do they skim? Do they read carefully? What parts do they read? Listen to what they say about the content.
Do usability testing to learn:
• Does the site have the content your site visitors want and need?
• Is the content presented with good information design – legible typography, good spacing, visible and useful headings?
• Is the content organized and broken up in a way that works for your site visitors?
• Does the writing help site visitors skim easily and read quickly?
• Are you using words that your site visitors understand immediately?
• Do site visitors interpret images to have the messages you meant?
Yes, you might ask them what they think of what they found. But also ask them to tell you what messages they got from the content. Test understand and use as well as find.
I hope this book has helped you create great content so that usability testing shows you are meeting your business goals by satisfying your site visitors’ conversations.