In-person lab research used to be the only game in town, and as with most industry practices, its procedures were developed, refined, and standardized, and then became entrenched in the corporate R&D product development cycle. Practically everything gets tested in a lab nowadays: commercial Web sites, professional and consumer software, even video games (see Figure 1-1).
Part of the appeal of lab-based user research was that it provided a seemingly scientific basis for making decisions by using observational data, instead of someone’s error-prone gut instincts. Stakeholders appreciated the firm protocol and apparent reliability of properly managed lab research. Lots of user research practitioners continue to perform lab research just because it’s what people have been doing for a long time.
Market Research vs. User Experience Research
Let’s make something clear. Focus groups are practically synonymous with user research in most people’s minds, and focus groups belong to the world of market research. But there’s a huge difference between market research and user experience (UX) research. Market research is much more common and comprises the lion’s share of research spending; UX research comprises just a fraction of that (see Figure 1-2).
Figure 1-2. The relationship between market research and UX research. The two fields seem similar, but they have different goals and take different forms.
However, this book is about user experience research, not market research. The main difference between the two fields is that market research focuses on opinions and preferences, whereas UX research focuses on behaviors. The distinction can be confusing, especially since a lot of online consumer research companies try to convince you they give you insight into “what your customers are doing on your Web site,” when they’re really just providing opinions.
A market research study might have goals like these:
“Determine how users respond to our branding.”
“Identify different segments’ color preferences for the homepage.”
“See if users like our new mascot.”
“Determine what users enjoy most and least about our site.”
While the goals of a UX study, on the other hand, would sound more like these:
“Can anyone actually use my interface?”
“Determine where users make errors in completing a purchase.”
“See whether users can successfully create a playlist.”
“Understand why users aren’t logging in.”
“See how users mentally organize different product categories.”
It’s important to keep in mind that market research is pretty useless over small sample sizes. Opinions can vary widely across demographics and location, are very sensitive to the phrasing of the research questions, and can change fairly quickly. Behavior, on the other hand, is fairly consistent across demographics and location for many tasks, and most usability flaws in a given software interface can be uncovered in a moderated study by a much smaller number of users.
How small? That’s a contentious question. UX luminary Jakob Nielsen (in)famously claimed that having five users was enough to uncover 80% of usability flaws in an interface, but others like Jared Spool insist that the number depends on factors such as user segmentation, risks associated with the errors, task complexity, and so on. At any rate, our point is that a moderated UX study usually requires much fewer people than a market research study.
Put it this way: ask 10 people what they think about how well a door is designed, and their comments might not overlap at all. One blames the condition of the hinges, another talks about the weight of the door, another complains about the color of the doorframe, and so on. But if you observe 10 people walking through the door, and the first two accidentally try to push when they ought to pull, then you’ve found your design flaw right there.
So, to put it all together: whether you go with market research or UX research depends on what you’re trying to find out. This book is about UX research, so it’s focused on user behaviors rather than opinions.
FURTHER READING ABOUT THE SAMPLE SIZE QUESTION
Turner, C. W., Lewis, J. R., and Nielsen, J. (2006). Determining usability test sample size. In W. Karwowski (ed.), International Encyclopedia of Ergonomics and Human Factors (pp. 3084–3088). Boca Raton, FL: CRC Press.
Lewis, J. (2001). Evaluation of procedures for adjusting problem-discovery rates estimated from small samples. The International Journal of Human–Computer Interaction 13(4), 445–479.
Lindgaard, G., and Chattratichart, J. (2007). Usability testing: What have we overlooked? In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (San Jose, CA, USA, April 28–May 03, 2007). CHI ’07. ACM, New York, NY, 1415–1424.
Heck no. Lab and remote research share the same broad purpose: to understand how people interact and behave with the thing you’ve made (from here on, let’s just call it “the interface”). There’s no need to set up a false opposition between the two approaches—one isn’t inherently better than the other. Despite the versatility of remote research, there are lots of reasons you might want to conduct an in-person study instead, most of which have to do with security, equipment, or the type of interaction you want to have with your research participants. More generally, lab research is appropriate when you need a high degree of control over some aspect of the session, such as the following situations.
Info security. Security is often a concern for institutions like banks and hospitals, which deal in sensitive information, or companies concerned with guarding certain types of intellectual property. If you’re testing a top-secret prototype, you obviously don’t want to let people access something from their home computer, where it could be saved or screen-captured. On the other hand, you might also be doing a study on users who would be secretive about sharing what’s on their screen—government employees, doctors, or lab technicians, for instance. Either way, you’ll want to test users in a controlled lab environment to keep things confidential, especially if what you’re testing is so hush-hush that you must have your users sign a nondisclosure form.
Inability to use screen sharing. You might also want to use a lab if your users are unable to share their screen over the Internet, for whatever reason. Some studies (of rural users, cybercafe patrons, etc.) may require you to talk to users who don’t have reliable high-speed Internet connections, who own computers too slow or unstable to use screen sharing services effectively, or who have operating systems incompatible with the screen sharing tools you’re using. These restrictions apply only to moderated studies, for which you need to see what’s on your users’ screens.
The need for special equipment. Depending on the interface you’re testing, you may require certain special software or physical equipment to run the study properly, which is most often the case with software that’s still under development. Getting users to install and configure tools to run elaborate software can be a pain (though that’s not unheard of), and requiring users to have certain equipment can make recruiting needlessly difficult.
The importance of seeing the user’s body. Some kinds of research will require you to study certain things about the user that are difficult to gather remotely. UX research has recently begun using eye-tracking studies, and for that kind of study, you’d need to bring the users to the eye-tracking device. Other studies might require you to attend to the participants’ physical movements, which may be difficult to capture with a stationary webcam. And then there are multiuser testing sessions, in which a single research moderator facilitates many participants at once; screen sharing is currently not well suited to sharing multiple desktops at once, though some tools (e.g., GoToMeeting) make it relatively painless to switch from one desktop to another. We want to emphasize, however, that for most studies, seeing the user at all is not actually important; we explain why in Chapter 5, (see Ain’t Nothing Wrong with Using the Phone) and Chapter 10.
Although these situations are all compelling reasons to conduct in-person research, part of what we want to demonstrate in this book is that remote research is very broad and adaptable, and even if a study is conducted in a lab, elements of remote methods can be adapted and incorporated to enhance in-person research methods. We’ll get to that in Chapter 9.
Again, most studies can successfully be done either in person or remotely, but, just as there are times when lab testing is more appropriate, there are also times when it makes more sense to use remote research methods.
Remote research is more appropriate when you want to watch people performing real tasks, rather than tasks you assign to them. The soul of remote research is that it lets you conduct what we call Time-Aware Research (TAR). By now, UX researchers are familiar with the importance of understanding the usage context of an interface—the physical environment where people are normally using an interface. Remote research opens the door to conducting research that also happens at the moment in people’s real lives when they’re performing a task of interest. This is possible because of live recruiting (the subject of Chapter 3), a method that allows you to instantly recruit people who are right in the middle of performing the task you’re interested in, using anything from the Web to text messages. Time-awareness in research makes all the difference in user motivation: it means that users are personally invested in what they’re doing because they’re doing it for their own reasons, not because you’re directing them to; they would have done it whether or not they were in your study.
Consider the difference between these two scenarios:
You’ve been recruited for some sort of computer study. The moderator shows you this online map Web app you’ve never heard of and asks you to use it to find some random place you’ve never heard of. This task is a little tricky, but since you’re sitting in this quiet lab and focusing—and you can’t collect your incentive check and leave until you finish—you figure it out eventually. Not so bad.
You’ve been planning a family vacation for months, but you’ve been busy at work so you procrastinated a bit on the planning, and now it’s the morning of the trip and you’re trying to quickly print out directions between finishing your packing and getting your kids packed. Your coworker told you about this MapTool Web site you’ve never used before, so you decide to give it a shot, and it’s not so bad—that is, until you get stuck because you can’t find the freaking button to print out the directions, and you’re supposed to leave in an hour, but you can’t until you print these damn directions, but your kids are jumping up and down on their suitcases and asking you where everything is. Why can’t they just make this stupid crap easy to use? Isn’t it obvious what’s wrong with it? Haven’t they ever seen a real person use it before?
Circumstances matter a lot in user research, and someone who’s using an interface in real life, for real purposes, is going to behave a lot differently—and give more accurate feedback—than someone who’s just being told to accomplish some little task to be able to collect an incentive check. Time-awareness is an important concept, so we’ll bring it up again throughout this book to demonstrate how the concept relates to different aspects of the remote research process (recruiting, moderating, and so on).
Note
TAR KEEPS YOU IN THE RIGHT 1985
Remember that diagram in Back to The Future II? Doc argues that messing with time has sent the world crashing hopelessly toward an alternate reality where things are horrible—the “wrong 1985.” And that’s sort of what happens when you try to assign people a hypothetical task to do at a time when they may or may not actually want to do it: you’re meddling with their time, and it’ll create results that look like the real thing but are all wrong.
When you schedule participants in advance and then ask them to pretend to care, you’re sending your research into the wrong 1985.
If you don’t want to create a time paradox— thereby ending the universe—you should do time-aware research.
Here are some additional benefits of remote research.
Geographic diversity. Even if you do have a lab, the users you want to talk to may not be able to get to it. This is actually the most common scenario: your interface, like most, is designed to be accessed and used all around the world, and you want to talk to users from around the world to get a range of perspectives. Will Chinese players like my video game? Is my online map widget intuitive even for users outside Silicon Valley? Big companies like Nokia and Microsoft are often able to conduct huge, ambitious research projects to address these questions, coordinating research projects in different labs around the world, flying researchers around in first-class. If you don’t have the cash for an international Gorillas-in-the-Mist project, then remote research is a no-brainer solution. If you can’t get to where your users are, test them remotely.
Ability to test almost anywhere. Remote research has comparatively minimal setup requirements and can reach anywhere that computers and the Internet can go: you can be anywhere; your participants can be anywhere. Lone-wolf consultants and start-up teams working out of cafés can have trouble finding the quiet office space they need to do in-person testing. If it’s too much bother to set up a proper lab, go remote; all you’ll need is a desk.
Some reduced costs. Beyond travel expenses, other costs associated with lab testing may be reduced or eliminated when you test remotely. With live recruiting methods, you can get around third-party recruiting costs, and because the recruiting pool is larger, you may not have to offer as much in the way of incentives as you might otherwise to attract enough participants. Because sessions are conducted through the computer, you can use relatively inexpensive software to replace costly testing accessories, such as video cameras, observation monitors, and screen recording devices. (Note, however, that the overall cost of a remote research study is often comparable to an in-person study for many reasons; see Chapter 10 for reasons why.)
Quicker setup. Closely related to the issue of money, as always, is time. Nearly all existing recruiting methods take many weeks. Recruiting agencies usually require a couple of weeks to gather recruits, and writing out precise recruiting requirements and explaining the study to them can eat up a lot of time. Getting users from your own mailing list can be faster and moderately effective, but what if you don’t have one? Or what if you’ve overfished the list from previous studies, or you don’t want to spam your customers, or you’re looking to test people who’ve never used your interface or heard of your company before? In any of these cases, recruiting your users online makes a lot of sense, since it allows you to do your recruiting as research sessions are ongoing. (We teach you how to do all this in Chapter 3.)
Context-dependent interfaces. Some interfaces just don’t make any sense to test outside their intended usage environment. If you need users to have their own photos and videos to use in a video editing tool, having them bring their laptop or media to a lab will be a tremendous hassle. Or, let’s say you’re testing a recipe Web site that guides users step-by-step through preparing a meal; it wouldn’t make much sense to take people out of their kitchen, where they’re unable to perform the task of interest. When this is the case, remote research is usually the most practical solution, unless the users also lack the necessary equipment.
If you have the gumption, you can test almost anything remotely. There are ways to get around nearly any obstacle, but the approach you take is all about what’s most practical and accurate. If it’s significantly cheaper, faster, or less of a hassle for you to just bring people into a lab, then by all means bring ’em in.
Sometimes this decision can be a tough call; users in the developing world may have limited access to the Internet, for instance, so you’d have to decide whether it’s worthwhile to fly over and talk to users in person, or to find people from that demographic in your area, or to arrange for the users to be at a workable Internet kiosk to test them remotely.
For clarity’s sake, let’s talk about some clear-cut cases of things you should and shouldn’t test remotely.
Remote testing is a no-brainer for Web sites, software, or anything that runs on a desktop computer—this is the kind of stuff remote research was practically invented to test. The only hitch is that the participants need to be able to use their own computer to access whatever’s being tested. Other Web sites besides your own are a cinch: just tell your users during the session to point their Web browsers to any address you want. If you’re testing prototype software, there needs to be a secure way to digitally deliver it to them; if it’s a prototype Web site, give them temporary password-protected access. If the testing is just too confidential to give them direct access on their computer, you can host the prototype on your own computer and use remote access software like VNC or GoToMeeting to let them have control over the computer. There’s almost always a way to do it.
The stuff you test doesn’t even have to be strictly functional. Wireframes, design comps, and static images are all doable; we’ve even tested drawings on napkins (really). Just scan them in to a standard image format and put them on a Web site. Make sure the user’s browser doesn’t automatically resize it by using a plain HTML wrapper around each image. There are also plenty of software solutions (like Axure and Fireworks) that can help you convert your images to HTML.
Can you test programs that require users to enter personal information? Yes, but make sure to give your participants a way to enter “dummy” information wherever they’re required to enter sensitive or personally identifying information. (According to Rolf Molich, people act differently when using dummy information, so bear that in mind.) If you require the participant to use real personal information, be sure to obtain explicit consent right at the beginning of the testing session (an issue covered in Chapter 4); you don’t want to spend 20 minutes on the phone with a user only to have to terminate the study over privacy issues.
Most remote research tools (screen sharing, recording, chat, etc.) are suited for a computer desktop environment, so physical products are harder to test remotely. We’re just beginning to see mobile device and mobile interface research become feasible, and we’ve researched interfaces like cars and computer games using some remote research methods. Plus, webcams, Web video streaming, and wireless broadband are all becoming more accessible, so there’s plenty of hope. But physical interfaces will require you to come up with some creative solutions and workarounds beyond the standard remote desktop testing approach. These approaches may or may not be worthwhile; see Chapter 9 for examples of some of these alternative remote approaches.
Case Study: Lab vs. Remote
By Julia Houck-Whitaker, Adaptive Path (and Bolt | Peters alum)
In 2002, Bolt | Peters conducted two remote studies on the corporate Web site of a Fortune 1000 software company. Both studies used identical test plans, but one was executed in a traditional usability lab, whereas the other was conducted remotely using an online screen sharing tool.
SUMMARY
Our comparison showed key differences in the areas of time, recruiting, and resource requirements, as well as the ability to test geographically distributed user audiences. Table 1-1 summarizes the key differences we found comparing the two methods. There appeared to be no significant differences in the quality and quantity of usability findings between remote and in-lab approaches.
Table 1-1. Overview Comparison of Lab and Remote Methods 
http://www.flickr.com/photos/rosenfeldmedia/4286397757/
Lab | Remote | |
|---|---|---|
Number of Users | 8 | 8 |
Recruiting Method | Recruiting agency | Online live recruiting |
Recruiting Duration | 12 days | 1 day |
Testing Duration | 2 days | 1 day |
Location | Pleasanton, CA | CA, OR, NY, UT |
Avg. Session Duration | 85.6 min | 51.5 min |
Total Key Findings | 98 | 114 |
Approximate Cost | $26,000 | $17,000 |
Deliverables | Report, highlight video | Report, highlight video, survey responses |
DETAILED COMPARISON OF METHODS
Table 1-2, Table 1-3, and Table 1-4 break down the process for each of the recruiting, testing, and analysis phases, respectively. The left-hand column describes the lab study details; the right-hand column describes the remote study details.
Table 1-2. Lab vs. Remote Recruiting http://www.flickr.com/photos/rosenfeldmedia/4287138014/
Lab | Remote | |
|---|---|---|
Recruiting Channel | Third-party recruiting agency schedules users | Pop-up screener on the Web site |
Selection Method | Agency selects users based on predetermined recruiting criteria | Researcher selects users based on responses to screener questions |
Recruiting Pool | Local users are selected to avoid travel expenses | Visitors to the Web site |
Duration | 12 days | 1 day |
Recruiting for the lab-based study was outsourced to a professional recruiting agency (see Table 1-3). Ten users were recruited, screened, and scheduled by G Focus Groups in San Francisco, including two extra recruits in case of no-shows. Recruiting eight users through the recruiting agency took 12 days. Agency-assisted recruiting successfully provided seven test subjects for the lab study; the eighth recruit did not fulfill the testing criteria.
Recruiting for the remote study was conducted using an online pop-up from the software company’s corporate Web site. The recruiting pop-up, hosted by the researchers, used the same questions as the G Focus Groups’ recruiting screener. Users in both studies were selected based on detailed criteria such as job title and annual company revenues. Respondents to the online screener who met the study’s qualifications were contacted in real time by the research moderators. The online recruiting method took one day and yielded eight users total from California, Utah, New York, and Oregon. Normally, the live screener requires four days of lead time to set up, but in this case it was completed for a previous project so setup was not necessary.
Table 1-3. Lab vs. Remote Environment http://www.flickr.com/photos/rosenfeldmedia/4287138116/
Lab | Remote | |
|---|---|---|
User Surroundings | User in controlled lab environment | User in native environment |
Researcher Surroundings | On location with user | Testing globally distributed users from one location |
User Screen Observation | Practitioner sees user screen on her computer | |
User Observation | Through a one-way mirror | None |
Communication Medium | Via microphone and speakers | Via telephone |
Client Observation of Users | Through a one-way mirror; user audio, screen, and facial expressions are captured for later viewing | Live session observation via computer screen sharing; user audio and screen are captured for later viewing |
The lab study was also conducted from the software company’s in-house usability lab. The recruits for the lab study went to the lab in Pleasanton, California to participate and used a Windows PC. In addition to users’ audio and screen movement capture, users’ facial expressions were also recorded.
The remote usability study was conducted using a portable lab from the software company’s headquarters in Pleasanton, California. The live recruits participated from their native environments and logged on to an online meeting allowing the moderators to view the participants’ screen movements. The users’ audio and screen movements were captured.
The lab study uncovered similar issues of similar quality and usefulness to the client when compared with the remote study results (see Table 1-4). The remote study uncovered usability issues of high value to the client. The lab method uncovered 98 key findings, compared with 116 findings in the remote study (not a statistically significant difference).
Table 1-4. Lab vs. Remote Findings http://www.flickr.com/photos/rosenfeldmedia/4286397971/
Lab | Remote | |
|---|---|---|
Quality of Findings | High | High |
Number of Usability Issues Uncovered | 98 | 116 |
Video Deliverables | Highlights video with picture-in-picture | Highlights video with picture-in-picture, audio |
So, you’ve decided it’s worth a shot to try a remote research study. Feels good, doesn’t it? The first thing to know is that remote research can be roughly divided into two very different categories: moderated and automated research.
Figure 1-3. Moderated research: a researcher (“moderator”) observes and speaks to a participant in another location. Outside observers can watch the session from yet a third location and communicate with the moderator as the session is ongoing.
In moderated research, a moderator (aka “facilitator”) speaks directly to the research participants (see Figure 1-3). One-on-one interviews, ethnographies, and group discussions (including the infamous focus group) are all forms of moderated research. All the parties involved in the study—researchers, participants, and observers—are in attendance at the same time, which is why moderated research is also sometimes known as “synchronous” research. Moderated research allows you to gather in-depth qualitative feedback: behavior, tone-of-voice, task and time context, and so on. Moderators can probe at new subjects as they arise over the course of a session, which makes the scope of the research more flexible and enables the researcher to explore behaviors that were unforeseen during the planning phases of the study. Researchers should pay close attention to these “emerging topics,” since they often identify issues that were overlooked during the planning of the study.
Automated (or “unmoderated”) research is the flip side of moderated research: the researcher has no direct contact or communication with the participant and instead uses some kind of tool or service to gather feedback or record user behaviors automatically (see Figure 1-4). Typically, automated research is used to gather quantitative feedback from a large sample, often a hundred or more. There’s all sorts of feedback you can get this way: users’ subjective opinions and responses to your site, user clicking behavior, task completion rates, how users categorize elements on your site, and even your users’ behavior on competitors’ Web sites. In contrast to moderated research, automated research is usually done asynchronously: first, the researcher designs and initiates the study; then the participants perform the tasks; then, once all the participants have completed the tasks, the researcher gathers and analyzes the data.
There’s plenty of overlap between automated and moderated methods, but Table 1-5 shows how it generally breaks down.
Table 1-5. Moderated vs. Automated Research http://www.flickr.com/photos/rosenfeldmedia/4286398073/
Moderated | Automated | |
|---|---|---|
Timing | Synchronous—moderator speaks while participant uses Web site | Asynchronous—participant performs task, researchers analyze the results later |
Sample Size | Small—about 5 to 30, depending on user segmentation and study goals | Large—usually 50 or more |
Incentives | Usually -$75 per participant, but can vary depending on who’s participating | Can offer 1 in 10 participants a normal incentive (-$75) or a smaller incentive (-$5) to all participants |
Types of Research Goals | Discovering errors and usability issues, formative research, Time-Aware Research, real-world contextual research | Assessing performance on narrowly defined tasks |
Type of Research | Qualitative and behavioral, but can also ask opinions and gather rich contextual info | Quantitative, both behavioral and opinion-based, but can also ask brief open-ended questions—less context |
Moderated research is qualitative; it allows you to observe how people use interfaces directly. You’ll want a moderated approach when testing an interface with many functions (Photoshop, most homepages) or a process with no rigid flow of tasks (browsing on Amazon, searching on Google) over a small pool of users. Since it provides lots of context and insight into exactly what users are doing and why, moderated methods are good for “formative research” when you’re looking for new ideas to come from behavioral observation. Moderated research can also be used to find usability flaws in an interface. We cover the nitty-gritty of remote moderated research in Chapter 5.
Automated research is nearly always quantitative and is good at addressing more specific questions (“What percentage of users can successfully log in?” “How long does it take for users to find the product they’re looking for?”), or measuring how users perform on a few simple tasks over a large sample. If all you need is raw performance data, and not why users behave the way they do, then automated testing is for you. (Suppose you just want to determine what color your text links should be: testing every different shade on a large sample size to see which performs best makes more sense than closely watching eight users use three different shades.) Also, some automated tools can be used to gather opinion-based market research data as well, so if you’re looking for both opinion-based and behavioral data, you can often gather both in a single study. And certain conceptual UX tasks, like card sorting and A/B testing, are well supported by automated tools. See Chapter 6 for a more thorough look at the various automated research methods.
You don’t necessarily have to choose between moderated and automated testing, or even between lab and remote methods. You can even conduct multiple studies on the same interface, using the findings from one study to add nuance to another. That’s probably excessive for the average study, but for really large-scale projects where you just want to gather every bit of information you can (a new version of a complex software program, an overhauled IA, etc.), being comprehensive can’t hurt.
After reading this chapter, you should have a good idea of whether or not remote research suits you. Give it a try—if it’s not your thing, you can always go back to lab testing. We won’t tell anyone.
Do a lab study when you need to use special equipment, keep the interface 100% secure, see the user’s physical movements, or when you can’t use screen sharing tools.
Remote research has its own strengths, the greatest of which is that it enables Time-Aware Research, in which you observe users performing tasks you’re interested in observing right at the moment they’d naturally perform them.
Remote methods can also give you greater user diversity, cut travel costs, allow you to test from anywhere, be quicker to set up, and can be used to test context-dependent interfaces that wouldn’t make sense in a lab.
Currently, remote methods work best for testing functional computer interfaces, prototypes, wireframes, design comps, and mockups. You can test physical products with the right setup, but it’s slightly harder.
There are two kinds of remote research: moderated and automated. Moderated research has the researcher communicating with and observing users as they perform tasks; automated research has researchers using online tools and services to collect behavioral data from users automatically.
Generally, moderated research is good for collecting rich, qualitative behavioral data from a small sample, while automated research is good for collecting quantitative data over a larger sample.