If you are a consultant, you may join a project the moment the need for user experience research is recognized. If you work for a large company, your job may be to ensure that user experience research is used to drive all new product decisions. If you are a student, you may propose to do user research as part of your thesis or dissertation, or if you are a faculty member, you may propose to do user research as part of an application for funding. Who you need to get involved will depend on your situation. In industry, you often need to get stakeholder input (see Chapter 1, “Getting Stakeholder Buy-in for Your Activity” section, page 16) in addition to buy-in from the product team, whereas on a thesis or dissertation project you need to get input from committee members. Either way, you need to find out what stakeholders want to get out of the user research you are going to plan and conduct. Depending on your preferences and the stakeholders’ availability, a meeting may be the most effective way to get input from everyone who needs to be involved and thinking about the project. The goal of the meeting should be to answer the question: what do we want to know at the conclusion of the user experience research?

Ask the Right Questions

Once you have input from stakeholders, you need to transform everyone’s input into questions that can be answered via user research. There are some questions that cannot be answered and others that should not be answered via user research. User research is not intended to answer questions about business decisions (e.g., companies to acquire to expand product offerings, timing of feature releases) or marketing (e.g., how to brand your offering, market segmentation, how much people are willing to spend on your offering). Market research is about understanding what people will buy and how to get them to buy it. User research is about understanding how people think, behave, and feel in order to design the best interaction experience. This includes questions such as the following:

■ What do users want and need?

■ How do users think about this topic or what is their mental model?

■ What terminology do users use to discuss this topic?

■ What are the cultural or other influences that affect how a user will engage with your product?

■ What are the challenges and pain points users are facing?

Know the Limits

Now that you know what the fundamental questions are that you should seek to answer during your user research, you need to understand what limits there are that will further guide your decisions. The most common limits we see in user research are time, money, access to appropriate users or participants, potential for bias, and legal/ethical issues. While at first, these limits can seem problematic, we like to think of them as constraints that help guide the choice of a particular research methodology. For example, if you are conducting research about a socially sensitive topic, it is important to be aware that social desirability bias will likely be at play, and therefore, you need to choose a method that decreases the likelihood of this bias (e.g., anonymous survey rather than in-person interviews). If you have six months to inform the redesign of your company’s washer and dryer models, you may choose to spend a few weeks conducting field studies across the country with different demographic groups of potential customers. However, if you have only six weeks to complete all of your research, a one-week diary study may be more appropriate. The key here is that whatever method you choose should be based on your research question and scaled to the size you can accomplish given your constraints. Knowing these limits in advance will help you choose the best method for your research question, timeline, budget, and situation (see Figure 5.1).

Diary Studies (Chapter 8)

Diary studies ask participants to capture information about their activities, habits, thoughts, or opinions as they go about their daily activities. These allow a researcher to collect in situ, typically longitudinal, data from a large sample. Diaries can be on paper or electronic and can be structured (i.e., participants are provided with specific questions) or unstructured (i.e., participants describe their experience in their own words and their own format). Both quantitative and qualitative data may be collected. Diary studies are often conducted along with other types of user research, such as interviews.

Interviews (Chapter 9)

Interviews are one of the most frequently used user research techniques. In the broadest sense, an interview is a guided conversation in which one person seeks information from another. There is a variety of different types of interviews you can conduct, depending on your constraints and needs. They are incredibly flexible and can be conducted as a solo activity or in conjunction with another user experience activity (e.g., following a card sort). Interviews can be leveraged when you want to obtain detailed information from users individually.

The end result of a set of interviews is an integration of perspectives from multiple users. If you conduct interviews with multiple user types of the same process/system/organization, you can obtain a holistic view. Finally, interviews can be used to guide additional user research activities.

Surveys (Chapter 10)

Surveys ask every user the same questions in a structured manner. Participants can complete them in their own time and from the comfort of their home or work. Since they can be distributed to a large number of users, you can typically collect much larger sample sizes than with interviews or focus groups. In addition, you can get feedback from people around the world; response rates can vary from 1% (charity surveys) to 95% (census surveys).

Card Sort (Chapter 11)

A card sort is most often used to inform or guide the development of the information architecture of a product. For example, it can help determine the hierarchy in applications. It can also provide information when deciding how to organize displays and controls on an interface.

To conduct the technique, each participant sorts cards describing objects or concepts in a product into meaningful groups. By aggregating the grouping created by several users, we can learn how closely related each of the concepts are. This method tells us how a product’s features should be structured to match the users’ expectations about how those features are related. This technique can be conducted with individuals or with a group of users working individually.

Focus Groups (Chapter 12)

In a focus group, six to ten users are brought together for an hour or two to provide information in response to a series of questions or to provide their subjective response to product demonstrations or concepts. Often, participants are given tasks to complete with prototypes of the product so they may have a better frame of reference from which to speak. Presenting the questions or product to a group sparks group discussion and can provide more information than interviewing individuals alone.

Focus groups are best suited for idea generation rather than formal evaluation and analysis. You can also discover problems, challenges, frustrations, likes, and dislikes among users; however, you cannot use focus groups to generalize the exact strength of users’ opinions, only that they are adverse to or in support of a concept or idea. If conducted well, a focus group can provide a wealth of useful information in a short period.

Field Studies (Chapter 13)

The term “field study” encompasses a category of activities that can include contextual inquiry, on-site interviews, simple observations, and ethnography. During a field study, a researcher visits users in their own environments (e.g., home or workplace) and observes them while they are going about their daily tasks. Field studies can last anywhere from a couple of hours to several days depending on the goals and resources of your study.

Using this technique, a UX professional gains a better understanding of the environment and context. By observing users in their own environment, you can capture information that affects the use of a product (e.g., interruptions, distractions, other task demands) and additional context that cannot be captured or replicated in a lab environment. Field studies can be used at any point during the product development life cycle but are typically most beneficial during the conceptual stage.

Evaluation Methods (Chapter 14)

Evaluation methods (e.g., usability tests) are a set of methods that focus on uncovering usability issues and determining whether new or redesigned products or services meet certain criteria. There are two general categories of evaluation methods: inspection methods, where trained UX/HCI professionals inspect an interface to assess its usability, and testing methods, where UX professionals observe users as they interact with a product or service. Evaluation criteria can include task success, time on task, page views, number of clicks, conversion, user satisfaction, ease of use, and usefulness. In addition, a list of issues uncovered is typically produced with proposed design improvements.

Behavioral vs. Attitudinal

While attitudes or beliefs and behaviors are related, for a variety of reasons, they do not always match (Fishbein & Ajzen, 1975). Because of this, it is important for you to decide which of these two, behaviors and attitudes, you are most interested in studying. If you want to know how people feel and talk about an organization or symbol, you are interested in their attitudes. If you want to know whether more people will click on a button that says “purchase flight” or “take me flying,” you are interested in behaviors. Some measures such as surveys are more appropriate for measuring attitudes, whereas other methods such as field studies allow you to observe behaviors. However, be certain to remember that attitudes and behavior are related; even if you or your stakeholders are primarily interested in behaviors for one study, you cannot disregard attitudes; attitudes can drive behaviors over the long term.

Research and Participant Roles

Related to issues of attitude and behavior are the roles of the researcher and participant in the research. Some methods rely entirely on self-report, which means that a participant provides information from memory about him or herself and his or her experiences. In contrast, other methods collect observations. Observational studies rely on the researcher watching, listening, and otherwise sensing what a participant does, rather than asking him or her to report what he or she does or thinks. Finally, some methods do not involve participants at all and instead rely on the experience of experts. These types of studies are called expert reviews.

Lab vs. Contextual

Conducting studies in a lab environment (see page 87) allows you to control many potential confounds in your study and is useful for isolating how one variable impacts another. However, the lab environment lacks contextual cues, information, people, and distractions that participants will have to contend with when they use your product in real life. When you are considering methods, realize that some methods, such as field studies, allow you to understand the users’ context. Others, such as lab-based usability tests, provide control over these contextual factors, so you can focus on the product in isolation.

Qualitative vs. Quantitative

Qualitative suggests data that include a rich verbal description, whereas quantitative suggests data that are numeric and measured in standard units. Qualitative data, such as open-ended interview responses, can be quantified. For example, you could use a content analysis to determine the number of times a certain word, phrase, or theme was expressed across participants. However, the root of the difference between these approaches is deeper and related to disciplinary traditions potentially making quantifying qualitative data unappealing. To answer many research questions thoroughly, you need to combine these approaches.

Formative vs. Summative

Formative evaluations are conducted during product development or as a product is still being formed. The goal is to influence design decisions as they are being made. In formative research, you can figure out what participants think about a topic, determine when a feature is not working well and why, and suggest changes based on those findings. Summative evaluations are conducted once a product or service is complete. The goal is to assess whether the product or service meets standards or requirements. In summative research, you can determine whether a product is usable by some standard measure, such as number of errors or time on task.

Number of Users

While some types of studies require a large participant population to provide the most useful information (e.g., surveys), others can be conducted with a relatively low N and still provide significant value (e.g., usability tests). Determining how many users you need for any study can seem like a daunting task. For any study, there are a number of considerations that come into play when you are trying to determine sample size, such as available resources (e.g., time to complete study, funds to pay participants, number of prototypes), size of the effect you are looking for, the population you would like to extrapolate your findings to, the confidence you need to have in your findings, the type of contribution you would like to make, and your experience, comfort, and preferences for determining sample size. There are a number of straightforward ways, however, that can help you determine how many participants you need, including power analysis, saturation, cost or feasibility analysis, and heuristics.

Sampling Strategies

In an ideal world, a user research activity should strive to represent the thoughts and ideas of the entire user population. Ideally, an activity is conducted with a representative random sample of the population so that the results are highly predictive of the entire population. This type of sampling is done through precise and time-intensive sampling methods. In reality, this is rarely done in industry settings and only sometimes done in academic, medical, pharmaceutical, and government research.

In industry user research, convenience sampling is often used. When employed, the sample of the population used reflects those who were available (or those you had access to) at a moment in time, as opposed to selecting a truly representative sample of the population. Rather than selecting participants from the population at large, you recruit participants from a convenient subset of the population.

The unfortunate reality of convenience sampling is that you cannot be positive that the information you collect is truly representative of your entire population. We are certainly not condoning sloppy data collection, but as experienced user research professionals are aware, we must strike a balance between rigor and practicality. For example, you should not avoid doing a survey because you cannot obtain a perfect sample. However, when using a convenience sample, still try to make it as representative as you possibly can. Other nonprobability-based sampling strategies we see used are snowball sampling, where previous participants suggest new participants, and purposive sampling, where participants are selected because they have a characteristic of interest to the researcher. One problem with snowball sampling is that you tend to get self-consistent samples because people often know and suggest other potential participants who are similar to themselves.

Figure 5.2 presents a graphical representation of general guidelines about how many participants are required for each type of study by study contextualization. An overview of sample size considerations is provided in this chapter, and a thorough discussion of number of participants is provided in each methods chapter.