Introduction
Why we wrote this book
Throughout the years that we’ve known each other, we’ve spent a lot of time thinking and talking about the unique challenges of moderating user research. In the usability lab between sessions, driving to and from field visits, at professional meetings, and in the local coffee shop, we picked each other’s brains. We shared stories about our experiences with participants, the approaches that our colleagues used, and any recent research on moderating techniques (although the latter always felt lacking). A book on this topic has been percolating for quite a while because, as you can probably guess, we’re big moderating geeks.
When we started to mentor new user researchers, we found ourselves sharing tips on more than just how to moderate, like how to handle anything that comes up during a session including what specifically to say in those situations. Even the most experienced researchers struggle to balance the competing and ever-changing needs of a session, which means less experienced researchers are often thrown off-balance by the unexpected as well as the expected.
While there are a lot of great resources on how to plan user research and how to be a good moderator, there is little written about how to handle unexpected, tricky, or sticky situations that occur during a user research session. Although we’ve seen hints of these guidelines in the literature and heard them in conversations with colleagues, we found almost nothing that codified this topic. Loring and Patel (2001) wrote a great paper about handling awkward usability testing situations, but we wanted something that went into more detail and incorporated other types of one-on-one user research. The closest full-length work we’ve found is Moderating Usability Tests by Dumas and Loring (2008), which is an important, helpful, and well-written book about moderating fundamentals. If you haven’t read it already, we recommend that you do so in conjunction with this book!
We wrote this book as a survival guide for anything that comes your way during a user research session. Our focus is on actionable and specific steps that you can take—including many examples of what to say—and how to decide what’s best for your situation. We also want to start a conversation about these topics and help encourage more mindful moderation.
Who this book is for
This book will be most useful for anyone who performs one-on-one user research. These roles typically include user researchers or other related professionals or students involved in the design and evaluation of systems or experiences.
We’re assuming that you already know the basics about moderating and running a user research session. You should also be familiar with logistical responsibilities such as recruiting participants, creating clear and concise informed consent forms, and welcoming the participant when he arrives for his session. If you need a refresher on the fundamentals, Appendix B lists some helpful resources for getting up to speed.
Beyond that, it doesn’t matter whether you’re a new moderator or a seasoned veteran; we believe there are nuggets of wisdom and food for thought in this book for everyone.
What you’ll learn
Our goal is that after reading this book, you’ll be inspired and empowered to make the best decisions for you, the participant, and your research. In many cases, your decisions will come down to “it depends” (as do so many things in user research), but we’ll help you think through and accommodate the things those decisions should depend on in various types of situations. This book should be a quick, handy resource for you—think of it as your moderating Swiss Army knife that you can employ as needed.
You’ll learn the steps involved in making these decisions and taking the appropriate action, including:
What to consider any time you encounter a common, unexpected, tricky, or sticky situation.
How to approach the situation and adapt your response as necessary depending on the study type, study goals, and the participant’s comfort level.
What kind of things you can say to help address the situation.
You can see specific examples of these principles in action in Part 2. In those chapters, we describe the spectrum of situations that you may encounter, ranging from the very common to the very rare. Note that we focus only on situations that occur during a research session and exclude any interactions that may occur in study preplanning, in-between sessions, or in postsession or poststudy debriefing. (That’s enough material for another book!) Obviously, we can’t provide a comprehensive list of every possible situation that you might encounter. At the same time, some of the situations listed here may never happen in your career. Essentially, these situations were selected to highlight how your responses need to adapt depending on what’s happening around you.
Through these examples you can envision what you might do in similar situations. Even if you never encounter, say, an earthquake during a session (and we hope you don’t), reading up on what to do in that situation will hopefully bring awareness and consideration for how to handle other types of emergencies.
One-on-one user research methods
This book focuses on user research methods where a moderator is interacting with a single participant, typically as an evaluation of or conversation about a product. We cover situations that occur during both in-person sessions (with the participant and moderator in the same location) and remote sessions (with the participant and moderator in different locations).
The three methods that we refer to are:
Usability studies. During a usability study, you watch a participant as he attempts to complete tasks using a product (or a prototype of a product). The goal of a usability study is to evaluate how easy or difficult that product is to use.
Contextual inquiry. During a contextual inquiry, you observe the participant in his native environment and ask the participant to show you how he does something. The participant guides the session, but you ask questions along the way to clarify and validate his observations. The goal of a contextual inquiry is usually to gain a deeper understanding of the user’s process, including what the user finds important and necessary.
Interview. During an interview, you ask a participant a series of questions. The goal of an interview is to gather detailed information from participants about a specific topic or set of topics. User research interviews, as opposed to market research interviews, usually focus on the participant’s experience using products with the goal of collecting requirements and feedback to improve the product or experience.
Language used in this book
For the sake of simplicity, we’ve made the following terminology choices that you’ll see throughout the book:
Product. The product, prototype, or service being evaluated as part of the user research. It can be a computer system, mobile device interface, physical product, or a process. In a usability study, the product can be in the form of a low- or high- fidelity prototype or live environment. Basically anything and everything is fair game—as long as it’s the focus of the user research.
User researcher. Synonymous with user experience researcher. This refers to you, the moderator, even if your title is something else.
Your organization. If you’re an employee of the organization (e.g., company or business) that you’re doing research for, your organization refers to your employer. If you are a consultant working on behalf of a client, your organization refers to your client.
Study and session. A study is made up of a number of sessions with participants. For example, a usability study of a product might include six individual sessions with participants. We most commonly refer to sessions in the context of being back-to-back in a full day, but sessions could also intermittently span the course of many days or weeks.
Study plan. The moderator’s guide, study protocol, or test script that a user researcher puts together for a study and follows for each session. This can be as formal as a 10-page typed document or as informal as a few handwritten notes on a piece of notebook paper.
Stakeholders. Your stakeholders are your key decision makers. If you’re a consultant, your stakeholders are your clients. If you are an in-house user researcher, your stakeholders could be business partners, developers, project managers, and/or anyone else you need to work with closely as you design your study. We refer to stakeholders when we’re talking about making decisions for your study.
Observers. Observers include your stakeholders and anyone else observing the session, whether they’re in the room with you, in another room (e.g., in a usability lab), or remote. We refer to observers when we mean anyone watching the study.
He/she. For ease of reading and equality for all, we alternate pronouns throughout this book between “he” (this introduction and even-numbered chapters) and “she” (odd-numbered chapters). Please keep this arbitrary rule in mind when reading, as the situations listed don’t have implicit gender-based assumptions. Every situation could happen with any combination of male and female moderators and participants.
How this book is organized
This book is structured in three parts.
Part 1 discusses your responsibilities as a moderator, provides high-level guidelines for handling situations during a session, and introduces moderation patterns. This part is broken into traditional chapters that you may want to read in their entirety.
Part 2 takes the principles from Part 1 and applies them to over 90 situations, grouped by topic. Each situation stands on its own, so Chapters 4–14 don’t necessarily need to be read end-to-end. You can look up particular situations, browse through types of situations, or read a bunch at a time. Each situation is organized the same way:
Description—what the situation is
Method(s)—the type of method that this situation occurs in (usability study, contextual inquiry, interview)
Frequency—how often a moderator might encounter this situation (frequent/occasional/rare)
Pattern(s) to apply—which moderation patterns may be appropriate and helpful
What to do—instructions for how to react and resolve the situation
What to say—specific wording guidelines for talking with the participant
What not to say or do—pitfalls to avoid
How to avoid—tips to minimize the chance of this situation happening or, if that’s not applicable, tips to help prepare you for the situation
See also—cross-references to other situations that are similar to this one
Part 3 offers tips on how to minimize the chances of certain types of situations occurring and how to sharpen your moderating skills, no matter your experience level!
The appendices include a handy aggregated checklist of the most useful phrases that a moderator can use during a session and a list of things that you can do before a session to minimize the chances of something unexpected or sticky happening. These checklists are also available on our website (http://www.modsurvivalguide.org).
Here is a detailed breakdown of what each part of our book covers:
Part 1: Your Moderation Toolkit
Chapter 1: Moderation Matters: Power, Responsibility, and Style. This chapter discusses the qualities that are most helpful for a moderator during a research session and the different moderating styles that you can choose depending on the needs of the session.
Chapter 2: In the Trenches: Six Steps for Handling Situations. This chapter provides a high-level view of the steps you should take to understand what type of situation the participant is dealing with and respond appropriately.
Chapter 3: Mix and Match: Your Moderation Patterns Toolbox. This chapter defines specific steps that you can take to change the course of a session and when each pattern works best.
Part 2: Your Survival Guide
Chapter 4: Recruiting Mishaps: Participants You Weren’t Expecting. This chapter covers situations where the participant does not fit your recruiting criteria or has some other feature that makes proceeding with the session a challenge.
Chapter 5: Participant Misconceptions: Not What the Participant Was Expecting. This chapter covers situations where there is a mismatch between the participant’s expectations for the sessions and what you’ve planned.
Chapter 6: Some Guidance Required: Participants in Need of Shepherding. This chapter covers situations where the participant or your observers are not fully engaged with the session or require some other type of guidance to ensure that you get the feedback you need.
Chapter 7: Make It Work: Handling Technical Obstacles. This chapter covers situations where you encounter challenges with the technical setup, including prototype and remote session difficulties.
Chapter 8: Is This Right? Responding to Uncertain Participants. This chapter covers situations where the participants seem hesitant or unsure about their feedback.
Chapter 9: What’s Going On? Recovering from External Interruptions. This chapter covers situations where your session is interrupted by an observer or someone else.
Chapter 10: Get on Track: Overcoming Momentum Blockers. This chapter covers situations that can slow down your session or throw it off track.
Chapter 11: Take the Wheel: Guiding Wayward Participants. This chapter covers situations where the participant becomes annoyed or defensive at you or the product.
Chapter 12: A Delicate Touch: Addressing Sensitive Situations. This chapter covers situations where something about the participant or his behavior has to be approached delicately.
Chapter 13: Uncomfortable Interactions: Responding to Awkward Situations. This chapter covers situations where the participant says or does something that makes you feel uncomfortable.
Chapter 14: Safety First: Minimizing Emotional and Physical Distress. This chapter covers situations that pose a risk to your and the participant’s emotional or physical well-being.
Part 3: Improving Your Skills
Chapter 15: An Ounce of Prevention: Avoiding and Mitigating Situations. This chapter highlights steps you can take before the session to increase the odds of everything running smoothly.
Chapter 16: Sharpening Steel: How to Improve Your Skills and Help Others Improve Theirs. This chapter provides actionable tips on how to improve your moderating skills.
Sidebars and survival stories
Scattered throughout the book you’ll find sidebars and survival stories:
Sidebars go into more detail about specific topics related to moderating and challenges you may encounter throughout a session.
Survival stories are written by other user researchers who are generously sharing experiences that they’ve had while moderating. We’re thrilled by the variety of stories and the interesting situations they’ve encountered and hope you’ll find them as entertaining and thought-provoking as we have.
Companion website and videos
Additional material, videos, and other bells and whistles are available on our website, http://www.modsurvivalguide.org.
We’ve put together seven videos, each of which shows a different situation or combination of situations. The videos do not show full sessions; instead, each video shows snippets of a session where a situation occurs and the moderator reacts. For dramatic effect, we show a few different snippets of the session to show the potential progression of a situation and how the moderator’s response changes.
These videos are all listed on this page of our website: http://www.modsurvivalguide.org/videos. You’ll also see these videos referenced throughout the book within certain situations. Each video reference will include a short description, the URL, and a QR code. If you’re so inclined, you can scan the code with a QR reader using your mobile device. This will bring you directly to the page that hosts and discusses that video in more detail.
If you’re an educator who is using this book for a class, please check out our page for educators on our website: http://www.modsurvivalguide.org/educators.
And, of course, we always welcome your feedback and questions, which you can submit by going to our contact page: http://www.modsurvivalguide.org/contact. You can also follow and talk with us on Twitter: @ModSurvivalUX.