An Ounce of Prevention
Avoiding and Mitigating Situations
Abstract
This chapter provides specific steps that you can take as part of your planning and preparation process to avoid or at least mitigate potential sticky and tricky situations that may occur during your user research. These steps include setting appropriate expectations with participants and observers, prioritizing your study plan so you can adapt it on-the-fly, and preparing the product, the study space, and any technology that you’ll be using.
Keywords
user research recruiting; user research study plan; user research study preparation; user research study troubleshooting; user research observers; user research observers ground rules
Many of the situations described in Part 2 can be mitigated earlier in the session or prevented entirely ahead of time. This chapter compiles the tips we provide in each situation’s “How to avoid” section to help reduce the chances of you encountering, or make it easier to deal with, many of the situations discussed earlier, especially the occasional and rare situations. This isn’t intended to be a comprehensive list of everything that you need to do to prepare for a successful study, but instead highlights the key topics—from expectation setting to technical tinkering to practicing and preparing your space.
15.1 Recruiting process
We hate that horrible feeling you get when you realize that the participant had completely different expectations for your session, or you have a room full of high-level observers who are antsy because the participant is running late. These tips help you set expectations with participants and plan ahead for situations where you absolutely cannot have a late or no-show participant.
Specify your recruit criteria
It doesn’t matter how carefully crafted your study plan is or how sharp your moderating skills are if the participant doesn’t meet the necessary criteria. As you prepare for your recruit, think carefully about the must-haves that your participants need to provide the desired feedback. Craft your list of criteria carefully to avoid any deal-breakers.
Potential must-haves may include participants who:
Have the desired experience or role. You can’t rely just on job titles to figure out if a participant is really who you’re looking for. Instead, think of the specific tasks or attributes that you need her to have. For example, if you need to get feedback on a new timesheet approval system, it will not be enough to recruit managers—you need to verify that participants have direct reports whose timesheets they approve.
Are representative users. Recruiting unrepresentative users often happens when you ask someone within your organization to recruit internal employees. For example, if you need to interview sales managers, the person scheduling participants may only find you the top performing managers. Communicate that you want truly representative users—an average sample.
Are set up to show you what you need. For example, if you’re visiting participants at home to watch their process for doing laundry, you need to make sure that participants will have dirty laundry at the time of the scheduled session.
By being specific about these criteria ahead of time, you’ll hopefully avoid any unpleasant surprises once your session starts!
Set participant expectations through the recruit
To avoid a mismatch between your expectations and those of your participants, it’s important to be crystal clear during the recruit about what participants will be doing during the session. Important elements to highlight include:
The type of research (e.g., a usability study, an interview), along with a brief description of what this means. Participants are sometimes intimidated to find out that they will be working through tasks in a usability study when they assumed they would just be chatting in an interview.
The length and location of the session. If the session is taking place at the participant’s location, confirm her contact information and address, and ask if there are any special things you need to know about parking, entering the building, or anything you might not be allowed to bring into the building (e.g., cameras or other equipment).
The session will be one-on-one, so there will only be one other person (you) interacting with the participant. If participants expect to be in a focus group, they may wrongly assume they would be able to hide behind others or cancel without advance warning.
If the session will be recorded, and what will be recorded (e.g., both the computer screen showing the product and a capture of the participant’s face).
The amount and type of compensation being provided, as well as how and when they’ll receive it. For example, some organizations compensate in cash immediately following or before a session, while other organizations use recruiting agencies and have the agency send a check in the mail after the session is complete.
Be sure to send a confirmation letter to participants either via mail or email so they have the study details in writing. The letter should also include your contact information on the day of the scheduled session (or whoever should be contacted in case the participant is running late or can’t make it). If possible, call the participant a day before the session to verify the details about the study and answer any last-minute questions she may have.
Take extra measures if you have internal employees as participants
When using internal employees of an organization for research, you’ll probably encounter a hornet’s nest of politics that have little, if anything, to do with your project. Internal employees may be chosen because they’re the target users of an internal product, or as surrogate users (who can play the role of your target users). They may be recruited as part of a convenience sample—they’re chosen because the person managing the recruit can easily find them rather than because they represent a diverse sample of users. Often they’re recruited by managers who tell them almost nothing about the project. As a result, they have no expectations or the wrong expectations (e.g., that it’s a training session). These participants may feel, or actually be, pressured into participating, so they may come into the session with a chip on their shoulder. Also, many organizations prohibit internal employees from receiving compensation for their participation, so participants may be doing your research for no additional incentive.
To help ensure a smooth session when you have internal employees as your participants:
If someone else recruited the participant for you, spend a bit of extra time at the beginning of her session explaining the purpose of the study and how the feedback will be used. Reinforce that you’ll keep her information confidential and will not use her name or any other specific identifying information, and that she is not required to participate. The participant may not have been told much, if anything, about the research.
Avoid having the participant’s direct manager, people from the same team as the participant, or employees from human resources observing a session. This can make the participant feel even more nervous or that she is being evaluated.
Disclose whether you have observers and that it’s possible any of them may recognize her, but emphasize the point that they’re all happy for any feedback she has to give and that she is not being evaluated at all. Reinforce that none of the observers are from her team. Also make sure to brief your observers on how to behave in a manner that respects the participant and maintains her confidentiality (see section 15.4 for more observer guidelines).
Sincerely thank the participant for her time, and stress that her feedback is extremely valuable.
Also, keep in mind that the same guidelines that apply to regular participants still count for internal employees! You still have an ethical responsibility to look out for their physical and emotional safety.
Consider scheduling backup or floater participants
Sometimes it’s critical not to have an empty timeslot during a study. Stakeholders may be traveling to observe sessions, or the project may be very time constrained so there is no way to schedule replacement sessions. In these situations, consider recruiting either:
Backup participants. A backup participant is double-booked for a timeslot, so you actually have two people scheduled for the same timeslot. Use the backup if the regular participant doesn’t show up or is late. If the regular participant shows up, compensate the backup and let her leave.
Floater participants. A floater participant is available as a backup for a period of time that stretches across multiple timeslots, and usually is compensated more for doing so. If you have floaters, but end up not needing them, you might still have a colleague conduct an interview to get some useful data, or give them a survey, an unmoderated study, or some other independent activity from which you can collect feedback.
When discussing backup and floater participants with your stakeholders and other user researchers, be sure to clarify exactly what you mean. Sometimes these two terms are used interchangeably.
It also helps to discuss with your stakeholders if surrogate users who are internal employees from your organization might be appropriate floaters or backups in case of a no-show participant. If you don’t have floaters or backups, set expectations that no-shows are a fact of life and plan accordingly (e.g., by recruiting ten participants to ensure that at least eight show up).
15.2 Your study plan
Sometimes you have to do some creative juggling within a session. Maybe you have to shift the focus of the research because your product is experiencing technical difficulties. Or, maybe the participant spent more time than expected answering your first few questions or tasks and you realize that you won’t get through everything else in your plan. You also need to think about how you’ll brief the participant, present your questions or tasks, and when and how you’ll provide compensation.
Prioritize the study plan and create a backup plan
As you’re writing your study plan, structure the protocol in such a way that it’s easy to identify the high-priority tasks and questions. Work with your stakeholders to determine what, if any, material can be comfortably cut if necessary. This way, if you know you need to end the session early for some reason, you can adapt on-the-fly while making sure you hit all the necessary areas.
Once you have your study plan in place, also think about additional questions or tasks that you can prepare as a backup. This backup content can be used if you need to shift the focus of a session or take an extended break, and will let you continue to get feedback related to your research. Depending on the length of your study plan, you may already have several lower-priority tasks and questions that can be easily repurposed. For example, if you’re planning a usability study or contextual inquiry, review your study plan and consider what additional questions you can ask if you need to shift the focus to an interview. These techniques are all ways to make use of a session that may otherwise have been wasted.
Another type of backup you can create is a paper or online questionnaire. The questionnaire can contain questions related to your project (e.g., demographics and product usage) or anything else that would be helpful, but not vital, to know about your participants. If appropriate, you can plan to give the questionnaire to all participants at the end of the session, but have it ready to use earlier in case something goes awry and you have to take an extended break.
If you’re asking participants to interact with a product, think about any extra tasks that you can keep in your “back pocket.” These may be lower-priority tasks related to the project or more general tasks that always apply to any user who participates. These tasks may come in handy if the product breaks in the middle of a session, or you realize that the participant doesn’t meet your criteria and would not provide the right kind of feedback on your intended tasks.
Prepare your presession briefing
When the participant arrives for her session, greet her warmly and make some casual conversation to help her feel comfortable. Once you arrive in your study space, transition to your presession briefing. The briefing should:
Explain your goals for the session.
Set clear expectations for how the session time will be used.
Let the participant know that she can take a break or end the session at any time.
Reiterate the points highlighted in your consent form including any recording that is being done and how the findings from this research will be used.
Set up the briefing in whatever format will be easiest for you to read without missing any key elements. Some researchers prefer to have a bulleted list, while others write everything out—find what is most comfortable for you. Even if you have your points memorized, make sure you refer back to the list often so that you aren’t forgetting important points; forgetting one important point may open the door for a tricky or sticky situation to occur later. To make things less awkward, you can preface the briefing by saying, “I’ll be referring to my notes here to make sure I don’t miss anything.”
If you have a detailed script, practice it ahead of time so that you can easily deliver it in a comfortable tone while making eye contact with the participant. If you read it word for word, looking down at your clipboard and with a monotone voice, participants may zone out, get irritated, or get more nervous. Newer moderators who feel more comfortable reading directly from a script can preface to the participant that you’ll do so, but may want to write in reminders for you to “LOOK UP!” after every few lines.
If you have a lot of points to cover in your briefing, find a way to keep it broken up and engaging. For example, ask some background questions in between sets of briefing points, stop periodically to ask if everything makes sense so far and whether the participant has questions, and use your intonation and delivery to keep the participant listening carefully. Sometimes it also helps to warn the participant by saying something like, “I have a lot of points to go over, so forgive me in advance for throwing a lot of information at you! I just want to make sure you have a clear understanding of what we’re doing here today.”
Consider keeping the tasks “close to your vest”
Letting usability study participants know ahead of time how many tasks you’ll be asking them to attempt may create complications. If the participant sees the number of tasks, she may start self-timing and rush through some tasks or take more time on others instead of approaching them naturally. Another challenge is that in some situations, like those described in sections 10.4 and 10.7, you may not get through everything you’d planned. She may then feel bad about not being able to give you all the feedback you wanted.
Divulging tasks one by one is a useful way to mitigate these issues. After the participant completes a task, you hand her the next task or move on to another part of your study plan. However, this method presents its own challenges: not being able to send tasks to remote participants to print ahead of time, having to read tasks to participants, or needing to be in the room with the participant to hand her the next task. Think about the needs of your study and choose accordingly. If the participant has to have the tasks ahead of time, it’s not the end of the world! As she starts each task, encourage her to go at her natural pace and not speed up or slow down for your sake (and write this into your study plan so you don’t forget to say it). Warn her that for the sake of time you may push her along sometimes, but that you don’t expect to get through everything (even if you do expect to), so she doesn’t feel bad if the session ends before all tasks have been attempted.
Decide when you’ll provide compensation
You’ll notice that in many situations, we recommend providing the full compensation amount to your participant even if she doesn’t stay for the full session or if she doesn’t meet your recruit criteria. We’ve found that it’s easier to provide the compensation no matter what than trying to define the specific instances when it’s not appropriate. It’s too difficult to figure out where to place the blame for a participant who does not meet your recruit criteria or why a participant arrived late, and we’d rather pay a little extra in compensation to keep participants satisfied than get into an argument with an uncompensated participant.
Obviously your mileage may vary on this recommendation. However, if you do implement a policy that would prevent the participant from receiving her compensation, be very clear with her about those circumstances during the recruiting process and allow a bit of tolerance. For example, if you let participants know that they won’t be compensated if they arrive more than 15 minutes late, and the participant arrives 17 minutes late, she should still receive her compensation. Be reasonable and understanding.
Also, think about when in the session—the beginning or end—you’ll provide compensation to participants. Most researchers provide compensation at the end of the session. However, providing compensation at the beginning of the session reassures the participant that she really is free to leave at any time and that she is not going to be penalized for leaving early.
15.3 The product, space, and technology
If you’re planning to have the participant interact with a product—whether a released/live version or a prototype—during the session, follow these tips to limit the possibility of technical or product-related situations.
Preparing a live website or application
If you’re getting feedback on a live website or application, talk with your development team to ensure that no updates will be applied during your research. Explain that you need participants to all give feedback on the same version of the product. If it’s a website and any A/B testing is being done, see if there is a way for you to either have that functionality turned off during your research or for you to consistently see the same version of pages (e.g., always the A design). If these guarantees cannot be made, see if you can run your research from a developer sandbox, quality assurance link, or some other staging area with the extensive functionality of the live product.
If you’re getting feedback on the released public-facing version of an application or software product, check to see if any updates are scheduled to be released between now and your study. For most research you want feedback on the most recent version, so you may be able to get access to a test or beta version of the upcoming release so you can plan ahead.
If you’re unable to talk with the development team (e.g., because you’re running a competitive study and have no way to contact a team), check the product every day for a couple of weeks before the study to see what, if anything, changes. This due diligence will help you get a feel for what you might expect on your study days. However, keep in mind that you might come in to find that a brand-new redesign has been rolled out. Be sure that your stakeholders are aware of this as a potential complication.
Preparing a prototype
The features needed in a prototype (typically used for a usability study) vary widely depending on the domain, product type, and goals of the study. However, the following tips will help make your prototype (i.e., an interactive environment beyond static images) be successful during a study:
Make the prototype as functional as possible. The situation described in section 11.4 is a good example of why: participants can be frustrated by a low-functioning prototype. Developers or prototypers new to user research often mistakenly think of the study as a demo that will be performed in a very structured and linear fashion. This mindset may lead them to create a prototype that has only the correct answer-paths working, and sometimes in only a set order. Participants will catch on to which areas are working and which aren’t, which can lead to frustration and bias. It may influence the way they think about the design and the actions they take.
If the prototype can’t be completely functional, make it adequately functional. In the real world, participants will try lots of crazy paths to an answer and sometimes you can’t anticipate all of them. So “adequately functional” means that the prototype includes more than just a linear flow of clicks or steps, and more than just the correct paths are working. The prototype should provide alternate and anticipated wrong paths or steps that the participant can then explore and attempt.
We recognize that most teams have time and resourcing constraints. If you can’t get more than the correct paths or steps working, provide the illusion that the prototype is more functional. One example of this is, in a website prototype, make all of the links on the page look clickable even if they’re just null links that don’t go anywhere. Doing this prevents the mouse “drive-by,” where participants can catch onto what is working based on where they get the hover-hand icon.
As with running your study on a live product, talk with your designers/developers to ensure that no design changes are made to the prototype during the study. Let them know that any changes they want to make need to be discussed with and approved by you first, as those changes may have an impact on your study protocol, the data you’re able to collect and compare to previous participants’ results, and the prototype’s stability.
Test and document the technical setup
To avoid, or at least minimize, technical difficulties during your study, do a full walkthrough of your technical setup for the session. This walkthrough should including turning on and off any cameras, screen-sharing applications, and recording devices. Test the order in which each item needs to be turned on or off and document the order that works best. Depending on the complexity of your setup, consider including a checklist of your setup steps in your study plan so you can easily reset if anything changes during a session.
If you’re running a remote session:
Familiarize yourself with options to mute others on the call or eject them from the screen-sharing session to avoid observers interrupting the session.
Do a dry run with participants to ensure that your screen-sharing software works on their computers, which will save time during their sessions.
In your technical setup checklist, include screenshots of what the participant will experience as she joins the remote session. These screenshots will help you provide instructions to the participant to perform basic troubleshooting.
For more ideas on what to look at in your technical setup, see the sidebar “Troubleshooting Skills Are a Lifesaver.”
TROUBLESHOOTING SKILLS ARE A LIFESAVER
A key skill for a user researcher is technical troubleshooting. Given how integrated technology is with our work, it’s in your best interest to learn as much as possible about:
Recording technologies, including software like Morae or Camtasia, and old-school recording technologies such as DVD burners and video cameras.
Screen-sharing technologies, such as WebEx and GoToMeeting.
Built-in and external web cameras, their drivers, and how they’re used through recording and screen-sharing programs.
Phone conferencing systems, including voiceover IP systems that integrate into programs like WebEx.
General computer basics, such as networking, extended and clone desktop, and changing screen resolution. Try to be at least familiar with both Macs and PCs.
Cables and connection types, including VGA, DVI, and HDMI.
Usability lab settings, the specifics of which depend on the lab but usually include changing camera views, routing screens, and sound.
Any other devices you plan to use to capture participant feedback, such as eye-tracking equipment.
No matter the problem, the most valuable troubleshooting tip we have is to isolate the variables! For example, if your audio is not working, avoid changing five settings at once. If the audio still is not working after making multiple changes, you don’t know anything more about the problem than when you started—and if it did work, you lost a chance to identify the most efficient solution. Change one thing at a time and see if it solves the problem. Only consider trying combinations of things if, based on prior experience, you think you need to and individual changes didn’t make a difference.
Knowing these basics will help you keep your sessions running smoothly without waiting for someone else to help you. Tinker with the equipment and settings before a study. Ask someone who knows the equipment to deliberately break something (e.g., disconnect a cable or change a camera setting) and try to troubleshoot it for practice. If you have a team of researchers, you can set up troubleshooting exercises for each other to help increase your competence and confidence. The next time something goes wrong during a session, you’ll have the skills to handle it!
Familiarize yourself with the space and its safety options
Before your study begins, familiarize yourself as much as possible with the space that your research is being performed in. Are there security guards? Where is the nearest fire exit? Don’t be afraid to ask questions, especially if you’re in an unfamiliar space. By knowing these answers ahead of time, you’ll be able to more quickly respond to anything that requires a bit of “native” knowledge.
If there is building security, notify them about any participants who may be coming for your research. Again, if something happens, their advance knowledge of what you’re doing may save precious time.
Also, make sure at least one other person from your team, whether a team member or stakeholder, is available to watch each session. We highly recommend avoiding running user research on your own because, if an uncomfortable or potentially litigious situation arises, you need another set of eyes, and hands.
Once you have another person who can join you for the session, consider creating a “safe phrase.” This is a phrase that you share with the person observing the session, and that you use only if you need that person to interrupt the session. For example, if you’ve been getting a very bad feeling about the participant, and the participant has positioned himself between you and the door, you may want to use the phrase to notify your observer that you need her to knock on the door and provide an excuse for you to leave the room. Since it may be difficult to tell exactly how serious a situation is from another room, the safe phrase is your way to request assistance.
Investigate cultural customs and taboos
If you’re doing research in other countries or cultural regions, do your due diligence and learn about the customs ahead of time. For example, some cultures require you to take off your shoes before entering. Others may take offense if you don’t accept tea, or if your knees and shoulders aren’t covered. Some cultures expect you to exchange business cards in a certain way. The more you can be prepared, the less uncomfortable or awkward the situation will be.
But also be careful not to assume anything solely based on what you read or hear—that could be offensive too. Try to take cues from your participants—for example, if the participant has her shoes off and there is a shoe rack at the door, that is probably a good sign that it’s important to respect that rule in the household. Put your empathy and observational skills to good use!
15.4 Your observers
By sharing your moderating approach and establishing ground rules ahead of time, you can decrease the odds of your observers disrupting your session.
Communicate your moderating approach
As you work with your team to create the study plan, set expectations with your potential observers about how you’re going to moderate during the session. The detail level you need to go into will vary depending on how familiar they are with user research in general, the specific method you’re using (e.g., contextual inquiry), and your particular moderating approach. Here are some things to consider going over:
You may not follow the study plan in its exact order, or use its exact wording (if you’re confident and experienced enough to take this approach). You might reprioritize goals on-the-fly based on what the participant says or does, or probe with follow-up questions wherever necessary.
You may let the participant go off on brief tangents and act interested in irrelevant things to let the participant feel heard.
You may not answer the participant’s question, or might pretend not to know an answer even if you do (and that is not an invitation for someone to come in and give you the correct answer!).
When probing for something specific—a goal, or understanding more behind something the participant did or didn’t do—you may start with broad questions and narrow down to more and more specific questions. For example: “What did you think of that?” “What did you think of how this area worked?” “What do you think this button does?”
You might drop a topic, redirect to another as a diversion, and come back to the first topic later (see Chapter 3 sidebar “The Diversionary Assist”).
If you’re doing a usability study:
You may let the participant struggle for the sake of uncovering findings, but ultimately the participant’s comfort is the first concern. There will be times you need to sacrifice “data” or be especially strict about observer behavior for the participant’s well-being.
Explain what an assist is, and when and how you give participants assists.
By setting these expectations, you can hopefully prevent an observer from making assumptions (e.g., assuming that you were being leading when you were giving an assist). It will also keep them from interrupting a session because they don’t understand what you’re doing and think you need additional information or assistance.
Provide ground rules
Another way to avoid or minimize observer interruptions is to provide a set of ground rules for observer behavior during a session. You should share these ground rules with your observers before the session so they have a chance to read them ahead of time. Also, print the list of rules and give it to your observers if they’re in the room with you during the session, or post the rules on the wall of their observation space.
Here are some ground rules for observers in another room (e.g., usability lab observation room):
Keep any conversation or discussion to a minimum. No loud laughing! Even supposedly soundproofed walls can be surprisingly permeable.
Keep the participant’s identity and personal information (e.g., company) confidential, especially if an observer knows her personally. The observers should also refrain from revealing the identity to others.
Avoid making fun of the participant.
Keep cell phones silenced (not on vibrate).
Leave the room quietly if a call must be taken.
Avoid talking about the research study outside of the observation room for as long as the sessions are running. This includes not talking to the participant about the study if observers run into her in the hallway or restroom before or after the session.
Stay for an entire session, and attend as many sessions as possible.
Follow agreed-on guidelines for asking questions to the participant. These guidelines should be based on a conversation you had with the observers before the session and should cover if, when, and how they can ask questions. However, be careful to set expectations that you may not always get to asking their questions, or ask their questions in the way they posed them.
Follow agreed-on guidelines for how to communicate with you. Again, these guidelines should be based on a conversation you’ve had before the session about if, when, and how to get in touch with you during the session. For example, you may set the expectation that if they see something that isn’t working in the prototype, they can quietly knock on the door of your study space so you can excuse yourself and talk to them, but otherwise they should hold their questions until you come talk to them toward the end of the session.
If you’ve asked them to take notes or help in some other way with data collection, include a short summary that observers can refer to throughout the session.
When an observer is in the room with you and the participant, most of the previous points apply. Some exceptions and additions include:
There should be no conversation or discussion, and absolutely no laughing or even throwing meaningful looks at other in-room observers.
Avoid multitasking (and the appearance of multitasking). Pay attention!
Avoid disruptions (stepping out, etc.) unless absolutely necessary.
Be careful of body language and any physical or audible reactions.
Know if, when, and how to interact with the participant, based on a conversation before the session. For example:
The observer should not help the participant complete tasks or answer questions.
Is the observer experienced talking to users? If so you may let her ask questions to the participant directly either during the session or at the end. Or, if you’re less comfortable with that, have her write down her questions on notes and (subtly) pass them to you.
When observers are observing remotely, include additional ground rules:
Keep the phone or voice over IP (VoIP) functionality on mute at all times unless told otherwise. This includes situations when it seems like the participant has dropped off the call or has hung up. As with in-person observing, the observer should know if she can interact with the participant at all, and when (e.g., at the end of the call when the moderator opens up the line for questions).
Know how and when to interact with you, as described in the preceding points. Communicating via an IM application can be very useful when you have remote observers. As discussed earlier, be clear with the observers about whether or how you might react to their IM requests and questions.
Watch Video 5 to see an example of a moderator dealing with an interrupting observer during a contextual inquiry. The moderator pulls the observer out of the room to reiterate the session ground rules.
Visit our website (http://www.modsurvivalguide.org/videos) or use your QR reader to scan this code. 
15.5 Your technique
Before you moderate your first session for a study, review:
The list of prefabricated phrases from Appendix A. If you’re new to moderating, jot a couple of the phrases in your study plan to reference throughout the study.
The guidelines for your language and tone from section 2.5.
The list of moderator behaviors to avoid from section 2.3.
Once you have a study plan almost finalized, you can also run a practice session with a colleague acting as your participant so you can get a better feel for how the session will go and any areas that you feel like you might struggle with. This practice will help you refine your study plan and help you feel more comfortable with the protocol by the time your first session comes around. Of course, also be sure to run a pilot session with an actual participant using your study plan and the product (if applicable) before your first full day of sessions. You’ll feel more confident and will appreciate the additional familiarity with the protocol in case anything unexpected happens and you need to adjust.