Chapter 11

Take the Wheel

Guiding Wayward Participants

Abstract

This chapter covers situations where the participant becomes agitated, annoyed, or disparaging toward you or your organization. These include situations such as a participant who reacts poorly to your neutrality, a participant who is frustrated by a prototype’s limited functionality, and a participant who seems to have an agenda against your organization. For each situation you’ll learn what to do, what to say, what not to do or say, and how you might be able to avoid it in the future.

Keywords

usability agitated participant; usability annoyed participant; usability angry participant; usability prototype; user research participant

If the participant becomes agitated, annoyed, or disparaging toward you or your organization, it’s time for you to take the wheel of the session. These situations require you to diffuse the participant’s frustration, and then monitor the session going forward to ensure the rest of her experience is not negatively affected.

11.1 Remote participant is obviously distracted

During a remote session, you hear the participant doing things that are obviously outside the realm of this study. Maybe you can hear her typing in a window that’s not being shared or is on a different monitor. There may be muffled conversations or noises in the background, and the participant may be slow to respond or entirely miss your questions.

Method(s): Any remote method

Frequency: Frequent

Pattern(s) to apply: Clarify the task/question; Build engagement; Take a break; End the session early

What to do

ent First, give the participant a subtle hint that you know something else is going on. Asking something like “Is this still a good time for you?” gives her an out in case she really does have something more important to deal with but is too polite to say so. It also gives the participant a chance to share any complications within her environment. For example, participants within corporate environments (if you’re talking to employees who are participating from work) often find it impossible to truly shut out the rest of the world for your research, and may be multitasking as you’re talking to them. The participant may tell you that she has to keep her email open during the session, or that it’s a very busy time and she may need to stop occasionally to talk with coworkers. In this case, you can offer to reschedule but you may need to accept that even so, she may still experience similar distractions.

ent If the participant is willing to continue and seems genuinely interested in providing feedback, but is dealing with distractions outside of her control, focus on the most important tasks/questions so you can make the most of her limited feedback.

ent If the participant doesn’t respond to your hint or you still get the sense that she is distracted, make a judgment call on whether or not to continue the session and, if you do continue, what you can adapt to fit these circumstances. If you think the session may be salvageable, consider asking the participant questions that are more directly targeted to what she does (e.g., how her process works today or what parts of the product she uses most often) instead of continuing with your preplanned tasks. Getting her to talk about herself may help increase her engagement in the session. If this works, you can then move back to your study plan, using the knowledge you’ve gained to tailor what comes next. However, if she still seems distracted, tell the participant that you’ve covered everything you’d planned, and then wrap it up early.

What to say

ent “<Participant>, is now still a good time for you?”

ent “It sounds like there’s something else going on. Do you need to take a quick break before we continue with <task/question>?”

ent “There seem to be a number of things competing for your attention right now. Are you able to continue with this study? If not, that’s fine, and we can reschedule.”

ent “I know you’re very busy, so I appreciate you taking the time to give us feedback.”

What not to do or say

ent Hide your frustration or annoyance with the participant. Be polite and respectful.

How to avoid

ent Be upfront about your expectations for the session so the participant can reschedule any competing priorities and plan accordingly. But, recognize that even this may not be enough and that interruptions could still come up.

ent During your recruiting process, ask the participant for the “best” time to conduct the session, and schedule her session for that time. Check with her the day before to ensure that the scheduled time still works.

Watch Video 3 to see an example of a moderator interacting with a distracted participant during a remote usability study. The participant ends up sharing more than she intends.

Visit our website (http://www.modsurvivalguide.org/videos) or use your QR reader to scan this code. image

11.2 Participant is distressed by a personal line of questioning

You’ve had to ask the participant some relatively personal questions and she starts to become uncomfortable and edgy. The tone of her voice changes, her voice gets louder or softer, and her answers become short and curt. She may tell you outright that she is not comfortable answering certain questions. The questions you’re asking are part of your study plan and may be domain-specific, like online dating behaviors as part of a usability study for a dating website or the frequency that she shaves as part of an interview about disposable razors. Or, you may be asking some specific questions about her job (e.g., how a factory worker breaks rules to get something done more safely, easily, or efficiently).

Method(s): Any

Frequency: Occasional

Pattern(s) to apply: Reassure the participant; Take a break

What to do

ent Think about the participant’s expectations. If the participant was expecting to only talk in general about financial websites and is distressed by your questions about estate planning, try to make your questions broader and less personal to be more in line with her expectations. If this isn’t possible, it’s best to just drop the topic even if it means losing that data point. If the participant signed up to participate in an interview about online dating, then you can soften the personal nature of the questions you’re asking, but she shouldn’t be too surprised about the topic you’re exploring.

ent Let the participant know that if there are any questions that she’s not comfortable answering, she should tell you.

ent Reassure the participant about how her answers will be used. For example, tell her that her feedback will be aggregated with the answers of other participants, and her name will not be associated with it. This reassurance will help make the participant feel as comfortable as possible sharing what she is willing to disclose.

ent If the participant seems afraid of getting in trouble because of her answers (e.g., because she is an internal employee and the research is for her organization), add to your reassurance a reminder that she’s not being evaluated in any way on how she approaches her work or any other aspect of her life.

ent If the participant is especially uncomfortable or agitated by your questions, offer to talk to her about it in private, with any recording equipment turned off. If she seems really rattled, take a break, and when the session resumes, soften the personal questions or move on to another area of your study plan. If the participant still seems upset or agitated, bring the session to an early end.

What to say

ent “Let me know if you’re uncomfortable answering any of my questions. While we want to learn as much as possible from you, I also want to respect your privacy and your comfort.”

ent “I’m sorry, I didn’t mean to pry with that question. Let me rephrase.…”

ent “Please know that anything you say will be kept confidential.”

ent “Thank you for your feedback so far. I know these topics may be difficult to talk about, but your feedback will help us understand how to make the product better for you and others like you.”

What not to do or say

ent Don’t ignore signs of the participant’s agitation or discomfort! Even if it means having to adjust your planned tasks or questions, be empathetic toward the participant and do your best to make her more comfortable.

How to avoid

ent Both during the recruiting process and at the beginning of the session, set appropriate expectations regarding the type of topics that will be covered during the session, especially if any of your topics are personal. This advanced warning should reduce a participant’s surprise at your questions.

11.3 Participant insists that she would never do something

When you ask the participant to show you how she does something or to complete a task, she tells you that it’s not something she would ever do. Or, she tells you that the task doesn’t fit into her current role or responsibilities.

Method(s): Usability study; Contextual inquiry

Frequency: Occasional

Pattern(s) to apply: Clarify the task/question

What to do

While this is similar to a participant who can’t or won’t do something that was a key condition of the recruit (see section 4.2), this participant does match your recruiting criteria.

ent Take advantage of this opportunity to learn more about why this falls outside of what the participant does or would expect to do. You might learn something that will be incredibly valuable for your team about their users, especially if this task is a key component of the product. Ask her to clarify (and demonstrate if possible) her current involvement in the task or process. What are the steps that she takes, and the tools she uses? If the person who normally does the task isn’t there, what would she do?

ent In a contextual inquiry, if you need feedback specifically on this task or process, ask the participant if she can point you toward the person who actually does the task that you’re interested in, and see if you can schedule time with that person. You may be able to adjust your study plan to get some other relevant information from the participant—especially since site visits can be tricky to coordinate—but even if you can’t, be sure to provide her with the promised compensation!

ent In a usability study:

ent Explain that while you understand that this task may not be something she does today, you’d still like her to attempt it so you can get feedback on the interface from the perspective of a new user.

ent If the participant pushes back (e.g., because she is more senior or more junior than the person who typically does this task), let her know that you understand, but her feedback would still be useful in making the product better by helping you understand what type of issues a new user may experience.

ent If she continues to push back or becomes obstinate about continuing, go ahead and skip to the next task. If time permits after you finish the rest of your study plan, you can ask the participant if she’d be willing to return to this task/process so you can get some feedback.

What to say

ent “Are you involved at all in <task/process>? If so, can you talk me through what you do, and what tools you use? Feel free to pull up any tools that you use as part of the process.”

ent “I understand that you don’t typically do this. However, for the sake of today’s study, I’d like you to play the role of someone who does <task/process> so we can understand what the experience would be like for a new user.”

ent “Okay, in that case, let’s move on to our next task….”

What not to do or say

ent Don’t move on to another task right away. Take advantage of the moment to understand more about the participant’s role, and how the task/process gets performed (and by whom).

ent Don’t push too hard if the participant is unwilling to play the role of a new user. Some participants have a harder time than others playing a role, and there may be political situations affecting her willingness to attempt this.

How to avoid

ent During your recruit, ensure that your criteria are detailed enough about the responsibilities and roles participants must have, especially for your key tasks (see section 15.1 for more on this). Also, reinforce these criteria in your communications with scheduled participants so they know what kind of responsibilities you’re expecting them to have. It’s better to discover a mismatch between user types and tasks before you begin a research study!

11.4 Participant is frustrated by the prototype’s limited functionality

The participant is using a prototype that isn’t completely built out. As she tries to complete tasks, you have to let her know over and over again that the functionality she’s trying to use isn’t working. She’s getting annoyed by this and may even make snarky comments like, “Oh, I bet this isn’t working either,” or “You ask me to give you feedback but the thing isn’t even working. How am I supposed to comment on it?”

Method(s): Usability study

Frequency: Occasional

Pattern(s) to apply: Redirect the participant; Reassure the participant

What to do

ent Refer back to what you told the participant in your briefing about her interacting with a prototype that has limited functionality instead of a finished product. Acknowledge that these limitations may be frustrating, especially if there are sections that she’s really interested in seeing or being able to provide feedback for. Gather her expectations for those missing areas; even if it’s outside the scope of your research, it will help the participant feel appreciated and may help reduce her frustration.

ent If the participant seems concerned that the missing or broken functionality indicates quality concerns with the product, reassure her that the prototype is not the same as the finished product. The prototype is just showing a limited set of features so you can get feedback on those specific areas.

ent Let the participant know that it’s helpful for you to see where she is trying to go, even though the functionality isn’t available. Emphasize that knowing her expectations for how an area of the prototype works is useful and ask for her to go into detail about how she would want those specific areas to behave.

ent Be patient with the participant, even if she has complained about the lack of functionality multiple times. Use an appreciative, sincere tone.

What to say

ent “I’m sorry that we couldn’t get everything built out in this prototype for today. But your expectations for how it should work are important too. Knowing where you expect things to be and what you’d expect to see if those areas were working is very useful for us.”

ent “As you can see, not too much is working in the prototype today. But thank you for being patient and just keep showing me where you’d go for these tasks and describing your expectations.”

What not to do or say

ent As much as you may want to snap back at the participant, don’t! At the end of the day, the participant is “always right” and you need to try to diffuse any negative feelings she’s having toward the study setup.

How to avoid

ent Make your prototype as functional as possible given your available time and resources. For more discussion about how to build successful prototypes, see section 15.3.

ent Set clear expectations with the participant at the beginning of the session about what she’ll be using. A participant who is expecting a fully functional product may be horrified by a medium fidelity prototype with what she perceives as “broken” functionality. Reinforce those expectations when a participant tries to do something that isn’t supported by the prototype by quickly acknowledging that the area isn’t built out right now, but you’d like to hear what she expects to see/happen. By being proactive and redirecting her energy, you can limit the time that a participant spends wondering if the prototype is broken.

11.5 Participant seems annoyed at your neutrality

The participant asks you tons of questions throughout the session, which you’ve responded to by deflecting the questions back to her and asking for her expectations. She seems to be getting more and more annoyed by your deflections and may even become snarky. For example, she might say, “You’re not going to tell me, are you?”

Method(s): Usability study

Frequency: Occasional

Pattern(s) to apply: Take responsibility; Reassure the participant; Build engagement

What to do

This annoyance typically happens with first-time usability participants who may not understand the reasoning behind your moderating approach. Participants may feel annoyed, perhaps because they’re genuinely looking for answers or help and aren’t getting any. Or they may feel patronized or challenged by what they perceive as you avoiding their questions.

ent Acknowledge the participant’s frustration and the artificiality of the situation with a friendly and understanding tone.

ent The next time you’re asked a question:

ent Consider your words carefully before answering instead of immediately turning it around with “What do you think?” or “What were you expecting?”

ent With each question, subtly change the way you respond. For example, one time ask, “So—I’ll turn that around on you again. What are you thinking?” and another time ask, “Tell me what you’re expecting?” If you’re not confident in your ability to rephrase on-the-fly, write down some responses ahead of time in your study plan that are similar in meaning but use different words.

ent Use paralanguage such as intonation to show that you’re genuinely interested in an answer. If you’re comfortable, you can use a bit of humor in your response. For example, “I know I sound like a broken record here! I really do want to understand your expectations though.”

ent If the participant is really annoyed, do one or both of the following things:

ent Reassure the participant that it’s really valuable for you to hear her questions. Remind her that if she was attempting these tasks on her own you wouldn’t be sitting next to her, so you’d like to see what she would do if you weren’t there.

ent Take responsibility by apologizing once for your constant neutral responses. Avoid apologizing more than once though, as that may create awkwardness and potentially bias the participant’s feedback.

What to say

ent “The feedback you’re giving is really valuable for our team. We need to know what questions people will have when they try to use this, so the product can do a better job of providing those answers.”

ent “I know it’s odd to have me keep turning the question around on you, but I really want to play a neutral role here and just understand your expectations. Thanks for being patient with me.”

ent “If you were doing this on your own and I wasn’t sitting next to you, what would you do?”

What not to do or say

ent Don’t be defensive about being neutral, or apologize more than once for your responses.

How to avoid

ent At the beginning of the session, set the expectation that you may not be able to answer questions during the session and that you may turn any questions back on the participant to gauge her expectations. Stress that even if you do so, you’re genuinely interested in hearing her ongoing feedback and any questions she’d have while using the product.

11.6 Participant does not seem to respect you or take you seriously

The participant does not seem to take you seriously. She seems to think that you’re really young or inexperienced, old and out of touch, or are operating at a different (lower) level than she is. Participants who have a lot of authority in their jobs (e.g., doctors, high–net worth investors, executives) sometimes act this way, especially if they feel like the session may be a waste of their time. Their feelings may come across as condescension (e.g., “You wouldn’t know the kind of things I have to go through”) or rudeness (e.g., “You probably weren’t even alive then”).

Method(s): Any

Frequency: Rare

Pattern(s) to apply: Redirect the participant; Build engagement

What to do

ent Be unfailingly polite. You can think of it as a challenge to get the participant to leave the session with a newfound respect (or at least a grudging acceptance) for you!

ent If the participant pointedly asks you how old you are, or for any other personal information about yourself, try to avoid directly answering the question. For example, if she asks how old you are, you can say that you’d rather not answer and quickly redirect to another topic. Providing personal information shifts the balance of the session away from your control and may take the participant further off topic. However, if you do not answer the question it may be perceived as rude, so you should provide a quick answer followed immediately by a redirect back to the task or question at hand. If you’re comfortable, you can also try using light humor. For example, if the participant asks what year you were born in, you can say, “I’m a baby boomer but that’s neither here nor there! Can you tell me more about.…”

ent Consider playing up the role that the participant seems to have cast you in by asking the participant to help explain things to you. For example, if the participant seems to think that you’re too young or inexperienced to understand the kind of issues she’s facing with retirement, respond immediately by asking her to tell you more about her situation and the challenges she’s facing. This approach has the added advantage of flattering her ego, and may help her “buy in” to the session.

What to say

ent “I’d rather not answer that question right now, but I would like you to tell me more about.…”

ent “You alluded to some of the factors that you’d consider important when making this decision. Tell me more about those factors and how they would influence the choice you’d make.”

ent “It’s really important for us to get the perspective from someone with your experience and expertise, so we’re really grateful you found the time to come in. We want to make sure that this product will be designed to meet your needs.”

What not to do or say

ent Don’t feel obliged to share personal information with the participant.

How to avoid

ent Establish your credentials and your role at the beginning of the session. Let the participant know that you’re a neutral observer, not a subject-matter expert.

ent Treat the participant with respect by using her title (e.g., Dr. Smith) until she tells you otherwise.

ent If the participant has a high-level role or a lot of authority in her job, acknowledge that the role she plays in her organization is a critical one so you’ll make good use of her time.

11.7 Participant becomes insulting or has an agenda

During the session, the participant says something insulting or disparaging that’s directed toward you, your stakeholders, or your organization. If the comment is directed toward your organization, she may be venting based on experiences she’s had outside of the session. She may have come to the session with an agenda or to nurse a grudge, and wants to make sure her opinion gets heard. This kind of behavior can happen both with an organization’s internal employees and its customers.

Method(s): Any

Frequency: Rare

Pattern(s) to use: Redirect the participant; Take a break; End the session early

What to do

ent If the participant seems to be venting about your stakeholders or your organization, provide her with a focused outlet for her venting. For example, you can acknowledge that she seems to be upset about bad experiences she’s had, and suggest taking the next few minutes to have her share what happened and then proceed with the rest of the session.

ent If the participant insults you directly (e.g., “You’re really bad at this”), take a moment before responding. You may be in shock but need to carefully think through your next step so you avoid escalating the situation. Try to either ignore the comment and move on without acknowledging it, or place that topic out of scope (e.g., “That’s not relevant to our session today”) before quickly moving on to another area of discussion.

ent Try your best to not react, and maintain a neutral expression and tone. This may be difficult to do, especially if you feel personally attacked, but do your best. Take a deep breath, control your face, and imagine the participant’s comments rolling off of you like water on a duck’s back—don’t let them stick.

ent Take a short break if necessary so you can recover and maintain neutrality in front of the participant. Use the break to consider your options for what you can do to improve the session once you resume.

ent If the participant’s insult was in the context of what you’re doing for the study (e.g., a derogatory remark when you ask her to use a website she strongly dislikes), clarify your role and responsibilities as a neutral observer and what your expectations are for what she’ll be doing. Apologize for any misunderstandings and ask if she has any more questions about how the session will work.

ent If the participant’s comment seemed sparked by what you were asking her to do or talk about (e.g., “Only an idiot would even try to do that”), change the topic of conversation or redirect the participant to the next task or section of your study plan.

ent If the participant continues to be insulting or disparaging toward you or becomes confrontational, end the session early and pay her any promised compensation. Remember that you need to look out for your emotional well-being and that additional data point you’d gain by sticking with the session is not worth it.

What to say

For a venting participant:

ent “You keep alluding to some really bad experiences with <organization>. Why don’t we take the next few minutes for you to share your experiences? However, after about five minutes, we need to get back to my list of questions. Does that sound okay?”

ent “Thank you for sharing that experience. Part of why this research is being performed is to help improve what users go through, so your feedback will hopefully keep others from going through bad experiences. So, let me ask you <return to your study plan>.”

For a participant who deliberately insults you:

ent “This topic is not relevant to our session today. Let’s go back to our questions/tasks, unless you’d like to take a short break first?”

To redirect a participant:

ent “I’d like to bring your attention now to <next task or question>.”

What not to do or say

ent Don’t get pulled into the participant’s agenda.

ent Don’t insult or get into an argument with the participant, even if you really want to! Remember to stay professional.

ent If the participant insulted a group of people who you strongly identify with, don’t bring up yourself as a counterpoint to whatever the participant said.

How to avoid

ent Be upfront with participants about the purpose of the session. This may not filter out everyone who has an agenda, but may reduce the odds.

ent Pay attention to the participant’s emotional state throughout the session and diffuse any earlier discomfort before it escalates to insults or disparagement.

11.8 Participant becomes agitated by a product’s usability issues

The participant seems really upset by the product she’s using. She may be frustrated at not being able to figure out how to accomplish a task or answer a question, and takes that anger out on the product or the people who created the product. While some agitation makes for powerful video clips, an extended period of anger ceases to be productive and could derail the session.

Method(s): Usability study; Contextual inquiry

Frequency: Rare

Pattern(s) to apply: Take responsibility; Redirect the participant; Take a break

What to do

ent Maintain a calm, neutral demeanor, even if you’re startled or nervous about the participant’s agitation. Keep the tone of your voice neutral and calm, and avoid any sudden shifts in body language. Avoid anything that might increase the level of agitation.

ent Acknowledge the participant’s frustration, and thank her for sharing her feedback. Slowly adjust your position to angle toward her and make eye contact. Reassure her that even though you know she’s irritated, it’s helpful to hear about these issues so the product can be improved.

ent Redirect her away from whatever is causing the extreme agitation. For example, if she’s frustrated because she can’t figure out how to accomplish a task, follow the recommendations in section 6.4 on giving the participant an assist to help her continue, or just move on to the next task/question that covers a different area of the product.

ent If the participant continues to be agitated, take a short break so she has a chance to calm down. When you resume the session, move on to a task/question that covers a different area of the product.

What to say

ent “It’s really helpful for us to hear what areas are confusing or frustrating so they can be improved in future versions. So, we really appreciate you going through all this with us!”

ent “Thank you for your feedback so far. Let’s take a quick five-minute break so I can see if any of our observers have any questions for you.”

What not to do or say

ent Don’t be impatient toward the participant or defensive about the product. Remember that the goal of your research is to understand a product’s usability issues, which is exactly what the participant has discovered.

How to avoid

ent At the beginning of the session, tell the participant that if she finds anything confusing or frustrating, you want to hear about it so the product can be improved so others don’t experience the same issues. Making sure the participant feels that she has permission to share her frustrations may reduce any agitation she feels later in the session.

See also

SURVIVAL STORY: “I WOULD HAVE TRUSTED MY GUT”

Christina York

The first time I had an unqualified participant slip through our screener, I was pretty green. I was conducting in-person usability studies in our lab. The participants would meet us at our office and do the usability assessment in a little lab we had set up in an empty office. I was both moderator and notetaker, with only an audio recorder as backup.

The participant criteria were pretty specific. Participants did not have to be familiar with our product, but they did have to have at least one semester of college under their belt and be currently enrolled in a degree program.

The moment Bobby entered the room, I had a gut feeling he had fooled our screener. He walked in with way too much confidence but wouldn’t look me in the eye. He had no paraphernalia that screamed, “I’m a student!”—no book bag, no computer, no notebook, no college colors, no case of acne. I didn’t know enough to trust my gut instinct and find out if he really was a student, so I pushed on with the session. When I asked him what browser he usually used, he deflected easily: “Oh, any old browse [sic] you got.” That is when I knew I was in trouble.

Bobby had difficulty with the tasks. He would ignore what I was asking him to do and dive into the website, gesturing with his hands at the screen, erratically moving the mouse (sometimes in midair), and all the while explaining why our website wasn’t working. He took on the attitude of teacher, and started explaining to me how search works, or “how to use your computer to read books on.” He confidently talked over any errors he was making or any confusion he had. He even explained to me how the mouse needed to be redesigned for “real people.” I tried desperately to glean some insight into his behavior and prompted him with questions like, “Tell me what it is about the mouse that makes completing this task difficult.” I didn’t know what to do. I took notes, I asked questions, I thanked him, and I paid him.

Knowing what I know now, I would have trusted my gut and ended the session before it began. I’ve learned that you can’t rely completely on any screener, and that confirming participant qualifications is necessary. I’ve also learned to have a retrospective on every study I conduct and examine my discomfort from situations like this so that I can plan how to handle future discomfort. Also, from that session on, I arranged for a “backup”—a person working nearby who will come into the lab to assist upon receiving my signal. That way, I would never feel like I was “on my own” conducting one-on-one research.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
52.14.239.155