Chapter 4

Recruiting Mishaps

Participants You Weren’t Expecting

Abstract

This chapter covers situations where the participant doesn't fit your recruiting criteria or doesn't meet your expectations in some other way. These include situations such as a participant who does not meet your recruit criteria, a participant who either refuses to or can't do a key task that was a condition of his recruit, and a participant who has a physical feature that does not align with your setup. For each situation you'll learn what to do, what to say, what not to do or say, and how you might be able to avoid it in the future.

Keywords

usability recruitment; usability misrecruiting; user research participant qualifications; user research participant low literacy

Recruiting mishaps can happen despite your best efforts. A participant who doesn’t meet your criteria or who has some other unexpected quality means that your session may immediately start with a go or no-go decision. These situations require you to make quick decisions about whether and how to proceed with the session, and to make any necessary accommodations for any unexpected participant characteristics.

4.1 Participant does not seem to meet a key recruit criteria

Your recruit criteria were very specific about certain qualifications that participants should have. However, when the participant arrives, he seems to not actually meet those qualifications.

Method(s): Any

Frequency: Frequent

Pattern(s) to apply: Take responsibility; Reassure the participant; Shift the focus

What to do

ent First, verify that the participant does not actually meet the desired criteria. One way to do this is to let the participant know that the information you have about him indicates that he has a specific qualification, and ask him directly if that is correct. If you’re looking for participants who perform a certain type of task, ask the participant to describe his role and, in particular, what he does regarding the task that you’re interested in. Another way is to be subtle with your line of questioning, so that it comes off as normal background questions. That way if you immediately take a break and/or end the session, the participant won’t feel like it’s his qualifications that caused the problem.

ent Keep in mind that once the participant arrives, you’re obligated to provide him with compensation (unless there was another agreement in place; see section 15.2 for more about your compensation decisions), so you might see if there is a way to adapt the session to include the participant’s feedback, or shift the focus of the session entirely to another method. To decide this, see if the participant’s actual qualifications would still be useful for your research. For example, based on your research goals, you may know that talking to a novice manager will provide useful feedback even though your primary target for recruiting was experienced managers. If your stakeholders are observing in person, you should take a moment to discuss this with them privately. Especially if your project has time or budget restrictions, you may not have a chance to replace the participant with someone who’s a better fit. Be prepared to argue the case for your observers as to why the participant still may yield useful and valid data.

ent If you need to let the participant go and not proceed with the session, consider being candid about why you’re unable to proceed. This approach may be appropriate if the criteria was role-based—for example, “I apologize for the misunderstanding. For the purpose of this study, we need to talk with managers who have direct reports.”—or any other reason where the participant won’t feel like he “failed.” However, if the reason is potentially sensitive, use a pretext such as having “technical difficulties.” Be appropriately apologetic to the participant, especially if he was excited about giving his feedback, and provide him with his full compensation.

ent If the participant is a very important person (VIP) or customer, or a colleague hooked in with your project team, you probably can’t get away with the technical failure pretext. Check with your observers/stakeholders on the best way to handle this particular participant. They may encourage you to proceed anyway so the participant feels like he has had a chance to provide feedback that will be taken seriously, even if it isn’t in line with your original research goals. One way to get this feedback is to adjust your study plan to learn more about what the participant does or is interested in doing.

What to say

ent “Before we continue, I just want to verify a few things about your <experience/expertise/role>. According to the information provided by our recruiter.…”

ent “I’m sorry, <participant name>, but for the purpose of this research, we need to talk with people who <criteria>. Thank you so much for your willingness to come in today, and here is your agreed-on compensation.”

Ending early:

ent “I’m sorry, but it looks like the prototype I wanted to get your feedback on isn’t working. I have a few more questions for you, but after that, we’ll end the session early. You’ll still get your full compensation though!”

What not to do or say

ent Don’t be accusatory about the participant’s qualifications (or lack thereof). He may take it personally. When verifying the participation criteria, do so in a subtle and nonjudgmental way.

ent Don’t assume that the participant lied about his qualifications. The screener questions could have been poorly worded, or your recruiter may have been more lenient than he should have been. After the session, go back and review the requirements with the recruiter.

ent Likewise, don’t assume that it was the recruiter who messed up. Until you’re able to look into what happened, you won’t know the truth of the situation. When interacting with the participant, always take the responsibility for the misunderstanding yourself, even if you’re pretty sure the fault lies with your recruiter.

How to avoid

ent Ensure that the recruiter has detailed information about each of your desired criteria and how to evaluate whether or not the participant matches the criteria. Avoid using “give-away” questions in a recruiting screener where a respondent can guess what the correct answer is. For example, instead of asking a yes/no questions such as, “Do you use an iPad?” ask the more open-ended, “Do you use any mobile devices or tablets? If so, which ones?”

ent Verify the participant’s qualifications at the beginning of the session, especially if any of your criteria are deal-breakers that would require you to end the session.

ent Consider having backup or floater participants available. This way, if a participant doesn’t meet your requirements, you can still get feedback from someone during the session time. For more on backups and floaters, see section 15.1.

ent Talk to your team ahead of time to understand their tolerance for misrecruits, and create an action plan for how to address them. It will be helpful to set expectations upfront, so that you can have a quick and efficient discussion during an actual session.

Watch Video 7 to see an example of a moderator realizing that the participant scheduled for an interview lacks the necessary experience. The moderator discusses what to do with a stakeholder and finds a way to adapt the session.

Visit our website (http://www.modsurvivalguide.org/videos) or use your QR reader to scan this code. image

4.2 Participant either refuses to or can’t do a key task

As part of the recruiting process, participants were told that they would be asked to do a specific task during the session. For example, you need the participant to log in to his bank account and perform nontransactional tasks, so the recruiter told him that he must be able to log in to his account for the session. When you ask the participant to do this, he either can’t (e.g., because he has forgotten his login information) or won’t (e.g., because he changed his mind).

Method(s): Usability study; Contextual inquiry

Frequency: Occasional

Pattern(s) to apply: Take responsibility; End the session early

What to do

ent If you were explicit about participants needing to do something specific during the session as a condition of the recruit, and the participant is unable or unwilling to do this, end the session. The exceptions are if there is a way for you to reschedule the participant, shift the focus of the session, or you have some other backup plan in place. You’ll need to provide the participant with his compensation unless you were crystal clear throughout the process that the compensation was contingent on performing this task. However, even in this circumstance, err on the side of the participant, especially if you have any sense that he misunderstood what he signed up for.

ent If the participant is willing to do the key task but is unable to (e.g., because he forgot his password), see if there is a way for you to help him get whatever information he needs to proceed. For example, if he forgot his password, ask him to go through the Forgot Password or Password Reset process. If this process involves him using the phone or answering sensitive questions on the screen to reset account information, leave the room to avoid overhearing personal identifying information and/or be sure that nothing is being screen-shared or recorded during that time. If the participant seems reluctant to make this attempt, do not force him, and instead end the session.

ent Offer to reschedule if the participant is genuinely disappointed about not being able to participate. Be sure to confirm and reconfirm your expectations for what he’ll need to do during the rescheduled time!

What to say

ent “I’m sorry, but we need participants to do <task>. Unfortunately, this means that we’re not going to be able to have you participate today. If you’re able and willing to get the information and come back, we’d be happy to reschedule you.”

What not to do or say

ent Don’t assume that the participant lied about his willingness to do this task when being recruited. The screener questions could have been poorly worded, or your recruiter may have been more lenient than he should have been. Take responsibility for any miscommunication.

How to avoid

ent If you’re making the participant’s compensation contingent on his ability to attempt the key task, confirm this task with him several times before the session. Be as explicit as possible about what he may need to have prepared for the session, especially if it will involve accessing potentially sensitive information. As soon as the session begins, verify that the participant will be able to attempt the key task before he signs the consent form (and thus before being recorded or observed by others).

ent Alternatively, you can avoid conflict altogether and compensate any participant who shows up for testing, regardless of whether he is willing or able to move forward.

ent Despite your best efforts, accept that you may have a participant arrive who still falls into this category. Have a backup plan in place for these participants in case any misunderstandings occur. Your backup plan may include having backup or floater participants available, shifting the focus to a different type of study, or setting up a test account that a participant can use to attempt tasks on his own. For more discussion on backup participants, floater participants, and backup plans, see sections 15.1 and 15.2.

4.3 Participant has an unexpected physical feature

Your room’s physical and technical setup works well until you get a participant who has a physical feature that requires an adjustment. For example, the participant may be left-handed, have an injured arm in a sling, or be a very large or small person.

Method(s): Any in-person method

Frequency: Occasional

Pattern(s) to apply: Take responsibility; Reassure the participant

What to do

ent If the feature is something you notice on your own (e.g., very tall, left-handed), make any adjustments to your facility as soon as possible. For example, show the participant how to lower the chair, or move the mouse from the right to left side). When you make the adjustments, place any blame on you or your facility for not being set up correctly.

ent If a participant is very large and has trouble fitting into his chair, offer to get him a different one without making a big deal about it. A chair without any arms can work very well. While most participants won’t take you up on your offer, those who do will be very grateful.

ent If the participant apologizes for inconveniencing you, reassure him that it’s no inconvenience and that you’re looking forward to his feedback.

What to say

ent “I apologize that we’re not set up as well as we could be. Let me show you how we can adjust the chair to a more comfortable height.”

ent “Is there anything I can do to make you more comfortable?”

ent “Sorry, but before we continue, I’m going to help you lower/raise the chair so our recording equipment can capture you.”

What not to do or say

ent Don’t call unnecessary attention to the participant’s physical attributes.

ent Occasionally, a participant will call attention to his own attribute to acknowledge it. If he does so, try not to follow up with personal questions. Focus any questions on understanding how you can better accommodate him. If the participant makes fun of himself for some sensitive attribute, do not laugh along. Smile politely, and then move on to your study plan.

ent Don’t complain about any extra work created by making the accommodations for the participant.

How to avoid

ent Familiarize yourself with the adjustments that your facility and equipment can support. For example, most chairs and desks can be lowered or raised, other types of chairs are usually available from somewhere, and a mouse (and its control panel settings) can be moved from the right side to the left side.

4.4 Participant is unfamiliar with the equipment

Your session requires the participant to interact with a computer, mobile device, or similar type of equipment. When asked to use this equipment, the participant indicates or demonstrates that he lacks basic familiarity with it. You can tell that proceeding with him will be impossible unless you spend some time acquainting him with the basics of the equipment.

Method(s): Usability study

Frequency: Rare

Pattern(s) to apply: Reassure the participant; Shift the focus; End the session early

What to do

ent First, figure out if the participant is just unfamiliar with the platform that is being used. For example, participants who are used to older Apple computers and mice may not understand how to use a Windows two-button mouse, but have an understanding of how a mouse is supposed to work. If this is the case, provide a short explanation of how the device works on this platform or, if possible, switch to what he is used to (e.g., if you’re testing an application on an Android operating system and he only knows how to use an iPhone with iOS, you may want to switch devices if you’re able to do so).

ent If the participant is truly unfamiliar with the equipment, experiment with a short five-minute explanation and demonstration of what to do. Depending on what your research is focused on, that short time period may be enough to get him to a level where he can proceed. Make it clear to the participant that you’re training him just for this first five minutes, after which you’ll no longer be training but rather observing.

ent Let the participant attempt the first task, with you providing minimal technical support as necessary. Ask him to think aloud so you can track whether any problems he runs into are based on the interface he is interacting with or his unfamiliarity with the equipment.

ent If the participant continues to struggle, shift the focus of the session to limit the use of this equipment and end early. Handle this as gracefully as possible. See section 10.8 for additional tips for dealing with struggling participants.

ent If at any point the participant seems like he is feeling bad or inadequate, reassure him that he is not being tested and that the feedback he is providing is helpful.

What to say

ent “You seem a bit uncomfortable with the <equipment>. Have you used anything like this before?”

ent “I’m going to take a few minutes to help get you set up on this <equipment>, since we’ll be using it later in the session.”

What not to do or say

ent Don’t show any surprise or shock regarding a participant’s experience level. Remember that you don’t know anything about the participant’s background, and it can be dangerous to make assumptions.

How to avoid

ent Consider adding some basic equipment experience into your recruit criteria. We’ve found that recruiting participants who spend at least five hours a week using the Internet doing varied activities (e.g., not just checking email but reading online, purchasing from e-commerce websites, etc.) typically ensures that they can use a computer and a mouse.

ent Depending on your study goals, this situation may not be something that you want to avoid. The perspective that a very new or inexperienced user provides can be extremely valuable. However, for a typical usability study, the extra time spent helping an inexperienced participant results in less time getting the feedback that you need, so you’ll have to weigh this trade-off on a study-by-study basis.

4.5 Participant has difficulty reading

The participant meets the recruiting criteria for the study, however, when you give him a task to read, he is hesitant and slow and struggles with the words. He has similar difficulties when trying to interact with the product, especially if there is a lot of text. You might notice that he talks about expectations instead of using the product, even when you explicitly ask him to do so. It seems that he may have low-literacy abilities, be a non-native English speaker, or have some kind of reading disability. As the participant struggles, he may grow increasingly nervous and uncomfortable, or seem embarrassed.

Method(s): Usability study

Frequency: Rare

Pattern(s) to apply: Reassure the participant; Shift the focus; End the session early

What to do

ent If the participant is having difficulty or seems nervous reading tasks aloud, read the task to him or ask him to read it quietly to himself and then verbally reiterate the task. You may find that despite having issues reading the tasks, he may approach the tasks with the product without issue.

ent If the participant continues to struggle with the tasks and is getting increasingly nervous or frustrated, shift the focus of the session to reduce the amount of reading required. For example, if he communicates well verbally, turn the session into an interview. If necessary, use a pretext to explain the shift (e.g., tell the participant that the prototype isn’t working as intended so you need to change the format).

ent Whether or not you shift the focus of the session to another method, end early. If possible, try to do it in such a way that the participant doesn’t realize that you’re ending things early (e.g., by saying that you got through everything faster than anticipated).

What to say

Switch over to reading tasks to participants:

ent “Let’s say that your next task is to.…”

ent “I apologize—I didn’t word this task very well. Let me try to rephrase it.…”

Change the focus to an interview:

ent “It looks like our prototype is not working as we planned it to, so if you don’t mind I’m going to just ask you some questions.”

End the session early:

ent At beginning of the session: “I’m sorry, but we’re going to have to cancel the session due to some technical difficulties. Thank you so much for coming in, and here’s your compensation.”

ent If a little later in session: “That is actually all I had for you today, so you’ll get some time back in your day! Thank you so much for your feedback, and here’s your compensation.”

What not to do or say

ent Don’t show any surprise or shock regarding a participant’s literacy level.

ent Avoid anything that might make the participant more nervous or embarrassed about his performance, such as pointing out that he seems to be having trouble reading.

How to avoid

ent During recruiting, let participants know that they will be interacting with a product rather than just answering questions.

ent If low-literacy participants are part of your target recruit, be sure to set up your study plan appropriately to accommodate their abilities. For example, you should not require them to read tasks.

4.6 Participant or others ask you to help

You’re onsite to conduct a contextual inquiry. Because they see you as an extra set of hands, the participant or your other contacts onsite ask you to pitch in with their work, provide training, or help out with something else. Their request may start as a quick favor, but they continue to rely on you to help as the session is beginning.

Method(s): Contextual inquiry

Frequency: Rare

Pattern(s) to apply: Clarify the task/question; Disengage with the participant

What to do

ent Clarify your purpose for being onsite and what you want to get out of the contextual inquiry. Emphasize that you wouldn’t typically be there to help, so you would rather see how the participant handles his work on his own.

ent The participant and your other contacts may perceive your session to be something informal, especially if you have a preexisting work relationship with them. If this is the case, disengage slightly and adapt to a more formal moderating style. For example, you may want to take a less social tone, and let them know that you need to start/resume your planned session goals.

ent If you have the resources available, offer to help after the session is over (if there is time) and/or to find them more help.

What to say

ent “I’m sorry, I would like to help—it actually looks fun/interesting. But my role is really to be a neutral outside observer here. If I’m helping, I won’t be able to focus on all that I need to.”

ent “If I helped you, you’d be dividing up labor that you usually do yourself, so I wouldn’t get an accurate depiction of your day-to-day work. I’d rather see what you’d do if I wasn’t here.”

ent “I wish I could be of help, but I can’t really leave to do <favor asked of you>. My role is to sit here and watch <the participant’s> end-to-end workflow. I’m happy to help after, though.”

If asked to train:

ent “I’m sorry, my role is not to train. It’s just the opposite—I don’t want to show you what to do! I’m here to sit back and observe the way you naturally do things on your own. If we have time at the end, I can try to answer any questions for you then.”

What not to do or say

ent Don’t be offensive or condescending about not helping. Remain professional and just explain your viewpoint.

ent Avoid helping participants do their work, especially in a business environment. Doing so may create liability issues if you give them bad advice and they lose or corrupt data. If you’re an expert in the systems used by the participant, you may offer some basic tips like shortcuts or ways to be more efficient at the end of the session. However, keep in mind that you don’t know all the details about their systems, so it may be better to direct the participant to someone who is in a better position to help.

How to avoid

ent While preparing for your field visit, set expectations that you’re there to watch what the participants do. Reiterate this point when you brief the participants at the beginning of a session.

See also

ent 5.1 Participant thinks that she is participating in a focus group

ent 5.5 Participant treats a contextual inquiry like an interview

SURVIVAL STORY: “AN UNEXPECTED PICTURE STARTED TO EMERGE”

Anonymous

In mid-2012, I was one of the user experience researchers for an interview-based study designed to inform new personas we were building. The main research goal of this exercise was to determine our customers’ emotional relationships with their money.

The day had been going well; I was viewing sessions remotely, taking notes for the moderator. At the beginning of the next session, the male participant who entered seemed “normal” enough—around 30 years old, informally, yet neatly dressed, and pretty articulate. As my colleague warmed him up during the background interview, an unexpected picture started to emerge.

“You just told me that the economic downturn necessitated moving back in with your family, is that right?” (Sadly, this was not an infrequently heard fact—even worse were stories of participants who didn’t have a “home” to go back to as they were on the brink of losing their houses and honestly did not know where they would turn next.)

“Well, yes, but I got kind of tired of that. So I moved after a few months. Into my … car.”

My colleague could not have handled this startling news with more grace. The participant went on to detail his grooming rituals, which included using baby wipes, mouthwash, and powder.

For the personas we were building, it was clear that his input would be excluded as an outlier within our target customer base. After a few minutes, our colleague pricked up her ears, saying she thought he heard a knock on the interview room door. She left the room for 30 seconds (which seemed an eternity over the phone) and returned with regret in her voice to tell the participant that the team was experiencing “technical difficulties with the recording apparatus” and would need to end early. The participant seemed to take it in stride.

What surprised and impressed me about the moderator’s reaction was how, once she had determined that the participant no longer qualified, she took a few minutes to gain composure and figure out what she would do next. She waited long enough past the critical moment to put his suspicion at bay, especially given her lack of reaction when she was listening to him.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.219.156.22