After your recruitment planning is complete, you may begin the process of actually contacting potential participants and scheduling them for sessions. Budget and staff permitting, you have a few options: 1) do the recruiting yourself (or within your organization), 2) hire an agency to recruit for you, or 3) have some other outside person do the recruiting.
The results of the NN/g recruiting survey indicate that a majority of companies were doing their own recruiting, as follows:
79% of respondents indicated that the person who will be running the study recruits participants for an average of 70% of the studies.
32% indicated that someone else within their company, usually clerical staff, recruits for an average of 32% of the studies.
12% indicated that a dedicated recruiter within their company recruits for an average of 67% of the studies.
The survey indicates that the remaining respondents hire a recruiting agency or have some other outside person recruit for them, as follows:
36% indicated that an outside recruiting agency recruits for an average of 61% of the studies.
13% indicated that some other person, usually the client, recruits for an average of 32% of the studies.
(Numbers total more than 100% because many companies use multiple methods. Also see Who Recruits Participants, on page 109.)
We also tested whether survey data would confirm the anecdotal data we received from usability professionals that the no-show rate is higher for professional recruiting agencies than for internal recruiters. The results from our survey tend to back up this claim, although the differences in no-show rates are not statistically significant. Three internal recruiter categories do only slightly better, while a fourth does only slightly worse—the person who runs the study—than the world average rate of 10.6%. Survey results indicated a no-show rate of 11.1% for people who run studies, and 10.8% for recruiting agencies. (Also see No-show rates by who recruits, on page 120.)
Whether you recruit yourself or use an outside source, don’t cut corners.
Remember that all your hard work planning the study itself will not produce reliable results unless you make the effort to find the right users. Spend the time and money needed to conduct rigorous recruiting. (Also see the article written by Susan M. Dray, Ph.D. and David A. Siegel, Ph.D., Penny Wise, Pound Wise: Making Smart Tradeoffs in Planning Usability Studies. http://www.dray.com/articles/pennywise.htm)
Even if your company is fortunate enough to have recruiters on staff, from time to time you may find it more expedient, or even necessary, to engage a professional, outside recruiting agency in order to meet your schedule. As noted above, the results of the NN/g recruiting survey indicate that 36% of the survey respondents hire an outside recruiting agency and that these companies use agencies for an average of 61% of their studies. Many good, professional recruiting agencies are available to help you with your recruiting needs. NN/g has used agencies all over the world with good results.
In addition to recruiting, many agencies offer a range of services, such as preparing screeners, hosting, and providing incentives to participants. Some agencies also have facilities you can rent for focus groups and usability testing, and often these facilities can provide lab techs and note-taking observers (for a fee, of course). This section provides advice for deciding when to use an outside agency, as well as finding, choosing, and working with agencies.
In a 1998 sidebar to an Alertbox column on the cost of user testing,[16] we discussed when to outsource recruiting and when to do your own recruiting. We’ve incorporated that advice into the following tips on outsourcing.
Avoid using a recruiting agency if you are conducting a usability study of an intranet.
For an intranet usability test, the representative users are your own employees, so it doesn’t make sense to ask an outside agency to recruit them. Instead, get your study users by randomly contacting employees in various departments or job categories. For example, you may ask department administrators to give you the names of people whose last names begin with a letter you chose arbitrarily from the alphabet. Change letters for each department you contact.
Avoid using a recruiting agency if you are conducting a usability study of a site that aims at a narrow group of external users.
Sometimes, the users are a specific, well-defined group (for example, an extranet may aim at the purchasing agents for major customers). Other times, the site has a list of registered users or existing customers. In these cases, you will probably have an easier time contacting these potential study users yourself, and they will be more willing to listen to you than to an external agency.
Use a recruiting agency if you are conducting a usability study of a site that aims at the general population or at a relatively broad group.
For these types of studies, it can be quite time consuming to call around to potential users and try to build a suitable schedule. It would be easier to have participants recruited by a company that specializes in recruiting. Go ahead and outsource the recruiting so that you can concentrate on the design of your site.
Use a recruiting agency if your study is tightly scheduled, and you simply do not have, or cannot afford, the resources to do the recruiting.
Occasionally, you will find that you just don’t have the time or affordable personnel to do the recruiting for a study. For example, a highly paid senior study facilitator who takes two or three days to recruit for a study may cost more in time than your budget can afford. Take the opportunity to try out an agency who may be able to recruit for a lot less than a senior user experience staff member.
Check with your marketing department for existing contracts with recruiting agencies.
They may already have a contract with a focus group company that specializes in your industry and has all the necessary procedures in place to recruit people for studies.
Ask usability colleagues which recruiting agencies they’ve used successfully.
This is probably the best endorsement you can get. If you enjoy good results working with a recommended recruiting agency, be sure to thank the colleague who recommended the agency and let the agency know you were satisfied with their work. Also be willing to recommend the agency to other usability colleagues.
Visit http://www.stcsig.org/usability/topics/recruiting-firms.html
The Usability Special Interest Group (SIG) area of the Society for Technical Communication (STC) website provides a good deal of information about participant recruiters and lab facilities. The site lists agencies and labs outside the US as well.
We encourage usability professionals everywhere who have had a successful experience using a recruiting agency or lab / focus group facility to contribute to this resource through [email protected].
STC asks for a range of useful information when you refer a business:
Facility or agency name and website (if available)
Address and contact information
Does the firm have lab facilities? If so, does the facility have:
Interview rooms (no mirrors or observation rooms)?
Market research rooms (mirrors, observation rooms and recording capability)?
Usability setup (record from screen, picture in picture)?
Does the firm provide audio or videotapes and / or other supplies?
Does the firm recruit participants?
Any restrictions (for example, recruiting only for sessions conducted in their facility)?
Does the firm provide usability study facilitators, participant hosting / greeting, or other services for conducting usability tests?
Please note that updates to this resource list on the STC website are made on an irregular schedule by a volunteer staff. STC reserves the sole right to determine the contents of this resource list.
Post an inquiry on a professional usability mailing list.
The usability community is always happy to help with an inquiry, and most usability mailing lists have members worldwide, which is especially helpful when you need to find an agency in another country. We recently posted an inquiry on a usability mailing list to help us find recruiters in London and received three excellent recommendations the same day.
Because many usability mailing lists are privately owned and maintained, we cannot provide information about them directly. Instead, ask any seasoned usability colleague to recommend his / her favorite usability-related mailing list, and join. But be prepared to receive email on a daily basis. You’ll quickly learn to scan the subject lines for the topics of interest to you. Be prepared to follow the posting and propriety guidelines of any email communities you join, or you could be expelled.
In the US, look up “Market Research” in the Yellow Pages of your telephone book. In other countries, try equivalent terms in the local language.
Carolyn Snyder of Snyder Consulting offers this advice:[17]
“For a preliminary source of potential companies that can provide recruiting services, look in the Yellow Pages under ‘Market Research.’ A physical phone book is often superior to an online one for this type of search, because an online directory may provide only the company name, address, and phone number.
“Company names may not be very descriptive—print ads are useful because they list services.
“At the time of this writing (early 2001), it would be quite unusual to see ‘usability testing’ advertised as a market research service, so look for firms that conduct focus groups. Focus groups are similar to usability testing in that participants who meet a certain profile must travel to a facility at a scheduled time.
“Thus, any company that’s accustomed to conducting focus groups should theoretically have the resources to handle recruitment for usability testing.”
To find recruiting agencies outside your local area, check online Yellow Pages.
Check the website of your local newspaper, which may have an online Yellow Pages. For example, “On Wisconsin,” the website of the Milwaukee Journal Sentinel, produces a good many search results for “market research.” It lists agencies for cities across the US, not just Milwaukee. http://www.switchboard.com/bin/cgidir.dll?MEM=1304
Note that, as Carolyn Snyder warns, you don’t get a description of services offered. You can enter the name of a company that sounds promising as a search string in Google to learn more about them, however. http://www.google.com
Outside the US, check the STC resource that lists recruiting agencies and labs outside the US. http://www.stcsig.org/usability/topics/recruiting-firms.html
See Tip 90 for ways to contribute to this resource through [email protected]
Try using the search strings: “usability research,” “market research,” “focus group recruiting,” “usability recruiting,” “participant recruiting,” “usability labs,” “usability testing,” “user testing” and other similar queries, followed by the name of the city where you want to test. If you will be evaluating an interface in non-English speaking countries, try equivalent terms in the local language, but often it is also worth trying the English query terms, because some recruiting agencies use them in an attempt to attract international customers. In July 2005, the search results were less than optimal, although we turned up a few good hits with Google.
Create a spreadsheet or text file to record your recruiting agency research and share it with your colleagues.
This list will save you from having to research the same thing again and again. When you’ve used one of the recruiting agencies successfully, record a short description of your experience for colleagues who may access your file.
Allow adequate lead time—at least two weeks.
Survey respondents, as well as the agencies we spoke with, indicated that agencies usually need at least two weeks’ lead time for recruiting. Some agencies may ask for more time, depending on the number of user groups you are targeting and the difficulty of finding the targeted groups (8% of the survey respondents who use agencies indicated they provide as much as five or six weeks lead time). Ask in advance how much lead time the agency requires and budget the time in your schedule accordingly. Some agencies may be able to handle emergency requests on shorter notice, but it’s best to avoid this situation whenever possible—the fees generally go up for emergency requests. Plan ahead.
To be safe, budget recruiting fees within 10% of what you expect to pay in incentives.
NN/g researched recruiting agencies in the US—West Coast, Midwest, and East Coast—and in London for a study on the investor relations section of corporate websites.[18] We researched the same four user-group profiles in all three places: individual investor, professional investor, financial advisor / planner, and financial journalist.
The following table compares the recruiting fees that agencies quoted with the participant incentives that recruiting agencies recommended, in US dollars.???
Table . Recruiting Fees and Recommended Incentives in US Dollars
Averages (15 agencies)
East Coast (8 agencies)
West Coast (1 agency)
Midwest (3 agencies)
London (3 agencies)
Profile
Fee
Incent.
Fee
Incent.
Fee
Incent.
Fee
Incent.
Fee
Incent.
Individual Investor
$182
$167
$104
$138
$150
$200
$197
$203
$278
$126
Financial Advisor
$185
$180
$114
$191
$150
$200
$197
$203
$278
$126
Professional Investor
$191
$178
$114
$184
$175
$200
$197
$203
$278
$126
Financial Journalist
$195
$182
$106
$200
$200
$200
$197
$203
$278
$126
Overall Average
$188
$177
$110
$178
$169
$200
$197
$203
$278
$126
As shown in the preceding table, for this sample of 15 recruiting agencies, the difference between the average fee charged by recruiting agencies and the average incentive they recommend is USD $11, or about 6%. In this example, the US East Coast fees are generally less than in the West Coast and Midwest, but the London recruiting agencies generally charge higher fees, and they recommend lower incentives. London recruiters told us that participants generally are hard to get. After reviewing the data in the preceding table, we wonder whether London recruiting agencies should just recommend higher incentives, which might make their job easier, and possibly, in turn, lower their recruiting fees.
As noted earlier, 33% of the NN/g recruiting survey respondents indicated they use recruiting agencies for some portion of their studies. In the following table, we present average fee requirements (listed in US dollars) charged by recruiting agencies around the world for internal and external participants, as indicated by 116 recruiting agency citations in our survey, listed by region.???
Table . Average Agency Fees in US Dollars by Region
Region
Recruitment Fee per Participant
Number of Responses
USA – West Coast (Washington, Oregon, and California)
$125
21 external, 3 internal
USA – Northeast (Maine to Washington DC)
$113
24 external, 2 internal
USA – Other
$124
21 external, 2 internal
Europe
$108
18 external, 4 internal
Unspecified
$ 75
2 external, 0 internal
Rest of the World
$ 57
14 external, 5 internal
Average World Fee:
$107
100 external, 16 internal
As shown in the preceding table, the overall world average fee that recruiting agencies charge per participant recruited is USD $107. As in the previous example, fees for the US East Coast are generally less than those on the West Coast and other US regions, but European and other non-US, non-European recruiting agencies generally charge lower recruiting fees.
In the following table, we present average fee requirements (in US dollars) for internal and external user categories.???
Table . Recruitment Fee in US Dollars per Participant
User Category
Fee (USD)
Internal – 16 citations
General Staff (example: “anybody” and “everybody”)
$84
Unspecified (no category given)
$73
Professional Staff (example: managers and executives)
$23
Technical Staff (example: system administrators)
$21
Support Staff (example: administrative assistants)
$10
External – 100 citations
Teachers / Professors
$250
Professional Staff (example: managers and executives)
$161
Technical Staff (example: system administrators)
$119
Medical Personnel
$117
Consumers
$90
Students / Young Adults
$84
Unspecified (no category given)
$51
Welfare Recipients
$42
As shown in the preceding table, professional categories of external participants generally warrant the highest recruiting agency fees. Be aware, however, that each recruiting agency sets its own fees and charges based on participant profile, number of user groups, and difficulty of finding people in the targeted groups.
Carefully screen a recruiting agency before engaging it.
To help determine whether a recruiting agency is suitable for your study needs, we recommend you ask them the following questions, which are based on Carolyn Snyder’s advice[19] and NN/g’s own experience using agencies:
Do you have a facility available? If so, will you recruit for projects that are not conducted at your facility? Many agencies will not recruit for you unless you use their facility too. If you need a facility, this arrangement could work very well—some facilities may offer a host to greet participants and hand out incentives at the end of sessions. Hosting service is especially useful if you have no receptionist available at your own study facility. If you decide to use the agency’s facility, be sure to find out what services like hosting, photocopying, video taping, and lab technicians will cost—are they included in the facility fee (often, upwards of US $1000 a day) or are they extra?
[If the agency does not have its own facility] Do you work with any facility, or can you recommend one? If you need a facility, chances are that a recruiting agency will be able to recommend a good one. If possible, visit the facility before engaging it, to ensure it meets your study needs.
What do you charge per recruited participant? Be prepared to explain the recruiting criteria you need for your study. Consider how experienced the agency is if they give you a quick answer without finding out more about what you need. If the agency does primarily consumer product testing, let them know your user profile may be more specialized than what they’re used to—for example, it is a lot more difficult to recruit bond traders or network administrators than it is to find people who use everyday household products. If you find yourself having to explain usability testing, refer to it as “in-depth, one-on-one interviews,” and save the detailed explanation for the agency you eventually engage.
Can we provide our own screener? Some agencies require that you provide your own screener. We think it’s best to do so, because your own carefully constructed screener will better ensure that the agency follows your desired criteria.
How much lead time do you require? The agency’s answer will depend on the difficulty of the user profile and how busy they are with other work. But, they should be able to tell you how much lead time they like to have for a typical project—usually two weeks.
What is the range and number of people in your database? If you have difficult criteria for the recruiter to work with, the agency may ask you to supplement the process with contacts of people that can possibly provide participants. Even though it is more work for you, don’t expect the agency to charge you any less, because it is still more work for them to recruit from outside their database.
Will you provide incentives to the participants? Many agencies are willing to provide participants with the incentives, especially if you are going to be using their facility. Be sure to find out how they handle payments to participants, and how much they charge for this service. Often, they ask for a deposit for the incentives after you engage them for recruiting. It’s also a good idea to review their receipt form, or better yet, provide your own. (See the sample receipt on page 85).
How do you handle no-shows? Although good recruiting firms have a lot of practice getting people to show up as scheduled, they cannot guarantee full attendance. Ask about their confirmation policies to make sure you are satisfied they will make a sufficient effort to get participants to attend. Let the agency know that you are unwilling to pay for participants who don’t appear and don’t call to reschedule, or who arrive unreasonably late. Discuss your tolerance for lateness, especially if your study schedule is tight—you may not have enough between-session time to accommodate latecomers. Some agencies double-book sessions to ensure you have participants, but you will have to pay for both, even though one will be excused (also see Scheduling Backups on page 75). In general, it’s best to let agencies do their job as they see fit, but it’s helpful to strategize together on a contingency plan, especially for high-visibility projects.
How do you handle participants who turn out to be unqualified for our study? Let the agency know that you are unwilling to pay for participants who appear for their sessions but turn out to be unqualified for the study (also see Dealing with Unqualified Participants on page 96.) The agencies that NN/g has dealt with have been willing to not charge for such participants.
How do you provide updates on reaching desired quotas? Look for an agency that is willing to provide you with a summary spreadsheet or text file of whom they have recruited so far and the key characteristics of each participant. Don’t expect to see participant contact information, however; they pledge confidentiality to their participants. (Also see Tip 111.) Getting an update every other day is not too much to ask. Many agency project managers also follow up with an email or phone call to tell you how it’s going.
Does your firm actually do the screening, or do you subcontract the work to another company? If the agency admits that they subcontract, you may wish to avoid them, unless someone whose opinion you trust has highly recommended the agency.
The following anecdote provides an additional caution about using recruiting agencies.???
Consider using a temporary employment agency.[20]
Most usability studies last one or two hours, which means that temporary employment agencies generally are not the best source of study participants. However, if you are running a study where you want to observe the same participants as they work with a system over a longer period of time (several days or even a few weeks), then you are moving into the area where temporary employment agencies are better than traditional recruiting agencies. People who are looking for temporary employment often have the time available to participate in a long-term study, and temporary employment agencies can help you find them.
It is possible to use temporary employment agencies for shorter studies. Usually you have to pay for four hours of service (about $50–$70 per person), even for a two-hour session. However, you still save money on the incentive and time to recruit. Just be sure to let the temporary employment agency know that your usability study is just a short, one-time commitment and you are not looking for temporary employees. (Also see Tip 102.)
Keep in mind that temporary employment agencies are often more successful in the summer months, when college students are readily available for short- and long-duration studies.
Don’t let a recruiting agency talk you into using more services than you really want or that your budget can afford.
Good agencies will tell you what they can and cannot do for you, and what they charge for what they do—then they will let you decide without pressure. If an agency tries too hard to push their “entire package,” be prepared to keep looking.
When you are not looking for temporary employees and generally do not want the same people they recruited previously, say so.
Unless you are specifically requesting return participants for long-term studies (see Reusing Participants on page 77), it’s a good idea to make clear that you are looking for one-time participants, not job applicants, as illustrated in the following anecdote from Singapore.???
Try explaining usability testing to the agency if you continue to have problems with getting job applicants rather than usability participants. If the situation doesn’t improve, you will need to find a different agency.
If you learn that an agency-recruited participant is unqualified only after the session begins, you should excuse the participant as you have planned in advance, and inform the agency.
Unless you need customers who are all from one particular company as participants, ask the agency to limit the number of recruits from any single company to one.
Even though you ask participants not to talk about the session—and they may even sign a nondisclosure agreeing not to discuss it—it is difficult for some people not to talk to their colleagues who may also be participants. This tendency increases if they’ve either really enjoyed participating, or found the system very difficult to use. They may think they’re helping you out by “training” subsequent participants from their company, as illustrated by the following anecdote from Switzerland.???
Marc’s anecdote also reiterates the advice from two other tips: the need to administer a background questionnaire (Tip 170) to learn of recent experience, and to caution participants not to discuss the session with co-workers or associates who might also be participants (Tip 179).
Give your own screener to the agency.
Because you would have to provide a participant profile, and then review the screener an agency would devise, you may as well take the next step and provide the screener yourself (see Preparing the Screening Script and Questionnaire on page 40). If the agency tells you that they must use their own screener with your participant profile edited in, you can still provide them with your screener to use as the basis for their own, as cautioned in the following anecdote.???
Make sure the recruiter thoroughly understands the criteria.
Even if you provide your own screener, ask to speak to the recruiter the agency assigns so you feel confident that s/he will be recruiting the right people. Go over the criteria to review any flexibility that you’ve built in, and reiterate which are “must-haves” and which can bend somewhat (see Tip 79 for advice on building in flexibility).???
Let the recruiter know s/he should feel free to call you any time there is a question on qualifying a participant, and be sure to provide a phone number at which you can easily be reached. If a few days have elapsed since your discussion, it’s a good idea to check in with the recruiter the day s/he is to begin making calls just to answer any last-minute questions.
Communicate the logistical details of the study.
Recruiting agencies are used to working with many different clients using many different methodologies—for example, telephone interviews, which do not require participants to leave their home or office.???
The anecdote above reiterates our advice to provide your own screener, which you can ensure provides logistical information for participants.
Review all recruiting progress updates to ensure the recruiting agency is meeting your criteria.
Speak with the recruiter right away if you feel they’ve scheduled someone who appears not to meet the criteria. Review the criteria again, and ask them to adhere to them more closely. If you find yourself having to reject numerous participants, cut your losses and look for an alternative recruiting agency.
Have the recruiting agency schedule sufficient backups.
NN/g recruiting survey respondents indicated that the average no-show rate for the recruiting agencies they hire is 10.8%, which is only slightly higher than the average world rate of 10.6%. Unfortunately, the observed no-show rate in any individual study will rarely be the exact average, but will fluctuate from 0% to 20% or more. In particularly unfortunate cases, we have encountered no-show rates of 50%, which can destroy a study that has only a small number of participants. We usually recommend having a recruiting agency recruit one or two backups per distinct user group. (Also see No-show rates by who recruits, on page 120, as well as Scheduling Backups on page 75).
Report in writing to the recruiting agency all latecomers, no-shows, and unqualified participants.
Send a return-receipt email to the recruiter to ensure that they know about latecomers, no-shows, and unqualified participants, and to ensure you have a record to check against the final invoice. (Not all email systems provide automatic return-receipt email—AOL for example does not confirm receipt. So, specifically ask for an acknowledgement in your email as well. If you do not get acknowledgement, send a certified or return-receipt letter to the agency.) As noted in Tip 99, find out in advance how the agency will handle latecomers, no-shows, and unqualified participants, and whether you will have to pay for them.
After the sessions are complete, provide feedback to the agency on the quality of their service.
If the agency performed less than satisfactorily, your feedback will help them improve. Assuming they did well, you will forge a good working relationship for the future. Let the agency know whether you will keep them in your files for future engagements. As suggested in Tip 89, freely recommend a good agency to other usability professionals. They will appreciate it.
Honor the agency’s database privacy (and thus, participants’).
Although you may ask participants recruited through an agency if they would be willing to participate again, you should never get or use agency-recruited participants’ contact information to recruit them yourself directly. Don’t even put them into your own database. Doing so would be an ethical and contractual violation because you are intruding on the recruiting agency’s proprietary database, which is their lifeblood.???
Asking agency-recruited participants for their contact information would also constitute a breach of the confidentiality that participants are guaranteed when they provide information to agencies. Instead, let the recruiting agency know which participants express interest in returning for future studies and remind the recruiting agency the next time you use them and want those participants back.
Agency-recruited participants may spontaneously volunteer the names of friends, family, or colleagues they think might be interested in participating in a similar study. This ethical area is more gray. Even if you rightly do not put the participant’s name in your database, consider carefully whether accepting an agency-recruited participant’s referrals could damage your relationship with that recruiting agency.
Our book, Usability Engineering,[21] advises: “Set up procedures that will allow you to recruit test users easily when they are needed. For example, cultivate relationships with major nearby customer sites and local colleges, set up contracts with temporary employment agencies, or build a database of interested volunteers (retired staff can often form a valuable source of volunteers with substantial domain knowledge). One of the major impediments to conducting user testing when it is needed is the time it may take to find appropriate users if one is not prepared.”
As noted earlier, the results of the NN/g recruiting survey indicate that a majority of companies are doing their own recruiting, as opposed to having a recruiting agency or some other outside person recruit for them. Doing the recruiting yourself can be a rewarding, if challenging, experience. By following the advice provided in this section, we think you will be better equipped to meet the challenge.
For internal studies, you need to recruit participants from within the study sponsors’ organization—their employees or closely partnered consultants. Of NN/g recruiting survey respondents, 83% indicated that they use internal people as pilot or dry-run participants and 67% said they use internal people as regular study participants.
For external studies, you need to recruit participants who are customers of the sponsors’ organization or their competitors’ customers (if you are running a competitive study). For external studies, you may decide to save some recruiting time and money by having internal people act as dry-run or pilot participants for your study. In our survey, 92% of respondents indicated they use external participants.
These tips apply to both kinds of studies:
If you are new to recruiting, find a recruiting mentor or consider investing in training.
Mentoring is one of the easiest ways to teach and being mentored is one of the fastest ways to learn. If your organization already has a recruiter on staff, try to arrange a period of training with that person. If you are going to be the first recruiter in your organization, rely on the experience of the usability team to guide you. Find out if other organizations in the area have recruiters on staff and would be willing to provide some training for a fee.
Budget enough time to find, screen, schedule, and coordinate with participants.
Even if you know exactly where to find representative participants, you still need to screen them to ensure they meet your study criteria, schedule them, and follow up with reminders. If you are new to recruiting, it would not be unreasonable to budget two or three hours per participant needed in the study. This allotment takes into account the time you spend screening people you do not accept into the study, communicating with the study facilitator, and following up with all accepted participants.
Responses to the NN/g recruiting survey indicate that organizations that do their own recruiting spend an average of 1.15 hour (or 69 minutes) per scheduled participant. 51% of respondents whose organizations do their own recruiting indicated that they spend more than one hour, and 2% more than six hours. (For a full summary of survey results, see Appendix A.)
Budget time to prepare all the participant forms you will need for the study.
In addition to screening and scheduling, keep in mind that you also will have to prepare all participant forms when you are doing the recruiting yourself. After you have them in place, however, it’s only a matter of editing them a bit for each study you do. (See Preparing Participant Session Forms on page 79.)
Ask study participants to suggest other study participants.[22]
Study participants often know other prospective study participants. Remember to ask for referrals. You might even suggest a deal in which everyone benefits. For example, “Usually we pay $75 to a study participant. If you get another qualified study participant for us, we’ll pay each of you $150.”???
Be very careful about asking agency-recruited participants for referrals, however, as cautioned in Tip 111, and consider when, and when not, to reuse participants, as advised in Reusing Participants on page 77.
Create recruiting brochures that your study participants could give to friends and colleagues.[23]
Your recruiting will be much easier if prospective study participants trust you because you have been recommended by someone they trust. But, see the cautions in Tips 111 and 115.
Whenever possible, avoid using participants from the same company, department, or family.
As noted in Tip 103, although you ask participants not to talk about the session—and they may even sign a nondisclosure agreeing not to talk about it—it is difficult for some people not to talk to their colleagues or even family members who also may be participants, especially if they’ve either really enjoyed participating or found the system very difficult to use. They may think they’re helping you out by “training” subsequent participants (also see the anecdote under Tip 103).
Ask for lists of people who have contacted customer service or the help desk.[24]
Often, such people are quite motivated to provide comments. We don’t recommend recruiting more than 40% of the study participants this way, however, because such people are often on the extremely knowledgeable and motivated end of the continuum. Watch out for highly dissatisfied customers who wish to use the study session as an outlet for their complaints—this is not the sort of motivation you want. One or two articulate, critical devil’s advocates can be quite useful in a study, however.
The following anecdote illustrates how a well-worded request by a public school system’s Web manager successfully motivated people to participate and, at the same time, provided a fruitful usability training exercise for budding student Web designers:???
Because internal participants are usually the employees or closely partnered contractors of the study sponsors, they are easier to find and are apt to be easier to recruit than external participants. Internal participants are also likely to better understand the direct impact usability evaluation will have on making their jobs easier—unless they’ve received promises of tool improvements in the past that didn’t come through. Unfortunately, in some organizations usability studies are commissioned but recommendations for improvements are not followed. If usability is known to be helping an organization, however, internal participants usually are willing, if not eager, to be involved in the usability evaluation process.
Do keep in mind that internal participants might worry about their performance being viewed by someone who might tell others (see Tip 185). In addition, although internal participants likely will not have to leave their place of work to participate, and therefore may not have to commute to the study, they may be more prone to work-related interruptions than external participants who leave their offices to participate in a study.
Talk to the product managers and developers.
The case study presented on page 17 summarizes an internal study. It describes how to work with product managers and developers to learn who the users of an internal system are. More than likely, the system team can provide you with names of users. Other times you may have to rely on additional resources to find the internal users you need. If so, ask a system team member to pave the way for you by letting you use his / her name when you talk with other people who may help you find internal participants.
Find out about internal special interest groups (SIGs) for your targeted users.
Larger organizations often have formal SIGs, especially for enterprise-wide systems with large numbers of users. Some organizations also have informal user groups. Your study sponsors should know of such groups. You may be able to get names of users from membership rosters, or you could ask the leader of the SIG to provide a recruiting notice to the group if s/he is protective of the membership roster. (See the sample script for preliminary contact by study sponsor on page 42.
You may have a better chance of borrowing an employee for a usability study session if you first talk to the employee’s manager. An employee also may be more willing to participant knowing his or her manager sanctions participation. Be sure to let the manager know that s/he will not be allowed to observe his / her employee’s session, however. (Also see Tip 185.)
Give a usability talk at employee / management meetings.
Make sure the talk is succinct but entertaining. At the end of your talk, ask for names of people who might be interested in participating in upcoming usability studies.
Talk to administrative assistants in the system team’s department.
Administrative assistants are often users and can be a source of a great deal of information about fellow employees who may meet your criteria.
Put up well-designed, succinct recruiting posters in strategic areas of the company.[25]
Secure permission to post such notices and make sure, as advised in other tips, that you pique people’s interest without giving away too much information about the study.
Meet with the human resources department.
They certainly know the names of current employees and their job titles. They also know about new hires that you may be able to recruit as new or potential users. Be aware, however, that new employees may feel sensitive about being watched and “tested.” Recruit them only if your study is evaluating systems designed for new hires. HR may also be able to provide the names of retirees who may fit an experienced-user profile. You might even ask HR to contact potential volunteers for you. Be prepared for HR security and privacy issues, though, and don’t persist if you encounter them.
For quick, informal feedback, meet people in the hallways.
If you need only short, specific feedback on something you can show to people quickly, set up a little station in a hallway or outside the cafeteria (with permission, of course) and let employees come to you with their curiosity.[26]
External participants may be somewhat more difficult to recruit than internal ones. External user groups might not be as well known to the study sponsors as internal groups. When you need to look for broad groups of people such as Palm technology users, online shoppers, stock market investors, and average consumers, your study sponsors might be able to point you in the general direction, but it will be up to you to find the targeted users.
As noted Tip 86, when the user profile is broad, you may have better luck recruiting by using a recruiting agency. If you have a recruiter on staff with time available, however, consider the tips in this section to help your recruiter find participants for your study.
Prepare a half-page description of usability and your current study that you can provide to people and organizations you approach.
The information sheet should closely match the introductory script of your screener. That is, without giving away specifics about your study (and, perhaps, the study sponsor), let people and organizations you approach know what usability is and why you are looking for people in their area. Emphasize how usability helps improve systems for users.
Contact the study sponsor’s sales department.[27]
Sales people often know many users of a system and may even be able to provide you with a list or database of names, phone numbers, and email addresses. Keep in mind that these customer lists usually are highly confidential and that you should offer assurance (or even sign a form) that you will use the list only for recruiting users for the study you will be conducting. Also, agree to destroy or give back the list as soon as the study is over. Be sure to follow through on your agreement with the sales department. Offer to let the sales department send your letter as an alternative.
Give a usability talk at a customer conference or trade show.[28]
At the end of your talk, ask for business cards from people who might be interested in supporting usability or design work. This method can make it easier to bypass marketing / sales groups that are nervous about letting usability specialists contact customers.
Whatever resource you are tapping, always ask for permission to talk to or contact potential participants.
In some of the next few tips, we suggest several places outside the study sponsor’s organization where you might find targeted participants. Wherever you go to recruit participants, however, it is very important not to interfere with establishments’ everyday business and to get the business management’s permission to speak with or contact their clientele.
Contact user and special interest groups (SIGs).
For example, if you are looking for participants with work-related injuries, contact SIGs for repetitive-stress injuries, hospital occupational-therapy departments, and organizations interested in accessibility compliance.
Specialized software often inspires the organization of SIGs or user groups. You may be able to get names of software users from membership rosters, or you could ask the leader of the SIG to send or post a recruiting notice to the group if s/he is protective of the membership roster. (Also see the sample script for preliminary contact by study sponsor on page 42).
Talk to the officers of professional organizations.
Discuss your project with an officer and then email your half-page information sheet with a request to distribute it to the organization if it is deemed an acceptable document. That way, the organization need not release its membership list, and only people who are actually interested will contact you.
Set up a booth at a strategic location.[29]
If you need “people off the street,” (users that do not have to meet a strict user profile) set up a booth at a flea market, shopping center, farmer’s market, or public transit hub. Depending on more-specific audience needs, you can set up your booth outside a campus computer lab, at an art fair, or a corporate customer visitor center. If you offer complimentary coffee and donuts, you will have an even better chance of getting people to stop and talk with you. It is sometimes even possible to conduct short studies in this very informal setting if you need only short, specific feedback on something you can show to people quickly.
Given the focus on increased security in public venues, however, be sure to discuss with the proper authorities where and when to set up a booth so that you won’t arouse anxiety.[30]
Approach civic and nonprofit organizations.[31]
Approach organizations that include the population you are seeking, such as churches, community service groups, and senior centers. Offer to donate money to the organization for each participant it recruits for you. Offering to make the donation in the name of the recruited participant would be a nice incentive for participants.
Look for users in their natural habitat.[32]
When evaluating a florist’s website, go to a local florist’s shop and talk with customers. If you are evaluating a library website, visit a branch of your local library.
For very specialized or unusual user profiles, look in specialized and unusual places to find participants. When looking for:
School children, contact school principals and teachers and get their help to recruit their students.
People with disabilities, contact hospital occupational and physical therapy departments, rehab centers, private physical therapists, accessibility compliance agencies, community support organizations, and nursing homes.
Senior citizens, contact senior centers, retirement homes, nursing facilities, community support organizations.
(Also see Tip 201 for additional resources on conducting usability studies with senior citizens, children, and people with disabilities.)
For other specialized groups, you can visit various places to look for participants, such as consulate offices, sports clubs, airline offices, chambers of commerce, and unemployment offices. You will think of several others as you further consider the characteristics of the people you are seeking as participants.
Place an ad in a professional journal or publication.
It is fairly inexpensive to place an ad in these publications, but because many of these types of publications are monthly, be aware that the deadline for ad submission may not be optimal for your study.
Consider additional resources for finding external participants.
Some additional resources to consider include:.
Newspaper ad. Place an ad in the business section of the newspaper local to your study locale. Keep in mind that an ad may cast too wide a net, however. The ad must be worded to give limited information, lest people research the system. (See the anecdote at the end of this tip.)
The ad should list your top two or three disqualifiers so you can eliminate people right away. We advise working with an expert copy writer to draft your ad. Note that you can pay up to USD $500 for a display ad you place in a newspaper, depending on the exact size of the ad and the length of time you run the ad.
Online services. Some usability professionals have had good luck with posting ads in Craig’s List jobs “et cetera” category (currently, no usability or market research categories are available). http://www.craigslist.org Other usability professionals have reported good results with Web Grrls and other online job-placement services. http://www.webgrrls.com
Expect to pay a fee of USD $70–$150 for an ad. Some sites let you post additional ads during the same period at a lower rate. However, people who read such online services tend to be among the more technologically literate parts of the population so they may not be suitable as the only participants for some types of studies.
Referrals. Ask your family, friends, and colleagues who may know of people in your targeted participant groups for referrals. For example, if you are conducting an online banking study, and your neighbor is a banker, ask him / her for suggestions on how to contact customers. Ask if it would be okay to use his / her name as a reference when contacting people. Just be careful when recruiting these referrals to avoid people who are too close to you, so they won’t skew the results. (Also see the anecdote below.)
You can also follow up with participants after a study to ask them for referrals to other people they know who might enjoy participating as much as they did. Be careful, however, when asking agency-recruited participants for referrals. (See Tip 111, as well as Reusing Participants on page 77.)
Phone book. We have successfully recruited for some studies using only the phone book. For example, for a study of a home version of a photo manipulation software product, we called several camera and printing shops listed in the Yellow Pages to find people who used digital cameras and were interested in such software.???
Reuse study participants from previous studies, when appropriate.
Keep a database of previous study participants and reuse them when appropriate. After each study, make a note of how expressive and articulate the study participants were and the depth of their comments, so you can target them for future studies. Use a participant no more than twice a year, however, and not on the same product or type of product.[33]
An exception to this advice would be for longitudinal studies where you want to track use over time. (See When it is Okay to Reuse Participants on page 77. Also see Building and Maintaining a Participant Database on page 98.)
After you have a pretty good sense of who targeted users are, and where you can reach them, you can get your materials organized, dry run your screener (and revise, if necessary), and then begin calling people.
Make a list of the people you will be calling; include phone numbers and other important tracking information.
Use this list to track who you have called and whether you need to call a candidate again. If you have a reference name, include it on the list so you will be sure to mention it when you call the prospective participant:
Name
Phone Number
Referred by
Screened
Call Back [date and time]
Prepare file folders labeled “Qualified,” “Maybe,” “Future Possibilities,” and “Not Qualified.”
File filled-in screeners in the appropriate folders and provide them to the project manager and / or study facilitator for review when requested. For participants you may have disqualified for a current study, decide if they might be qualified for a future study. If so, file their screeners in “Future Possibilities” and enter their information into your participant spreadsheet or database for future use.
Create a table on which to summarize participants and their characteristics.
This table should reflect the screening criteria. Fill in each accepted participant’s information and provide the most recent update to the project manager and / or study facilitator at the end of each day of recruiting. (See the following sample (filled-in) participant summary table, which summarizes the participants for the usability study described in the sample screener on page 47.)???
Table . Sample Filled-In Participant Summary Table
Participants Scheduled for the Aztech.com Usability Test Wednesday, July 8, and Thursday, July 9, 2005
Phil Pilot
Robert
Sam
Betty
David
Linda
Jane Backup
Greg Backup
Ellen Backup
Gender
Male
Male
Male
Female
Male
Female
Female
Male
Female
Job title
Store Manager
Banker
Underwriter
Fitness Instructor
Dentist
Executive Secretary
Programmer
Head Librarian
Financial Analyst
Age
31–40
41–50
31–40
20–30
41–50
31–40
41–50
20–30
31–40
Income (in USD $1000)
$46–$60
$61–$75
$46–$60
$46–$60
$61–$75
$35–$45
$46–$60
$35–$45
$61–$75
Renter or Homeowner
Renter
Owner
Owner
Renter
Owner
Renter
Owner
Renter
Owner
Computer use (hours per week)
10
20
40
15
5
30
60
25
20
Web use (hours per week / total)
5 4 years
7 2 years
10 1.5 years
10 3 years
5 .5 year
5 1 year
10 5 years
12 4 years
10 2.5 years
Online purchases
Books Tickets Music
Music Tickets Art Travel
Tools Music Tickets
Equipment Books Vitamins Travel
Drugs Dental supplies Tickets
Clothes Books DVDs
Books Garden tools Pet food
Office supplies DVDs Music
Books Tickets Music Travel
Session
Wed. 9 am
Wed. 1 pm
Wed. 3 pm
Wed. 5 pm
Thurs. 9 am
Thurs. 11 am
Thurs. 1 pm
Thurs. 3 pm
Thurs. 5 pm
First, practice reading your script aloud.
Don’t have a prospective participant on the line the very first time you read your script aloud. First, read it aloud a few times to yourself, to see where you may stumble over phrasing. Edit or practice it until it flows smoothly for you. Be sure, however, that your revisions don’t change the intent of the content or add clues as to the right answers.
Test the script and screener questions with a couple of co-workers or friends.
If possible, pick people who come close to the profile you are screening for. If that’s not possible, your screening questionnaire will still benefit from your asking objective third parties the questions aloud to see if they make sense.
Aspire to make a good first impression when speaking with candidates.
Aim to sound professional, but cordial.
Use adequate volume without forcing your voice and speak reasonably slowly.
Repeat questions for hearing impaired individuals, without impatience.
Excuse unqualified people graciously and ask if they’d be interested in a future study if they seem to have basic experience.
Respect people’s current schedule constraints, if they say this is not a good time to answer questions, and arrange to call back at a time more convenient for them.
When scheduling, be sure to let the participant know of all available time slots, so s/he can pick the one most convenient for him / her. Try to get a second choice as well, in case the project manager or study facilitator must adjust the schedule later.
When possible and appropriate, let backup participants know they are designated as backups and what they can expect. (Also see Tip 158.)
Before ending the phone call with an accepted participant, reconfirm the date and time you both agreed on for that person’s session, and verify contact information so you can send directions by mail.
Acquire the “radar” needed to recognize undesirable candidates.
This skill comes with practice. As you gain more experience, you will be able to recognize job seekers, technology spies or competitors, and others who provide exaggerated or “iffy” information to get into a study. The more experienced the recruiter, the less likely s/he will schedule unacceptable people.
Listen for responses, non-responses, and implications that will help you weed out unacceptable candidates. As soon as you decide a candidate is undesirable, excuse them politely and quickly, and file their screener in the “Not Qualified” folder.
Many of the following cautions are adapted from author Deborah Sova’s paper Challenges in Recruiting for Usability Testing:[34]
Beware of:
Job seekers, especially if you posted a recruitment ad to the local paper or an online job service. Job seekers may ask you:
How many other people will be there (vying for a job).
How to dress for the session (wanting their attire to be appropriate for a job interview).
Extended questions about the system, the study sponsor, and where the study is being held (cross-checking their experience against what the sponsor may be looking for in an employee).
If they should bring their resume (an obvious clue).
How many days or months the study lasts (considering usability studies as a possible short-term job).
Technology spies or competitors, who may:
Indicate too much knowledge about the system being tested, especially when the system is a prototype.
Ask extended questions about the system, possibly to compare how their own system measures up.
Try to find out how many participants will be there so they can bring enough business cards or samples—not all responses are this obvious, however.
We are pleased to report that the vast majority (98%) of our survey respondents do some kind of follow-up with participants before the study begins to ensure participants show up for sessions, and 28% perform more than one follow-up activity.
92% provide driving instructions and / or a map to the sessions
81% make a telephone call
71% send an email
25% send a written letter
Send a confirmation email, fax, or letter a couple of days before the study to confirm session dates and times.
Include a map, and driving directions, if appropriate (also see Tip 51). Then, follow up with the phone call the day before to reconfirm and make sure participants know how to get to the study locale (see the anecdote under Tip 147). The following sample paraphrases the content of the invitation.
Hello [participant name]:
Thank you very much for agreeing to participate in the [study name] [if as a backup, say “as a backup participant” and include wording about what is expected of a backup]. As I explained when we spoke on [day], your evaluation session will be held at [facility name], [address], in [city] on [date and time]. Please plan to arrive about 10 minutes before your scheduled session time to fill in some participant forms.
Because we are scheduling very few people for this study, it is extremely important that you keep your appointment with us. If for any reason you cannot make it, or you find you will be unavoidably late, please contact [name of person who can resolve last-minute changes], at [phone number] as soon as you know. Please do not send anyone else in your place.
If you wear glasses while using the computer, please bring them with you to your session. You will use [generic description of system—remember you probably don’t want the participant to practice in advance] to do a few tasks. Then the evaluation staff will ask you a few follow-up questions to clarify their observations to ensure they thoroughly understand your actions with the [system] and obtain any additional feedback you feel is important.
At the completion of your session, you will receive [incentive] as our thanks for your participation. [unless you will be sending the incentive later. See paragraph below.]
[If you are videotaping session]. I’d like to remind you that, with your permission, your session will be videotaped, and that these tapes will be used by the project team within the company for analysis purposes only; all personal information, including your identity, will remain company confidential.
[If you will administer a nondisclosure]. In addition, [we, or study sponsor] need you to sign a nondisclosure agreement that says you agree not to talk to anyone about the [system] you are evaluating. You may tell them that you participated in a usability study, however.
[If you will provide a meal or refreshments]. We will be providing food and beverages. The food will consist of [describe, for example, small sandwiches and cookies].
[If the incentive will not be provided at the session]. We will provide [incentive, such as the next release of software] approximately four weeks after your session. We will need the address to which you would like us to send the [incentive].
[Below, or attached] is a map and driving directions to [facility]. If you have any questions, please feel free to call me at [phone number].
Regards,
[Name]
——End of Sample Confirmation Message——
Reconfirm by phone the day before.
Also verify that participants know how to get to the study locale and resend a map and directions, if necessary.???
Decide in advance who will recruit and schedule the dry-run participant.
Sometimes the study facilitator will enlist a co-worker on the spur of the moment to dry run the study protocol, and the recruiter need not be involved. But when the study is complex and the facilitator wants to dry run the protocol several days in advance, the recruiter likely will need to recruit and schedule a person, who can be internal, for a dry run of the session materials. (Also see Tip 15.)
Provide regular updates to the project manager and study facilitator.
Provide copies of the most recent update of the participant summary table (see example on page 72) at the end of each day of recruiting. Also, when requested, provide review copies of filled-in screeners (see screener example on page 47) in their appropriate folders (see Tip 140).
Discuss with the study sponsor and project team any difficulty you experience getting the desired number of participants.
If necessary, the team can discuss ways to relax criteria without compromising the study. (Also see Tip 79.)
Let the project manager and study facilitator decide which participants will be designated as pilots and backups.
If all participants are equally qualified, this choice will be a matter of scheduling people in the time slots that are most convenient for them. If you are having difficulty getting enough fully qualified people, however, you can generally use less-qualified people as the pilot and backup users. Let the project manager or study facilitator make the final decision. If this process affects the schedule, you may have to call participants to adjust their appointment times.
Plan ahead for inevitable cancellations and no-shows.
Increase the likelihood of your getting a final pool of representative participants by scheduling backup users. The average no-show rate reported by respondents of the NN/g Recruiting Survey is 10.6% (a full summary of survey responses is provided in Appendix A). When your budget allows, plan to over-recruit by your own typical no-show rate, to help fill all participant sessions. If you don’t know your typical no-show rate, we suggest you use the averages provided by the NN/g recruiting survey respondents, or schedule one backup for each user group represented. (Also see No-Show Rates By Who Recruits on page 120.)
You can schedule backups in a few different ways:
On-call participants agree to be available to participate at any time on a certain day or half-day and will come to the lab when called.
Floater participants agree to come to the lab at a time that overlaps two other sessions so that they can fill in for either session should it become necessary. You must pay both, though, if they both show up.
Double booking participants involves scheduling two participants for the same time slot, and releasing one if both show up. Pay both for showing up.
Include extra time slots in which you can schedule all your backups. Make them the last appointments in your schedule, and cancel them with as much notice as possible if they are not needed.
Backups also can cover for sessions in which you find it necessary to excuse a participant during a session for one reason or another.
Ask people to be “on call” only when you know they have flexibility and that they can reach the study locale quickly, when needed.
The on-call arrangement tends to work better for internal participants, who are more likely to be in the same building or office complex as the study team.
Designate “floaters” only when your budget allows and people are willing.
Because floaters come to the lab and stay for a designated period of time overlapping two sessions, you should plan to pay a higher incentive for their greater time spent. If possible, provide distractions such as television or magazines and refreshments for floaters.
Plan to “double book” only when you need the guarantee of 100% attendance under a tight time constraint.
Because some participants may be disappointed to make the effort to come to a session, only to be excused as soon as they arrive, we advise avoiding double booking. Even though you pay the excused person, s/he may still feel dissatisfied if s/he was eager to participate, as illustrated in the following anecdote.???
Consider preparing an in-depth user questionnaire for the second person to fill out, so they can still make a contribution to the study. If you have the resources and personnel, also consider reserving a place where you can run a partial session with the second person. If you can do neither of these things, we think it’s better to let people know they may not be called to participate. Avoid handing people a built-in excuse not to show up, by letting them know they will still receive the incentive if they appear for their appointment.
Whenever possible, schedule backups as your last participants of the overall schedule.
We think this is the most humane way to treat backups, and it wastes the least amount of time. Tell scheduled backup users that you might cancel their session, with notice, and that they will still receive an incentive (you might be able to negotiate a lower rate in the event of adequate notification).
Whenever possible, use fully qualified participants as backups.
You want the person who steps in for a no-show to be representative of the system users so that you get the best possible data. Still, if you have had to relax your screening criteria for a particularly difficult recruit, it is obviously better to designate the more marginal participants to serve as the backups.
Let participants know that they are backups, what they can expect, and that, as backups, they are important to the study.
Avoid giving backups participants the impression that they are less important than other participants. Let them know that you regularly schedule a full complement of qualified participants to cover for no-shows and that they will be performing a valuable service as backups.
Plan to compensate backups whether or not they participate.
Plan to compensate backup participants even if you do not call them to substitute for a no-show; they’ve made room in their schedule for the study and have fulfilled their commitment. For backups you cancel, even with notice, expect to provide at least half the regular incentive.
Reusing participants has advantages, but it also poses problems when you use the same base group repeatedly. This section provides advice for when, and when not, to reuse study participants.
You can reuse participants for studies that do not focus on ease of learning.
If you want to find out how well a system performs during extended use, or how much iterative designs improve a system over time, you can reuse the same participants. It’s a good idea to evaluate with novice users to make sure the system meets their needs, but when your study does not need to take into account how easy it is to learn to use the system, then reusing participants is okay.
You can reuse participants when the entire user base is from one company, or a specific department of one company, but don’t use any one person more than twice in a year.
If a system is designed for a narrowly targeted group, such as insurance agents, and no one else will use the system, you will probably have to reuse participants to evaluate new versions of the system. You will not be able to recruit agents from competing insurance firms. If the pool of agents is large enough, you may be able to use each person no more than twice in one year.
You can reuse participants when you need to schedule people very quickly, and you know they fit your recruiting criteria.
If a study participant is good at thinking aloud and sparkling with catchy quotes, you can reuse that study participant, but, again, don’t use a study participant more than twice a year and not on the same system or type of system.[35]
One advantage of reusing participants is that you can skip some of the preliminaries and dive right into the study, which can save you a little time., Don’t skip taking care of participant needs in the process, however. (See Honoring the “Participant Bill of Rights” on page 88.)
Avoid reusing participants for iterative tests of the same system if you want to study ease of learning or first exposure to a design.
People cannot erase their experience. A person who has participated in a previous study of an earlier version of a certain design will remember some things from that session and will not approach the new version with a completely fresh perspective.
Avoid reusing participants who have already participated in any usability study twice in one year.
You want to prevent developing a pool of “professional participants.” After a few sessions, participants can become accustomed to the types of problems they think they are expected to find, and you lose the fresh perspective you need, as illustrated by the following anecdote from Switzerland.???
Avoid reusing participants whom you have had to excuse from a previous study because they demonstrated questionable motives.
The best way to avoid this situation is to mark suspect or undesirable participants in your spreadsheet or database: those who misrepresented their qualifications, were interested only in the incentives, or were job-seekers or technology spies who beat your screener.
Avoid reusing participants whom you know from previous studies have not honored the nondisclosure agreement.
Again, the best way to avoid this situation is to mark these participants in your spreadsheet or database after you learn they disregarded their pledge of nondisclosure.
[16] “When to Outsource the Recruiting of Test Users,” sidebar for Jakob Nielsen’s Alertbox for May 3, 1998: Cost of User Testing a Website. http://www.useit.com/alertbox/outsource_recruiting.html
[17] Carolyn Snyder, Finding a User Recruitment Vendor (2001), Snyder Consulting white paper http://www.snyderconsulting.net/booksarticles.htm
[18] The report from this study is available at http://www.nngroup.com/reports/ir
[19] Carolyn Snyder, Finding a User Recruitment Vendor, Snyder Consulting white paper, 2001. http://www.snyderconsulting.net/booksarticles.htm
[20] 230 Tips and Tricks for a Better Usability Test, a Nielsen Norman Group report, http://www.nngroup.com/reports/tips/usertest
[21] Jakob Nielsen, Usability Engineering (1993), Academic Press.
[22] 230 Tips and Tricks for a Better Usability Test, a Nielsen Norman Group report, http://www.nngroup.com/reports/tips/usertest/
[23] 230 Tips and Tricks for a Better Usability Test, a Nielsen Norman Group report, http://www.nngroup.com/reports/tips/usertest/
[24] Ibid.
[25] Chauncey Wilson, Director, Bentley College Design and Usability Testing Center, provided this tip.
[26] 230 Tips and Tricks for a Better Usability Test, a Nielsen Norman Group report, http://www.nngroup.com/reports/tips/usertest/
[27] Ibid.
[28] 230 Tips and Tricks for a Better Usability Test, a Nielsen Norman Group report, http://www.nngroup.com/reports/tips/usertest/
[29] Ibid.
[30] Chauncey Wilson, Director, Bentley College Design and Usability Testing Center, provided the security caveat we added to this tip.
[31] 230 Tips and Tricks for a Better Usability Test, a Nielsen Norman Group report, http://www.nngroup.com/reports/tips/usertest
[32] 230 Tips and Tricks for a Better Usability Test, a Nielsen Norman Group report, http://www.nngroup.com/reports/tips/usertest
[33] 230 Tips and Tricks for a Better Usability Test, a Nielsen Norman Group report, http://www.nngroup.com/reports/tips/usertest
[34] Deborah Hinderer (Sova), Challenges in Participant Recruiting for Usability Tests (1998), Tec-Ed, Inc. and IPCC.
[35] 230 Tips and Tricks for a Better Usability Test, a Nielsen Norman Group report, http://www.nngroup.com/reports/tips/usertest