Planning Your Recruiting Needs

14

The Cardinal Rule for Recruiting

14

1.

Know thy users.

14

Developing Recruiting Criteria

14

Learning About the Users

15

2.

Budget some time to meet early on with stakeholders.

15

3.

Make a list in advance of the topics and questions that you want to cover with stakeholders.

15

4.

Understand the study goals.

15

5.

Interview the system team.

15

6.

Get a demonstration of the system.

15

7.

Talk to current users.

16

8.

Find out about the legacy or manual system.

16

9.

Talk to the marketing group.

16

10.

Find out about competitive or similar systems.

16

11.

Ask as many questions as you need to.

16

Deciding How Many Participants to Recruit

19

12.

For your study results to produce statistical significance, plan to evaluate with 10 to 12 participants per condition.

19

13.

For a less formal usability study, plan to evaluate with 4 to 5 participants per distinctly different user group.

19

14.

Consider other factors that might affect the number of participants you choose to recruit.

19

15.

Don’t forget to count the dry run, pilot, and backups in the number of participants you will need to recruit.

20

Drafting the Screening Criteria

20

16.

Ensure that recruiting criteria reflect specific user characteristics.

20

17.

Use the recruiting criteria as the basis for your participant screener.

21

18.

If you are evaluating a website, plan to screen participants for prior experience using the Web.

21

19.

If you are evaluating an application that is used by people new to a company, recruit new hires as participants.

22

20.

If you are evaluating an intranet, plan to recruit as participants mainly people who have been employees for a few months.

22

Determining the Appropriate Incentives

22

21.

Research potential incentives for participants as you would user characteristics.

22

22.

Be as generous as your budget will allow.

23

23.

Do not let an incentive take the place of verbally thanking a user for participating.

23

24.

Be sure the incentive is commensurate with the time involvement of the participants.

23

Monetary Incentives

23

25.

Offer monetary incentives to participants who otherwise would be paid for their time.

24

26.

Avoid offering monetary incentives to internal participants who already are receiving pay for their time to participate.

24

27.

Find out in advance whether certain participants are prohibited from receiving money.

24

28.

Consider mitigating factors when deciding how much of a monetary incentive to offer.

24

29.

Handle cash incentives prudently.

26

30.

Consider the drawbacks, as well as the benefits, of dispensing checks.

27

Non-monetary Incentives

27

31.

Offer a non-monetary incentive to participants who otherwise would not be paid for their time.

28

32.

Offer a non-monetary incentive for internal participants who already are being paid for their time.

28

33.

Find out in advance whether certain participants are prohibited from receiving any incentive.

28

34.

Consider rewarding the technologically curious with technology-oriented incentives.

29

35.

Be imaginative about other non-monetary incentives you could offer.

29

36.

Always offer food and beverages.

29

37.

When your study involves school children, check with their teacher and principal to find out what an appropriate incentive would be.

29

How Incentives May Affect Participant Behavior

31

38.

Ensure that the incentive amount is appropriate for the users’ job category and the time involved.

31

39.

Be aware that incentives can motivate people to exaggerate their qualifications when answering screening questions.

32

How and When to Provide Incentives

32

40.

Assign the job of distributing incentives to the team member for whom it makes the most sense to do so and have the same person do that for the whole study.

32

41.

Enclose monetary incentives, whether check or cash, in an envelope and place the envelope directly into the participant’s hand.

32

42.

Consider whether to provide the incentive at the beginning or the end of the session.

33

43.

Plan to give incentives to participants who don’t finish their sessions.

33

44.

If an incentive will not be available by the time the study session takes place, provide an incentive voucher.

33

Considering the Study Locale

34

When Participants Come to You

34

45.

Choose a location that is easy to get to.

34

46.

Plan to offer a premium for travel time and expenses.

34

47.

Designate someone to greet arriving participants.

34

48.

Find out in advance about venue security.

34

49.

If you are using a client’s lab or other testing facility (not your own lab) tape a sign, prominently bearing your company name, to the external door.

35

50.

If you are using your own facility, provide a direct-line phone at the external door with a sign saying that it is for study participants to use.

35

51.

Provide a decent map to the study locale.

35

52.

Understand that climate and surroundings also can affect attendance.

36

53.

Take traffic into consideration for metropolitan venues.

36

When You Must Go to the Participants

36

54.

Plan to hold sessions with children at their school.

37

55.

Plan to hold sessions with senior citizens or people with disabilities where it’s convenient for them.

37

56.

When planning to visit participants’ offices, be sure to let the management staff know that you will be visiting and when.

37

57.

Plan times that are convenient for participants, but not disruptive to the organization.

37

58.

Find out about security and after-hours regulations.

38

59.

Communicate to the study team the importance of respecting participants’ hospitality, surroundings, and privacy.

38

60.

Let participants and their management know what to expect from your site visit.

38

61.

For international studies, carefully research logistical and cultural issues for visiting participant sites outside your own country.

38

Planning for Training and Orientation

39

62.

Decide early on whether and how much orientation or training participants need in order to provide effective feedback.

39

63.

Get someone to provide training, preferably someone involved with or supportive of the study.

40

64.

Arrange for participants to get the training in a timely fashion.

40

65.

Be sure that the study sponsor is involved in planning the training.

40

66.

Expect to provide additional incentives when participants must commit time to orientation and training.

40

Preparing the Screening Script and Questionnaire

40

Cover Page and Opening Script

41

67.

Prepare a cover page for your screener that includes quick reference information about the respondent.

41

68.

Keep the opening script succinct and polite.

41

69.

Whenever possible, provide the name of someone known to the respondent as a lead-in.

41

70.

Avoid saying too much about what you are evaluating so participants can’t prepare in advance—especially when you will be evaluating with new users.

41

71.

Avoid words that make a usability study sound like a scientific experiment or some type of psychological study.

42

72.

Have someone on the study sponsor’s team make the first contact by email or phone.

42

Screening Questions

43

73.

Put quick disqualifier questions first in your screening questionnaire.

43

74.

Screen out employees of competing companies.

43

75.

Screen out the study sponsors’ employees and their family members for studies evaluating systems designed for external users.

43

76.

Use polite, socially acceptable phrasing when screening for possibly sensitive information.

43

77.

Avoid “giving away” the profile you are targeting.

44

78.

Ensure your screening questions accurately reflect the intent of your criteria.

45

79.

Decide which criteria can be relaxed and which cannot be.

46

Invitation and Schedule

47

80.

Include script for inviting the respondent to participate and provide some information on what to expect from the session.

47

81.

Include a master schedule on which the recruiter can record participant appointments.

47

82.

Include your screener as an appendix in your usability study report.

47

Screening and Scheduling Participants

53

83.

Whether you recruit yourself or use an outside source, don’t cut corners.

53

Working With an Outside Recruiting Agency

53

When to Outsource Recruiting

54

84.

Avoid using a recruiting agency if you are conducting a usability study of an intranet.

54

85.

Avoid using a recruiting agency if you are conducting a usability study of a site that aims at a narrow group of external users.

54

86.

Use a recruiting agency if you are conducting a usability study of a site that aims at the general population or at a relatively broad group.

54

87.

Use a recruiting agency if your study is tightly scheduled, and you simply do not have, or cannot afford, the resources to do the recruiting.

54

Finding a Recruiting Agency

54

88.

Check with your marketing department for existing contracts with recruiting agencies.

54

89.

Ask usability colleagues which recruiting agencies they’ve used successfully.

54

90.

Visit http://www.stcsig.org/usability/topics/recruiting-firms.html.

55

91.

Post an inquiry on a professional usability mailing list.

55

92.

In the US, look up “Market Research” in the Yellow Pages of your telephone book. In other countries, try equivalent terms in the local language.

55

93.

To find recruiting agencies outside your local area, check online Yellow Pages.

56

94.

Outside the US, check the STC resource that lists recruiting agencies and labs outside the US. http://www.stcsig.org/usability/topics/recruiting-firms.html.

56

95.

Use Web search engines.

56

96.

Create a spreadsheet or text file to record your recruiting agency research and share it with your colleagues.

56

What to Expect from a Recruiting Agency

56

97.

Allow adequate lead time—at least two weeks.

56

98.

To be safe, budget recruiting fees within 10% of what you expect to pay in incentives.

57

Choosing a Recruiting Agency

58

99.

Carefully screen a recruiting agency before engaging it.

58

100.

Consider using a temporary employment agency.

60

Managing a Recruiting Agency to Your Best Benefit

60

101.

Don’t let a recruiting agency talk you into using more services than you really want or that your budget can afford.

60

102.

When you are not looking for temporary employees and generally do not want the same people they recruited previously, say so.

60

103.

Unless you need customers who are all from one particular company as participants, ask the agency to limit the number of recruits from any single company to one.

61

104.

Give your own screener to the agency.

61

105.

Make sure the recruiter thoroughly understands the criteria.

61

106.

Communicate the logistical details of the study.

62

107.

Review all recruiting progress updates to ensure the recruiting agency is meeting your criteria.

62

108.

Have the recruiting agency schedule sufficient backups.

62

109.

Report in writing to the recruiting agency all latecomers, no-shows, and unqualified participants.

63

110.

After the sessions are complete, provide feedback to the agency on the quality of their service.

63

111.

Honor the agency’s database privacy (and thus, participants’).

63

Doing Your Own Recruiting

63

Internal vs. External Studies

64

112.

If you are new to recruiting, find a recruiting mentor or consider investing in training.

64

113.

Budget enough time to find, screen, schedule, and coordinate with participants.

64

114.

Budget time to prepare all the participant forms you will need for the study.

64

115.

Ask study participants to suggest other study participants.

64

116.

Create recruiting brochures that your study participants could give to friends and colleagues.

65

117.

Whenever possible, avoid using participants from the same company, department, or family.

65

118.

Ask for lists of people who have contacted customer service or the help desk.

65

Finding Participants for Internal Studies

66

119.

Talk to the product managers and developers.

66

120.

Find out about internal special interest groups (SIGs) for your targeted users.

66

121.

Talk to management.

66

122.

Give a usability talk at employee / management meetings.

67

123.

Talk to administrative assistants in the system team’s department.

67

124.

Put up well-designed, succinct recruiting posters in strategic areas of the company.

67

125.

Meet with the human resources department.

67

126.

For quick, informal feedback, meet people in the hallways.

67

Finding Participants for External Studies

67

127.

Prepare a half-page description of usability and your current study that you can provide to people and organizations you approach.

67

128.

Contact the study sponsor’s sales department.

68

129.

Give a usability talk at a customer conference or trade show.

68

130.

Whatever resource you are tapping, always ask for permission to talk to or contact potential participants.

68

131.

Contact user and special interest groups (SIGs).

68

132.

Talk to the officers of professional organizations.

68

133.

Set up a booth at a strategic location.

68

134.

Approach civic and nonprofit organizations.

69

135.

Look for users in their natural habitat.

69

136.

Place an ad in a professional journal or publication.

69

137.

Consider additional resources for finding external participants.

69

138.

Reuse study participants from previous studies, when appropriate.

71

Making the Calls and Tracking Recruitment Progress

71

139.

Make a list of the people you will be calling; include phone numbers and other important tracking information.

71

140.

Prepare file folders labeled “Qualified,” “Maybe,” “Future Possibilities,” and “Not Qualified.”.

71

141.

Create a table on which to summarize participants and their characteristics.

71

142.

First, practice reading your script aloud.

72

143.

Test the script and screener questions with a couple of co-workers or friends.

72

144.

Aspire to make a good first impression when speaking with candidates.

72

145.

Acquire the “radar” needed to recognize undesirable candidates.

73

146.

Send a confirmation email, fax, or letter a couple of days before the study to confirm session dates and times.

74

147.

Reconfirm by phone the day before.

75

148.

Decide in advance who will recruit and schedule the dry-run participant.

75

149.

Provide regular updates to the project manager and study facilitator.

75

150.

Discuss with the study sponsor and project team any difficulty you experience getting the desired number of participants.

75

151.

Let the project manager and study facilitator decide which participants will be designated as pilots and backups.

75

152.

Plan ahead for inevitable cancellations and no-shows.

75

153.

Ask people to be “on call” only when you know they have flexibility and that they can reach the study locale quickly, when needed.

76

154.

Designate “floaters” only when your budget allows and people are willing.

76

155.

Plan to “double book” only when you need the guarantee of 100% attendance under a tight time constraint.

76

156.

Whenever possible, schedule backups as your last participants of the overall schedule.

77

157.

Whenever possible, use fully qualified participants as backups.

77

158.

Let participants know that they are backups, what they can expect, and that, as backups, they are important to the study.

77

159.

Plan to compensate backups whether or not they participate.

77

Reusing Participants

77

When It Is Okay to Reuse Participants

77

160.

You can reuse participants for studies that do not focus on ease of learning.

77

161.

You can reuse participants when the entire user base is from one company, or a specific department of one company, but don’t use any one person more than twice in a year.

77

162.

You can reuse participants when you need to schedule people very quickly, and you know they fit your recruiting criteria.

78

When to Avoid Reusing Participants

78

163.

Avoid reusing participants for iterative tests of the same system if you want to study ease of learning or first exposure to a design.

78

164.

Avoid reusing participants who have already participated in any usability study twice in one year.

78

165.

Avoid reusing participants whom you have had to excuse from a previous study because they demonstrated questionable motives.

78

166.

Avoid reusing participants whom you know from previous studies have not honored the nondisclosure agreement.

78

Preparing Participant Session Forms

79

167.

Print multi-page forms back-to-back and number the pages using a “1 of x” format.

79

168.

Create all participant forms in plain, understandable language, not “legalese.”

79

169.

If a participant refuses to sign a form that you or your study sponsor require, excuse that participant from the study.

79

170.

Administer a background questionnaire to the participant just before the session begins.

80

171.

Avoid mailing the background questionnaire to participants in advance.

80

172.

Base the background questionnaire on the essential questions from the screening questionnaire.

80

173.

For studies with minors, administer a consent form that requires the signature of the minor’s parent or guardian.

81

174.

Have participants (or parents or guardians of minors) sign recording consent forms before sessions you plan to video- or audio-tape.

84

175.

Have participants sign receipts for incentives they receive.

85

176.

Give participants an incentive voucher if their incentive will not be available by the time study session takes place.

85

177.

Avoid nondisclosure agreements whenever possible.

86

178.

If corporate policy requires that participants sign a nondisclosure agreement, ensure that participants sign one.

86

179.

Even if you do not administer a nondisclosure form, ask participants not to discuss the session.

87

180.

In the US, check a current-year tax guide to find out how much any one person may be paid in a calendar year before you must produce and mail a 1099 form.

87

181.

Avoid using any one participant so often that you must report his or her incentive earnings to the government.

87

182.

Consult with a tax expert and the CFO in the study sponsor’s company to learn about any additional tax information you should collect or know about when providing incentives to participants.

87

Honoring the “Participant Bill of Rights”

88

Treating Participants as Human Beings

88

183.

Remember that participants are people, not “test subjects.”

88

184.

Chat with participants to help them feel comfortable with the study session.

88

185.

Don’t allow a participant’s manager or co-workers to observe sessions.

89

186.

Be sensitive to cultural diversity.

89

187.

Be sensitive to individuals who are mentally or physically challenged, without feeling awkward.

89

188.

Make the first task very simple.

89

189.

Consider gracefully ending a session if you sense that a participant is uncomfortable and hesitates to ask to be excused.

90

190.

Express gratitude, and follow up with a thank-you note.

90

191.

In your study results, be sure to quote participants and report their data accurately and anonymously.

90

192.

Consider using memory aids so that you can easily recall each participant.

91

Attending to Participants’ Physical Comfort

91

193.

Provide an adequate number of breaks in the session—with food and beverages available.

91

194.

Be sure participants know where to find the closest lavatories (toilettes).

91

195.

Provide comfortable, ergonomic chairs for participants.

91

196.

Position the monitor, mouse, and keyboard a comfortable distance from the participant.

91

197.

Make sure the room environment is physically comfortable.

92

198.

Help make the room visually appealing and less lab-like.

92

199.

Provide coloring books and reading material to keep child participants occupied while they wait for their parents or guardians to retrieve them.

92

200.

Respect participants’ personal space.

92

201.

Research additional resources when planning studies with seniors, children, or people with disabilities as participants.

92

Ensuring Participants’ Safety

93

202.

Make sure you know who is dropping off and retrieving a child participant to and from your session.

93

203.

When sessions take place in the evening, provide an appropriate escort for participants who would feel unsafe, or even uneasy, without it.

93

204.

When sessions take place in the evening, make sure you have more than one team member.

93

205.

Remove obstacles in the study room that may hamper or endanger the mobility of participants, especially people with physical challenges.

93

Ensuring Participants’ Privacy

94

206.

Provide repeated assurance that you will report participant data anonymously.

94

207.

Don’t ask agency-recruited participants for their contact information.

94

208.

Don’t put non-agency participants’ names into your recruiting database or spreadsheet without their knowledge or consent.

94

209.

Observe strict rules for handling videotapes.

94

210.

Define rules for data retention.

95

211.

Consult with your organization’s legal department to learn about the legal ramifications of usability testing and evaluating with participants.

95

Dealing with Unqualified Participants

96

212.

Try to discover unqualified participants before the study session begins, with a background questionnaire.

96

213.

Try to determine the cause of a qualification discrepancy, and act accordingly.

96

214.

If during the session, you discover a job seeker, competitor, technology spy, or incentive-only seeker, deal quickly and firmly with the situation.

96

215.

If, during the session, you become uncomfortable with a participant’s behavior, or it is clear that the participant is ill with something like the flu or a cold, excuse that participant.

97

216.

If unqualified or abusive participants were agency-recruited, let the agency know.

97

217.

Review and edit your screener to prevent participant-qualification problems in the future.

97

Future Planning

98

Building and Maintaining a Participant Database

98

218.

Keep the data simple.

98

219.

Use readily available spreadsheet or database software.

98

220.

Whenever possible, get referrals and add them to your participant file.

98

221.

Include participants you may have disqualified for a current study, especially if they might be qualified for a future study.

98

222.

Audit your participant database yearly.

99

223.

Put into place a security policy for the database.

99

Building and Managing a Recruiting Staff

99

Assessing Recruiting Skills

99

224.

Choose as your recruiter or recruiter-trainee a person who is multi-talented.

99

225.

Avoid candidates who are more interested in a usability engineering job.

100

Interviewing Potential Recruiters

100

226.

Schedule a phone interview with a potential recruiter, even if s/he is someone you already know.

100

227.

Provide a generic screener to a potential recruiter and have him/her screen you and other colleagues.

100

228.

Show a potential recruiter your current recruiting and participant forms and ask him/her to suggest improvements.

100

229.

Ask an experienced recruiter about participant follow-up, scheduling backups, and handling no-shows.

100

230.

Ask all potential recruiters to demonstrate or discuss the skills listed in Tip 224.

100

Managing Recruiters

101

231.

Expect to pay recruiters a salary commensurate with a skilled administrative assistant in your organization.

101

232.

Consider other tasks your recruiters can do.

101

233.

Help recruiters feel involved in usability.

101