Nielsen Norman group conducted a worldwide email survey to gather specific information about the usability community’s collective recruiting experience. We received 201 responses, indicating great interest in the recruiting aspect of usability studies. We thank those who participated and provided valuable information.
We created a plain-text survey that we could send in the body of an email message, which people could fill in as a reply to the email. Jakob Nielsen posted a notice about the survey on the Useit.com website and sent it to the Alertbox mailing list, inviting people who conduct user studies to request a copy of the survey: We also posted a similar notice on a widely accessed usability mailing list.
The survey introduction explained the research NN/g was conducting and the incentives respondents would receive for responding to the survey, as follows:
===============================================================
Nielsen Norman Group (NN/g) is preparing a practical report on recruiting participants for usability studies and we would like to ask you to take part in our preliminary research. Please take a few minutes to fill in the survey below, which asks about the participant recruiting process for your usability studies. You needn’t be the person who actually does the recruiting to provide valuable information.
To thank you for your time. NN/g will:
Provide all respondents with a summary of the survey results.
Offer all respondents a 50% discount on the published NN/g participant recruiting report.
Randomly select 10 respondents to receive a complimentary copy of any NN/g usability report, up to a value of $250.
If you do not wish to receive any of the items listed above, please indicate your preference at the end of the survey. Simply reply to this email and type your responses directly into the survey in your reply.
NN/g will not use respondents’ email addresses or other identifying information in reporting survey results or in the published NN/g participant recruiting report; all respondents will remain anonymous.
Thank you.
===============================================================
The survey contained 10 recruiting questions—some with multiple parts. We collected information on:
The geographical areas from which respondents recruit participants
Who recruits participants: the study facilitator, a staff recruiter, an outside recruiting agency, or some other person
Internal time spent recruiting
Agency lead time required and agency fees
User profiles, incentives offered, and session length
No-show rates
Pre-session interaction with participants
Participant documents
Post-session follow up with participants
Recruiting anecdotes (from respondents who agreed to be contacted; anecdotes are provided within the body of this report, not in the survey results).
NN/g also provided assurance of privacy with the following wording at the end of the survey:
“NN/g will use respondents’ email addresses only to send a summary of the survey results, to contact the winners of our drawing, to inform respondents of the procedure for getting the 50% discount, or to contact respondents who answered “Yes” to providing NN/g with a recruiting anecdote. Afterwards, NN/g will delete all respondents’ email addresses. NN/g will not sell or disclose respondents’ email addresses to anyone else.
“NN/g also will not use respondents’ email addresses or other identifying information in reporting survey results, or in the published NN/g participant recruiting report; all respondents will remain anonymous.”
We received 350 email messages requesting a copy of the survey.
The survey was completed and returned by 201 people, for a response rate of 57%
Of the 201 responses, 63 (or 31%) were received as the result of our having sent a reminder to the requestor to fill in and return his or her survey.
The 201 responses came from at least 25 countries in the following geographical areas:
54% US – 108
22% Europe – 44 (UK–16, Germany–6, Denmark–4, Sweden–3, Switzerland–3, France–2, Italy–2, Netherlands–2, Spain–2, Austria–1, Belgium–1, Finland–1, Romania–1)
21% Rest of the World – 43 (Canada–15, Australia–11, India–3, New Zealand–3; South Africa–3, Mexico–2, Singapore–2, China–1, Brazil–1, Ecuador–1, Israel–1)
3% Unspecified – 6.
We divided the 108 US responses into three major regions as follows:
30% Northeast (Maine to Washington DC) – 32
32% West Coast (California, Oregon, and Washington) – 35
38% All other US areas – 41.???
The 201 survey respondents specified 296 locations, which included city, region (state or province), and country. We designated five regions—areas that contained 17% or more of the total regional citations. Under the regions below, we list the three most-often-cited cities within each area.
USA Northeast (Maine to Washington DC) – 17% of locations specified (51 of 296).
New York City – 14
Boston – 11
Washington DC – 6
USA West Coast (California, Oregon, Washington) – 17% of locations specified (51 of 296)
San Francisco – 14
San Jose – 7
Seattle – 7
USA, all other states – 20% of locations specified (59 of 296)
Chicago – 9
Dallas – 4
Denver – 4
Europe – 22% of locations specified (65 of 296)
London – 10
Zurich – 3
Paris – 2
Rest of the world – 24% of locations specified (70 of 296)
Toronto – 8
Sydney – 7
Melbourne – 5
The three most-often-cited cities in the world are New York and San Francisco, with 14 citations each, and London, with 10 citations.
In the following table, responses are sorted by larger region (shaded, centered horizontal bar), country (in alphabetical order), region (in alphabetical order), and city (in alphabetical order).???
USA Northeast (Maine to Washington DC): 17% | |||
---|---|---|---|
Country | State, Province, or Region | City | Number of Citations |
USA | Connecticut | Hartford | 1 |
District of Columbia | Washington | 6 | |
Massachusetts | Boston | 11 | |
Maine | Unspecified | 1 | |
Maryland | Baltimore | 1 | |
New Hampshire | Manchester | 1 | |
New Hampshire | Unspecified | 2 | |
New Jersey | Newark | 1 | |
New Jersey | Whippany | 1 | |
New York | New Plats | 1 | |
New York | New York | 14 | |
New York | Potsdam | 1 | |
New York | Rochester | 1 | |
New York | Unspecified | 1 | |
Pennsylvania | King of Prussia | 1 | |
Pennsylvania | Philadelphia | 3 | |
Virginia | Arlington | 1 | |
Virginia | Danville | 1 | |
Virginia | Fairfax | 1 | |
Virginia | Falls Church | 1 | |
17% – USA Northeast TOTAL: 51 |
USA West Coast (California, Oregon, and Washington): 17% | |||
---|---|---|---|
Country | State, Province, or Region | City | Number of Citations |
USA | California | Bay Area | 1 |
California | Brisbane | 1 | |
California | Los Angeles | 2 | |
California | Palo Alto | 2 | |
California | Pleasanton | 2 | |
California | San Diego | 3 | |
California | San Francisco | 14 | |
California | San Jose | 7 | |
California | Santa Barbara | 1 | |
California | Santa Cruz | 1 | |
California | Silicon Valley | 1 | |
California | Sunnyvale | 1 | |
California | Unspecified | 1 | |
Oregon | Portland | 5 | |
Washington | Olympia | 1 | |
Washington | Seattle | 7 | |
Washington | Unspecified | 1 | |
17% – USA West Coast TOTAL: 51 |
USA – All Other States: 20% | |||
---|---|---|---|
Country | State, Province, or Region | City | Number of Citations |
USA | Arizona | Phoenix | 1 |
Arizona | Unspecified | 1 | |
Colorado | Denver | 4 | |
Florida | Boca Raton | 1 | |
Florida | Fort Lauderdale | 1 | |
Florida | Jacksonville | 1 | |
Georgia | Atlanta | 2 | |
Georgia | Rowell | 1 | |
Illinois | Bloomington | 1 | |
Illinois | Chicago | 9 | |
Illinois | Unspecified | 1 | |
Kansas | Kansas City | 1 | |
Kentucky | Bowling Green | 1 | |
Michigan | Ann Arbor | 1 | |
Michigan | Detroit | 1 | |
Michigan | Lansing | 2 | |
Minnesota | Minneapolis / St. Paul | 3 | |
Missouri | Joplin area | 1 | |
Missouri | Kansas City | 2 | |
Missouri | St. Louis | 2 | |
North | Carolina Chapel Hill | 1 | |
Ohio | Cleveland | 1 | |
Ohio | Columbus | 1 | |
Ohio | Dayton | 1 | |
Ohio | Findlay | 1 | |
Texas | Austin | 3 | |
Texas | Dallas | 4 | |
Texas | Houston | 1 | |
Texas | Unspecified | 1 | |
Utah | Logan | 1 | |
Wisconsin | Madison | 3 | |
19% – USA Non-Northeast and Non-West Coast TOTAL: 56 | |||
Unspecified | Metropolitan Areas | 1 | |
Unspecified | Unspecified | 2 | |
1% – USA Unspecified Total: 3 | |||
54% – USA Total: 161 |
Europe: 22% | |||
---|---|---|---|
Country | State, Province, or Region | City | Number of Citations |
Austria | Unspecified | Vienna | 1 |
Unspecified | Unspecified | 1 | |
Austria Total: 2 | |||
Belgium | Unspecified | Brussels | 1 |
Belgium Total: 1 | |||
Denmark | Eastern Jutland | Unspecified | 1 |
Unspecified | Copenhagen | 3 | |
Denmark Total: 4 | |||
Finland | Unspecified | Helsinki | 2 |
Unspecified | Tampere | 1 | |
Finland Total: 3 | |||
France | Ile de France | Paris | 2 |
Other than Ile de France | Small cities | 1 | |
Unspecified | Unspecified | 2 | |
France Total: 5 | |||
Germany | Baden–Württemberg | Karlsruhe | 1 |
Baden–Württemberg | Tübingen | 1 | |
Bavaria | Munich | 1 | |
Berlin–Brandenburg | Potsdam | 1 | |
Hamburg | Hamburg | 1 | |
Hessen | Frankfurt | 1 | |
Unspecified | Berlin | 2 | |
Unspecified | Unspecified | 1 | |
Germany Total: 9 | |||
Italy | Lazio | Rome | 1 |
Lombardia | Milan | 1 | |
Sicily | Palermo | 1 | |
Italy Total: 3 | |||
Netherlands | Noord–Holland | Amsterdam | 1 |
Unspecified | The Hague | 1 | |
Netherlands Total: 2 | |||
Romania | Unspecified | Unspecified | 1 |
Romania Total: 1 | |||
Spain | Unspecified | Madrid | 1 |
Unspecified | Barcelona | 1 | |
Unspecified | Valencia | 1 | |
Spain Total: 3 | |||
Sweden | Unspecified | Stockholm | 2 |
Unspecified | Unspecified | 2 | |
Sweden Total: 4 | |||
Switzerland | Unspecified | Bern | 2 |
Unspecified | Zurich | 3 | |
Switzerland Total: 5 | |||
United Kingdom | England | Bristol | 1 |
England | Liverpool | 2 | |
London | London | 10 | |
England | Wirral | 1 | |
Hampshire | Southampton | 1 | |
Norfolk | Norwich | 1 | |
Sussex | Brighton | 1 | |
Tyne | Sunderland | 1 | |
Unspecified | Milton Keynes | 1 | |
Unspecified | Unspecified | 2 | |
United Kingdom Total: 21 | |||
Unspecified | Europe | Unspecified Europe: 2 | |
22% – Europe TOTAL: 65 |
Rest of the World: 24% | |||
---|---|---|---|
Country | State, Province, or Region | City | Number of Citations |
Australia | Queensland | Brisbane | 1 |
Australian Capital Territory | Canberra | 3 | |
Hunter | New Castle | 1 | |
New South Wales | Lismore | 1 | |
New South Wales | Sydney | 7 | |
Victoria | Geelong | 1 | |
Victoria | Melbourne | 5 | |
Unspecified | Unspecified | 1 | |
Australia Total: 20 | |||
Brazil | Unspecified | Sao Paulo | 1 |
Brazil Total: 1 | |||
Canada | British Columbia | Vancouver | 2 |
British Columbia | Victoria | 2 | |
British Columbia | Unspecified | 1 | |
Ontario | Ottawa | 2 | |
Ontario | Toronto | 8 | |
Ontario | Waterloo | 1 | |
Prince Edward Island | Charlottetown | 1 | |
Quebec | Montreal | 1 | |
Saskatchewan | Regina City | 1 | |
Unspecified | Unspecified | 1 | |
Canada Total: 20 | |||
China | Unspecified | Hong Kong | 1 |
Unspecified | Shanghai | 1 | |
China Total: 2 | |||
Ecuador | Guayas | Guayaquil | 1 |
Pichincha | Quito | 1 | |
Ecuador Total: 2 | |||
India | Delhi | New Delhi | 1 |
Maharashtra | Mumbai | 1 | |
Tamilnadu | Chennai | 2 | |
India Total: 4 | |||
Israel | Unspecified | Tel Aviv | 2 |
Israel Total: 2 | |||
Mexico | Unspecified | Mexico City | 2 |
Mexico Total: 2 | |||
New Zealand | Unspecified | Auckland | 2 |
Unspecified | Wellington | 2 | |
Unspecified | Hamilton | 1 | |
New Zealand Total: 5 | |||
Singapore | Unspecified | Unspecified | 2 |
Singapore Total: 2 | |||
South Africa | Eastern Cape | Unspecified | 1 |
Gauteng | Johannesburg | 1 | |
Gauteng | Pretoria | 1 | |
Unspecified | Cape Town | 1 | |
South Africa Total: 4 | |||
Unspecified: 6 | |||
24% – Rest of the World TOTAL: 70 |
Regional Citations GRAND TOTAL: 296 |
79% of respondents indicated that the person who runs the study recruits participants for an average of 70% of the studies, and 28% of respondents indicated that the person who runs the study recruits 100% of the studies.
36% of respondents indicated that an outside recruiting agency recruits participants for an average of 61% of the studies, and 9% indicated that an agency recruits 100% of the studies.
32% respondents indicted that someone else within their company recruits participants for an average of 32% of the studies. “Clerical staff” was cited most often (six times), and other support staff, such as customer service or receptionists, were cited five times.
13% of respondents indicated that some other person recruits participants for an average of 32% of the studies. “The client” was cited most often (16 times).
12% of respondents indicated that a dedicated recruiter within their company recruits participants for an average of 67% of the studies, and 6% indicated that dedicated recruiters recruit 80%–100% of the studies.
Percentages total more than 100% because we allowed multiple responses. In fact, it seems that most companies combine two different approaches to recruiting, depending on the project.
The following five tables provide the breakdown of percentages for each recruiter type cited by respondents. For example, in the Recruiter 1 table directly below, 57 respondents indicated that the person who runs the study recruits 100% of the studies in their organizations.???
As shown in the preceding table, 79% of respondents (158 of 201) indicated that the person who runs the study recruits participants for an average of 70% of the studies, and 28% of respondents (57 of 201) indicated that the person who runs the study recruits for 100% of the studies.???
As shown in the preceding table, 36% of respondents (73 of 201) indicated that an outside recruiting agency recruits participants for an average of 61% of the studies, and 9% (19 of 201) indicated that an agency recruits for 100% the studies.???
Table . Recruiter 3: Someone else within my company
Recruiter Cited | Number of Citations | Percentage of Studies Cited |
---|---|---|
Customer Service Project Manager Development Manager Consulting Manager | 1 1 1 1 | 100% |
Project Manager or Development Manager | 1 | 95% |
Test Center Board Room Concierge | 1 1 | 90% |
Not specified | 1 | 80% |
Associate Producer Project Manager | 1 1 | 75% |
Not specified | 1 | 65% |
Clerical Staff | 2 | 50% |
Communications Sales Rep Not specified | 1 1 1 | 40% |
Not specified Marketing Staff Product Manager Customer Service | 7 1 1 1 | 30% |
Clerical Staff Not specified | 3 2 | 25% |
Not specified Account Rep Other UE Specialist Website Manager Web Developer Clerical Staff | 7 1 1 1 1 1 | 20% |
Other UE Specialist | 1 | 15% |
Not specified Other UE Specialist Visual Designer Business Manager Website Manager Project Manager Client Services Instructional Designer | 5 3 1 1 1 1 1 1 | 10% |
Not specified | 4 | 5% |
Receptionist Not specified | 1 1 | 1% |
Total Citations: 64 | Average: 32% |
As shown in the preceding table, 32% of respondents (64 of 201) indicted that someone else within their company recruits participants for an average of 32% of the studies. “Clerical staff” (highlighted in blue) was cited most often (six times), and other support staff, such as “Customer Service” or “Receptionist,” were cited five times (also highlighted in blue).???
Table . Recruiter 4: Some other person
Recruiter Cited | Number of Citations | Percentage of studies cited |
---|---|---|
Research Office | 1 | 100% |
Client | 1 | 80% |
Client | 1 | 70% |
Client Not specified | 2 2 | 50% |
Client | 1 | 45% |
Client | 1 | 40% |
Client | 1 | 35% |
Client Usability Firm | 2 1 | 30% |
Team Manager | 1 | 25% |
Account Manager Data Logger Client Not specified | 1 1 1 1 | 20% |
Client Observer Person Playing Computer | 6 1 1 | 10% |
Student | 1 | 5% |
Total Citations: 27 | Average: 32% |
As shown in the preceding table, 13% of respondents (27 of 201) indicated that some other person recruits participants for an average of 32% of the studies. “Client” was cited most often (16 times).???
As shown in the preceding table, 12% of respondents (24 of 201) indicated that a dedicated recruiter within their company recruits participants for an average of 67% of the studies, and 6% (13 of 201) indicated dedicated recruiters recruit 80%–100% of the studies.??????
As shown in the preceding table, 62% of respondents (125 of 201) indicted that internal people who spend time recruiting participants spend an average of 1.15 hour (or 69 minutes) per participant recruited. 59% of these 125 respondents spend .5 to 1.5 hours per participant recruited, and 24% spend 2 to 6.5 hours per participant recruited.??????
As shown in the preceding table, 33% of respondents (67 of 201) indicated that agencies require average of 1.93, or nearly two weeks’ lead time.
As noted in the preceding table, 33% of respondents (67 of 201) indicated they use recruiting agencies for 8% – 40% of their studies. In the following table, we present average fee requirements (listed in US dollars) charged by agencies around the world for internal and external participants, as indicated by 116 agency citations generated by the 67 positive responses in our survey, and listed by region.???
Table . Agency fees by region
Region | Recruitment Fee per Participant | Number of Internal Citations | Number of External Citations |
---|---|---|---|
USA – West Coast (California, Oregon, and Washington) | $125 | 3 | 21 |
USA – Other | $124 | 2 | 21 |
USA – Northeast (Maine to Washington DC) | $113 | 2 | 24 |
Europe | $108 | 4 | 18 |
Unspecified | $75 | 0 | 2 |
Rest of the World | $57 | 5 | 14 |
Average: $107 | Total: 16 | Total: 100 |
As shown in the preceding table, the overall world average fee that agencies charge per recruited participant is $107. The US Northeast is generally lower than the West Coast and other US regions, and European and other non-US, non-European agencies generally charge lower recruiting fees than their US counterparts.
In the following table, we present average fee requirements (listed in US dollars) for internal and external user categories.???
Table . Agency fees by user category
User Category | Recruitment Fee per Participant |
---|---|
Internal – 16 citations | |
General Staff (“anybody” and “everybody,” for example) | $84 |
Unspecified (no category given) | $73 |
Professional Staff (managers and executives, for example) | $23 |
Technical Staff (system administrators, for example) | $21 |
Support Staff (admin assistants, for example) | $10 |
External – 100 citations | |
Teachers / Professors | $250 |
Professional Staff (managers and executives, for example) | $161 |
Technical Staff (system administrators, for example) | $119 |
Medical Personnel | $117 |
Consumers (buys readily available goods and services) | $90 |
Students / Young Adults | $84 |
Unspecified (no category given) | $51 |
Welfare Recipient | $42 |
Average World Fee: | $107 |
As shown in the preceding table, professional categories of external participants generally warrant the highest agency fees???
The following descriptions of the user groups and incentives respondents cited will provide context for the results for Question 5.
We organized all internal user groups that respondents cited using as dry run, pilot, or regular study participants into four separate categories—general, professional, support / clerical, and technical—as described below. Many respondents provided data about usability studies they conduct with internal users, but did not specify an internal user group.
General (nonspecific): anyone, colleague, employee, everyone, friend, general user, new hire, non-developer, non-project personnel, staff, student, subject-matter expert, volunteer.
Professional: academic, administrator, business analyst, consultant, faculty, journalist, manager, market or other researcher, study stakeholder, travel agent.
Support / clerical: administrative assistant, call center staff, clerical staff, customer service staff, human resources staff, librarian, secretary, operational staff.
Technical: developer, documentation specialist, electrical engineer, HCI student, help desk staff, instructional designer, network engineer, network administrator, product manager, scientist, software engineer, system administrator, trainer, UI designer, usability staff.
Unspecified: Respondents did not specify a user group.
We organized external user groups that respondents cited into six separate categories—high-end professional, medium professional, low-end professional, technical, consumer, and unpaid / nonprofessional—as described below. A few respondents provided data about usability studies they conduct with external users, but did not specify a user group.
High-end professional: architect, attorney / lawyer, CEO, consultant, executive, physician, surgeon, professional investor, real-estate agents / brokers.
Medium professional: academic, administrator, banker, business analyst, consultant, exporter, faculty, financial advisor / planner, graphic designer, insurance agent / broker, manager, market or other researcher, marketing / salesperson, nurse, journalist, professor, public-health official, teacher, travel agent.
Low-end professional: administrative assistant, artist (performing, visual), call center staff, clerical staff, construction worker, customer-service staff, farmer, government worker, human-resources staff, librarian, secretary, office manager, operational staff, truck driver.
Technical: developer, documentation specialist, electrical engineer, help-desk staff, instructional designer, network engineer, network administrator, product manager, scientist, software engineer, system administrator, trainer.
Consumer (person who purchases goods or services available to the general public): cross-section of general public, electronics (including cell phone, computer, PDAs, internet), big-ticket items (furniture, autos, large appliances), services (banking, financial, repair, utilities, telephone, travel and leisure).
Special needs or nonprofessional: children, homemaker, medical patient, person with disabilities, retiree, senior citizen, student, public-assistance recipient, volunteer.
Unspecified: Respondents did not specify a user group.
We organized incentives into the broad categories of monetary and non-monetary. Non-monetary incentives fell into four separate subcategories—certificates for merchandise, food and beverages, “intangible rewards,” and tangible gifts—as described below. Some respondents indicted that they provide a non-monetary incentive, but did not specify what it was.
Monetary incentives: we present all monetary incentive information in US dollars and, for easier comparison, monetary incentives reflect a dollars-per-hour rate, not dollars-per-session.
Non-monetary incentives: for purposes of presenting data, we have placed all cited, non-monetary incentives into the following categories:
Certificates: redeemable certificates and coupons in denominations ranging from $5 – $50 for book shops, department stores, restaurants, corporate cafeterias, local merchants and coffee shops.
Food and beverages: bottle of champagne, liquor, or wine; cake, coffee or tea, dinner or lunch, prepackaged snacks, refreshments or treats.
Intangible rewards: airline frequent-flyer miles, charging time to project (not overhead), job performance, extra credit (for students), internet access, members-only website privileges, merit certificates, opportunity to provide input for improvement, study results, thanks, time off, training, and travel expenses for study covered.
Tangible gifts: merchandise ranging in value from around USD $5 – $100, including coloring books (for children), compact discs, flowers, logo merchandise, movie passes, office supplies, software, theatre tickets, and toys and “trinkets” (for both children and adults).
Unspecified: respondents did not specify a non-monetary incentive.
83% of respondents (166 of 201) indicated they use the following internal groups (as described above) for dry-run and / or pilot sessions:
27% Unspecified (45 of 166)
25% General (41 of 166)
24% Technical (40 of 166)
16% Professional (26 of 166)
8% Support (14 of 166).
57 minutes is the average length of time for internal dry-run / pilot sessions.
67% of respondents (134 of 201) indicated they use the following internal groups as actual participants in usability studies:
28% Unspecified (37 of 134)
23% General (31 of 134)
22% Professional (29 of 134)
16% Technical (22 of 134)
11% Support (15 of 134)
65 minutes is the average length of time for internal participant sessions.
92% of respondents (184 of 201) indicated they use external participants. Some cited up to three user groups each for a total of 260 citations comprising the following user groups as participants in usability studies:
37% Consumer (96 of 260)
16% Special Need / Nonprofessional (42 of 260)
15% Medium Professional (39 of 260)
13% Technical (34 of 260)
11% High-End Professional (29 of 260)
5% Low-End Professional (13 of 260)
3% Unspecified (7 of 260).
72 minutes, or about 1.25 hours, is the average length of time for external participant sessions.
Respondents indicated the monetary and non-monetary incentives that they provide to participants. From that, we extrapolated the number of participants who receive no incentive at all. As shown in the following table, 67% of internal pilot / dry-run participants, 55% of actual internal participants, and 9% of external participants receive no incentive, for an overall average of 37%—a little more than a third of the sample.???
Table . Types of incentives by user group
User Group | Monetary Incentives | Non-monetary Incentives | No Incentive | Total Citations |
---|---|---|---|---|
Internal Pilots / Dry Runs | 11 | 43 | 112 | 166 |
Internal Actual Participants | 13 | 47 | 74 | 134 |
External Participants (35 offered both monetary and non-monetary and are counted in both columns) | 165 | 106 | 24 | 295 (Overlap) – 35 |
Total | 189 | 196 | 210 | 560 |
As shown in the preceding table, survey respondents indicated that not everyone provides monetary incentives to participants, especially not to internal participants (generally, employees or closely related contractors of the study sponsors).
Of the 166 respondents who use internal people as dry-run or pilot participants, only 7% provide a monetary incentive.
Of the 134 respondents who use internal people as regular study participants, only 10% provide a monetary incentive.
The worldwide average rate paid to internal participants (pilot, dry-run, and actual) is USD $32 per session hour.
184 respondents who use external participants recruit multiple user groups—for a total of 260 citations. Of the total participants, 63% received monetary incentives, in marked contrast to internal participants.
The worldwide average rate paid to external participants is USD $64 per session hour—twice that of the internal rate.
The following table provides survey responses for monetary incentives provided, sorted by user group.???
Table . Monetary incentives in US dollars by user group
User Group | Average USD Paid Per Session Hour | Total Citations |
---|---|---|
Internal Pilots / Dry runs | $34 | 11 |
General | $44 | 2 |
Professional | $0 | 0 |
Support | $38 | 2 |
Technical | $5 | 2 |
Unspecified | $46 | 5 |
Internal Participants | $31 | 13 |
General | $33 | 2 |
Professional | $37 | 3 |
Support | $0 | 0 |
Technical | $11 | 3 |
Unspecified | $37 | 5 |
External Participants | $64 | 165 |
Consumer | $54 | 68 |
Higher Professional | $118 | 23 |
Medium Professional | $69 | 22 |
Lower Professional | $50 | 6 |
Special Needs or Nonprofessional | $32 | 20 |
Technical | $67 | 22 |
Unspecified | $55 | 4 |
Total | 189 |
As shown in the preceding table, of internal staff who receive monetary incentives, technical staff receive the lowest incentive rate. (Technical people often are motivated to participate for the opportunity to be involved in technology in a new and different way.) As pilot and dry-run participants, professional staff receive no monetary incentive—perhaps because they may have some stake in the study. Interestingly, support staff are paid as pilot and dry-run participants but not as actual participants.
The above table also indicates that external monetary rates are double that for internal participants, with high-end professionals garnering the highest rate.
The following table summarizes the monetary incentives offered to participants by geographical region and user group.???
Table . Monetary incentives in US dollars by region
Region | Average USD Paid Per Session Hour to Internal Pilots / Dry runs | Average USD Paid Per Session Hour to Internal Ppts | Average USD $ Per Hour Paid to External Ppts | Total Citations |
---|---|---|---|---|
USA $73 | 126 | |||
Northeast $70 | $33 | $13 | $77 | |
West Coast $79 | $55 | • | $81 | |
Rest of USA $69 | $17 | $7 | $78 | |
Europe $34 | $23 | $58 | $34 | 36 |
Rest of the World $27 | • | $18 | $29 | 27 |
Total | 189 |
As shown in the preceding table, monetary payment trends for internal participants by region are unpredictable. Monetary payments are highest on the US West Coast ($79 per session-hour on average), and lowest in non-European, non-US countries ($27 per session-hour).
Why are incentives twice as high in the United States as compared with Europe? One likely explanation is that Americans tend to feel over-surveyed and require more of an incentive to participate in studies. In contrast, Europeans may not yet be as tired of market research and may show more appreciation for being asked to influence the future direction of technology. It is understandable that the West Coast in the US is more expensive than the rest of the country (because the cost of living tends to be higher on the coasts), but why isn’t the Northeast more expensive as well? It might be because of a similar phenomenon on the West Coast. With its higher number of computer and high-tech companies, the West Coast’s population may simply feel more over-surveyed than people on the East Coast.
The 201 respondents generated 196 non-monetary incentive citations within a total of 560 distinct user-group citations (166 internal dry-run / pilot citations, 134 internal participant citations, and 260 external participant citations).
We provide non-monetary citations by user group and by geographical location.???
Table . Non-monetary incentives by user group
User Group | Gift | Redeemable Certificate | Food | Intangible | Total Citations |
---|---|---|---|---|---|
Internal Pilots / Dry Runs | 18 | 6 | 12 | 7 | 43 |
General | 8 | 3 | 1 | 2 | 14 |
Professional | 1 | 1 | 3 | 2 | 7 |
Support | 0 | 1 | 3 | 0 | 4 |
Technical | 3 | 0 | 4 | 1 | 8 |
Not specified | 6 | 1 | 1 | 2 | 10 |
Internal Participants | 19 | 13 | 10 | 5 | 47 |
General | 7 | 4 | 2 | 2 | 15 |
Professional | 4 | 2 | 2 | 1 | 9 |
Support | 2 | 2 | 3 | 0 | 7 |
Technical | 0 | 3 | 3 | 2 | 8 |
Not specified | 6 | 2 | 0 | 0 | 8 |
External Participants | 58 | 24 | 13 | 11 | 106 |
Consumer | 22 | 11 | 4 | 2 | 39 |
Higher Professional | 3 | 2 | 1 | 1 | 7 |
Medium Professional | 10 | 3 | 1 | 4 | 18 |
Lower Professional | 2 | 1 | 1 | 0 | 4 |
Special Needs / Nonprofessional | 13 | 6 | 3 | 1 | 23 |
Technical | 8 | 1 | 1 | 3 | 13 |
Not specified | 0 | 0 | 2 | 0 | 2 |
Total | 95 | 43 | 35 | 23 | 196 |
As shown in the preceding table, gifts and gift certificates are the preferred incentives, being offered two times as often as other non-monetary incentives for both internal and external groups.
The following table summarizes the non-monetary incentives offered to participants by geographical region.??????
As shown in the following table, the overall average no-show rate for our survey sample is 10.6%.???
Note that respondents did not provide ranges; rather we combined cited ranges so this table would not be inordinately long.
We tested whether survey data would confirm the anecdotal data we received from usability professionals that the no-show rate is higher for professional recruiting agencies than for internal recruiters. The results from our survey tend to back up this claim, although the differences in no-show rates are not statistically significant. Three internal recruiter categories do slightly better, while a fourth—the person who runs the study (presumably a usability professional)—does slightly worse than the world average rate of 10.6%, as described below. The following bullet points summarize the no-show rates from best to worst, for internal and agency recruiters:
Someone else within the company: 10.1%
32% of respondents indicated someone else from within their company, usually support staff, recruits for an average of 32% of the studies.
Some other person: 10.4%
13% of respondents indicated that some other person, usually “the client,” recruits for an average of 32% of the studies.
Dedicated recruiter with the company: 10.5%
12% of respondents indicated that a recruiter within their company recruits for an average of 67% of the studies.
Recruiting agencies: 10.8%
36% of respondents indicated that an outside recruiting agency recruits for an average of 61% of the studies.
Person who runs the study: 11.1%
79% of respondents indicated that the person who runs the study also recruits for an average of 70% of the studies.??????
Table . Pre-session interaction
Items Provided Before Session | Number of Citations | Percentage of Respondents |
---|---|---|
Telephone call to confirm session date and time | 162 | 81% |
Email message to confirm session date and time | 143 | 71% |
Driving directions to session location | 94 | 47% |
A map to session location | 91 | 45% |
Some amount of orientation or training before the session | 89 | 44% |
A written letter to confirm session date and time | 51 | 25% |
Transportation to session location | 19 | 9% |
Other (some respondents cited more than one item):
| 14 | 7% |
Nothing | 4 | 2% |
As shown in the preceding table:
Table . Documents provided to participants
Documents Provided Before Session Ends | Number of Citations | Percentage of Respondents |
---|---|---|
Background questionnaire | 136 | 68% |
Nondisclosure agreement | 115 | 57% |
Audio or video recording consent form | 91 | 45% |
Receipt for incentives received | 67 | 33% |
Participant “Bill of Rights” | 29 | 14% |
Nothing | 24 | 12% |
Other:
| 21 | 10% |
Tax form, such as W4 (USA) or other | 6 | 3% |
As shown in the preceding table:
Table . Follow-up with participants
Follow-up Activities with Participants | Number of Citations | Percentage of Respondents |
---|---|---|
Thank-you email or note | 112 | 56% |
None | 68 | 34% |
Request for participant referrals | 37 | 18% |
Other (some respondents cited more than one item):
| 37 | 18% |
As shown in the preceding table:
More than half the respondents send a thank-you email or note – 56%.
A third of the respondents send nothing – 34%.
Almost a fifth request participant referrals – 18%.
Some follow up with participants in other ways – 16%.
18.219.22.169