Appendix A. Participant Recruiting Survey

Nielsen Norman group conducted a worldwide email survey to gather specific information about the usability community’s collective recruiting experience. We received 201 responses, indicating great interest in the recruiting aspect of usability studies. We thank those who participated and provided valuable information.

About the Survey

Background

We created a plain-text survey that we could send in the body of an email message, which people could fill in as a reply to the email. Jakob Nielsen posted a notice about the survey on the Useit.com website and sent it to the Alertbox mailing list, inviting people who conduct user studies to request a copy of the survey: We also posted a similar notice on a widely accessed usability mailing list.

Contents

The survey introduction explained the research NN/g was conducting and the incentives respondents would receive for responding to the survey, as follows:

===============================================================

Nielsen Norman Group—Participant Recruiting Survey

Nielsen Norman Group (NN/g) is preparing a practical report on recruiting participants for usability studies and we would like to ask you to take part in our preliminary research. Please take a few minutes to fill in the survey below, which asks about the participant recruiting process for your usability studies. You needn’t be the person who actually does the recruiting to provide valuable information.

To thank you for your time. NN/g will:

  • Provide all respondents with a summary of the survey results.

  • Offer all respondents a 50% discount on the published NN/g participant recruiting report.

  • Randomly select 10 respondents to receive a complimentary copy of any NN/g usability report, up to a value of $250.

If you do not wish to receive any of the items listed above, please indicate your preference at the end of the survey. Simply reply to this email and type your responses directly into the survey in your reply.

NN/g will not use respondents’ email addresses or other identifying information in reporting survey results or in the published NN/g participant recruiting report; all respondents will remain anonymous.

Thank you.

===============================================================

Survey Questions

The survey contained 10 recruiting questions—some with multiple parts. We collected information on:

  1. The geographical areas from which respondents recruit participants

  2. Who recruits participants: the study facilitator, a staff recruiter, an outside recruiting agency, or some other person

  3. Internal time spent recruiting

  4. Agency lead time required and agency fees

  5. User profiles, incentives offered, and session length

  6. No-show rates

  7. Pre-session interaction with participants

  8. Participant documents

  9. Post-session follow up with participants

  10. Recruiting anecdotes (from respondents who agreed to be contacted; anecdotes are provided within the body of this report, not in the survey results).

Privacy Statement

NN/g also provided assurance of privacy with the following wording at the end of the survey:

“NN/g will use respondents’ email addresses only to send a summary of the survey results, to contact the winners of our drawing, to inform respondents of the procedure for getting the 50% discount, or to contact respondents who answered “Yes” to providing NN/g with a recruiting anecdote. Afterwards, NN/g will delete all respondents’ email addresses. NN/g will not sell or disclose respondents’ email addresses to anyone else.

“NN/g also will not use respondents’ email addresses or other identifying information in reporting survey results, or in the published NN/g participant recruiting report; all respondents will remain anonymous.”

Responses

  • We received 350 email messages requesting a copy of the survey.

  • The survey was completed and returned by 201 people, for a response rate of 57%

  • Of the 201 responses, 63 (or 31%) were received as the result of our having sent a reminder to the requestor to fill in and return his or her survey.

World Responses

The 201 responses came from at least 25 countries in the following geographical areas:

  • 54% US – 108

  • 22% Europe – 44 (UK–16, Germany–6, Denmark–4, Sweden–3, Switzerland–3, France–2, Italy–2, Netherlands–2, Spain–2, Austria–1, Belgium–1, Finland–1, Romania–1)

  • 21% Rest of the World – 43 (Canada–15, Australia–11, India–3, New Zealand–3; South Africa–3, Mexico–2, Singapore–2, China–1, Brazil–1, Ecuador–1, Israel–1)

  • 3% Unspecified – 6.

United States Responses

We divided the 108 US responses into three major regions as follows:

  • 30% Northeast (Maine to Washington DC) – 32

  • 32% West Coast (California, Oregon, and Washington) – 35

  • 38% All other US areas – 41.???

Survey Results

Geographical Areas From Which Participants are Recruited

Results for Participant Geographical Areas

The 201 survey respondents specified 296 locations, which included city, region (state or province), and country. We designated five regions—areas that contained 17% or more of the total regional citations. Under the regions below, we list the three most-often-cited cities within each area.

  • USA Northeast (Maine to Washington DC) – 17% of locations specified (51 of 296).

    New York City – 14

    Boston – 11

    Washington DC – 6

  • USA West Coast (California, Oregon, Washington) – 17% of locations specified (51 of 296)

    San Francisco – 14

    San Jose – 7

    Seattle – 7

  • USA, all other states – 20% of locations specified (59 of 296)

    Chicago – 9

    Dallas – 4

    Denver – 4

  • Europe – 22% of locations specified (65 of 296)

    London – 10

    Zurich – 3

    Paris – 2

  • Rest of the world – 24% of locations specified (70 of 296)

    Toronto – 8

    Sydney – 7

    Melbourne – 5

The three most-often-cited cities in the world are New York and San Francisco, with 14 citations each, and London, with 10 citations.

In the following table, responses are sorted by larger region (shaded, centered horizontal bar), country (in alphabetical order), region (in alphabetical order), and city (in alphabetical order).???

Geographical Areas from which Participants are Recruited

USA Northeast (Maine to Washington DC): 17%

Country

State, Province, or Region

City

Number of Citations

USA

Connecticut

Hartford

1

 

District of Columbia

Washington

6

 

Massachusetts

Boston

11

 

Maine

Unspecified

1

 

Maryland

Baltimore

1

 

New Hampshire

Manchester

1

 

New Hampshire

Unspecified

2

 

New Jersey

Newark

1

 

New Jersey

Whippany

1

 

New York

New Plats

1

 

New York

New York

14

 

New York

Potsdam

1

 

New York

Rochester

1

 

New York

Unspecified

1

 

Pennsylvania

King of Prussia

1

 

Pennsylvania

Philadelphia

3

 

Virginia

Arlington

1

 

Virginia

Danville

1

 

Virginia

Fairfax

1

 

Virginia

Falls Church

1

17% – USA Northeast TOTAL: 51

USA West Coast (California, Oregon, and Washington): 17%

Country

State, Province, or Region

City

Number of Citations

USA

California

Bay Area

1

 

California

Brisbane

1

 

California

Los Angeles

2

 

California

Palo Alto

2

 

California

Pleasanton

2

 

California

San Diego

3

 

California

San Francisco

14

 

California

San Jose

7

 

California

Santa Barbara

1

 

California

Santa Cruz

1

 

California

Silicon Valley

1

 

California

Sunnyvale

1

 

California

Unspecified

1

 

Oregon

Portland

5

 

Washington

Olympia

1

 

Washington

Seattle

7

 

Washington

Unspecified

1

17% – USA West Coast TOTAL: 51

USA – All Other States: 20%

Country

State, Province, or Region

City

Number of Citations

USA

Arizona

Phoenix

1

 

Arizona

Unspecified

1

 

Colorado

Denver

4

 

Florida

Boca Raton

1

 

Florida

Fort Lauderdale

1

 

Florida

Jacksonville

1

 

Georgia

Atlanta

2

 

Georgia

Rowell

1

 

Illinois

Bloomington

1

 

Illinois

Chicago

9

 

Illinois

Unspecified

1

 

Kansas

Kansas City

1

 

Kentucky

Bowling Green

1

 

Michigan

Ann Arbor

1

 

Michigan

Detroit

1

 

Michigan

Lansing

2

 

Minnesota

Minneapolis / St. Paul

3

 

Missouri

Joplin area

1

 

Missouri

Kansas City

2

 

Missouri

St. Louis

2

 

North

Carolina Chapel Hill

1

 

Ohio

Cleveland

1

 

Ohio

Columbus

1

 

Ohio

Dayton

1

 

Ohio

Findlay

1

 

Texas

Austin

3

 

Texas

Dallas

4

 

Texas

Houston

1

 

Texas

Unspecified

1

 

Utah

Logan

1

 

Wisconsin

Madison

3

19% – USA Non-Northeast and Non-West Coast TOTAL: 56

 

Unspecified

Metropolitan Areas

1

 

Unspecified

Unspecified

2

1% – USA Unspecified Total: 3

54% – USA Total: 161

Europe: 22%

Country

State, Province, or Region

City

Number of Citations

Austria

Unspecified

Vienna

1

 

Unspecified

Unspecified

1

Austria Total: 2

Belgium

Unspecified

Brussels

1

Belgium Total: 1

Denmark

Eastern Jutland

Unspecified

1

 

Unspecified

Copenhagen

3

Denmark Total: 4

Finland

Unspecified

Helsinki

2

 

Unspecified

Tampere

1

Finland Total: 3

France

Ile de France

Paris

2

 

Other than Ile de France

Small cities

1

 

Unspecified

Unspecified

2

France Total: 5

Germany

Baden–Württemberg

Karlsruhe

1

 

Baden–Württemberg

Tübingen

1

 

Bavaria

Munich

1

 

Berlin–Brandenburg

Potsdam

1

 

Hamburg

Hamburg

1

 

Hessen

Frankfurt

1

 

Unspecified

Berlin

2

 

Unspecified

Unspecified

1

Germany Total: 9

Italy

Lazio

Rome

1

 

Lombardia

Milan

1

 

Sicily

Palermo

1

Italy Total: 3

Netherlands

Noord–Holland

Amsterdam

1

 

Unspecified

The Hague

1

Netherlands Total: 2

Romania

Unspecified

Unspecified

1

Romania Total: 1

Spain

Unspecified

Madrid

1

 

Unspecified

Barcelona

1

 

Unspecified

Valencia

1

Spain Total: 3

Sweden

Unspecified

Stockholm

2

 

Unspecified

Unspecified

2

Sweden Total: 4

Switzerland

Unspecified

Bern

2

 

Unspecified

Zurich

3

Switzerland Total: 5

United Kingdom

England

Bristol

1

 

England

Liverpool

2

 

London

London

10

 

England

Wirral

1

 

Hampshire

Southampton

1

 

Norfolk

Norwich

1

 

Sussex

Brighton

1

 

Tyne

Sunderland

1

 

Unspecified

Milton Keynes

1

 

Unspecified

Unspecified

2

United Kingdom Total: 21

Unspecified

Europe

Unspecified Europe: 2

22% – Europe TOTAL: 65

Rest of the World: 24%

Country

State, Province, or Region

City

Number of Citations

Australia

Queensland

Brisbane

1

 

Australian Capital Territory

Canberra

3

 

Hunter

New Castle

1

 

New South Wales

Lismore

1

 

New South Wales

Sydney

7

 

Victoria

Geelong

1

 

Victoria

Melbourne

5

 

Unspecified

Unspecified

1

Australia Total: 20

Brazil

Unspecified

Sao Paulo

1

Brazil Total: 1

Canada

British Columbia

Vancouver

2

 

British Columbia

Victoria

2

 

British Columbia

Unspecified

1

 

Ontario

Ottawa

2

 

Ontario

Toronto

8

 

Ontario

Waterloo

1

 

Prince Edward Island

Charlottetown

1

 

Quebec

Montreal

1

 

Saskatchewan

Regina City

1

 

Unspecified

Unspecified

1

Canada Total: 20

China

Unspecified

Hong Kong

1

 

Unspecified

Shanghai

1

China Total: 2

Ecuador

Guayas

Guayaquil

1

 

Pichincha

Quito

1

Ecuador Total: 2

India

Delhi

New Delhi

1

 

Maharashtra

Mumbai

1

 

Tamilnadu

Chennai

2

India Total: 4

Israel

Unspecified

Tel Aviv

2

Israel Total: 2

Mexico

Unspecified

Mexico City

2

Mexico Total: 2

New Zealand

Unspecified

Auckland

2

 

Unspecified

Wellington

2

 

Unspecified

Hamilton

1

New Zealand Total: 5

Singapore

Unspecified

Unspecified

2

Singapore Total: 2

South Africa

Eastern Cape

Unspecified

1

 

Gauteng

Johannesburg

1

 

Gauteng

Pretoria

1

 

Unspecified

Cape Town

1

South Africa Total: 4

Unspecified: 6

24% – Rest of the World TOTAL: 70

Regional Citations GRAND TOTAL: 296

Who Recruits Participants

Results for Who Recruits

  • 79% of respondents indicated that the person who runs the study recruits participants for an average of 70% of the studies, and 28% of respondents indicated that the person who runs the study recruits 100% of the studies.

  • 36% of respondents indicated that an outside recruiting agency recruits participants for an average of 61% of the studies, and 9% indicated that an agency recruits 100% of the studies.

  • 32% respondents indicted that someone else within their company recruits participants for an average of 32% of the studies. “Clerical staff” was cited most often (six times), and other support staff, such as customer service or receptionists, were cited five times.

  • 13% of respondents indicated that some other person recruits participants for an average of 32% of the studies. “The client” was cited most often (16 times).

  • 12% of respondents indicated that a dedicated recruiter within their company recruits participants for an average of 67% of the studies, and 6% indicated that dedicated recruiters recruit 80%–100% of the studies.

Percentages total more than 100% because we allowed multiple responses. In fact, it seems that most companies combine two different approaches to recruiting, depending on the project.

Recruiter tables

The following five tables provide the breakdown of percentages for each recruiter type cited by respondents. For example, in the Recruiter 1 table directly below, 57 respondents indicated that the person who runs the study recruits 100% of the studies in their organizations.???

Table . Recruiter 1: The person who runs the study

Number of Citations

Percentage of Studies Cited

57

100%

2

95%

15

90%

15

80%

7

75%

6

70%

6

60%

13

50%

2

40%

5

30%

3

25%

8

20%

12

10%

6

5%

1

1%

Total Citations: 158

Average: 70%

As shown in the preceding table, 79% of respondents (158 of 201) indicated that the person who runs the study recruits participants for an average of 70% of the studies, and 28% of respondents (57 of 201) indicated that the person who runs the study recruits for 100% of the studies.???

Table . Recruiter 2: An outside recruiting agency

Number of Citations

Percentage of studies cited

19

100%

8

90%

1

85%

8

80%

1

75%

2

70%

2

60%

8

50%

4

40%

1

25%

3

20%

1

15%

8

10%

7

5%

Total Citations: 73

Average: 61%

As shown in the preceding table, 36% of respondents (73 of 201) indicated that an outside recruiting agency recruits participants for an average of 61% of the studies, and 9% (19 of 201) indicated that an agency recruits for 100% the studies.???

Table . Recruiter 3: Someone else within my company

Recruiter Cited

Number of Citations

Percentage of Studies Cited

Customer Service

Project Manager

Development

Manager

Consulting Manager

1

1

1

1

100%

Project Manager or Development Manager

1

95%

Test Center

Board Room

Concierge

1

1

90%

Not specified

1

80%

Associate Producer

Project Manager

1

1

75%

Not specified

1

65%

Clerical Staff

2

50%

Communications

Sales Rep

Not specified

1

1

1

40%

Not specified

Marketing Staff

Product Manager

Customer Service

7

1

1

1

30%

Clerical Staff

Not specified

3

2

25%

Not specified

Account Rep

Other UE Specialist

Website Manager

Web Developer

Clerical Staff

7

1

1

1

1

1

20%

Other UE Specialist

1

15%

Not specified

Other UE Specialist

Visual Designer

Business Manager

Website Manager

Project Manager

Client Services

Instructional

Designer

5

3

1

1

1

1

1

1

10%

Not specified

4

5%

Receptionist

Not specified

1

1

1%

Total Citations: 64

Average: 32%

As shown in the preceding table, 32% of respondents (64 of 201) indicted that someone else within their company recruits participants for an average of 32% of the studies. “Clerical staff” (highlighted in blue) was cited most often (six times), and other support staff, such as “Customer Service” or “Receptionist,” were cited five times (also highlighted in blue).???

Table . Recruiter 4: Some other person

Recruiter Cited

Number of Citations

Percentage of studies cited

Research Office

1

100%

Client

1

80%

Client

1

70%

Client

Not specified

2

2

50%

Client

1

45%

Client

1

40%

Client

1

35%

Client

Usability Firm

2

1

30%

Team Manager

1

25%

Account Manager

Data Logger

Client

Not specified

1

1

1

1

20%

Client

Observer

Person Playing

Computer

6

1

1

10%

Student

1

5%

Total Citations: 27

Average: 32%

As shown in the preceding table, 13% of respondents (27 of 201) indicated that some other person recruits participants for an average of 32% of the studies. “Client” was cited most often (16 times).???

Table . Recruiter 5: A dedicated recruiter within my company

Number of Citations

Percentage of studies cited

7

100%

2

95%

2

90%

2

80%

2

60%

2

50%

1

40%

1

30%

1

25%

2

20%

2

10%

Total Citations: 24

Average: 67%

As shown in the preceding table, 12% of respondents (24 of 201) indicated that a dedicated recruiter within their company recruits participants for an average of 67% of the studies, and 6% (13 of 201) indicated dedicated recruiters recruit 80%–100% of the studies.??????

Internal Time Spent Recruiting Participants

Results for Internal Time Spent Recruiting

Table . Internal time spent recruiting participants

Time Range in Hours

Number of Citations

Percentage of Citations

Low

High

0.25

0.33

21

17%

0.50

0.75

40

32%

1.00

1.50

34

27%

2.00

2.50

22

18%

3.00

4.59

5

4%

5.00

6.50

3

2%

Average Time Spent: 1.15 hours

Total Citations: 125

Total: 100%

As shown in the preceding table, 62% of respondents (125 of 201) indicted that internal people who spend time recruiting participants spend an average of 1.15 hour (or 69 minutes) per participant recruited. 59% of these 125 respondents spend .5 to 1.5 hours per participant recruited, and 24% spend 2 to 6.5 hours per participant recruited.??????

Agency Lead Time Required and Fees Charged

Results for Agency Lead Time

Table . Lead time required by agencies

Time Range in Weeks

Number of Citations

Percentage of Citations

Low End

High End

.24

.33

8

12%

1.00

1.50

21

31%

2.00

2.50

30

40%

3.00

4.00

6

9%

5.00

6.00

5

8%

Average Lead Time: 1.93 weeks

Total Citations: 67

Total: 100%

As shown in the preceding table, 33% of respondents (67 of 201) indicated that agencies require average of 1.93, or nearly two weeks’ lead time.

Results for Agency Fees

As noted in the preceding table, 33% of respondents (67 of 201) indicated they use recruiting agencies for 8% – 40% of their studies. In the following table, we present average fee requirements (listed in US dollars) charged by agencies around the world for internal and external participants, as indicated by 116 agency citations generated by the 67 positive responses in our survey, and listed by region.???

Table . Agency fees by region

Region

Recruitment Fee per Participant

Number of Internal Citations

Number of External Citations

USA – West Coast (California, Oregon, and Washington)

$125

3

21

USA – Other

$124

2

21

USA – Northeast (Maine to Washington DC)

$113

2

24

Europe

$108

4

18

Unspecified

$75

0

2

Rest of the World

$57

5

14

Average: $107

Total: 16

Total: 100

As shown in the preceding table, the overall world average fee that agencies charge per recruited participant is $107. The US Northeast is generally lower than the West Coast and other US regions, and European and other non-US, non-European agencies generally charge lower recruiting fees than their US counterparts.

In the following table, we present average fee requirements (listed in US dollars) for internal and external user categories.???

Table . Agency fees by user category

User Category

Recruitment Fee per Participant

Internal – 16 citations

General Staff (“anybody” and “everybody,” for example)

$84

Unspecified (no category given)

$73

Professional Staff (managers and executives, for example)

$23

Technical Staff (system administrators, for example)

$21

Support Staff (admin assistants, for example)

$10

External – 100 citations

Teachers / Professors

$250

Professional Staff (managers and executives, for example)

$161

Technical Staff (system administrators, for example)

$119

Medical Personnel

$117

Consumers (buys readily available goods and services)

$90

Students / Young Adults

$84

Unspecified (no category given)

$51

Welfare Recipient

$42

Average World Fee:

$107

As shown in the preceding table, professional categories of external participants generally warrant the highest agency fees???

Participant User Groups, Incentives, and Session Length

User Group and Incentive Descriptions

The following descriptions of the user groups and incentives respondents cited will provide context for the results for Question 5.

Internal User Groups

We organized all internal user groups that respondents cited using as dry run, pilot, or regular study participants into four separate categories—general, professional, support / clerical, and technical—as described below. Many respondents provided data about usability studies they conduct with internal users, but did not specify an internal user group.

  • General (nonspecific): anyone, colleague, employee, everyone, friend, general user, new hire, non-developer, non-project personnel, staff, student, subject-matter expert, volunteer.

  • Professional: academic, administrator, business analyst, consultant, faculty, journalist, manager, market or other researcher, study stakeholder, travel agent.

  • Support / clerical: administrative assistant, call center staff, clerical staff, customer service staff, human resources staff, librarian, secretary, operational staff.

  • Technical: developer, documentation specialist, electrical engineer, HCI student, help desk staff, instructional designer, network engineer, network administrator, product manager, scientist, software engineer, system administrator, trainer, UI designer, usability staff.

  • Unspecified: Respondents did not specify a user group.

External User Groups

We organized external user groups that respondents cited into six separate categories—high-end professional, medium professional, low-end professional, technical, consumer, and unpaid / nonprofessional—as described below. A few respondents provided data about usability studies they conduct with external users, but did not specify a user group.

  • High-end professional: architect, attorney / lawyer, CEO, consultant, executive, physician, surgeon, professional investor, real-estate agents / brokers.

  • Medium professional: academic, administrator, banker, business analyst, consultant, exporter, faculty, financial advisor / planner, graphic designer, insurance agent / broker, manager, market or other researcher, marketing / salesperson, nurse, journalist, professor, public-health official, teacher, travel agent.

  • Low-end professional: administrative assistant, artist (performing, visual), call center staff, clerical staff, construction worker, customer-service staff, farmer, government worker, human-resources staff, librarian, secretary, office manager, operational staff, truck driver.

  • Technical: developer, documentation specialist, electrical engineer, help-desk staff, instructional designer, network engineer, network administrator, product manager, scientist, software engineer, system administrator, trainer.

  • Consumer (person who purchases goods or services available to the general public): cross-section of general public, electronics (including cell phone, computer, PDAs, internet), big-ticket items (furniture, autos, large appliances), services (banking, financial, repair, utilities, telephone, travel and leisure).

  • Special needs or nonprofessional: children, homemaker, medical patient, person with disabilities, retiree, senior citizen, student, public-assistance recipient, volunteer.

  • Unspecified: Respondents did not specify a user group.

Incentives Provided

We organized incentives into the broad categories of monetary and non-monetary. Non-monetary incentives fell into four separate subcategories—certificates for merchandise, food and beverages, “intangible rewards,” and tangible gifts—as described below. Some respondents indicted that they provide a non-monetary incentive, but did not specify what it was.

  • Monetary incentives: we present all monetary incentive information in US dollars and, for easier comparison, monetary incentives reflect a dollars-per-hour rate, not dollars-per-session.

  • Non-monetary incentives: for purposes of presenting data, we have placed all cited, non-monetary incentives into the following categories:

    • Certificates: redeemable certificates and coupons in denominations ranging from $5 – $50 for book shops, department stores, restaurants, corporate cafeterias, local merchants and coffee shops.

    • Food and beverages: bottle of champagne, liquor, or wine; cake, coffee or tea, dinner or lunch, prepackaged snacks, refreshments or treats.

    • Intangible rewards: airline frequent-flyer miles, charging time to project (not overhead), job performance, extra credit (for students), internet access, members-only website privileges, merit certificates, opportunity to provide input for improvement, study results, thanks, time off, training, and travel expenses for study covered.

    • Tangible gifts: merchandise ranging in value from around USD $5 – $100, including coloring books (for children), compact discs, flowers, logo merchandise, movie passes, office supplies, software, theatre tickets, and toys and “trinkets” (for both children and adults).

    • Unspecified: respondents did not specify a non-monetary incentive.

Results for Participant User Groups Cited

Internal dry-run / pilot participants
  • 83% of respondents (166 of 201) indicated they use the following internal groups (as described above) for dry-run and / or pilot sessions:

    • 27% Unspecified (45 of 166)

    • 25% General (41 of 166)

    • 24% Technical (40 of 166)

    • 16% Professional (26 of 166)

    • 8% Support (14 of 166).

  • 57 minutes is the average length of time for internal dry-run / pilot sessions.

Actual internal participants
  • 67% of respondents (134 of 201) indicated they use the following internal groups as actual participants in usability studies:

    • 28% Unspecified (37 of 134)

    • 23% General (31 of 134)

    • 22% Professional (29 of 134)

    • 16% Technical (22 of 134)

    • 11% Support (15 of 134)

  • 65 minutes is the average length of time for internal participant sessions.

External participants
  • 92% of respondents (184 of 201) indicated they use external participants. Some cited up to three user groups each for a total of 260 citations comprising the following user groups as participants in usability studies:

    • 37% Consumer (96 of 260)

    • 16% Special Need / Nonprofessional (42 of 260)

    • 15% Medium Professional (39 of 260)

    • 13% Technical (34 of 260)

    • 11% High-End Professional (29 of 260)

    • 5% Low-End Professional (13 of 260)

    • 3% Unspecified (7 of 260).

  • 72 minutes, or about 1.25 hours, is the average length of time for external participant sessions.

Results for Incentives Provided

Respondents indicated the monetary and non-monetary incentives that they provide to participants. From that, we extrapolated the number of participants who receive no incentive at all. As shown in the following table, 67% of internal pilot / dry-run participants, 55% of actual internal participants, and 9% of external participants receive no incentive, for an overall average of 37%—a little more than a third of the sample.???

Table . Types of incentives by user group

User Group

Monetary Incentives

Non-monetary Incentives

No Incentive

Total Citations

Internal Pilots / Dry Runs

11

43

112

166

Internal Actual Participants

13

47

74

134

External Participants (35 offered both monetary and non-monetary and are counted in both columns)

165

106

24

295 (Overlap) – 35

Total

189

196

210

560

Monetary Incentives

As shown in the preceding table, survey respondents indicated that not everyone provides monetary incentives to participants, especially not to internal participants (generally, employees or closely related contractors of the study sponsors).

  • Of the 166 respondents who use internal people as dry-run or pilot participants, only 7% provide a monetary incentive.

  • Of the 134 respondents who use internal people as regular study participants, only 10% provide a monetary incentive.

  • The worldwide average rate paid to internal participants (pilot, dry-run, and actual) is USD $32 per session hour.

  • 184 respondents who use external participants recruit multiple user groups—for a total of 260 citations. Of the total participants, 63% received monetary incentives, in marked contrast to internal participants.

  • The worldwide average rate paid to external participants is USD $64 per session hour—twice that of the internal rate.

The following table provides survey responses for monetary incentives provided, sorted by user group.???

Table . Monetary incentives in US dollars by user group

User Group

Average USD Paid Per Session Hour

Total Citations

Internal Pilots / Dry runs

$34

11

General

$44

2

Professional

$0

0

Support

$38

2

Technical

$5

2

Unspecified

$46

5

Internal Participants

$31

13

General

$33

2

Professional

$37

3

Support

$0

0

Technical

$11

3

Unspecified

$37

5

External Participants

$64

165

Consumer

$54

68

Higher Professional

$118

23

Medium Professional

$69

22

Lower Professional

$50

6

Special Needs or Nonprofessional

$32

20

Technical

$67

22

Unspecified

$55

4

Total

189

As shown in the preceding table, of internal staff who receive monetary incentives, technical staff receive the lowest incentive rate. (Technical people often are motivated to participate for the opportunity to be involved in technology in a new and different way.) As pilot and dry-run participants, professional staff receive no monetary incentive—perhaps because they may have some stake in the study. Interestingly, support staff are paid as pilot and dry-run participants but not as actual participants.

The above table also indicates that external monetary rates are double that for internal participants, with high-end professionals garnering the highest rate.

The following table summarizes the monetary incentives offered to participants by geographical region and user group.???

Table . Monetary incentives in US dollars by region

Region

Average USD Paid Per Session Hour to Internal Pilots / Dry runs

Average USD Paid Per Session Hour to Internal Ppts

Average USD $ Per Hour Paid to External Ppts

Total Citations

USA $73

126

Northeast $70

$33

$13

$77

 

West Coast $79

$55

$81

Rest of USA $69

$17

$7

$78

Europe $34

$23

$58

$34

36

 

Rest of the World $27

$18

$29

27

Total

189

As shown in the preceding table, monetary payment trends for internal participants by region are unpredictable. Monetary payments are highest on the US West Coast ($79 per session-hour on average), and lowest in non-European, non-US countries ($27 per session-hour).

Why are incentives twice as high in the United States as compared with Europe? One likely explanation is that Americans tend to feel over-surveyed and require more of an incentive to participate in studies. In contrast, Europeans may not yet be as tired of market research and may show more appreciation for being asked to influence the future direction of technology. It is understandable that the West Coast in the US is more expensive than the rest of the country (because the cost of living tends to be higher on the coasts), but why isn’t the Northeast more expensive as well? It might be because of a similar phenomenon on the West Coast. With its higher number of computer and high-tech companies, the West Coast’s population may simply feel more over-surveyed than people on the East Coast.

Non-monetary Incentives

The 201 respondents generated 196 non-monetary incentive citations within a total of 560 distinct user-group citations (166 internal dry-run / pilot citations, 134 internal participant citations, and 260 external participant citations).

We provide non-monetary citations by user group and by geographical location.???

Table . Non-monetary incentives by user group

User Group

Gift

Redeemable Certificate

Food

Intangible

Total Citations

Internal Pilots / Dry Runs

18

6

12

7

43

General

8

3

1

2

14

Professional

1

1

3

2

7

Support

0

1

3

0

4

Technical

3

0

4

1

8

Not specified

6

1

1

2

10

Internal Participants

19

13

10

5

47

General

7

4

2

2

15

Professional

4

2

2

1

9

Support

2

2

3

0

7

Technical

0

3

3

2

8

Not specified

6

2

0

0

8

External Participants

58

24

13

11

106

Consumer

22

11

4

2

39

Higher Professional

3

2

1

1

7

Medium Professional

10

3

1

4

18

Lower Professional

2

1

1

0

4

Special Needs / Nonprofessional

13

6

3

1

23

Technical

8

1

1

3

13

Not specified

0

0

2

0

2

Total

95

43

35

23

196

As shown in the preceding table, gifts and gift certificates are the preferred incentives, being offered two times as often as other non-monetary incentives for both internal and external groups.

The following table summarizes the non-monetary incentives offered to participants by geographical region.??????

Table . Non-monetary incentives by region

Region

Gift

Redeemable Certificate

Food

Intangible

Total Citations

USA

47

24

8

13

92

Northeast

5

3

5

4

17

West Coast

19

13

0

5

37

Rest of USA

23

8

3

4

38

Europe

24

7

9

3

43

 

Rest of the World

24

12

18

7

61

Total

95

43

35

23

196

No-Show Rates

Results for No-Show Rates

As shown in the following table, the overall average no-show rate for our survey sample is 10.6%.???

Table . No-show rates

No-Show Rate Range

Number of Citations

0% – 1%

45

2% – 5%

40

6% – 8%

7

10%

46

12% – 15%

20

20%

21

25%

12

30% – 50%

8

60%

2

Average No-Show Rate: 10.6%

Total Citations: 201

Note that respondents did not provide ranges; rather we combined cited ranges so this table would not be inordinately long.

No-show rates by who recruits

We tested whether survey data would confirm the anecdotal data we received from usability professionals that the no-show rate is higher for professional recruiting agencies than for internal recruiters. The results from our survey tend to back up this claim, although the differences in no-show rates are not statistically significant. Three internal recruiter categories do slightly better, while a fourth—the person who runs the study (presumably a usability professional)—does slightly worse than the world average rate of 10.6%, as described below. The following bullet points summarize the no-show rates from best to worst, for internal and agency recruiters:

  • Someone else within the company: 10.1%

    32% of respondents indicated someone else from within their company, usually support staff, recruits for an average of 32% of the studies.

  • Some other person: 10.4%

    13% of respondents indicated that some other person, usually “the client,” recruits for an average of 32% of the studies.

  • Dedicated recruiter with the company: 10.5%

    12% of respondents indicated that a recruiter within their company recruits for an average of 67% of the studies.

  • Recruiting agencies: 10.8%

    36% of respondents indicated that an outside recruiting agency recruits for an average of 61% of the studies.

  • Person who runs the study: 11.1%

    79% of respondents indicated that the person who runs the study also recruits for an average of 70% of the studies.??????

Pre-Session Interaction

Results for Pre-session Interaction

Table . Pre-session interaction

Items Provided Before Session

Number of Citations

Percentage of Respondents

Telephone call to confirm session date and time

162

81%

Email message to confirm session date and time

143

71%

Driving directions to session location

94

47%

A map to session location

91

45%

Some amount of orientation or training before the session

89

44%

A written letter to confirm session date and time

51

25%

Transportation to session location

19

9%

Other (some respondents cited more than one item):

  • Brief description of project objectives – 3

  • Parking – 3

  • Info on local transportation – 2

  • Demo passwords – 1

  • Description of food and incentives – 1

  • Outlook meeting invitations (for internals) – 1

  • Phone call to supervisor – 1

  • Poster in organization – 1

  • Preview interview – 1

14

7%

Nothing

4

2%

As shown in the preceding table:

  • 98% of respondents (197 of 201) provide at least one of the eight queried items.

  • The top three items cited were:

    • Phone confirmation – 81%

    • Email confirmation – 71%

    • Directions to session location – 45%

  • 2% of respondents (4 of 201) provide nothing.??????

Participant Session Documents

Results for Documents Provided to Participants

Table . Documents provided to participants

Documents Provided Before Session Ends

Number of Citations

Percentage of Respondents

Background questionnaire

136

68%

Nondisclosure agreement

115

57%

Audio or video recording consent form

91

45%

Receipt for incentives received

67

33%

Participant “Bill of Rights”

29

14%

Nothing

24

12%

Other:

  • Post test questionnaire – 6

  • Participant’s instructions form – 3

  • Simple project description – 2

  • Interface rating – 2

  • Photo release form – 1

  • Critical incident survey – 1

  • Receipt for expenses only; non-income – 1

  • Brochure and documentation – 1

  • Quantitative survey – 1

  • Thinking Aloud – 1

  • Verbal audio or video consent – 1

  • We own the invention rights – 1

21

10%

Tax form, such as W4 (USA) or other

6

3%

As shown in the preceding table:

  • 76% of respondents (177 of 201) provide at least one of the seven items we asked about.

  • The top three items cited were:

    • Background Questionnaire – 68%

    • Nondisclosure agreement – 57%

    • Audio-video consent form – 45%.

  • 12% of respondents (24 of 201) provide no documents.??????

Participant Follow-Up

Results for Participant Follow-Up

Table . Follow-up with participants

Follow-up Activities with Participants

Number of Citations

Percentage of Respondents

Thank-you email or note

112

56%

None

68

34%

Request for participant referrals

37

18%

Other (some respondents cited more than one item):

  • Ask if participants want to participate again – 6

  • Send or personally deliver incentives – 6

  • Debrief or summary document – 5

  • Notify when the site goes live or undergoes a revision – 4

  • Send follow-up questionnaire – 4

  • Send results of study – 3

  • Clarify data – 3

  • Tell how participant input was used – 3

  • Offer a limited license edition – 1

  • Answer questions that could not be answered by the session facilitator – 1

  • Unspecified follow up – 1

37

18%

As shown in the preceding table:

  • More than half the respondents send a thank-you email or note – 56%.

  • A third of the respondents send nothing – 34%.

  • Almost a fifth request participant referrals – 18%.

  • Some follow up with participants in other ways – 16%.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.219.22.169