23
CHAPTER 4
Faceted Framework of IIR User
Studies
Faceted approach as a widely used tool in conceptual analysis and library science classication
research can help researchers deconstruct a complex concept or entity into several specic, interre-
lated facets and dimensions and thereby can facilitate theorization, typology development, as well
as the associated empirical studies. One of the most relevant faceted frameworks in the area of IIR
is Li and Belkin’s (2008) faceted classication scheme of task in information seeking and retrieval.
is chapter presents the faceted framework which was gradually developed and iteratively revised
during the paper coding process, as our response to the very rst research goal that we proposed in
Chapter 1: develop a faceted framework of user study characterization and evaluation and identify
easily ignored user-side facets and factors. To fully address this problem, we rst discuss the main
facets and the associated subfacets (e.g., research focus, task, system feature) here as they jointly
serve as the basis for user study design. en, we identify a series of easily ignored, under-reported
facets that may signicantly aect the process and result of user study.
In the following sections, the identied user study facets or dimensions will be discussed in
detail and be illustrated with the associated examples or representative user studies from selected
venues. e nal version of faceted user study framework is shown in Figure 4.1, and will be
explained piece by piece in this chapter. Built upon the collection of recently published IIR user
studies from a variety of top venues, our faceted framework can oer a comprehensive picture of
the-state-of-the-art user study practice and facilitate faceted analysis (e.g., generating standard
metadata of user studies for summarization, comparative analysis, and evaluation).
4.1 STRUCTURE OF THE FACETED FRAMEWORK
Despite the uniqueness of every user study (especially in terms of the specic techniques and sys-
tem features), it is still possible and also critical for researchers to extract main common facets and
subfacets from a pool of specic IIR user studies and to gure out how these facets are dened,
designed, and manipulated according to the nature of the proposed research problems and goals.
We believe that a comprehensive, faceted framework of user study can serve as a checklist for IIR
researchers to reect on when they design, report, and evaluate various user studies. Based on our
qualitative coding, iterative discussion and continuous revision of the coding scheme, we developed
a faceted framework of IIR user study which consists of nine main facets: (1) the root or starting
24 4. FACETED FRAMEWORK OF INTERACTIVE IR USER STUDIES
Figure 4.1: Faceted evaluation framework of interactive IR user studies.
meta-evaluation of evaluation metrics
system/interface features, evaluation
understanding search behavior and experience
independent variables
quasi-independent variables
dependent variables
recruitment methods (e.g., crowdsourcing, social network, y
ers)
sample size
demographics
language (i.e., user’s native language, language used in study)
incentives
search behavior measures
relevance judgment
neuro-physiological measures
search performance
task perception
search experience and system evaluation
quality control and data ltering
desktop
mobile device
regular incentives for participation
extra incentives (bonus)
controlled lab
eld
work task type
do work task
work task
search task
time per task
amount of tasks
additional control of tasks
number of tasks in study
number of tasks per person
statistical and prediction models
results
Research Focus
Variable
System Feature
Behavior and Experience
Data Analysis and Results
Task
Experimental Design
Participant
Faceted Framework
Study Procedure
search task source
search task topic
search task type
answer search task
within-subjects
between-subjects
mixed design
study environment
search interface elements varied
search assistant tool
ranking algorithm
search collection/corpora
device
pre-study control
within-study control
post-study control
length of study
pilot study
pre-study training
⊝
⊝
⊝
⊝
⊝
⊝
⊝
⊝
⊝
⊝
⊝
⊝
⊝
⊝
⊝
⊝
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.