xv
Preface
According to the cognitive viewpoint of information retrieval (IR) research, IR systems are essen-
tially interactive and thus users should not be abstracted out of system evaluations. To understand
the role of users in IR and to evaluate systems with users, interactive IR (IIR) researchers usually
design user studies in dierent contexts (i.e., controlled laboratory and naturalistic settings) to
examine various elements, facets, and stages of information search and evaluation process. Since
user study approach has been widely applied in search behavior and IR systems evaluation studies,
an eective evaluation framework of IIR user studies is critical for sharpening the instruments of
IIR research and improving the reliability and validity of the conclusions drawn from IIR user
studies. Despite the longstanding emphasis on evaluation in the IR community, there is still a lack
of evaluation of user studies as such, which hinders the fusion of the rigor of classical IR experi-
mentation and the richness of users’ search interactions. is book is an attempt to highlight this
research gap, synthesize literature around it, and engage the readers in thinking through some of
the ways forward.
We start by presenting a faceted framework for user study evaluation based on a system-
atic review of the state-of-the-art IIR research papers published in ACM SIGIR conferences
(2000–2018), CHIIR conferences (2016–2018), Journal of the Association for Information Science and
Technology (JASIST), Information Processing and Management (IP&M), CHI conferences, and ACM
Transactions on Information Systems (ACM TOIS) in Chapters 2 and 3. Note that we only included
the user studies where researchers proposed IIR-related research questions, recruited participants,
and clearly articulated the major components of study design (e.g., experimental system, task, test
collection), aiming to ensure that the detailed information concerning user study design can be
properly collected and analyzed.
Within the proposed faceted evaluation framework, in Chapter 4 we identify three major
types of research focuses based on the literature review in Chapter 2 (i.e., understanding user be-
havior and experience, system/interface features evaluation, and meta-evaluation of evaluation met-
rics) and explain how dierent facets and factors are employed and combined to jointly support the
explorations on dierent research problems. Also, we highlight the under-reported user study facets
(e.g., language, participant incentives, pilot study), which may signicantly aect the results and the
associated ndings of research. en, in Chapter 5 we employ the proposed faceted framework and
examine the strengths and weaknesses of a series of IIR user studies by (1) evaluating the value and
quality of each individual facet values (e.g., task type, system components), and (2) evaluating the
connections and “collaborations” among dierent facets (e.g., the relationship between participant
xvi PREFACE
and task context; the connection between study design and statistical methods). In addition, we
emphasize that for evaluation-oriented user studies, it is also critical for researchers to evaluate the
ground truth measures in order to make sure the evaluation of system features or evaluation metrics
are built upon solid empirical basis.
Although the faceted evaluation of IIR user studies can help improve the instruments of
user-oriented IIR studies and thereby broaden the territory of IIR research, it does not mean that
there is a single best toolkit for user studies which can be universally applied for all IIR-related
research questions. As Stephen Robertson notes in his Journal of Information Science paper titled
“On the History of Evaluation in IR” (2008, p.447), “In the end, any experimental design is a
compromise, a matter of balancing those aspects which are felt to be necessary for a good result.”
e ultimate goal of this work is to evaluate dierent types of design decisions and compromises in
user studies with respect to their contributions to answering the proposed corresponding research
questions.
When deconstructing a user study as a complex entity via a faceted approach, we can con-
ceptualize every specic user study as a particular combination of interrelated facet values. rough
bridging diverse combinations of facet values with the study design decisions and compromises
made for addressing research problems, this book makes the following contributions.
• It oers a comprehensive picture of various facets of the state-of-the-art IIR user
study design in a variety of research contexts.
• e faceted evaluation framework sheds light on both user study evaluation and future
standard IIR study practice and resource (e.g., data, technique) preservation.
• is work highlights the roles of and the relationships among dierent facets and
factors of user studies, and thus may help improve user study reporting practices in
IIR community.
Chapters 6 and 7 discuss in detail the implications and challenges associated with the
proposed faceted framework on multiple aspects of user study practices. By carefully examining
the facets of user studies and the underlying connections among them, this work can encourage
students, researchers, and practitioners in the IIR community to reect on the roles and potential
of user studies in understanding the interactions between people, information, and search systems.
Also, it can enhance our understanding of the structure and the related impacts of the “decisions
in compromise” and better support the dialectic around IIR user study design, reporting, and eval-
uation practices.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.