2 1. INTRODUCTION
them in IR studies? How can we properly evaluate user studies of dierent types based on the
corresponding research problems they are designed to address? Answering these questions and
building a framework for characterizing and evaluating IIR user studies can help us better reect on
the roles and eects of dierent components, methods, and techniques in user-oriented IR studies
and thus can further improve our understanding of human information interaction in IR context.
1.1 BACKGROUND: USER STUDY IN IIR RESEARCH
e term user study in the area of information science originally refers to the studies that explore
human information seeking needs (Kelly, 2009). In contemporary research studies, due to the
growth of information behavior community and the diversication of user-oriented research prob-
lems and techniques, user study is dened more generally to describe any study that involves human
participants. Within this broad scope of research, dierent types of user studies can be placed at
dierent positions in a research continuum that is anchored by human focused studies and system
focused studies (cf. Kelly, 2009), depending on the nature of their main research focuses. is wide
range of user study type demonstrates that user study as a research methodology has the potential to
help address a variety of research questions which are directly or indirectly related to user-oriented
search system design and evaluation. Note that some of the user studies published in IR community
are not directly associated with any of the system components in IIR evaluation (e.g., Moshfeghi,
Triantallou, and Pollick, 2016). Instead, they focused on some more fundamental concepts (e.g.,
information need, relevance, anomalous state of knowledge) that form the theoretical and empirical
basis of user modeling and user-centered system design and evaluation.
According to the cognitive viewpoint of IR research, information search and retrieval is es-
sentially interactive, thus users should not be abstracted out of IR system evaluation (Belkin, 1990;
Ingwersen, 1996). Built upon this premise, IIR user study brings user factors into system evaluation
and puts IR problems into the broader research context of human information interaction. Because
of the complexity of user behavior, many dierent facets of user study are considered and examined
in study design, reporting, and result interpretation. Also, a series of new techniques and methods
(e.g., eye tracker, functional magnetic resonance imaging) are employed to capture user features
(e.g., Moshfeghi, Triantallou, and Pollick, 2016; Cole et al., 2013). Given the role and potential of
user studies in understanding users’ interactions with IR systems, examining and synthesizing the
facets of user studies can enhance our understanding of IIR research methodology, help us address
the aforementioned questions associated with user study evaluation.
1.2 AIM OF THE WORK
As Robertson notes, “In the end, any experimental design is a compromise, a matter of balancing
those aspects which the experimenter feels should be realistic with those controls which are felt
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.