4 1. INTRODUCTION
One may argue that this type of user study did deal with the issue of study design compromise in
the sense that the researchers faced the trade-o between the richness of user features and data
suciency in large-scale search logs. However, this is not the type of compromise we are interested
in here. More details regarding paper selection and the development of faceted framework are re-
ported in Chapters 3 and 4.
When the paper coding process was completed, we applied the faceted framework in charac-
terizing and evaluating IIR user studies of dierent types. e user study evaluation roughly follows
three steps.
1. Given a specic user study case, we rst identify the research focus (e.g., understand-
ing user behavior, evaluating new interface component) as well as the specic research
problems.
2. Based on the step 1, we identify the main facets and the associated sub-facets and
factors from the faceted evaluation framework. Specically, we mainly focus on the
factors that are manipulated in the study (e.g., size of the search engine result page,
search assistance) and the facets that are associated with the variables of interest (e.g.,
task features, user characteristics, mode of search, stage of search session). Also, we
look at the facets that need to be controlled given the research context.
3. More importantly, we explore the connections among dierent facets and evaluate
the compromises made in user study design based on the research questions proposed
in the paper. e decisions and compromises in study design are usually made for
the purpose of obtaining benets from other aspects and facets. Our hope is that the
faceted approach proposed here can at least help researchers qualitatively understand
and evaluate the role of dierent facets and the compromises made by researchers in
balancing dierent facets or dimensions and ultimately in answering the proposed
research questions. More details regarding the faceted scheme of IIR user studies are
provided with specic examples in Chapters 5 and 6.
1.4 IMPLICATIONS OF THE FACETED FRAMEWORK
When deconstructing a user study as a complex entity via a faceted or multidimensional approach,
we can dene every specic user study as a unique vector of dierent facet values corresponding to
dierent user study components. rough building a faceted framework for depicting and evalua-
tion IIR user study, this book makes the following contributions.
• It oers a comprehensive picture of various facets of the state-of-the-art user study
design in a variety of research contexts.
5
• e faceted framework sheds light on user study reporting, evaluation and future stan-
dard IIR study practice and resource (e.g., data, system components and techniques,
study procedure) preservation.
• Most importantly, this work highlights the roles of and the relationships among dif-
ferent facets of user studies, and thus may help improve user study reporting practices
in the IR community.
It is worth noting that the comprehensiveness of our framework refers to the broad cover-
age and representativeness of the facets extracted from existing IIR user studies, rather than the
exhaustiveness with our coverage of the past literature. In other words, we are not trying to review
and annotate all available user study papers published in IIR-related venues so far. Despite the
dierence in specic research problems and subjects, a large body of IR research actually adopted
a very similar set of study protocols and methodologies. In this work we seek to develop a faceted
framework that can cover and capture most, if not all, of the facets and factors of user study as well
as the connections among these facets.
As Robertson (2008) points out, “a eld advances not by deciding on a single best com-
promise, but through dierent researchers taking dierent decisions, and the resulting dialectic”
(p. 447). In the recent two decades, the diversity in user study design decisions (e.g., participant
recruitment, task design, study device, data collection) largely supports the explorations of users’
interactions with IR systems in context and signicantly expands the entire scope of IIR research.
By carefully examining the facets of user studies and the relations among them, this work can en-
hance our understanding of the structure and the related impacts of the “decision in compromise”
and better support the dialectic around IIR user study design, reporting, and evaluation practices.
1.4 IMPLICATIONS OF THE FACETED EVALUATION FRAMEWORK
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.