57
CHAPTER 7
Conclusion and Future Directions
According to the cognitive viewpoint of IR research, search systems are essentially interactive and
hence users should not be abstracted out of system design and evaluation processes (Belkin, 2015;
Ingwersen, 1996). To understand the role of users in search interaction and to evaluate IR systems
with users, IIR researchers often design user studies of dierent types to examine various elements,
dimensions, and stages of the information search and evaluation process. erefore, to sharpen
the instruments of IIR user studies and to improve the reliability and validity of the associated
conclusions, it is critical to develop a comprehensive, generalizable framework for characterizing,
reporting, evaluating IIR user studies.
To this end, we developed a faceted framework for user study evaluation based on a system-
atic review and coding analysis of 462 state-of-the-art IIR research papers recently published in
several top venues, including ACM SIGIR conferences (2000-2018), CHIIR conferences (2016-
2018), Journal of the Association for Information Science and Technology (JASIST), Information Pro-
cessing and Management (IP&M), CHI conferences, and ACM Transactions on Information Systems
(ACM TOIS). Our paper reviewing and coding procedure was introduced in Chapter 3. Built upon
the systematic user study review, In Chapters 4 and 5, we illustrated the main facets and factors
of the framework with specic user study examples and also explained in detail how our frame-
work can shed lights on the evaluation of IIR user studies of dierent types. e proposed faceted
framework can oer a relatively comprehensive picture of the facets of state-of-the-art IIR user
studies and help researchers understand the underlying connections among dierent facets when
evaluating the decisions and compromises made in user study design.
Compared to the existing works on the summarization and evaluation of IR user studies
(e.g., Kelly, 2009; Sakai, 2016), our work includes new facets and factors extracted from recently
published IR literature and highlights the importance of understanding the connections and “col-
laborations” between dierent facets and factors (especially the factors under dierent facet cate-
gories) for characterizing and evaluating IIR user studies. In addition, we use several representative
user studies as examples and illustrate how the components and connections embedded in a user
study can be deconstructed and analyzed in the light of divergent research questions and contexts.
Despite the strengths and implications of the faceted evaluation framework discussed in the
previous chapters, it still has limitations in several aspects (e.g., the scope of paper reviewing in our
work, the qualitative nature of paper coding, current reporting practice in IIR community). e
limitations are explained in detail in Chapter 6. We believe that these limitations can potentially
58 7. CONCLUSION AND FUTURE DIRECTIONS
create new opportunities for researchers in the IR community to further improve IIR study design,
reporting, and evaluation practices.
With respect to future directions, there are several ways to improve our faceted scheme and
to further enhance our understanding of the roles and potential of IIR user studies. For instance, fu-
ture research can expand the scope of user study review and include more relevant venues for paper
collection and facet extraction, such as ASIS&T, CIKM, JCDL, and so on. By doing so, researchers
may be able to identify new subfacets and factors (e.g., new system and/or interface features and
search assistant tools, new data collection techniques and associated features, new models for data
analysis) and thereby can improve the current version of faceted evaluation framework.
Besides, it is also important to develop new scales, measures, and indicators to quantify the
faceted evaluation framework and to improve the generalizability of the framework in evaluating a
wide range of IIR user studies. is quantication should include both the measurements of dier-
ent facets and dimensions (e.g., how many widely-ignored, under-reported subfacets are discussed,
which critical statistics are reported) and the quantication of baselines (e.g., a series of interrelated,
quantied user study facet measures) for user study evaluation. Due to the fundamental dierences
among dierent research focuses and problems, researchers should employ dierent baselines when
evaluating dierent types of IIR user studies. In addition, as it was discussed in Chapter 6, estab-
lishing new platforms (e.g., paper submission tracks, workshops) for reporting replication studies
and unexpected results can also deepen our understanding of the potential eects of dierent facets
and dimensions (e.g., task facets, participant or user characteristics, system or interface features) on
the results of user studies.
As Robertson (2008) points out, “a eld advances not by deciding on a single best compro-
mise, but through dierent researchers taking dierent decisions, and the resulting dialectic” (p.
447). In the past two decades, we have witnessed the diversity in user study design manipulations
and decisions (e.g., participant recruitment, task design, study device, study procedure, type of data
collected) which has largely supported the explorations of users’ interactions with search systems in
context and signicantly expanded the territory of IIR research. By carefully examining and evalu-
ating IIR user studies via the faceted approach, our hope is that the work presented here could add
more knowledge about the structure and the related impacts of the “decision in compromise” and
better facilitate the dialectic around IIR user study design, reporting, and evaluation.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.