15
CHAPTER 3
Methodology: Paper Selection and
Coding Scheme
Compared to the traditional system-focused IR research, IIR studies often appear to be rather
complicated in that they usually involve a series of dynamic user characteristics (e.g., cognitive
variation, emotional state, behavioral pattern) and rich contextual features (e.g., task facets, see Li
and Belkin, 2008; environments of search interaction, such as home and workplace; task mode, such
as mono-tasking and multi-tasking), part of which are fairly dicult to predict or control. Mean-
while, this complexity also creates sizable room and unprecedented opportunities for IR researchers
and system designers to personalize users’ search interactions (Belkin, 2008; Teevan, Dumais, and
Horvitz, 2007, 2010). As it is discussed in previous chapters, a comprehensive, faceted framework
of user study can be of help for IIR study design, reporting, and evaluation in multiple streams of
research (i.e., understanding user behavior and experience, system/interface features evaluation,
meta-evaluation of evaluation metrics).
To obtain a deeper understanding of the state-of-the-art user study practices and develop
an up-to-date faceted evaluation framework, we followed Cooper (1989)’s approach and conducted
a systematic review on recently published IIR research literature. Cooper’s systematic review ap-
proach consists of several major steps: (1) state and clarify research problems/focuses (in this case,
identify the facets of IIR studies and develop a comprehensive framework to illustrate the role of
and, more importantly, the connections between the facets); (2) develop guidelines for collecting
literature, especially the criteria for literature inclusion and exclusion; (3) develop a comprehensive
search plan for nding literature (in our case, select the most relevant venues where high-quality
IIR research are published); (4) develop code form and coding scheme for classifying and char-
acterizing collected literature; (5) code the literature; and (6) synthesize the literature. Since step
1 is already completed in the previous chapters, in this chapter, we will explain steps 2–3 in paper
selection (Section 3.1) and discuss steps 4–6 in the coding scheme (Section 3.2).
Overall, in this study, we manually examined 462 IIR user study papers published between
2000 and 2018. Specically, we focused on the following major venues: ACM SIGIR Conference
on Research and Development in Information Retrieval (SIGIR, 2000–2018, n=109), ACM
SIGIR Conference on Human Information Interaction and Retrieval (CHIIR, 2016–2018, n=48),
ACM Conference on Human Factors in Computing Systems (CHI, 2000–2018, n=126), Journal of
the Association for Information Science and Technology (JASIST, 2010–2018, n=102), Information Pro-
16 3. METHODOLOGY: PAPER SELECTION AND CODING SCHEME
cessing and Management (IP&M, 2010–2018, n=61), and ACM Transactions on Information Systems
(TOIS, 2010–2018, n=10).
We chose these venues as our sources for paper selection primarily for three reasons.
1. ey are major international venues for the publication of research papers on current
IIR research problems and advances. erefore, the IIR research published in these
venues can jointly serve as a good representation of the state-of-the-art of user study
practices.
2. ese venues are relatively strict on study design reporting practices compared to
other relevant venues. us, user study papers published in these venues are more
likely to oer suciently detailed information on study design and procedure and
hence can serve as appropriate raw materials for developing the scheme of user study.
3. Despite the similarity in broad research problems, dierent venues focus on dierent
specic topics and approaches and thus can oer diverse perspectives on IIR user
study design. Specically, for example, papers published in SIGIR usually focus more
on system-side features (e.g., algorithm, system latency) evaluation, whereas CHIIR
community is more interested in understanding user characteristics and behaviors
that are manifested in search interactions.
IIR papers presented in CHI community usually address problems related to the role of
innovative interface components (e.g., embedded tool and search assistant) in supporting the inter-
action between human and computing systems. It is worth noting that the growth of IIR research
in multiple reputable venues conrms the importance of IIR user studies, and also demonstrates
the IR community’s acceptance of including users, interacting with information, as one of the major
research focuses (Belkin, 2015).
Note that we did not extend the scope of journal paper reviewing to 2000 as we did for
all conference venues because we noticed that the majority of the user study papers published in
the three journals before 2010 (i.e., JASIST, IP&M, TOIS) reported widely adopted user study
procedures and techniques (e.g., survey, semi-structured in-depth interview) which were already
extracted from the most recent journal articles and conference papers (especially papers presented
in CHIIR). erefore, including them in paper coding would not add any main facet or lead to any
major revision of the faceted framework that emerged from the existing IIR literature. We believe
that narrowing down the time frame of journal paper reviewing in this way can help us reduce the
unnecessary workload of paper coding without losing any major factors and better focus on the
identied facets and factors that were adopted, manipulated, and discussed in a wide range of IIR
user studies.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.