53
expand the research problem space in which the user study approach as an enriched, useful meth-
odology can be applied and evaluated in meaningful ways.
6.2 IMPLICATIONS FOR IIR USER STUDY REPORTING
PRACTICES
In addition to the aforementioned implications for study design and evaluation, the idea of examin-
ing, connecting, and organizing facets can also oer insights for IIR user study reporting practices.
Report the design choices in dierent facets and their roles: Reporting practice is the basis of
user study evaluation. Our faceted evaluation framework was constructed based upon a series of
state-of-the-art IIR user studies reported in the research papers recently published in several top
venues. Also, to perform a comprehensive faceted evaluation, researchers need to obtain enough
details regarding user study design from published research papers and reports. Poor reporting
practices in published papers can cause huge issues for meta-evaluations (Kelly and Sugimoto,
2013; O’Brien and McCay-Peet, 2017; Sakai, 2016), not to mention the diculty in interpreting
the study results and implications. To clarify the structure of a user study and the roles of dierent
components as well as the connections among them, researchers need to consider and discuss the
three aforementioned questions associated with the design and manipulation in facets. By doing so,
future researchers can better understand the rationales of the specic design choices and evaluate
these choices based on the specic research questions and problem spaces.
Report the commonly under-reported facets and factors. e under-reported facets and
factors identied in the previous sections need to be reported in user study papers (especially
when these facets are directly related to the research questions and study design compromises)
as they may oer important cues for interpreting the variations in results. For example, when the
research goal is measuring participants’ knowledge gain and search performance in a controlled
experimental setting, it is critical to also examine the possible eects of language (e.g., participants’
native language, language used in presenting study materials), topic-independent search skills, and
participants’ motivations to participate (e.g., the structure of incentives). With respect to the task
facet, to fully understand the eects of search task facets (e.g., task goal, task product, task stage,
task complexity and prior determinability), researchers need to rst investigate the overarching
work tasks or problematic situations and their associations with the search tasks dened in the user
study (Belkin, 2008). Examining and reporting the critical under-reported facets can help expand
the research in IR to take a broader perspective of the information seeking and search processes and
to include more user characteristics and contextual features. As a result, we may be able to obtain
a deeper, more comprehensive understanding on the dynamic associations between user character-
istics, behavioral patterns, and contextual factors identied in the past user studies (e.g., Jiang, He,
and Allan, 2017; Liu et al., 2010, 2019).
6.2 IMPLICATIONS FOR INTERACTIVE IR USER STUDY REPORTING PRACTICES
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.