34 4. FACETED FRAMEWORK OF INTERACTIVE IR USER STUDIES
4.1.5 USER BEHAVIOR, EXPERIENCE, DATA ANALYSIS, AND RESULTS
To capture and evaluate the products of user studies, researchers have developed a variety of user
behavioral, performance, and experience measures, including both online measures collected within
search sessions and oine measures consisting of user annotations, document relevance and use-
fulness judgments, and other self-reported metrics (e.g., NASA Task Load Index, user engagement
indicators). To extract useful insights and actionable implications from the empirical evidence
emerged from user studies, researchers also have to feed raw data into mathematical models and
wait for the results on the output side. As is illustrated in Figures 4.1 and 4.2, the adoption of
behavioral measures and analytical models is often determined based upon the research problems
identied as well as the variables and associated main facets dened in user study design.
e metric of user behaviors and interaction experiences (e.g., satisfaction, cognitive load,
frustration) is a fundamental aspect of the user studies that aim at understanding users and applying
user-centered approach in search systems design and evaluation. With respect to user behaviors,
past user studies have explored various possible approaches to measuring multiple components of
information seeking and search episodes, such as querying (e.g., query formulation and re-formula-
tion, use of query suggestion, query diversity and complexity), browsing (e.g., dwell time on SERP
and landing/content pages), and result examination behavior (e.g., clicks, scrolls, mouse hovering).
In addition, researchers also employed oine user judgments (e.g., usefulness and relevance judg-
ment), annotations, and performance measures (e.g., average precision, normalized discounted
cumulative gain in search result examination) and evaluated the usefulness of these metrics in IR
system evaluation against a series of user-centered ground truth measures (e.g., satisfaction). Chen
et al. (2017) provides a summary of existing online and oine behavioral metrics and compared the
usefulness and eectiveness of these distinct measures in a user-centered system evaluation. Aside
from the explicit search behaviors, performances, and user annotations, some researchers have also
explored the possibility of measuring users’ cognitive and aective variations and examined the ex-
tent to which neuro-physiological measures of dierent types (e.g., eye movement, pupil size, heart
rate, skin conductance rate, EEG and fMRI signals) are associated with traditional behavioral and
experience measures (e.g., Cole et al., 2013; Edwards and Kelly, 2017; Eickho, Dungs, and Tran,
2015; Eugster et al., 2014; Gwizdka and Zhang, 2015; Moshfeghi, Triantallou, and Pollick, 2016).
In IIR user studies, the behavioral and interaction experience measures discussed above have served
as useful sensors for detecting the features and behavioral patterns of users’ search interactions.
Combining these “behavioral pixels” in modeling, we can obtain a relatively comprehensive picture
about how users’ interaction with search systems in tasks of dierent types and how they assess the
eectiveness of the system as well as their own search experience in dierent contexts.
In addition to data collection and measures, data analyses and result presentation methods
are also determined by the research questions proposed at the beginning of many user studies.
Given the nature of the research problems and goals (e.g., examining the correlation between
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.