Series ISSN: 1947-945X
Interactive IR
User Study Design,
Evaluation, and
Reporting
Jiqun Liu
Chirag Shah
Series Editor: Gary Marchionini, University of North Carolina at Chapel Hill
Interactive IR User Study Design, Evaluation, and Reporting
Jiqun Liu, Rutgers University
Chirag Shah, Rutgers University
Since user study design has been widely applied in search interactions and information retrieval
(IR) systems evaluation studies, a deep reection and meta-evaluation of interactive IR (IIR)
user studies is critical for sharpening the instruments of IIR research and improving the
reliability and validity of the conclusions drawn from IIR user studies. To this end, we developed
a faceted framework for supporting user study design, reporting, and evaluation based on a
systematic review of the state-of-the-art IIR research papers recently published in several top
IR venues (n=462). Within the framework, we identify three major types of research focuses,
extract and summarize facet values from specic cases, and highlight the under-reported user
study components which may signicantly aect the results of research. en, we employ the
faceted framework in evaluating a series of IIR user studies against their respective research
questions and explain the roles and impacts of the underlying connections and “collaborations”
among dierent facet values. rough bridging diverse combinations of facet values with the
study design decisions made for addressing research problems, the faceted framework can shed
light on IIR user study design, reporting, and evaluation practices and help students and young
researchers design and assess their own studies.
store.morganclaypool.com
About SYNTHESIS
This volume is a printed version of a work that appears in the Synthesis
Digital Library of Engineering and Computer Science. Synthesis
books provide concise, original presentations of important research and
development topics, published quickly, in digital and print formats.
INTERACTIVE IR USER STUDY DESIGN, EVALUATION, AND REPORTING
Series ISSN: 1947-945X
LIU • SHAH
MORGAN & CLAYPOOL
Interactive IR User Study Design,
Evaluation, and
Reporting
iii
Synthesis Lectures on Information
Concepts, Retrieval, and Services
Editor
Gary Marchionini, University of North Carolina at Chapel Hill
Synthesis Lectures on Information Concepts, Retrieval, and Services publishes short books on
topics pertaining to information science and applications of technology to information discovery,
production, distribution, and management. Potential topics include: data models, indexing theory
and algorithms, classication, information architecture, information economics, privacy and iden-
tity, scholarly communication, bibliometrics and webometrics, personal information management,
human information behavior, digital libraries, archives and preservation, cultural informatics, in-
formation retrieval evaluation, data fusion, relevance feedback, recommendation systems, question
answering, natural language processing for retrieval, text summarization, multimedia retrieval,
multilingual retrieval, and exploratory search.
Interactive IR User Study Design, Evaluation, and Reporting
Jiqun Liu and Chirag Shah
Predicting Information Retrieval Performance
Robert M. Losee
Framing Privacy in Digital Collections with Ethical Decision Making
Virginia Dressler
Mobile Search Behaviors: An In-depth Analysis Based on Contexts, APPs, and Devices
Dan Wu and Shaobo Liang
Images in Social Media: Categorization and Organization of Images and eir Collections
Susanne Ørnager and Haakon Lund
Exploring Context in Information Behavior: Seeker, Situation, Surroundings, and Shared Identities
Naresh Kumar Agarwal
Researching Serendipity in Digital Information Environments
Lori McCay-Peet and Elaine G. Toms
Social Monitoring for Public Health
Michael J. Paul and Mark Dredze
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.