Copyright © 2019 by Morgan and Claypool
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in
any form or by any means—electronic, mechanical, photocopy, recording, or any other except for brief quota-
tions in printed reviews, without the prior permission of the publisher.
Interactive IR User Study Design, Evaluation, and Reporting
Jiqun Liu and Chirag Shah
www.morganclaypool.com
ISBN: 9781681735795 Paperback
ISBN: 9781681735801 eBook
ISBN: 9781681735818 Hardcover
DOI 10.2200/S00923ED1V01Y201905ICR067
A Publication in the Morgan and Claypool Publishers series
SYNTHESIS LECTURES ON INFORMATION CONCEPTS, RETRIEVAL, AND SERVICES
Lecture #67
Series Editor: Gary Marchionini, University of North Carolina, Chapel Hill
Series ISSN 1947-945X Print 1947-9468 Electronic
Interactive IR User Study Design,
Evaluation, and Reporting
Jiqun Liu and Chirag Shah
Rutgers University
SYNTHESIS LECTURES ON INFORMATION CONCEPTS, RETRIEVAL,
AND SERVICES #67
M
&C
MORGAN
&
CLAYPOOL PUBLISHERS
xii
ABSTRACT
Since user study design has been widely applied in search interactions and information retrieval
(IR) systems evaluation studies, a deep reection and meta-evaluation of interactive IR (IIR) user
studies is critical for sharpening the instruments of IIR research and improving the reliability and
validity of the conclusions drawn from IIR user studies. To this end, we developed a faceted frame-
work for supporting user study design, reporting, and evaluation based on a systematic review of the
state-of-the-art IIR research papers recently published in several top IR venues (n=462). Within
the framework, we identify three major types of research focuses, extract and summarize facet values
from specic cases, and highlight the under-reported user study components which may signi-
cantly aect the results of research. en, we employ the faceted framework in evaluating a series
of IIR user studies against their respective research questions and explain the roles and impacts of
the underlying connections and “collaborations” among dierent facet values. rough bridging
diverse combinations of facet values with the study design decisions made for addressing research
problems, the faceted framework can shed light on IIR user study design, reporting, and evaluation
practices and help students and young researchers design and assess their own studies.
KEYWORDS
interactive information retrieval, user studies, faceted approach, evaluation, controlled laboratory
study, eld study, human-information interaction, information seeking and retrieval, user study
reporting practice
xiii
Contents
Preface ............................................................ xv
Acknowledgments .................................................. xvii
1 Introduction ....................................................... 1
1.1 Background: User Study in IIR Research ............................ 2
1.2 Aim of the Work ............................................... 2
1.3 Systematic Review and User Study Evaluation ........................ 3
1.4 Implications of the Faceted Framework .............................. 4
2 Interactive Information Retrieval ...................................... 7
2.1 Background ................................................... 7
2.2 Understanding User Behavior and Experience ....................... 10
2.3 System/Interface Features Evaluation .............................. 12
2.4 Meta-Evaluation of Evaluation Metrics ............................ 13
2.5 Summary .................................................... 14
3 Methodology: Paper Selection and Coding Scheme ....................... 15
3.1 Paper Selection: SIGIR, CHIIR, CHI, JASIST, IP&M, and TOIS ....... 17
3.2 Coding Scheme ............................................... 20
4 Faceted Framework of IIR User Studies ................................. 23
4.1 Structure of the Faceted Framework ............................... 23
4.1.1 Research Focus and Variables .............................. 25
4.1.2 Participant and Task ..................................... 27
4.1.3 System Feature ......................................... 31
4.1.4 Study Procedure and Experimental Design ................... 32
4.1.5 User Behavior, Experience, Data Analysis, and Results .......... 34
4.1.6 Example of Faceted Annotation ............................ 35
4.2 Under-Reported Facets and Factors ............................... 37
4.3 Summary .................................................... 40
5 Evaluating IIR User Studies of Dierent Types ........................... 41
5.1 Understanding User Behavior and Experience ....................... 42
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.