Access the full text.
Sign up today, get DeepDyve free for 14 days.
P. Vakkari, Nanna Hakala (2000)
Changes in relevance criteria and problem stages in task performanceJ. Documentation, 56
I. Ruthven (2005)
Integrating approaches to relevance
A. Turpin, W. Hersh (2001)
Why batch and user evaluations do not give the same results
Catherine Smith, P. Kantor (2008)
User adaptation: good results from poor systems
Mark Smucker, Chandra Jethani (2010)
Human performance and retrieval precision revisitedProceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval
Mu-Hsuan Huang, Hui-yu Wang (2004)
The influence of document presentation order and number of documents judged on users' judgments of relevanceJ. Assoc. Inf. Sci. Technol., 55
E. Voorhees (2001)
Evaluation by highly relevant documents
V. Florance, G. Marchionini (1995)
Information processing in the context of medical care
T. Saracevic, P. Kantor (1988)
A study of information seeking and retrieving. II. Users, questions, and effectivenessJ. Am. Soc. Inf. Sci., 39
N. Ford, David Miller, N. Moss (2001)
The role of individual differences in Internet searching: An empirical studyJ. Assoc. Inf. Sci. Technol., 52
E. Voorhees (1998)
Variations in relevance judgments and the measurement of retrieval effectivenessInf. Process. Manag., 36
Loraine Lau-Gesk, Joan Meyers-Levy (2009)
Emotional Persuasion: When the Valence versus the Resource Demands of Emotions Influence Consumers’ AttitudesJournal of Consumer Research, 36
J. Gwizdka, Irene Lopatovska (2009)
The role of subjective factors in the information search processJ. Assoc. Inf. Sci. Technol., 60
S. Harter (1996)
Variations in Relevance Assessments and the Measurement of Retrieval EffectivenessJ. Am. Soc. Inf. Sci., 47
Falk Scholer, A. Turpin, M. Sanderson (2011)
Quantifying test collection quality based on the consistency of relevance judgementsProceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval
I. Ruthven, Mark Baillie, David Elsweiler (2006)
The Relative Effects of Knowledge, Interest and Confidence in Assessing Relevance
H. Chu (2011)
Factors affecting relevance judgment: a report from TREC Legal trackJ. Documentation, 67
E. Abels, D. Nitecki (1976)
Relevance : A Review of the Literature and a Framework for Thinking on the Notion in Information Science . Part II : Nature and Manifestations of Relevance *
(2007b)
Relevance: a review of the literature and a framework for thinking on the notion in information science. Part III: behavior and effects of relevanceJournal of the American Society for Information Science and Technology, 58
(2003)
Paying attention to emotion: an fmri investigation of cognitive and emotional stroop tasks”, Cognitive, Affective
D. Kelly, Xin Fu, C. Shah (2010)
Effects of position and number of relevant documents retrieved on users' evaluations of system performanceACM Trans. Inf. Syst., 28
Youngok Choi, E. Rasmussen (2002)
Users' relevance criteria in image retrieval in American historyInf. Process. Manag., 38
A. Spink, Howard Greisdorf, Judy Bateman (1998)
From Highly Relevant to Not Relevant: Examining Different Regions of RelevanceInf. Process. Manag., 34
, (2004)
The influence of document presentation order and number of documents judged on users’ judgements of relevanceJournal of the American Society for Information Science and Technology, 55
Alissa Sherry, R. Henson (2005)
Conducting and Interpreting Canonical Correlation Analysis in Personality Research: A User-Friendly PrimerJournal of Personality Assessment, 84
Eero Sormunen (2002)
Liberal relevance criteria of TREC -: counting on negligible documents?
N. Ford (1999)
The Growth of Understanding in Information Science: Towards a Developmental ModelJ. Am. Soc. Inf. Sci., 50
Karen Jones (2006)
What's the value of TREC: is there a gap to jump or a chasm to bridge?SIGIR Forum, 40
, , (2001)
The role of individual differences in internet searching: an empirical studyJournal of the American Society for Information Science and Technology, 52
Michael Eisenberg, Carol Barry (1988)
Order Effects: A Study of the Possible Influence of Presentation Order on User Judgments of Document Relevance.Journal of the Association for Information Science and Technology, 39
Jimmy Lin, Pengyi Zhang (2007)
Deconstructing nuggets: the stability and reliability of complex question answering evaluation
, (2009)
The role of subjective factors in the information search processJournal of the American Society for Information Science and Technology, 60
(1996)
Relevance reconsidered. Information science: Integration in perspectives
Michael Eisenberg, Carol Barry (1988)
Order effects: A study of the possible influence of presentation order on user judgments of document relevanceJ. Am. Soc. Inf. Sci., 39
James Allan (2003)
HARD Track Overview in TREC 2003: High Accuracy Retrieval from Documents
J. González, Jaime Gómez (2007)
TREC: Experiment and evaluation in information retrievalJ. Assoc. Inf. Sci. Technol., 58
(2005)
TREC: Experiment and Evaluation
(2003)
Paying attention to emotion : an fmri investigation of cognitive and emotional stroop tasks ”
Judy Bateman (1998)
Changes in Relevance Criteria: A Longitudinal Study., 35
Carol Barry, Linda Schamber (1998)
Users' Criteria for Relevance Evaluation: A Cross-situational ComparisonInf. Process. Manag., 34
(2001)
Web searching behavior: Selection of search terms
Mark Smucker, Chandra Jethani (2011)
Measuring assessor accuracy: a comparison of nist assessors and user study participantsProceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval
(1996)
Information science: Integration in perspectives
T. Saracevic (2007)
Relevance: A review of the literature and a framework for thinking on the notion in information science. Part III: Behavior and effects of relevanceJ. Assoc. Inf. Sci. Technol., 58
– The purpose of this paper is to examine how various types of TREC data can be used to better understand relevance and serve as test-bed for exploring relevance. The author proposes that there are many interesting studies that can be performed on the TREC data collections that are not directly related to evaluating systems but to learning more about human judgements of information and relevance and that these studies can provide useful research questions for other types of investigation. Design/methodology/approach– Through several case studies the author shows how existing data from TREC can be used to learn more about the factors that may affect relevance judgements and interactive search decisions and answer new research questions for exploring relevance. Findings– The paper uncovers factors, such as familiarity, interest and strictness of relevance criteria, that affect the nature of relevance assessments within TREC, contrasting these against findings from user studies of relevance. Research limitations/implications– The research only considers certain uses of TREC data and assessment given by professional relevance assessors but motivates further exploration of the TREC data so that the research community can further exploit the effort involved in the construction of TREC test collections. Originality/value– The paper presents an original viewpoint on relevance investigations and TREC itself by motivating TREC as a source of inspiration on understanding relevance rather than purely as a source of evaluation material.
Journal of Documentation – Emerald Publishing
Published: Oct 7, 2014
Keywords: Relevance behaviour; TREC; Test collections
Read and print from thousands of top scholarly journals.
Already have an account? Log in
Bookmark this article. You can see your Bookmarks on your DeepDyve Library.
To save an article, log in first, or sign up for a DeepDyve account if you don’t already have one.
Copy and paste the desired citation format or use the link below to download a file formatted for EndNote
Access the full text.
Sign up today, get DeepDyve free for 14 days.
All DeepDyve websites use cookies to improve your online experience. They were placed on your computer when you launched this website. You can change your cookie settings through your browser.