Access the full text.
Sign up today, get DeepDyve free for 14 days.
Loading next page...
/lp/emerald-publishing/exploring-fact-focused-relevance-and-novelty-detection-xAEeFPZhfG
References (39)
-
H. Halteren, Simone Teufel (2003)
Examining the consensus between human summaries: initial experiments with factoid analysis -
James Allan (2002)
Topic detection and tracking: event-based information organization -
Donald Bren, E. Tchetgen (2022)
Nonparametric Statistics for the Behavioral SciencesThe SAGE Encyclopedia of Research Design
-
Barbara Eugenio (2000)
On the Usage of Kappa to Evaluate Agreement on Coding Tasks -
James Allan, C. Wade, A. Bolivar (2003)
Retrieval and novelty detection at the sentence levelProceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval
-
J. Carletta (1996)
Assessing Agreement on Classification Tasks: The Kappa StatisticComput. Linguistics, 22
-
A. Nenkova, R. Passonneau (2004)
Evaluating Content Selection in Summarization: The Pyramid Method -
E. Voorhees (1998)
Variations in relevance judgments and the measurement of retrieval effectivenessInf. Process. Manag., 36
-
I. Soboroff, Donna Harman (2003)
Overview of the TREC 2003 Novelty Track -
J. Fiscus, G. Doddington (2002)
Topic detection and tracking evaluation overview -
James Allan, J. Carbonell, G. Doddington, J. Yamron, Yiming Yang (1998)
Topic Detection and Tracking Pilot Study Final Report -
S. Harter (1996)
Variations in Relevance Assessments and the Measurement of Retrieval EffectivenessJ. Am. Soc. Inf. Sci., 47
-
I. Soboroff, D. Harman (2005)
Novelty Detection: The TREC Experience -
M. Moens (2002)
The MIT Press Journals -
E. Voorhees, Dawn Tice (2000)
The TREC-8 Question Answering Track Evaluation, 3
-
Linda Schamber (1994)
Relevance and Information Behavior., 29
-
(1999)
Topic-based novelty detection 1999 summer workshop at Center for Language and Speech Processing final report, -
D. Harman (2002)
Overview of the TREC 2002 Novelty Track -
10.1002/(SICI)1097-4571(199709)48:9<810::AID-ASI6>3.0.CO;2-U
-
10.1002/(SICI)1097-4571(199601)47:1<37::AID-ASI4>3.0.CO;2-3
-
(1994)
Relevancy reconsidered – towards an agenda for the 21 st century : introduction to the special topic issue ” -
P. Vakkari, Eero Sormunen (2004)
The influence of relevance levels on the effectiveness of interactive information retrievalJ. Assoc. Inf. Sci. Technol., 55
-
D. Marcu, Abdessamad Echihabi (2002)
An Unsupervised Approach to Recognizing Discourse Relations -
10.1162/089120102762671936
-
K. Krippendorff (1980)
Content Analysis: An Introduction to Its Methodology -
10.1016/S0306-4573(00)00010-8
-
T. Froehlich (1994)
Relevance reconsidered—towards an agenda for the 21st century: introduction to special topic issue on relevance researchJournal of the Association for Information Science and Technology, 45
-
James Allan, Rahul Gupta (2001)
Topic Models for Summarizing Novelty -
James Allan, Ben Carterette, Joshua Lewis (2005)
When will information retrieval be "good enough"? -
C. Cuadra, R. Katter (1967)
OPENING THE BLACK BOX OF ‘RELEVANCE’Journal of Documentation, 23
-
Barry Schiffman, K. McKeown (2005)
Learning to identify new information -
J. Landis, G. Koch (1977)
The measurement of observer agreement for categorical data.Biometrics, 33 1
-
K. Krippendorff (1980)
Krippendorff, Klaus, Content Analysis: An Introduction to its Methodology . Beverly Hills, CA: Sage, 1980. -
Stefano Mizzaro (1997)
Relevance: The Whole HistoryJ. Am. Soc. Inf. Sci., 48
-
Yi Zhang, Jamie Callan, T. Minka (2002)
Novelty and redundancy detection in adaptive filtering -
Daniel Jones (2004)
Words in the mind: An introduction to the mental lexiconMachine Translation, 4
-
Enrique Amigó, Julio Gonzalo, V. Peinado, Anselmo Peñas, M. Verdejo (2004)
An Empirical Study of Information Synthesis Task -
Jade Goldstein-Stewart, M. Kantrowitz, Vibhu Mittal, J. Carbonell (1999)
Summarizing text documents: sentence selection and evaluation metrics -
Jacob Cohen (1960)
A Coefficient of Agreement for Nominal ScalesEducational and Psychological Measurement, 20
- Publisher
- Emerald Publishing
- Copyright
- Copyright © 2008 Emerald Group Publishing Limited. All rights reserved.
- ISSN
- 0022-0418
- DOI
- 10.1108/00220410810884057
- Publisher site
- See Article on Publisher Site
Abstract
Purpose – Automated sentence‐level relevance and novelty detection would be of direct benefit to many information retrieval systems. However, the low level of agreement between human judges performing the task is an issue of concern. In previous approaches, annotators were asked to identify sentences in a document set that are relevant to a given topic, and then to eliminate sentences that do not provide novel information. This paper aims to explore a new approach in which relevance and novelty judgments are made within the context of specific, factual information needs, rather than with respect to a broad topic. Design/methodology/approach – An experiment is conducted in which annotators perform the novelty detection task in both the topic‐focused and fact‐focused settings. Findings – Higher levels of agreement between judges are found on the task of identifying relevant sentences in the fact‐focused approach. However, the new approach does not improve agreement on novelty judgments. Originality/value – The analysis confirms the intuition that making sentence‐level relevance judgments is likely to be the more difficult of the two tasks in the novelty detection framework.
Journal
Journal of Documentation – Emerald Publishing
Published: Jul 25, 2008
Keywords: Information retrieval; Text retrieval; Information searches; Semantics