Summary of the SIGIR 2003 Workshop on Defining Evaluation Methodologies for Terabyte-Scale Test Collections Ian Soboroff and Ellen Voorhees National Institute of Standards and Technology Gaithersburg, MD ian .soboroff@nist .gov Nick .CraswelAcsiro .au Nick Craswell CSIRO Canberra, ACT, Australia Introduction Early retrieval test collections were small, allowing relevance judgments to be based on an exhaustive examination of the documents, but limiting the general applicability of the findings . Karen Sparck Jones and Keith van Rijsbergen proposed a way of building significantly larger test collections by using pooling, a procedure adopted and subsequently validated by TREC . Now TREC-sized collections (several gigabytes of text and a few million documents) are small for some realistic tasks, but current po$ling practices do not scale to substantially larger document sets . This article summarizes a workshop held at SIGIR 2003-in Toronto, Canada, the goal of which was to develop an evaluation methodology for terabyte-scale document collections . The outcome of the workshop was a proposal for a new TREC track to investigate ad hoc retrieval on a collection of 100M web pages. In particular, we began by assuming the existence of a collection of several hundred million web pages, and discussed methods
End of preview. The entire article is 4 pages. To view the full-text, please rent this article to continue.
/lp/association-for-computing-machinery/summary-of-the-sigir-2003-workshop-on-defining-evaluation-CFt8w10p4m
Welcome to DeepDyve! Rent Premier Research Articles and Save Up to 90%
Summary of the SIGIR 2003 workshop on defining evaluation methodologies for terabyte-scale test collections
Soboroff, Ian; Voorhees, Ellen; Craswell, Nick
Follow ACM SIGIR Forum
, Volume 37 (2)
Association for Computing Machinery – Sep 1, 2003
More Info
More Like This Article
View All dataSource[]=actageo&dataSource[]=aspet&dataSource[]=aaos&dataSource[]=aacc&dataSource[]=aacr&dataSource[]=aea&dataSource[]=aip&dataSource[]=ajnr&dataSource[]=ams&dataSource[]=aps_physical&dataSource[]=appi_book&dataSource[]=appi_journal&dataSource[]=apha&dataSource[]=asip&dataSource[]=asm&dataSource[]=asn&dataSource[]=aspb&dataSource[]=avs&dataSource[]=annual_reviews&dataSource[]=arxiv&dataSource[]=acm&dataSource[]=berghahn&dataSource[]=cabi&dataSource[]=clinical_trials&dataSource[]=dailymed&dataSource[]=degruyter&dataSource[]=du_press&dataSource[]=esa&dataSource[]=eu_press&dataSource[]=elsevier&dataSource[]=emerald&dataSource[]=ejtr&dataSource[]=emea&dataSource[]=epo&dataSource[]=faseb&dataSource[]=gsa&dataSource[]=health_affairs&dataSource[]=hindawi&dataSource[]=imanager&dataSource[]=imedpub&dataSource[]=informa_healthcare&dataSource[]=informs&dataSource[]=iop&dataSource[]=iucr&dataSource[]=iospress&dataSource[]=jbjs&dataSource[]=leftcoast&dataSource[]=lu_press&dataSource[]=mesharpe&dataSource[]=mary_ann_liebert&dataSource[]=medline&dataSource[]=mit_press&dataSource[]=nature&dataSource[]=oxford&dataSource[]=pier_professional&dataSource[]=pnas&dataSource[]=portlandpress&dataSource[]=psyc_articles&dataSource[]=psyc_books&dataSource[]=psyc_critiques&dataSource[]=plos_journal&dataSource[]=pubmed_central&dataSource[]=rsna&dataSource[]=rockefeller&dataSource[]=rcn&dataSource[]=ria&dataSource[]=rsc&dataSource[]=sage&dataSource[]=spie&dataSource[]=springer_journal&dataSource[]=springer&dataSource[]=taylor_francis&dataSource[]=aps&dataSource[]=the_scientist&dataSource[]=uc_press&dataSource[]=uspto_abstract&dataSource[]=wiley&dataSource[]=pct
© 2012 DeepDyve, Inc. All rights reserved.
Terms of Service | Privacy Policy
