Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Minimal test collections for low-cost evaluation of Audio Music Similarity and Retrieval systems

Minimal test collections for low-cost evaluation of Audio Music Similarity and Retrieval systems Reliable evaluation of Information Retrieval systems requires large amounts of relevance judgments. Making these annotations is not only tedious but also complex for many Music Information Retrieval tasks. As a result, performing such evaluations usually requires too much effort. A low-cost alternative is the application of Minimal Test Collections algorithms, which offer very reliable results while significantly reducing the required annotation effort. The idea is to represent effectiveness scores as random variables that can be estimated, iteratively selecting which documents to judge so that we can compute accurate estimates with a certain degree of confidence and with the least effort. In this paper we show the application of Minimal Test Collections to the evaluation of the Audio Music Similarity and Retrieval task, run by the annual MIREX evaluation campaign. An analysis with the MIREX 2007, 2009, 2010 and 2011 data shows that with as little as 2 % of the total judgments we can obtain accurate estimates of the ranking of systems. We also present a method to rank systems without making any annotations, which can be successfully used when little or no resources are available. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png International Journal of Multimedia Information Retrieval Springer Journals

Minimal test collections for low-cost evaluation of Audio Music Similarity and Retrieval systems

Download PDF
12 pages

Loading next page...
 
/lp/springer-journals/minimal-test-collections-for-low-cost-evaluation-of-audio-music-2lhnAAP0Ph
Publisher
Springer Journals
Copyright
Copyright © 2012 by Springer-Verlag London
Subject
Computer Science; Multimedia Information Systems; Information Storage and Retrieval; Information Systems Applications (incl. Internet); Data Mining and Knowledge Discovery; Image Processing and Computer Vision; Computer Science, general
ISSN
2192-6611
eISSN
2192-662X
DOI
10.1007/s13735-012-0030-4
Publisher site
See Article on Publisher Site

Abstract

Reliable evaluation of Information Retrieval systems requires large amounts of relevance judgments. Making these annotations is not only tedious but also complex for many Music Information Retrieval tasks. As a result, performing such evaluations usually requires too much effort. A low-cost alternative is the application of Minimal Test Collections algorithms, which offer very reliable results while significantly reducing the required annotation effort. The idea is to represent effectiveness scores as random variables that can be estimated, iteratively selecting which documents to judge so that we can compute accurate estimates with a certain degree of confidence and with the least effort. In this paper we show the application of Minimal Test Collections to the evaluation of the Audio Music Similarity and Retrieval task, run by the annual MIREX evaluation campaign. An analysis with the MIREX 2007, 2009, 2010 and 2011 data shows that with as little as 2 % of the total judgments we can obtain accurate estimates of the ranking of systems. We also present a method to rank systems without making any annotations, which can be successfully used when little or no resources are available.

Journal

International Journal of Multimedia Information Retrieval – Springer Journals

Published: Jan 1, 2013

References

  • The scientific evaluation of music information retrieval systems: foundations and future
    Downie, JS
  • Effects of album and artist filters in audio similarity computed for very large music databases
    Flexer, A; Schnitzer, D
  • Information retrieval evaluation
    Harman, DK
  • The analysis of ordered categorical data: an overview and a survey of recent developments
    Liu, I; Agresti, A
  • Variations in relevance judgments and the measurement of retrieval effectiveness
    Voorhees, EM
  • The VGAM package for categorical data analysis
    Yee, T
  • Vector generalized additive models
    Yee, T; Wild, C