Mach Learn (2018) 107:675–702
Robust Plackett–Luce model for k-ary crowdsourced
· Yuangang Pan
· Ivor W. Tsang
Received: 31 March 2017 / Accepted: 16 September 2017 / Published online: 25 October 2017
© The Author(s) 2017
Abstract The aggregation of k-ary preferences is an emerging ranking problem, which plays
an important role in several aspects of our daily life, such as ordinal peer grading and online
product recommendation. At the same time, crowdsourcing has become a trendy way to
provide a plethora of k-ary preferences for this ranking problem, due to convenient platforms
and low costs. However, k-ary preferences from crowdsourced workers are often noisy, which
inevitably degenerates the performance of traditional aggregation models. To address this
challenge, in this paper, we present a RObust PlAckett–Luce (ROPAL) model. Speciﬁcally,
to ensure the robustness, ROPAL integrates the Plackett–Luce model with a denoising vector.
Based on the Kendall-tau distance, this vector corrects k-ary crowdsourced preferences with
a certain probability. In addition, we propose an online Bayesian inference to make ROPAL
scalable to large-scale preferences. We conduct comprehensive experiments on simulated and
real-world datasets. Empirical results on “massive synthetic” and “real-world” datasets show
that ROPAL with online Bayesian inference achieves substantial improvements in robustness
and noisy worker detection over current approaches.
Keywords Ranking · k-Ary crowdsourced preferences · Robust Plackett–Luce model ·
Online Bayesian inference
Bo Han and Yuangang Pan have contributed equally to this work.
Editors: Wee Sun Lee and Robert Durrant.
Ivor W. Tsang
Yua ng an g P a n
Centre for Artiﬁcial Intelligence (CAI), University of Technology Sydney, Sydney, Australia