Qual Quant (2014) 48:2225–2240
DOI 10.1007/s11135-013-9888-3
A fuzzy set theory based computational model
to represent the quality of inter-rater agreement
Enrico Ciavolino · Sergio Salvatore · Antonio Calcagnì
Published online: 6 July 2013
© Springer Science+Business Media Dordrecht 2013
Abstract In this paper we present a method to evaluate the quality of a rater’s judgement,
which can integrate and enrich the use of inter-rater agreement as a reliability measure. Our
proposal is an integrative one and evaluates the quality of a rater’s performance through an
analysis of the profile of that individual rater’s performance. We discuss its rationale on the
basis of the interpretation of inter-rater agreement, highlighting some critical issues. For this
purpose, we adopt a computational model based on fuzzy set theory, demonstrating its main
characteristics with an exemplary case study.
Keywords Inter-rater agreement · Fuzzy set theory based models ·
Performance measurement · Quality measurement
1 Introduction
In the field of psychology, and in the social sciences generally, rating scales (e.g., PQS) and
coding systems (e.g.: the GMI, Auletta et al. 2012; for the social science see: Marradi 1981;
Corbetta 1999) are widely used. Such measures are generally based on evaluations performed
by independent competent raters. As a consequence of such an involvement, these measures
have to be shown not to suffer from the reliability problems potentially associated with any
human judgement (Gigerenzer and Todd 1999). To this end, researchers are accustomed
E. Ciavolino (
B
) · S. Salvatore
Department of History, Society and Human Studies, University of Salento,
Palazzo Parlangeli, Via V.M. Stampacchia, 45, 73100 Lecce, Italy
e-mail: enrico.ciavolino@unisalento.it
S. Salvatore
e-mail: sergio.salvatore@unisalento.it
A. Calcagnì
Department of Psychology and Cognitive Science, University of Trento,
Corso Bettini, 84, 38068 Rovereto, TN, Italy
e-mail: antonio.calcagni@unitn.it
123
@Phil_Robichaud
@deepthiw
@JoseServera