Qual Quant (2013) 47:553–559
Survey error in an international context: an empirical
assessment of cross-cultural differences regarding
Hilde Tobi · Jarl K. Kampen
Published online: 17 March 2011
© Springer Science+Business Media B.V. 2011
Abstract Likert-type of questions are prone to differences in response styles. The aim of
the present study is to examine possible cross-cultural differences regarding scale effects
resulting from different answer categories. This study aims to extend the literature on scale
effects to a foreign language. An English questionnaire containing two experimental ques-
tions was presented to students enrolled in international MSc programs. In addition to the
experimental questions, the questionnaire contained ﬁller questions, demographic questions,
and questions on English proﬁciency. The questionnaires differed only from each other with
respect to answer categories and scale presented with the experimental questions. The ques-
tionnaire was presented during class, to 526 students coming from over 70 different countries.
These countries were classiﬁed according to continent and language into 10 groups of which
nine were large enough to be included in the analysis. After considering possible confounding
factors, like age, gender, English language proﬁciency, and disciplinary background, scale
effects were shown to be present across groups of countries. These differences seriously
affect rankings. Concludingly, scale effects persist in non-native language questionnaires,
and the effect sizes differ across cultures.
Keywords Scale effect · Cross-cultural differences · International education
It is well-established in the scientiﬁc literature that the rating of Likert scales is prone to cross-
cultural differences in response style (Van Herk et al. 2004; Baumgartner and Steenkamp
2001; Johnson et al. 2005; De Jong et al. 2008; Smith and Fischer 2008; Van Dijk et al.
2009). These differences in response style refer to differences in the provision of extreme
responses, and the tendency to provide a positive or favorable response.
H. Tobi (
) · J. K. Kampen
Wageningen University and Research Centre, Research Methodology Group, Hollandseweg 1,
6706 KN Wageningen, The Netherlands