Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Fast, Cheap, and Unethical? The Interplay of Morality and Methodology in Crowdsourced Survey Research

Fast, Cheap, and Unethical? The Interplay of Morality and Methodology in Crowdsourced Survey... Crowdsourcing is an increasingly popular method for researchers in the social and behavioral sciences, including experimental philosophy, to recruit survey respondents. Crowdsourcing platforms, such as Amazon’s Mechanical Turk (MTurk), have been seen as a way to produce high quality survey data both quickly and cheaply. However, in the last few years, a number of authors have claimed that the low pay rates on MTurk are morally unacceptable. In this paper, I explore some of the methodological implications for online experimental philosophy research if, in fact, typical pay practices on MTurk are morally impermissible. I argue that the most straightforward solution to this apparent moral problem—paying survey respondents more and relying only on “high reputation” respondents—will likely increase the number of subjects who have previous experience with survey materials and thus are “non-naïve” with respect to those materials. I then discuss some likely effects that this increase in experimental non-naivete will have on some aspects of the “negative” program in experimental philosophy, focusing in particular on recent debates about philosophical expertise. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Review of Philosophy and Psychology Springer Journals

Fast, Cheap, and Unethical? The Interplay of Morality and Methodology in Crowdsourced Survey Research

Haug, Matthew
Review of Philosophy and Psychology , Volume 9 (2) – Dec 8, 2017
Download PDF
17 pages

Loading next page...
 
/lp/springer_journal/fast-cheap-and-unethical-the-interplay-of-morality-and-methodology-in-ogw1sHsziC
Publisher
Springer Journals
Copyright
Copyright © 2017 by Springer Science+Business Media B.V., part of Springer Nature
Subject
Philosophy; Philosophy of Mind; Cognitive Psychology; Neurosciences; Epistemology; Developmental Psychology; Philosophy of Science
ISSN
1878-5158
eISSN
1878-5166
DOI
10.1007/s13164-017-0374-z
Publisher site
See Article on Publisher Site

Abstract

Crowdsourcing is an increasingly popular method for researchers in the social and behavioral sciences, including experimental philosophy, to recruit survey respondents. Crowdsourcing platforms, such as Amazon’s Mechanical Turk (MTurk), have been seen as a way to produce high quality survey data both quickly and cheaply. However, in the last few years, a number of authors have claimed that the low pay rates on MTurk are morally unacceptable. In this paper, I explore some of the methodological implications for online experimental philosophy research if, in fact, typical pay practices on MTurk are morally impermissible. I argue that the most straightforward solution to this apparent moral problem—paying survey respondents more and relying only on “high reputation” respondents—will likely increase the number of subjects who have previous experience with survey materials and thus are “non-naïve” with respect to those materials. I then discuss some likely effects that this increase in experimental non-naivete will have on some aspects of the “negative” program in experimental philosophy, focusing in particular on recent debates about philosophical expertise.

Journal

Review of Philosophy and Psychology – Springer Journals

Published: Dec 8, 2017

References

  • Comparisons of online recruitment strategies for convenience samples: Craigslist, Google AdWords, Facebook, and Amazon mechanical Turk
    Antoun, C; Zhang, C; Conrad, FG; Schober, MF
  • Income security in the on-demand economy: Findings and policy lessons from a survey of Crowdworkers
    Berg, J
  • Social science for pennies
    Bohannon, J
  • Work experiences on MTurk: Job satisfaction, turnover, and information sharing
    Brawley, AM; Pury, CLS
  • Intuition fail: Philosophical activity and the limits of expertise
    Buckwalter, W
  • Amazon’s mechanical Turk: A new source of inexpensive, yet high-quality, data?
    Buhrmester, M; Kwang, T; Gosling, SD
  • Nonnaïvéte among Amazon mechanical Turk workers: Consequences and solutions for behavioral researchers
    Chandler, J; Mueller, P; Paolacci, G
  • Using nonnaive participants can reduce effect sizes
    Chandler, J; Paolacci, G; Peer, E; Mueller, P; Ratliff, KA
  • Conducting clinical research using crowdsourced convenience samples
    Chandler, J; Shapiro, D
  • Evaluating Amazon’s mechanical Turk as a tool for experimental behavioral research
    Crump, MJC; McDonnell, JV; Gureckis, TM
  • Working the crowd: Employment and labor law in the crowdsourcing industry
    Felstiner, A
  • Amazon mechanical Turk: Gold mine or coal mine?
    Fort, K; Adda, G; Bretonnel Cohen, K
  • Cognitive reflection and decision making
    Frederick, S
  • Payment of clinical research subjects
    Grady, C
  • Ethics in human subjects research
    Grant, RW; Sugarman, J
  • Survey satisficing inflates reliability and validity measures: An experimental comparison of college and Amazon mechanical Turk samples
    Hamby, T; Taylor, W
  • Attentive Turkers: MTurk participants perform better on attention checks than do subject pool participants
    Hauser, DJ; Schwarz, N
  • Analyzing the Amazon mechanical Turk marketplace
    Ipeirotis, PG
  • The relationship between motivation, monetary compensation, and data quality among US- and India-based workers on mechanical Turk
    Litman, L; Robinson, J; Rosenzweig, C
  • Conducting behavioral research on Amazon’s mechanical Turk
    Mason, W; Suri, S
  • Financial incentives and the ‘performance of crowds
    Mason, W; Watts, DJ
  • Paying people to participate in research: Why not?
    McNeill, P
  • Measuring the prevalence of problematic respondent behaviors among MTurk, campus, and community participants
    Necka, EA; Cacioppo, S; Norman, GJ; Cacioppo, JT
  • Inside the Turk: Understanding mechanical Turk as a participant pool
    Paolacci, G; Chandler, J
  • Running experiments on Amazon mechanical Turk
    Paolacci, G; Chandler, J; Ipeirotis, PG
  • Reputation as a sufficient condition for data quality on Amazon mechanical Turk
    Peer, E; Vosgerau, J; Acquisti, A
  • Social heuristics shape intuitive cooperation
    Rand, DG; Peysakhovich, A; Kraft-Todd, GT
  • How not to test for philosophical expertise
    Rini, RA
  • Expertise in moral reasoning? Order effects on moral judgment in professional philosophers and non-philosophers
    Schwitzgebel, E; Cushman, F
  • Philosophers’ biased judgments persist despite training, expertise and reflection
    Schwitzgebel, E; Cushman, F
  • A multi-group analysis of online survey respondent data quality: Comparing a regular USA consumer panel to MTurk samples
    Smith, SM; Roster, CA; Golden, LL; Albaum, GS
  • The average laboratory samples a population of 7,300 Amazon mechanical Turk workers
    Stewart, N; Ungemach, C; Harris, AJL; Bartels, DM; Newell, BR; Paolacci, G; Chandler, J
  • A companion to experimental philosophy
    Stich, S; Tobia, KP
  • Methodology of longitudinal surveys
    Sturgis, P; Allum, N; Brunton-Smith, I
  • Investigating an alternate form of the cognitive reflection test
    Thomson, KS; Oppenheimer, DM
  • Moral intuitions: Are philosophers experts?
    Tobia, K; Buckwalter, W; Stich, S
  • Inducement in research
    Wilkinson, M; Moore, A
  • A companion to experimental philosophy
    Williamson, T
  • On the ethics of crowdsourced research
    Williamson, V