Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

The effect of insufficient effort responding on the validity of student evaluation of teaching

The effect of insufficient effort responding on the validity of student evaluation of teaching The purpose of this paper is to investigate the effect of insufficient effort responding (IER) on construct validity of student evaluations of teaching (SET) in higher education.Design/methodology/approachA total of 13,340 SET surveys collected by a major Jordanian university to assess teaching effectiveness were analyzed in this study. The detection method was used to detect IER, and the construct (factorial) validity was assessed using confirmatory factor analysis (CFA) and principal component analysis (PCA) before and after removing detected IER.FindingsThe results of this study show that 2,160 SET surveys were flagged as insufficient effort responses out of 13,340 surveys. This figure represents 16.2 percent of the sample. Moreover, the results of CFA and PCA show that removing detected IER statistically enhanced the construct (factorial) validity of the SET survey.Research limitations/implicationsSince IER responses are often ignored by researchers and practitioners in industrial and organizational psychology (Liu et al., 2013), the results of this study strongly suggest that higher education administrations should give the necessary attention to IER responses, as SET results are used in making critical decisionsPractical implicationsThe results of the current study recommend universities to carefully design online SET surveys, and provide the students with clear instructions in order to minimize students’ engagement in IER. Moreover, since SET results are used in making critical decisions, higher education administrations should give the necessary attention to IER by examining the IERs rate in their data sets and its consequences on the data quality.Originality/valueReviewing the related literature shows that this is the first study that investigates the effect of IER on construct validity of SET in higher education using an IRT-based detection method. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Journal of Applied Research in Higher Education Emerald Publishing

The effect of insufficient effort responding on the validity of student evaluation of teaching

Download PDF
12 pages

Loading next page...
 
/lp/emerald-publishing/the-effect-of-insufficient-effort-responding-on-the-validity-of-CiJPv3NlDu
Publisher
Emerald Publishing
Copyright
© Emerald Publishing Limited
ISSN
2050-7003
DOI
10.1108/jarhe-03-2018-0034
Publisher site
See Article on Publisher Site

Abstract

The purpose of this paper is to investigate the effect of insufficient effort responding (IER) on construct validity of student evaluations of teaching (SET) in higher education.Design/methodology/approachA total of 13,340 SET surveys collected by a major Jordanian university to assess teaching effectiveness were analyzed in this study. The detection method was used to detect IER, and the construct (factorial) validity was assessed using confirmatory factor analysis (CFA) and principal component analysis (PCA) before and after removing detected IER.FindingsThe results of this study show that 2,160 SET surveys were flagged as insufficient effort responses out of 13,340 surveys. This figure represents 16.2 percent of the sample. Moreover, the results of CFA and PCA show that removing detected IER statistically enhanced the construct (factorial) validity of the SET survey.Research limitations/implicationsSince IER responses are often ignored by researchers and practitioners in industrial and organizational psychology (Liu et al., 2013), the results of this study strongly suggest that higher education administrations should give the necessary attention to IER responses, as SET results are used in making critical decisionsPractical implicationsThe results of the current study recommend universities to carefully design online SET surveys, and provide the students with clear instructions in order to minimize students’ engagement in IER. Moreover, since SET results are used in making critical decisions, higher education administrations should give the necessary attention to IER by examining the IERs rate in their data sets and its consequences on the data quality.Originality/valueReviewing the related literature shows that this is the first study that investigates the effect of IER on construct validity of SET in higher education using an IRT-based detection method.

Journal

Journal of Applied Research in Higher Education – Emerald Publishing

Published: Jun 18, 2019

Keywords: Construct validity; Insufficient effort responding (IER); Student evaluation of teaching (SET)

References

  • Detection of random responding on the MMPI-A
    Baer, R.A.; Ballenger, J.; Berry, D.T.R.; Wetter, M.W.
  • Detecting insufficient effort responding: an item response theory approach
    Barnes, T.D.
  • Are they paying attention? Students’ lack of motivation and attention potentially threaten the utility of course evaluations
    Bassett, J.; Cleveland, A.; Acorn, D.; Nix, M.; Snyder, T.
  • MMPI-2 Random responding indices: validation using a self-report methodology
    Berry, D.T.R.; Wetter, M.W.; Baer, R.A.; Larsen, L.; Clark, C.; Monroe, K.
  • Constructing validity: basic issues in objective scale development
    Clark, L.A.; Watson, D.
  • Detection of back random responding: effectiveness of MMPI-2 and personality assessment inventory validity indices
    Clark, M.E.; Gironda, R.J.; Young, R.W.
  • Best practices in explanatory factor analysis: four recommendations for getting the most from your analysis
    Costello, A.B.; Osborne, J.W.
  • Web surveys: perceptions of burden
    Crawford, S.D.; Couper, M.P.; Lamias, M.J.
  • Improving person‐fit assessment by correcting the ability estimate and its reference distribution
    De La Torre, J.; Deng, W.
  • Dirty data: the effects of screening respondents who provide low-quality data in survey research
    DeSimone, J.A.; Harms, P.D.
  • Appropriateness measurement with polychotomous item response models and standardized indices
    Drasgow, F.; Levine, M.V.; Williams, E.A.
  • Bootstrap methods: another look at the Jackknife
    Efron, B.
  • Detection and diagnosis of person misfit from patterns of summed polytomous item scores
    Emons, W.H.
  • Assessing inconsistent responding in E and N measures: an application of person-fit analysis in personality
    Ferrando, P.J.
  • Invalid responding in questionnaire-based research: Implications for the study of schizotypy
    Fervaha, G.; Remington, G.
  • Confirmatory factor analytic procedures for the determination of measurement invariance
    French, B.F.; Finch, W.H.
  • Personality and value correlates of careless and erratic questionnaire responses
    Furnham, A.; Hyde, G.; Trickey, G.
  • Students as partners in evaluation: student and teacher perspectives
    Giles, A.; Martin, S.C.; Bryce, D.; Hendry, G.D.
  • Person fit test for IRT models for polytomous items
    Glas, С.A.W.; Dagohoy, A.V.T.
  • Measuring teaching quality in higher education: assessing selection bias in course evaluations
    Goos, M.; Salomons, A.
  • Structural equation modelling: guidelines for determining model fit
    Hooper, D.; Coughlan, J.; Mullen, M.R.
  • Detecting insufficient effort responding with an infrequency scale: evaluating validity and participant reactions
    Huang, J.L.; Bowling, N.A.; Liu, M.; Li, Y.
  • Detecting and deterring insufficient effort responding to surveys
    Huang, J.L.; Curran, P.G.; Keeney, J.; Poposki, E.M.; DeShon, R.P.
  • Can a reliability coefficient be too high?’
    Hulin, C.; Netemeyer, R.; Cudeck, R.
  • Assessment of response pattern aberrancy in Eysenck Personality Inventory
    Jelínek, M.; Květoň, P.; Vobořil, D.
  • Ascertaining the validity of individual protocols from web-based personality inventories
    Johnson, J.A.
  • The application of electronic computers to factor analysis
    Kaiser, H.F.
  • Comparing the aberrant response detection performance of thirty six person-fit statistics
    Karabatsos, G.
  • Detecting random responders with infrequency scales using an error-balancing threshold
    Kim, D.S.; McCabe, C.J.; Yamasaki, B.L.; Louie, K.A.; King, K.M.
  • Student evaluations of teaching: perceptions of faculty based on gender, position and rank
    Kogan, L.R.; Schoenfeld-Tacher, R.; Hellyer, P.W.
  • Measuring the appropriateness of multiple-choice test scores
    Levine, M.V.; Rubin, D.B.
  • Insufficient effort responding to surveys as a threat to validity: the perceptions and practices of SIOP members
    Liu, M.; Bowling, N.A.; Huang, J.L.; Kent, T.A.
  • Insufficient effort survey responding: an under‐appreciated problem in work and organisational health psychology research
    McGonagle, A.K.; Huang, J.L.; Walsh, B.M.
  • Evidence for response bias as a source of error variance in applied assessment
    McGrath, R.E.; Mitchell, M.; Kim, B.H.; Hough, L.
  • Caring about carelessness: participant inattention and its effects on research
    Maniaci, M.R.; Rogge, R.D.
  • Students’ evaluations of university teaching: dimensionality, reliability, validity, potential baises, and utility
    Marsh, H.W.
  • Making students’ evaluations of teaching effectiveness effective: the critical issues of validity, bias, and utility
    Marsh, H.W.; Roche, L.A.
  • Identifying careless responses in survey data
    Meade, A.W.; Craig, S.B.
  • Person-fit research: an introduction
    Meijer, R.R.
  • Methodology review: evaluating person fit
    Meijer, R.R.; Sijtsma, K.
  • Detecting careless respondents in web-based questionnaires: which method to use?
    Niessen, A.S.M.; Meijer, R.R.; Tendeiro, J.N.
  • Psychometric quality of a student evaluation of teaching survey in higher education
    Oon, P.T.; Spencer, B.; Kam, C.
  • Random responding from participants is a threat to the validity of social science research results
    Osborne, J.W.; Blanchard, M.R.
  • On the use of college students in social science research: insights from a second-order meta-analysis
    Peterson, R.A.
  • The number of feedbacks needed for reliable evaluation: a multilevel analysis of the reliability, stability and generalisability of students’ evaluation of teaching
    Rantanen, P.
  • A systematic review of the methodology for person fit research in item response theory: lessons about generalizability of inferences from the design of simulation studies
    Rupp, A.A.
  • Beyond alpha: an empirical examination of the effect of different sources on measurement error on reliability estimates for measures of individual differences constructs
    Schmidt, F.; Le, H.; Llies, R.
  • On the equivalence of a likelihood ratio of Drasgow, Levine, and Zickar (1996) and the statistic based on the Neyman–Pearson Lemma of Belov
    Sinharay, S.
  • Starting at the beginning: an introduction to coefficient alpha and internal consistency
    Streiner, D.L.
  • Assessing goodness of fit in confirmatory factor analysis
    Sun, J.
  • PerFit: person fit: R package version 1.4.1
    Tendeiro, J.N.
  • The lz(p)* person-fit statistic in an unfolding model context
    Tendeiro, J.N.
  • PerFit: an R package for person-fit analysis in IRT
    Tendeiro, J.N.; Meijer, R.R.; Niessen, A.S.M.
  • The effect of examinee motivation on test construction within an IRT framework
    van Barneveld, C.
  • Identifying the types of insufficient effort responders
    Wertheimer, M.E.
  • Low examinee effort in low-stakes assessment: problems and potential solutions
    Wise, S.L.; DeMars, C.E.
  • The effects of the promise of test feedback on examinee performance and motivation under low-stakes testing conditions
    Wise, V.L.
  • Careless responding to reverse-worded items: implications for confirmatory factor analysis
    Woods, C.M.