What do GCSE examiners think of 'thinking aloud'? Findings from an exploratory study
Abstract
Background: 'Thinking aloud' is a well-established method of data collection in education, assessment, and other fields of research. However, while many researchers have reported their views on its usage, the first-hand experiences of research participants have received less attention. Purpose: The aim of this exploratory study was to obtain the perspectives of GCSE examiners on thinking aloud as a means of investigating the cognitive strategies they use to mark examinations. Sample: The study focused on the experimental marking of two contrasting GCSE examination papers from 2004: mathematics and business studies. Twelve experienced examiners (six for each subject) participated in the study as paid volunteers. Design and methods: All examiners were individually asked to think aloud while concurrently marking up to five examination scripts each. Afterwards, they participated in one-to-one, semi-structured interviews about their experiences of the method. These interviews were recorded, transcribed and analysed qualitatively. Results: There were no clear differences between the mathematics and business studies examiners. However, views on the validity of thinking aloud, and advantages and disadvantages, varied considerably among individuals, possibly reflecting diverse personal marking techniques within each subject. Examiners varied in the kinds of questions they found it hardest to think aloud about. Conclusions: The findings have implications for further research and marking practice. They challenge two contrasting views reported in the literature: (1) that the simplest judgements are always hardest to verbalise; and (2) that it is always hardest to think aloud when engaging in complex thought processing. This indicates apparent variation in validity of the think aloud method among individuals. However, thinking aloud may also be useful in the training of new examiners.