psychometrika—vol. 83, no. 1, 67–88
MARKOV DECISION PROCESS MEASUREMENT MODEL
Michelle M. LaMar
EDUCATIONAL TESTING SERVICE
Within-task actions can provide additional information on student competencies but are challenging
to model. This paper explores the potential of using a cognitive model for decision making, the Markov
decision process, to provide a mapping between within-task actions and latent traits of interest. Psychome-
tric properties of the model are explored, and simulation studies report on parameter recovery within the
context of a simple strategy game. The model is then applied to empirical data from an educational game.
Estimates from the model are found to correlate more strongly with posttest results than a partial-credit
IRT model based on outcome data alone.
Key words: latent-trait model, cognitive model, performance assessment.
Complex performance tasks often provide potentially useful information in the series of
problem-solving actions taken to complete the task; however, these data are challenging to model
with traditional psychometric models. Continuous latent-trait measurement models rely on a
meaningful mapping between performance indicators and the latent constructs which are the
target of inference. For most item response theory (IRT) models, this mapping is provided by
expert judgment via a scoring matrix or answer key, where all possible responses to each discrete
item are assigned a score for one or more of the latent-trait dimensions. When performances
are more complex, however, meaningful student actions are embedded in a larger context which
make their ultimate value dependent upon both the current state of the problem and the student’s
subsequent actions as they implement a strategic plan. For example, if one wishes to measure
ability in a board game, the ﬁnal outcome is easy to score, but the value of a particular move
might vary greatly depending upon the current conﬁguration of the board and potential future
moves. To utilize the information contained in such within-task actions, a model is needed that
can both provide action scores and account for the dependence inherent in the action sequencing.
For many complex tasks, a dynamic problem state can be deﬁned that includes all of the
factors that are relevant to the within-task decisions which make up the complex performance.
In board games, for example, the conﬁguration of the pieces on the board is often a complete
description the problem state. The problem state is then sufﬁcient to give the within-task actions
both value and independence, and as such, the state can be considered analogous to an item in a
traditional assessment framework. Thus, rather than identifying responses by sequence, they can
be associated with the problem state in which they were chosen. The number of possible “items”
would then be as large as the state space, and each student would only have responses for a small
number of them. Given this approach, the probability of student j taking an action in a particular
state s can be modeled using an IRT framework by including the scoring parameter within the
model as in the nominal response model (Bock, 1972),
∼ N (μ, σ
Electronic supplementary material The online version of this article (doi:10.1007/s11336-017-9570-0) contains
supplementary material, which is available to authorized users.
Correspondence should be made to Michelle M. LaMar, Educational Testing Service, Princeton, NJ, USA.
© 2017 The Psychometric Society