psychometrika—vol. 82, no. 3, 693–716
A VARIATIONAL MAXIMIZATION–MAXIMIZATION ALGORITHM FOR
GENERALIZED LINEAR MIXED MODELS WITH CROSSED RANDOM EFFECTS
UNIVERSITY OF CALIFORNIA, LOS ANGELES
AMERICAN INSTITUTES FOR RESEARCH, WASHINGTON D.C.
UNIVERSITY OF CALIFORNIA, BERKELEY
We present a variational maximization–maximization algorithm for approximate maximum likelihood
estimation of generalized linear mixed models with crossed random effects (e.g., item response models with
random items, random raters, or random occasion-speciﬁc effects). The method is based on a factorized
variational approximation of the latent variable distribution given observed variables, which creates a
lower bound of the log marginal likelihood. The lower bound is maximized with respect to the factorized
distributions as well as model parameters. With the proposed algorithm, a high-dimensional intractable
integration is translated into a two-dimensional integration problem. We incorporate an adaptive Gauss–
Hermite quadrature method in conjunction with the variational method in order to increase computational
efﬁciency. Numerical studies show that under the small sample size conditions that are considered the
proposed algorithm outperforms the Laplace approximation.
Key words: variational approximation, lower bound, Kullback–Leibler divergence, EM algorithm, VMM
algorithm, adaptive quadrature, GLMM, crossed random effects.
Variational approximation methods are an emerging class of analytical techniques for approx-
imating high-dimensional integrals. The key to the variational method is to approximate the inte-
grals with a simple tractable form—creating a lower bound to the marginal likelihood. The now
tractable integration then becomes a simpliﬁed problem of bound optimization, or making the
bound as close as possible to the true value.
Variational methods have been utilized in both Bayesian and maximum likelihood (ML)
inference. In the ML framework, the conventional likelihood L (or equivalently, log-likelihood
l) is replaced by its lower bound L
(or l). Maximization of l with respect to model parameters is
then replaced by the maximization of l
with respect to model parameters as well as an auxiliary,
variational distribution. In the Bayesian context, the intractable posterior distribution over model
parameters and latent variables (or random effects) is approximated with a relatively simple,
factorized variational distribution form.
Variational approximation methods are deterministic in nature and involve additional alge-
braic calculations. However, variational methods are signiﬁcantly faster than Markov chain Monte
Electronic supplementary material The online version of this article (doi:10.1007/s11336-017-9555-z) contains
supplementary material, which is available to authorized users.
Correspondence should be made to Minjeong Jeon, Department of Education, University of California, Los Angeles,
405 Hilgard Avenue, Los Angeles, CA 90095, USA. Email: firstname.lastname@example.org
© 2017 The Psychometric Society