Population Research and Policy Review 18: 299–322, 1999.
© 1999 Kluwer Academic Publishers. Printed in the Netherlands.
On the validity of MAPE as a measure of population forecast
& DAVID A. SWANSON
San Diego Association of Governments;
Science Applications International Corporation,
Las Vegas, Nevada, USA;
Helsinki School of Economics and Business Administration,
International Program, Mikkeli, Finland
Abstract. The mean absolute percent error (MAPE) is the summary measure most often used
for evaluating the accuracy of population forecasts. While MAPE has many desirable criteria,
we argue from both normative and relative standpoints that the widespread practice of exclus-
ively using it for evaluating population forecasts should be changed. Normatively, we argue
that MAPE does not meet the criterion of validity because as a summary measure it overstates
the error found in a population forecast. We base this argument on logical grounds and support
it empirically, using a sample of population forecasts for counties. From a relative standpoint,
we examine two alternatives to MAPE, both sharing with it, the important conceptual fea-
ture of using most of the information about error. These alternatives are symmetrical MAPE
(SMAPE) and a class of measures known as M-estimators. The empirical evaluation suggests
M-estimators do not overstate forecast error as much as either MAPE or SMAPE and are,
therefore, more valid measures of accuracy. We consequently recommend incorporating M-
estimators into the evaluation toolkit. Because M-estimators do not meet the desired criterion
of interpretative ease as well as MAPE, we also suggest another approach that focuses on
nonlinear transformations of the error distribution.
Keywords: Population, Forecast, MAPE, Validity
Any summary measure of error should meet ﬁve highly desirable criteria.
In addition to the all-important criterion of measurement validity, they in-
clude reliability, ease of interpretation, clarity of presentation, and support of
statistical evaluation (NRC 1980). In attempting to meet these criteria, the
summary measure of population forecast error most often used is MAPE, the
mean absolute percent error (e.g., Ahlburg 1995; Isserman 1977; Murdock
et al. 1984; Smith 1987; Smith & Sincich 1990, 1992; Tayman et al. 1998).
We argue that MAPE satisfactorily meets at least four of the aforementioned
criteria, but is less satisfactory in meeting the criterion of validity when used
in evaluating the accuracy of population forecasts.
The MAPE’s lack of
validity frames the normative component of our argument. We also have a