Appl Math Optim 44:105–129 (2001)
2001 Springer-Verlag New York Inc.
Nearly Optimal Control of Singularly Perturbed
Markov Decision Processes in Discrete Time
R. H. Liu,
and G. Yin
Department of Mathematics, Boyd GSRC,
The University of Georgia,
Athens, GA 30602-7403, USA
Department of Mathematics, Wayne State University,
Detroit, MI 48202, USA
Abstract. This work develops asymptotically optimal controls for discrete-time
singularly perturbed Markov decision processes (MDPs) having weak and strong
interactions. The focus is on ﬁnite-state-space-MDP problems. The state space of
the underlying Markov chain can be decomposed into a number of recurrent classes
or a number of recurrent classes and a group of transient states. Using a hierarchical
control approach, continuous-time limit problems that are much simpler to handle
than the original ones are derived. Based on the optimal solutions for the limit
problems, nearly optimal decisions for the original problems are obtained. The
asymptotic optimality of such controls is proved and the rate of convergence is
provided. Inﬁnite horizon problems are considered; both discounted costs and long-
run average costs are examined.
Key Words. Markov decision process, Dynamic programming, Singular pertur-
bation, Asymptotically optimal control.
AMS Classiﬁcation. 90C40, 60J10, 60J27, 93E20.
The research of R. H. Liu and Q. Zhang was supported in part by USAF Grant F30602-99-2-0548
and ONR Grant N00014-96-1-0263. The research of G. Yin was supported in part by the National Science
Foundation under Grant DMS-9877090.