ISSN 0032-9460, Problems of Information Transmission, 2007, Vol. 43, No. 4, pp. 367–379.
Pleiades Publishing, Inc., 2007.
Original Russian Text
B.Ya. Ryabko, 2007, published in Problemy Peredachi Informatsii, 2007, Vol. 43, No. 4, pp. 109–123.
Application of Data Compression Methods
to Nonparametric Estimation of Characteristics
of Discrete-Time Stochastic Processes
B. Ya. Ryabko
Siberian State University of Telecommunication and Information Science
Institute of Computational Technologies, Siberian Branch of the RAS, Novosibirsk
Received April 9, 2007; in ﬁnal form, July 31, 2007
Abstract—Discrete-time stochastic processes generating elements of either a ﬁnite set (al-
phabet) or a real line interval are considered. Problems of estimating limiting (or stationary)
probabilities and densities are considered, as well as classiﬁcation and prediction problems.
We show that universal coding (or data compression) methods can be used to solve these prob-
Though methods of encoding data sources (often referred to as data compression) and methods
of information theory in the whole have been widely used in problems of mathematical statistics
since the middle of the past century , recent years were marked with novel result and directions in
this area. It was found that commonly used archivers (i.e., computer programs that implement data
compression methods) can be directly applied to testing statistical hypotheses [2,3] and prediction
of stochastic processes . In the present paper we continue these studies and show how one can
apply universal codes (or universal data compression methods) to estimate limiting probabilities
and densities of time series. (Let us note here that by deﬁnition, a universal code asymptotically
“compresses” a sequence of n symbols generated by a stationary ergodic source to nH bits, where H
is the Shannon entropy.) The obtained estimators are applied to the construction of prediction
methods, solution of classiﬁcation problems (called sometimes classiﬁcation with side information),
etc. Note also that universal coding methods are applied to predicting stochastic processes starting
with the paper , where the case of ﬁnite-alphabet processes was considered, which after that was
generalized to sources generating elements of metric spaces [6–11].
Let us give a precise statement of the problem. We consider stationary ergodic processes gen-
erating sequences of elements x
of a set (alphabet) A; two cases are considered: A is ﬁnite
or A is a real line interval. We consider so-called nonparametric methods; i.e., we assume that there
is no other information on probabilistic characteristics of the process. In mathematical statistics,
getting information on a process is usually formulated as either a hypothesis testing problem or
estimating parameters of the process, such as limiting probabilities, distributions, regression, etc.
We ﬁrst brieﬂy dwell on the hypothesis testing problem, studied in [2, 3], in order to present
main ideas underlying the use of universal coding methods in problems of mathematical statistics.
Consider the problem of testing the hypothesis H
that a sequence of zeros and ones x
Supported in part by the Russian Foundation for Basic Research, project no. 06-07-89025.