ISSN 0032-9460, Problems of Information Transmission, 2013, Vol. 49, No. 2, pp. 99–110.
Pleiades Publishing, Inc., 2013.
Original Russian Text
M. Kovaˇcevi´c, I. Stanojevi´c, V.
Senk, 2013, published in Problemy Peredachi Informatsii, 2013, Vol. 49, No. 2, pp. 3–16.
Some Properties of R´enyi Entropy
over Countably Inﬁnite Alphabets
M. Kovaˇcevi´c, I. Stanojevi´c, and V.
Department of Electrical Engineering, Faculty of Technical Sciences,
University of Novi Sad, Serbia
email@example.com vojin firstname.lastname@example.org
Received December 3, 2012; in ﬁnal form, January 30, 2013
Abstract—We study certain properties of R´enyi entropy functionals H
probability distributions over Z
. Primarily, continuity and convergence issues are addressed.
Some properties are shown to be parallel to those known in the ﬁnite alphabet case, while others
illustrate a quite diﬀerent behavior of the R´enyi entropy in the inﬁnite case. In particular, it is
shown that for any distribution P and any r ∈ [0, ∞] there exists a sequence of distributions P
converging to P with respect to the total variation distance and such that lim
R´enyi entropies are an important family of functionals deﬁned on the space of discrete or con-
tinuous probability distributions. They were introduced by A. R´enyi [1,2] on axiomatic grounds as
a generalization of the Shannon entropy, and have been studied extensively ever since.
For a probability distribution P =(p
), the R´enyi entropy of order α, α ≥ 0, is deﬁned as
1 − α
where it is understood that 
= H(P) (1.2)
which is precisely the Shannon entropy of P. (When α = 0, the convention 0
= 0 is used. The base
of the logarithm in (1.1), b>1, is arbitrary and will not be speciﬁed.) Hence, the R´enyi entropy can
be thought of as a more fundamental concept, of which the Shannon entropy is an important special
case. The fact that this is not a mere mathematical generalization has been seen afterwards when
R´enyi entropies found applications in many scientiﬁc disciplines such as information and coding
theory [3–6], statistical physics [7, 8], multifractal systems , etc. Related concepts of the R´enyi
divergence  (see also, e.g., [10, 11]) and conditional R´enyi entropy [12, 13]—again appropriate
generalizations of the corresponding Shannon measures—are also studied in a variety of contexts.
Supported by the Ministry of Science and Technological Development of the Republic of Serbia, grant
nos. TR32040 and III44003.