Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

On estimation error using maximum entropy density estimates

On estimation error using maximum entropy density estimates Purpose – In many problems involving decision‐making under uncertainty, the underlying probability model is unknown but partial information is available. In some approaches to this problem, the available prior information is used to define an appropriate probability model for the system uncertainty through a probability density function. When the prior information is available as a finite sequence of moments of the unknown probability density function (PDF) defining the appropriate probability model for the uncertain system, the maximum entropy (ME) method derives a PDF from an exponential family to define an approximate model. This paper, aims to investigate some optimality properties of the ME estimates. Design/methodology/approach – For n > m , when the exact model can be best approximated by one of an infinite number of unknown PDFs from an n parameter exponential family. The upper bound of the divergence distance between any PDF from this family and the m parameter exponential family PDF defined by the ME method are derived. A measure of adequacy of the model defined by ME method is thus provided. Findings – These results may be used to establish confidence intervals on the estimate of a function of the random variable when the ME approach is employed. Additionally, it is shown that when working with large samples of independent observations, a probability density function (PDF) can be defined from an exponential family to model the uncertainty of the underlying system with measurable accuracy. Finally, a relationship with maximum likelihood estimation for this case is established. Practical implications – The so‐called known moments problem addressed in this paper has a variety of applications in learning, blind equalization and neural networks. Originality/value – An upper bound for error in approximating an unknown density function, f ( x ) by its ME estimate based on m moment constraints, obtained as a PDF p ( x , α ) from an m parameter exponential family is derived. The error bound will help us decide if the number of moment constraints is adequate for modeling the uncertainty in the system under study. In turn, this allows one to establish confidence intervals on an estimate of some function of the random variable, X , given the known moments. It is also shown how, when working with a large sample of independent observations, instead of precisely known moment constraints, a density from an exponential family to model the uncertainty of the underlying system with measurable accuracy can be defined. In this case, a relationship to ML estimation is established. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Kybernetes Emerald Publishing

On estimation error using maximum entropy density estimates

J.P. Noonan; Prabahan Basu
Kybernetes , Volume 36 (1): 13 – Feb 20, 2007
Download PDF
13 pages

Loading next page...
 
/lp/emerald-publishing/on-estimation-error-using-maximum-entropy-density-estimates-lLDjzoUmJj

References (18)

Publisher
Emerald Publishing
Copyright
Copyright © 2007 Emerald Group Publishing Limited. All rights reserved.
ISSN
0368-492X
DOI
10.1108/03684920710741143
Publisher site
See Article on Publisher Site

Abstract

Purpose – In many problems involving decision‐making under uncertainty, the underlying probability model is unknown but partial information is available. In some approaches to this problem, the available prior information is used to define an appropriate probability model for the system uncertainty through a probability density function. When the prior information is available as a finite sequence of moments of the unknown probability density function (PDF) defining the appropriate probability model for the uncertain system, the maximum entropy (ME) method derives a PDF from an exponential family to define an approximate model. This paper, aims to investigate some optimality properties of the ME estimates. Design/methodology/approach – For n > m , when the exact model can be best approximated by one of an infinite number of unknown PDFs from an n parameter exponential family. The upper bound of the divergence distance between any PDF from this family and the m parameter exponential family PDF defined by the ME method are derived. A measure of adequacy of the model defined by ME method is thus provided. Findings – These results may be used to establish confidence intervals on the estimate of a function of the random variable when the ME approach is employed. Additionally, it is shown that when working with large samples of independent observations, a probability density function (PDF) can be defined from an exponential family to model the uncertainty of the underlying system with measurable accuracy. Finally, a relationship with maximum likelihood estimation for this case is established. Practical implications – The so‐called known moments problem addressed in this paper has a variety of applications in learning, blind equalization and neural networks. Originality/value – An upper bound for error in approximating an unknown density function, f ( x ) by its ME estimate based on m moment constraints, obtained as a PDF p ( x , α ) from an m parameter exponential family is derived. The error bound will help us decide if the number of moment constraints is adequate for modeling the uncertainty in the system under study. In turn, this allows one to establish confidence intervals on an estimate of some function of the random variable, X , given the known moments. It is also shown how, when working with a large sample of independent observations, instead of precisely known moment constraints, a density from an exponential family to model the uncertainty of the underlying system with measurable accuracy can be defined. In this case, a relationship to ML estimation is established.

Journal

Kybernetes – Emerald Publishing

Published: Feb 20, 2007

Keywords: Cybernetics; Error analysis; Decision making

There are no references for this article.