Abstract
Motivated from the bandwidth selection problem in local likelihood density estimation and from the problem of assessing a final model chosen by a certain model selection procedure, we consider estimation of the Kullback–Leibler divergence. It is known that the best bandwidth choice for the local likelihood density estimator depends on the distance between the true density and the ‘vehicle’ parametric model. Also, the Kullback–Leibler divergence may be a useful measure based on which one judges how far the true density is away from a parametric family. We propose two estimators of the Kullback-Leibler divergence. We derive their asymptotic distributions and compare finite sample properties.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Akaike, H. (1973). Information theory and an extension of the maximum likelihood principle. In: B. N. Petrov., F. Csaki (Eds.) 2nd International symposium on information Theory. pp 267–81. Budapest: Akademiai Kiado. (Reproduced (1992) In: S. Kotz & N. L. Johnson (Eds.) Breakthroughs in Statistics 1, Sringer-Verlag, New York, pp. 610–624.
Akaike H. (1974). A new look at the statistical model selection. IEEE Transactions on Automatic Control 19:716–723
Azzalini A. (1985). A class of distributions which includes the normal ones. Scandinavian Journal of Statististics 12:171–178
Copas J.B. (1995). Local likelihood based on kernel censoring. Journal of the Royal Statistical Society, Series B 57:221–235
Efron B. (1982). The Jackknife, the Bootstrap and Other Resampling Plans. SIAM, Philadelphia
Eguchi S., Copas J.B. (1998). A class of local likelihood methods and near-parametric asymptotics. Journal of the Royal Statistical Society, Series B 60:709–724
Eguchi S., Kim T.Y., Park B.U. (2003). Local likelihood method: a bridge over parametric and nonparametric regression. Journal of Nonparametric Statistics 15:665–683
Hjort N.L., Jones M.C. (1996). Locally parametric nonparametric density estimation. The Annals of Statistics 24:1619–1647
Konishi S., Kitagawa G. (1996). Generalised information criteria in model selection. Biometrika 83:875–890
Kullback S., Leibler R.A. (1951). On information and sufficiency. The Annals of Mathematical Statistics 22:79–86
Park B.U., Kim W.C., Jones M.C. (2002). On local likelihood density estimation. The Annals of Statistics 30:1480–1495
Park B.U., Lee Y.K., Kim T.Y., Park C., Eguchi S. (2006). On local likelihood density estimation when the bandwidth is large. Journal of Statistical Planning and Inference 136:839–859
Ripley B.D. (1996) Pattern Recognition and Neural Networks. Cambridge University Press, Cambridge
Stone M. (1977). An asymptotic equivalence of choice of model by cross-validation and Akaike’s criterion. Journal of the Royal Statistical Society, Series B 39:44–47
Author information
Authors and Affiliations
Corresponding author
Additional information
Research of Young Kyung Lee was supported by the Brain Korea 21 Projects in 2004. Byeong U. Park’s research was supported by KOSEF through Statistical Research Center for Complex Systems at Seoul National University.
About this article
Cite this article
Lee, Y.K., Park, B.U. Estimation of Kullback–Leibler Divergence by Local Likelihood. Ann Inst Stat Math 58, 327–340 (2006). https://doi.org/10.1007/s10463-005-0014-8
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10463-005-0014-8