Skip to main content
SpringerLink
Log in
Book cover

International Conference on Geometric Science of Information

GSI 2015: Geometric Science of Information pp 417–426Cite as

Generalized EM Algorithms for Minimum Divergence Estimation

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 9389))

Abstract

Minimum divergence estimators are derived through the dual form of the divergence in parametric models. These estimators generalize the classical maximum likelihood ones. Models with unobserved data, as mixture models, can be estimated with EM algorithms, which are proved to converge to stationary points of the likelihood function under general assumptions. This paper presents an extension of the EM algorithm based on minimization of the dual approximation of the divergence between the empirical measure and the model using a proximal-type algorithm. The algorithm converges to the stationary points of the empirical criterion under general conditions pertaining to the divergence and the model. Robustness properties of this algorithm are also presented. We provide another proof of convergence of the EM algorithm in a two-component gaussian mixture. Simulations on Gaussian and Weibull mixtures are performed to compare the results with the MLE.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    More investigation is needed here since we may use asymmetric kernels to overcome this difficulty.

  2. 2.

    Normally, \(\mu _2^l\) is bounded; still, we can extract a subsequence which converges.

References

  1. Al Mohamad, D.: Towards a better understanding of the dual representation of phi divergences. ArXiv e-prints (2015)

    Google Scholar 

  2. Broniatowski, M., Keziou, A.: Minimization of divergences on sets of signed measures. Studia Sci. Math. Hungar. 43(4), 403–442 (2006)

    MathSciNet  MATH  Google Scholar 

  3. Broniatowski, M., Keziou, A.: Parametric estimation and tests through divergences and the duality technique. J. Multivar. Anal. 100(1), 16–36 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  4. Chretien, S., Hero, A.O.: Acceleration of the EM algorithm via proximal point iterations. In: Proceedings of the 1998 IEEE International Symposium on Information Theory, 1998, p. 444 (1998)

    Google Scholar 

  5. Chrétien, S., Hero, A.O.: Generalized proximal point algorithms and bundle implementations. Technical report, Department of Electrical Engineering and Computer Science, The University of Michigan (1998)

    Google Scholar 

  6. Chrétien, S., Hero, A.O.: On EM algorithms and their proximal generalizations. ESAIM: Probab. Stat. 12, 308–326 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  7. Csiszár, I.: Eine informationstheoretische Ungleichung und ihre anwendung auf den Beweis der ergodizität von Markoffschen Ketten. Publications of the Mathematical Institute of Hungarian Academy of Sciences 8, 95–108 (1963)

    MATH  Google Scholar 

  8. Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the EM algorithm. J. Roy. Stat. Soc. Ser. B 39(1), 1–38 (1977)

    MathSciNet  MATH  Google Scholar 

  9. Goldstein, A.A., Russak, I.B.: How good are the proximal point algorithms? Numer. Funct. Anal. Optim. 9(7–8), 709–724 (1987)

    Article  MathSciNet  MATH  Google Scholar 

  10. Liese, F., Vajda, I.: On divergences and informations in statistics and information theory. IEEE Trans. Inf. Theor. 52(10), 4394–4412 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  11. McLachlan, G., Krishnan, T.: The EM Algorithm and Extensions. Wiley Series in Probability and Statistics. Wiley, Hoboken (2007)

    MATH  Google Scholar 

  12. Ostrowski, A.M.: Solution of Equations and Systems of Equations. Pure and Applied Mathematics. Academic Press, New York (1966)

    MATH  Google Scholar 

  13. Rockafellar, R.T., Wets, R.J.-B.: Variational Analysis. Die Grundlehren der mathematischen Wissenschaften in Einzeldarstellungen, 3rd edn. Springer, Heidelberg (1998)

    Book  MATH  Google Scholar 

  14. Toma, A., Broniatowski, M.: Dual divergence estimators and tests: robustness results. J. Multivar. Anal. 102(1), 20–36 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  15. Tseng, P.: An analysis of the EM algorithm and entropy-like proximal point methods. Math. Oper. Res. 29(1), 27–44 (2004)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Laboratoire de Statistique Théorique et Appliquée, Université Pierre et Marie Curie, 4 Place Jussieu, 75005, Paris, France

    Diaa Al Mohamad & Michel Broniatowski

Authors
  1. Diaa Al Mohamad
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michel Broniatowski
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Diaa Al Mohamad .

Editor information

Editors and Affiliations

  1. Bâtiment Alan Turing, CS35003, École Polytechnique, Palaiseau, France

    Frank Nielsen

  2. Thales Land\& Air Systems, Limours, France

    Frédéric Barbaresco

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Al Mohamad, D., Broniatowski, M. (2015). Generalized EM Algorithms for Minimum Divergence Estimation. In: Nielsen, F., Barbaresco, F. (eds) Geometric Science of Information. GSI 2015. Lecture Notes in Computer Science(), vol 9389. Springer, Cham. https://doi.org/10.1007/978-3-319-25040-3_45

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-25040-3_45

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-25039-7

  • Online ISBN: 978-3-319-25040-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation