Abstract
The EM algorithm is a popular method for computing maximum likelihood estimates or posterior modes in models that can be formulated in terms of missing data or latent structure. Although easy implementation and stable convergence help to explain the popularity of the algorithm, its convergence is sometimes notoriously slow. In recent years, however, various adaptations have significantly improved the speed of EM while maintaining its stability and simplicity. One especially successful method for maximum likelihood is known as the parameter expanded EM or PXEM algorithm. Unfortunately, PXEM does not generally have a closed form M-step when computing posterior modes, even when the corresponding EM algorithm is in closed form. In this paper we confront this problem by adapting the one-step-late EM algorithm to PXEM to establish a fast closed form algorithm that improves on the one-step-late EM algorithm by insuring monotone convergence. We use this algorithm to fit a probit regression model and a variety of dynamic linear models, showing computational savings of as much as 99.9%, with the biggest savings occurring when the EM algorithm is the slowest to converge.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Albert J.H. and Chib S. 1993. Bayesian analysis of binary and polychotomous response data. Journal of the American Statistical Association 88: 669–679.
Dempster A.P., Laird N.M., and Rubin D.B. 1977. Maximum likelihood from incomplete data via the EM algorithm (with discussion). Journal of the Royal Statistical Society, Series B, Methodological 39: 1–37.
Foulley J.-L. and van Dyk D.A. 2000. The PX-EM algorithm for fast stable fitting of Henderson's mixed model. Genetics Selective Evolution 32: 143–163.
Gelfand A.E., Sahu S.K., and Carlin B.P. 1995. Efficient parameterization for normal linear mixed models. Biometrika 82: 479–488.
Goodrich R. and Caines P. 1979. Linear system identification from nonstationary cross-sectional data. IEEE Trans. Aut. Cont. AC-24: 403–411.
Green P.J. 1990. On use of the EM algorithm for penalized likelihood estimation. Journal of the Royal Statistical Society, Series B, Methodological 52: 443–452.
Gupta N. and Mehra R. 1974. Computational aspects of maximum likelihood estimation and reduction in sensitivity function calculations. IEEE Trans. Aut. Cont. AC-19: 774–783.
Haas M. 1994. Igg subclass deposits in glomeruli of lupus and nonlupus membranous nephropathies. American Journal of Kidney Disease 23: 358–364.
Haas M. 1998. Value of igg subclasses and ultrastructural markers in predicting latent membranous lupus nephritis. Modern Pathology 11: 147A.
Harrison P. 1967. Exponential smoothing and short-term forecasting. Man. Sci. 13: 821–842.
Harrison P.J. 1965. Short-term sales forecasting. Applied Statistics 14: 102–139.
Harvey A. 1989. Forecasting, Structural Time Series Models and the Kalman Filter. Cambridge University Press, New York.
Higdon D.M. 1998. Auxiliary variable methods for Markov chain Monte Carlo with applications. Journal of the American Statistical Association 93: 585–595.
Imai K. and van Dyk D.A. 2003. A Bayesian analysis of the multinomial probit model using marginal augmentation. Submitted to The Journal of Econometrics.
Jazwinski A. 1970. Stochastic Processes and Filtering Theory. Academic Press, New York.
Kalman R. 1960. A new approach to linear filtering and prediction problems. Trans. ASME J. of Basic Eng. 8: 35–45.
Kalman R. and Bucy R. 1961. New results in linear filtering and prediction theory. Trans. ASME J. of Basic Eng. 83: 95–108.
Ledolter J. 1979. A recursive approach to parameter estimation in regression and time series models. Communications in Statistics, Part A-Theory and Methods 8: 1227–1246.
Little R.J. and Rubin D.B. 1987. Statistical AnalysisWith Missing Data. John Wiley & Sons, New York.
Liu C. and Rubin D.B. 1994. The ECME algorithm: A simple extension of EM and ECM with faster monotone convergence. Biometrika 81: 633–648.
Liu C. and Rubin D.B. 1995. ML estimation of the t distribution using EM and its extensions, ECM and ECME. Statistica Sinica 5: 19–39.
Liu C., Rubin D.B., and Wu Y.N. 1998. Parameter expansion for EM acceleration-The PXEM algorithm. Biometrika 75: 755–770.
Liu J.S. and Wu Y.N. 1999. Parameter expansion scheme for data augmentation. Journal of the American Statistical Association 94: 1264–1274.
McCulloch R. and Rossi P. 1994. An exact likelihood analysis of the multinomial probit model. Journal of Econometrics 64: 207–240.
Meng X.-L. and van Dyk D.A. 1997. The EM algorithm-An old folk song sung to a fast newtune (with discussion). Journal of the Royal Statistical Society, Series B, Methodological 59: 511–567.
Meng X.-L. and van Dyk D.A. 1998. Fast EM implementations for mixed-effects models. Journal of the Royal Statistical Society, Series B, Methodological 60: 559–578.
Meng X.-L. and van Dyk D.A. 1999. Seeking efficient data augmentation schemes via conditional and marginal augmentation. Biometrika 86: 301–320.
Pilla R.S. and Lindsay B.G. 1999. Alternative EM methods in highdimensional finite mixtures. Biometrika submitted.
Pole A., West M., and Harrison J. 1994. Applied Bayesian Forecasting and Time Series Analysis. Chapman & Hall, London.
Shumway R.H. and Stoffer D.S. 1982. An approach to time series smoothing and forecastin using the EM algorithm. Journal of Time Series Analysis 3: 253–264.
van Dyk D.A. 2000a. Fast new EM-type algorithms with applications in astrophysics. Technical Report.
van Dyk D.A. 2000b. Fitting mixed-effects models using efficient EMtype algorithms. Journal of Computational and Graphical Statistics 9: 78–98.
van Dyk D.A. 2000c. Nesting EM algorithms for computational efficiency. Statistical Sinica 10: 203–225.
van Dyk D.A. and Meng X.-L. 2000. Algorithms based on data augmentation. In: Pourahmadi K. and Berk K. (Eds.), Computing Science and Statistics: Proceedings of the 31st Symposium on the Interface, 230–239. Interface Foundation of North America, Fairfax Station, VA.
van Dyk D.A. and Meng X.-L. 2001. The art of data augmentation. The Journal of Computational and Graphical Statistics 10: 1–111.
Watson W. and Engle R. 1983. Alternative algorithm for the estimation of dynamic factor, mimic and varying coefficient regression. Journal of Econometrics 23: 385–400.
West M. and Harrison J. 1997. Bayesian Forecasting and Dynamic Models (2nd edn.). Springer-Verlag, New York.
Wu C.F.J. 1983. On the convergence properties of the EM algorithms. The Annals of Statistics 11: 95–103.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Van Dyk, D.A., Tang, R. The one-step-late PXEM algorithm. Statistics and Computing 13, 137–152 (2003). https://doi.org/10.1023/A:1023256509116
Issue Date:
DOI: https://doi.org/10.1023/A:1023256509116