Skip to main content

Advertisement

Log in

Inference on high-dimensional implicit dynamic models using a guided intermediate resampling filter

  • Published:
Statistics and Computing Aims and scope Submit manuscript

Abstract

We propose a method for inference on moderately high-dimensional, nonlinear, non-Gaussian, partially observed Markov process models for which the transition density is not analytically tractable. Markov processes with intractable transition densities arise in models defined implicitly by simulation algorithms. Widely used particle filter methods are applicable to nonlinear, non-Gaussian models but suffer from the curse of dimensionality. Improved scalability is provided by ensemble Kalman filter methods, but these are inappropriate for highly nonlinear and non-Gaussian models. We propose a particle filter method having improved practical and theoretical scalability with respect to the model dimension. This method is applicable to implicitly defined models having analytically intractable transition densities. Our method is developed based on the assumption that the latent process is defined in continuous time and that a simulator of this latent process is available. In this method, particles are propagated at intermediate time intervals between observations and are resampled based on a forecast likelihood of future observations. We combine this particle filter with parameter estimation methodology to enable likelihood-based inference for highly nonlinear spatiotemporal systems. We demonstrate our methodology on a stochastic Lorenz 96 model and a model for the population dynamics of infectious diseases in a network of linked regions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Acevedo, W., de Wiljes, J., Reich, S.: Second-order accurate ensemble transform particle filters. SIAM J. Sci. Comput. 39, A1834–A1850 (2017)

    MathSciNet  MATH  Google Scholar 

  • Ades, M., Van Leeuwen, P.J.: The equivalent-weights particle filter in a high-dimensional system. Q. J. R. Meteorol. Soc. 141, 484–503 (2015)

    Google Scholar 

  • Andrieu, C., Doucet, A., Holenstein, R.: Particle Markov chain Monte Carlo methods. J. R. Stat. Soc. Ser. B (Stat. Methodol.) 72, 269–342 (2010)

    MathSciNet  MATH  Google Scholar 

  • Bakker, K.M., Martinez-Bakker, M.E., Helm, B., Stevenson, T.J.: Digital epidemiology reveals global childhood disease seasonality and the effects of immunization. Proc. Natl. Acad. Sci. 113, 6689–6694 (2016)

    Google Scholar 

  • Becker, A.D., Birger, R.B., Teillant, A., Gastanaduy, P.A., Wallace, G.S., Grenfell, B.T.: Estimating enhanced prevaccination measles transmission hotspots in the context of cross-scale dynamics. Proc. Natl. Acad. Sci. 113, 14595–14600 (2016)

    Google Scholar 

  • Bengtsson, T., Bickel, P., Li, B.: Curse-of-dimensionality revisited: collapse of the particle filter in very large scale systems. In: Probability and Statistics: Essays in Honor of David A. Freedman, pp. 316–334. Institute of Mathematical Statistics (2008)

  • Beskos, A., Crisan, D.O., Jasra, A., Whiteley, N.: Error bounds and normalising constants for sequential Monte Carlo samplers in high dimensions. Adv. Appl. Probab. 46, 279–306 (2014a)

    MathSciNet  MATH  Google Scholar 

  • Beskos, A., Crisan, D., Jasra, A.: On the stability of sequential Monte Carlo methods in high dimensions. Ann. Appl. Probab. 24, 1396–1445 (2014b)

    MathSciNet  MATH  Google Scholar 

  • Beskos, A., Crisan, D., Jasra, A., Kamatani, K., Zhou, Y.: A stable particle filter for a class of high-dimensional state-space models. Adv. Appl. Probab. 49, 24–48 (2017)

    MathSciNet  MATH  Google Scholar 

  • Bickel, P.J., Doksum, K.A.: Mathematical Statistics: Basic Ideas and Selected Topics, vol. 117. CRC Press, Boca Raton (2015)

    MATH  Google Scholar 

  • Billingsley, P.: Convergence of Probability Measures. Wiley Series in Probability and Statistics. Wiley, New York (1999)

    MATH  Google Scholar 

  • Bjørnstad, O.N., Grenfell, B.T.: Noisy clockwork: time series analysis of population fluctuations in animals. Science 293, 638–643 (2001)

    Google Scholar 

  • Blackwood, J.C., Streicker, D.G., Altizer, S., Rohani, P.: Resolving the roles of immunity, pathogenesis, and immigration for rabies persistence in vampire bats. Proc. Natl. Acad. Sci. 110, 20837–20842 (2013)

    Google Scholar 

  • Blake, I.M., Martin, R., Goel, A., Khetsuriani, N., Everts, J., Wolff, C., Wassilak, S., Aylward, R.B., Grassly, N.C.: The role of older children and adults in wild poliovirus transmission. Proc. Natl. Acad. Sci. 111, 10604–10609 (2014)

    Google Scholar 

  • Bloem-Reddy, B., Orbanz, P.: Random-walk models of network formation and sequential Monte Carlo methods for graphs. J. R. Stat. Soc. Ser. B (Stat. Methodol.) 80, 871–898 (2018)

    MathSciNet  MATH  Google Scholar 

  • Bretó, C., He, D., Ionides, E.L., King, A.A.: Time series analysis via mechanistic models. Ann. Appl. Stat. 3, 319–348 (2009)

    MathSciNet  MATH  Google Scholar 

  • Bunch, P., Godsill, S.: Approximations of the optimal importance density using Gaussian particle flow importance sampling. J. Am. Stat. Assoc. 111, 748–762 (2016)

    MathSciNet  Google Scholar 

  • Cappé, O., Godsill, S.J., Moulines, E.: An overview of existing methods and recent advances in sequential Monte Carlo. Proc. IEEE 95, 899–924 (2007)

    Google Scholar 

  • Chen, R., Wang, X., Liu, J.S.: Adaptive joint detection and decoding in flat-fading channels via mixture Kalman filtering. IEEE Trans. Inf. Theory 46, 2079–2094 (2000)

    MathSciNet  MATH  Google Scholar 

  • Cheng, Y., Reich, S.: Assimilating data into scientific models: an optimal coupling perspective. In: Van Leeuwen, P.J., Cheng, Y., Reich, S. (eds.) Nonlinear Data Assimilation, Frontiers in Applied Dynamical Systems: Reviews and Tutorials, vol. 2, pp. 75–118. Springer, Cham (2015)

  • Chopin, N.: Central limit theorem for sequential Monte Carlo methods and its application to Bayesian inference. Ann. Stat. 32, 2385–2411 (2004)

    MathSciNet  MATH  Google Scholar 

  • Chopin, N., Jacob, P.E., Papaspiliopoulos, O.: SMC\({}^2\): an efficient algorithm for sequential analysis of state space models. J. R. Stat. Soc. Ser. B (Stat. Methodol.) 75, 397–426 (2013)

    MathSciNet  MATH  Google Scholar 

  • Chorin, A.J., Tu, X.: Implicit sampling for particle filters. Proc. Natl. Acad. Sci. 106, 17249–17254 (2009)

    Google Scholar 

  • Chorin, A.J., Morzfeld, M., Tu, X.: A survey of implicit particle filters for data assimilation. In: Zeng, Y., Wu, S. (eds.) State-Space Models, pp. 63–88. Springer, Berlin (2013)

    Google Scholar 

  • Clapp, T., Godsill, S.: Fixed-lag smoothing using sequential importance sampling. In: Bayesian statistics 6: Proceeding of the Sixth Valencia International Meeting, vol. 6, pp. 743–752 (1999)

  • Cleveland, W.S., Grosse, E., Shyu, W.M.: Local regression models. In: Chambers, J., Hastie, T. (eds.) Statistical Models in S, pp. 309–376. Chapman and Hall, London (1992)

    Google Scholar 

  • Dalziel, B.D., Bjørnstad, O.N., van Panhuis, W.G., Burke, D.S., Metcalf, C.J.E., Grenfell, B.T.: Persistent chaos of measles epidemics in the prevaccination United States caused by a small change in seasonal transmission patterns. PLoS Comput. Biol. 12, e1004655 (2016)

    Google Scholar 

  • Del Moral, P.: Feynman-Kac Formulae: Genealogical and Interacting Particle Systems with Applications. Springer, New York (2004)

    MATH  Google Scholar 

  • Del Moral, P., Guionnet, A.: On the stability of interacting processes with applications to filtering and genetic algorithms. Ann. l’Inst. Henri Poincaré (B) Probab. Stat. 37, 155–194 (2001)

    MathSciNet  MATH  Google Scholar 

  • Del Moral, P., Jacod, J.: Interacting particle filtering with discrete observations. In: Doucet, A., de Freitas, N., Gordon, N. (eds.) Sequential Monte Carlo Methods in Practice, pp. 43–75. Springer, Berlin (2001)

    Google Scholar 

  • Del Moral, P., Murray, L.M.: Sequential Monte Carlo with highly informative observations. SIAM/ASA J. Uncertain. Quantif. 3, 969–997 (2015)

    MathSciNet  MATH  Google Scholar 

  • Diggle, P.J., Gratton, R.J.: Monte Carlo methods of inference for implicit statistical models. J. R. Stat. Soc. Ser. B (Methodol.) 46, 193–212 (1984)

    MathSciNet  MATH  Google Scholar 

  • Douc, R., Cappé, O., Moulines, E.: Comparison of resampling schemes for particle filtering. In: Proceedings of the 4th International Symposium on Image and Signal Processing and Analysis, 2005, pp. 64–69. IEEE (2005)

  • Doucet, A., Johansen, A.M.: A tutorial on particle filtering and smoothing: Fifteen years later. In: Crisan, D., Rozovskii, B. (eds.) Oxford Handbook of Nonlinear Filtering. Oxford University Press, Oxford (2011)

    Google Scholar 

  • Doucet, A., Godsill, S., Andrieu, C.: On sequential Monte Carlo sampling methods for Bayesian filtering. Stat. Comput. 10, 197–208 (2000)

    Google Scholar 

  • Doucet, A., De Freitas, N., Gordon, N.: Sequential Monte Carlo Methods in Practice. Springer, Berlin (2001)

    MATH  Google Scholar 

  • Doucet, A., Briers, M., Sénécal, S.: Efficient block sampling strategies for sequential Monte Carlo methods. J. Comput. Graph. Stat. 15, 693–711 (2006)

    MathSciNet  Google Scholar 

  • Doucet, A., Pitt, M., Deligiannidis, G., Kohn, R.: Efficient implementation of Markov chain Monte Carlo when using an unbiased likelihood estimator. Biometrika 102, 295–313 (2015)

    MathSciNet  MATH  Google Scholar 

  • Eggo, R.M., Cauchemez, S., Ferguson, N.M.: Spatial dynamics of the 1918 influenza pandemic in England, Wales and the United States. J. R. Soc. Interface 8, 233–243 (2010)

  • Evensen, G.: Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. J. Geophys. Res. Oceans 99, 10143–10162 (1994)

    Google Scholar 

  • Farchi, A., Bocquet, M.: Comparison of local particle filters and new implementations. Nonlinear Process. Geophys. 25, 765–807 (2018)

    Google Scholar 

  • Fasiolo, M., Pya, N., Wood, S.N.: A comparison of inferential methods for highly nonlinear state space models in ecology and epidemiology. Stat. Sci. 31, 96–118 (2016)

    MathSciNet  MATH  Google Scholar 

  • Giraud, F., Del Moral, P.: Nonasymptotic analysis of adaptive and annealed Feynman-Kac particle models. Bernoulli 23, 670–709 (2017)

    MathSciNet  MATH  Google Scholar 

  • Gordon, N.J., Salmond, D.J., Smith, A.F.: Novel approach to nonlinear/non-Gaussian Bayesian state estimation. IEE Proc. F (Radar Signal Process.) 140, 107–113 (1993)

    Google Scholar 

  • Guarniero, P., Johansen, A.M., Lee, A.: The iterated auxiliary particle filter. J. Am. Stat. Assoc. 112, 1636–1647 (2017)

    MathSciNet  Google Scholar 

  • He, D., Ionides, E.L., King, A.A.: Plug-and-play inference for disease dynamics: measles in large and small populations as a case study. J. R. Soc. Interface 7, 271–283 (2009)

  • Houtekamer, P.L., Mitchell, H.L.: A sequential ensemble Kalman filter for atmospheric data assimilation. Mon. Weather Rev. 129, 123–137 (2001)

    Google Scholar 

  • Hunt, B.R., Kostelich, E.J., Szunyogh, I.: Efficient data assimilation for spatiotemporal chaos: a local ensemble transform Kalman filter. Phys. D Nonlinear Phenomena 230, 112–126 (2007)

    MathSciNet  MATH  Google Scholar 

  • Ionides, E.L., Nguyen, D., Atchadé, Y., Stoev, S., King, A.A.: Inference for dynamic and latent variable models via iterated, perturbed Bayes maps. Proc. Natl. Acad. Sci. 112, 719–724 (2015)

    MathSciNet  MATH  Google Scholar 

  • Ionides, E.L., Breto, C., Park, J., Smith, R.A., King, A.A.: Monte Carlo profile confidence intervals for dynamic systems. J. R. Soc. Interface 14, 20170126 (2017)

    Google Scholar 

  • Johansen, A.M.: On blocks, tempering and particle MCMC for systems identification. In: Proceedings of 17th IFAC Symposium on System Identification, pp. 969–974 (2015)

  • Kevrekidis, I.G., Gear, C.W., Hummer, G.: Equation-free: the computer-aided analysis of complex multiscale systems. AIChE J. 50, 1346–1355 (2004)

    Google Scholar 

  • King, A.A., Nguyen, D., Ionides, E.L.: Statistical inference for partially observed Markov processes via the R package pomp. J. Stat. Softw. 69, 1–43 (2016)

    Google Scholar 

  • King, A.A., Ionides, E.L., Bretó, C.M., Ellner, S.P., Ferrari, M.J., Kendall, B.E., Lavine, M., Nguyen, D., Reuman, D.C., Wearing, H., Wood, S.N.: pomp: Statistical inference for partially observed Markov processes. R package, version 2.4 (2019). https://kingaa.github.io/pomp/. Accessed 10 Mar 2020

  • Kitano, H.: Computational systems biology. Nature 420, 206 (2002)

    Google Scholar 

  • Kong, A., Liu, J.S., Wong, W.H.: Sequential imputations and Bayesian missing data problems. J. Am. Stat. Assoc. 89, 278–288 (1994)

    MATH  Google Scholar 

  • Le Gland, F., Oudjane, N.: Stability and uniform approximation of nonlinear filters using the Hilbert metric and application to particle filters. Ann. Appl. Probab. 14, 144–187 (2004)

    MathSciNet  MATH  Google Scholar 

  • Lei, J., Bickel, P., Snyder, C.: Comparison of ensemble Kalman filters under non-Gaussianity. Mon. Weather Rev. 138, 1293–1306 (2010)

    Google Scholar 

  • Lin, M., Chen, R., Mykland, P.: On generating Monte Carlo samples of continuous diffusion bridges. J. Am. Stat. Assoc. 105, 820–838 (2010)

    MathSciNet  MATH  Google Scholar 

  • Lin, M., Chen, R., Liu, J.S.: Lookahead strategies for sequential Monte Carlo. Stat. Sci. 28, 69–94 (2013)

    MathSciNet  MATH  Google Scholar 

  • Liu, J.S., Chen, R.: Blind deconvolution via sequential imputations. J. Am. Stat. Assoc. 90, 567–576 (1995)

    MathSciNet  MATH  Google Scholar 

  • Lorenz, E.N.: Predictability: a problem partly solved. Proc. Seminar Predict. 1, 1–18 (1996)

    Google Scholar 

  • Marjoram, P., Molitor, J., Plagnol, V., Tavaré, S.: Markov chain Monte Carlo without likelihoods. Proc. Natl. Acad. Sci. 100, 15324–15328 (2003)

    Google Scholar 

  • Miller, R.N., Carter, E.F., Blue, S.T.: Data assimilation into nonlinear stochastic models. Tellus A Dyn. Meteorol. Oceanogr. 51, 167–194 (1999)

    Google Scholar 

  • Morzfeld, M., Tu, X., Atkins, E., Chorin, A.J.: A random map implementation of implicit filters. J. Comput. Phys. 231, 2049–2066 (2012)

    MathSciNet  MATH  Google Scholar 

  • Neal, R.M.: Annealed importance sampling. Stat. Comput. 11, 125–139 (2001)

    MathSciNet  Google Scholar 

  • Owen, J., Wilkinson, D.J., Gillespie, C.S.: Scalable inference for Markov processes with intractable likelihoods. Stat. Comput. 25, 145–156 (2015)

    MathSciNet  MATH  Google Scholar 

  • Palmer, T.N.: Towards the probabilistic Earth-system simulator: a vision for the future of climate and weather prediction. Q. J. R. Meteorol. Soc. 138, 841–861 (2012)

    Google Scholar 

  • Papadakis, N., Mémin, É., Cuzol, A., Gengembre, N.: Data assimilation with the weighted ensemble Kalman filter. Tellus A Dyn. Meteorol. Oceanogr. 62, 673–697 (2010)

    Google Scholar 

  • Pitt, M.K., Shephard, N.: Filtering via simulation: auxiliary particle filters. J. Am. Stat. Assoc. 94, 590–599 (1999)

    MathSciNet  MATH  Google Scholar 

  • Pons-Salort, M., Grassly, N.C.: Serotype-specific immunity explains the incidence of diseases caused by human enteroviruses. Science 361, 800–803 (2018)

    Google Scholar 

  • Ranjeva, S.L., Baskerville, E.B., Dukic, V., Villa, L.L., Lazcano-Ponce, E., Giuliano, A.R., Dwyer, G., Cobey, S.: Recurring infection with ecologically distinct HPV types can explain high prevalence and diversity. Proc. Natl. Acad. Sci. 114, 13573–13578 (2017)

    Google Scholar 

  • Rebeschini, P., Van Handel, R.: Can local particle filters beat the curse of dimensionality? Ann. Appl. Probab. 25, 2809–2866 (2015)

    MathSciNet  MATH  Google Scholar 

  • Reich, S.: A nonparametric ensemble transform method for Bayesian inference. SIAM J. Sci. Comput. 35, A2013–A2024 (2013)

    MathSciNet  MATH  Google Scholar 

  • Sisson, S.A., Fan, Y., Tanaka, M.M.: Sequential Monte Carlo without likelihoods. Proc. Natl. Acad. Sci. 104, 1760–1765 (2007)

    MathSciNet  MATH  Google Scholar 

  • Snyder, C., Bengtsson, T., Bickel, P., Anderson, J.: Obstacles to high-dimensional particle filtering. Mon. Weather Rev. 136, 4629–4640 (2008)

    Google Scholar 

  • Snyder, C., Bengtsson, T., Morzfeld, M.: Performance bounds for particle filters using the optimal proposal. Mon. Weather Rev. 143, 4750–4761 (2015)

    Google Scholar 

  • Van Leeuwen, P.J.: Nonlinear data assimilation in geosciences: an extremely efficient particle filter. Q. J. R. Meteorol. Soc. 136, 1991–1999 (2010)

    Google Scholar 

  • Vergé, C., Dubarry, C., Del Moral, P., Moulines, E.: On parallel implementation of sequential Monte Carlo methods: the island particle model. Stat. Comput. 25, 243–260 (2015)

    MathSciNet  MATH  Google Scholar 

  • Whiteley, N.: Stability properties of some particle filters. Ann. Appl. Probab. 23, 2500–2537 (2013)

    MathSciNet  MATH  Google Scholar 

  • Whiteley, N., Lee, A.: Twisted particle filters. Ann. Stat. 42, 115–141 (2014)

    MathSciNet  MATH  Google Scholar 

  • Wilks, D.S.: Effects of stochastic parametrizations in the Lorenz 96 system. Q. J. R. Meteorol. Soc. 131, 389–407 (2005)

    Google Scholar 

  • Xia, Y., Bjørnstad, O.N., Grenfell, B.T.: Measles metapopulation dynamics: a gravity model for epidemiological coupling and dynamics. Am. Nat. 164, 267–281 (2004)

    Google Scholar 

  • Xiu, D., Kevrekidis, I.G., Ghanem, R.: An equation-free, multiscale approach to uncertainty quantification. Comput. Sci. Eng. 7, 16 (2005)

    Google Scholar 

Download references

Acknowledgements

The authors thank Aaron King for the discussions motivating this research and for insightful feedback. Comments on the manuscript by Kidus Asfaw, Yves Atchadé, and two anonymous referees have led to improvements. This work was supported by National Science Foundation Grants DMS-1308919, DMS-1761603, and DMS-1513040, and National Institutes of Health Grants 1-U54-GM111274 and 1-U01-GM110712.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Joonha Park.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 337 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Park, J., Ionides, E.L. Inference on high-dimensional implicit dynamic models using a guided intermediate resampling filter. Stat Comput 30, 1497–1522 (2020). https://doi.org/10.1007/s11222-020-09957-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11222-020-09957-3

Keywords

Navigation