Abstract
We consider the problem of learning Variational Autoencoders (VAEs), i.e., a type of deep generative model, from data with missing values. Such data is omnipresent in real-world applications of machine learning because complete data is often impossible or too costly to obtain. We particularly focus on improving a VAE’s amortized posterior inference, i.e., the encoder, which in the case of missing data can be susceptible to learning inconsistent posterior distributions regarding the missingness. To this end, we provide a formal definition of posterior consistency and propose an approach for regularizing an encoder’s posterior distribution which promotes this consistency. We observe that the proposed regularization suggests a different training objective than that typically considered in the literature when facing missing values. Furthermore, we empirically demonstrate that our regularization leads to improved performance in missing value settings in terms of reconstruction quality and downstream tasks utilizing uncertainty in the latent space. This improved performance can be observed for many classes of VAEs including VAEs equipped with normalizing flows.
This work was funded in parts by the Federal Ministry of Education, Science and Research (BMBWF), Austria [Digitize! Computational Social Science in the Digital and Social Transformation].
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Allen, A., Li, W.: Generative adversarial denoising autoencoder for face completion (2016). www.cc.gatech.edu/hays/7476/projects/Avery_Wenchen/
Berchtold, A.: Treatment and reporting of item-level missing data in social science research. vol. 22, pp. 431–439. Routledge (2019). https://doi.org/10.1080/13645579.2018.1563978
Burda, Y., Grosse, R.B., Salakhutdinov, R.: Importance Weighted Autoencoders. In: International Conference on Learning Representations (ICLR) (2016), arxiv.org/abs/1509.00519
van Buuren, S., Groothuis-Oudshoorn, K.: mice: Multivariate imputation by chained equations in r. vol. 45, pp. 1–67. Foundation for Open Access Statistics (2011)
Cheng-Xian Li, S., Jiang, B., Marlin, B.: MisGAN: Learning from Incomplete Data with Generative Adversarial Networks. In: International Conference on Learning Representations (ICLR) (2019), arxiv.org:1902.09599
Collier, M., Nazabal, A., Williams, C.: VAEs in the presence of missing data. In: ICML Workshop on the Art of Learning with Missing Values (Artemiss) (2020). www.openreview.net/forum?id=PnZT5EWoB7
Cremer, C., Li, X., Duvenaud, D.: Inference suboptimality in variational autoencoders. In: International Conference on Machine Learning (ICML), pp. 1078–1086. PMLR (2018)
Denton, E.L., Chintala, S., Szlam, A., Fergus, R.: Deep generative image models using a laplacian pyramid of adversarial networks. In: Advances in Neural Information Processing Systems (NeurIPS). Curran Associates, Inc. (2015). www.proceedings.neurips.cc/paper/2015/file/aa169b49b583a2b5af89203c2b78c67c-Paper.pdf
Dua, D., Graff, C.: UCI machine learning repository (2017), www.archive.ics.uci.edu/ml
Ghahramani, Z., Jordan, M.: Supervised learning from incomplete data via an em approach. In: Advances in Neural Information Processing Systems (NeurIPS) (1993). www.proceedings.neurips.cc/paper/1993/file/f2201f5191c4e92cc5af043eebfd0946-Paper.pdf
Ghahramani, Z., Jordan, M.I.: Learning from incomplete data. Technical Report AIM-1509CBCL-108, Massachusetts Institute of Technology (1995)
Ghalebikesabi, S., Cornish, R., Kelly, L.J., Holmes, C.: Deep generative pattern-set mixture models for nonignorable missingness. arXiv preprint arXiv:2103.03532 (2021)
Goodfellow, I., et al.: Generative Adversarial Nets. In: Advances in Neural Information Processing Systems (NeurIPS) (2014). www.proceedings.neurips.cc/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf
Goodfellow, I.J.: NIPS 2016 tutorial: Generative adversarial networks. CoRR abs/1701.00160 (2017), arxiv.org:1701.00160
Ipsen, N.B., Mattei, P.A., Frellsen, J.: not-MIWAE: Deep Generative Modelling with Missing not at Random Data. In: International Conference on Learning Representations (ICLR) (2021)
Kingma, D.P., Ba, J.: Adam: A Method for Stochastic Optimization (2014)
Kingma, D.P., Welling, M.: Auto-encoding variational bayes. In: International Conference on Learning Representations (ICLR) (2014)
Li, Y., Liu, S., Yang, J., Yang, M.H.: Generative face completion. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3911–3919. IEEE (2017)
Lim, D.K., Rashid, N.U., Oliva, J.B., Ibrahim, J.G.: Unsupervised imputation of non-ignorably missing data using importance-weighted autoencoders (2021). https://doi.org/10.48550/ARXIV.2101.07357, arxiv.org/abs/2101.07357
Little, R., Rubin, D.: Statistical Analysis with Missing Data. Wiley Series in Probability and Statistics, Wiley (2019). www.books.google.at/books?id=BemMDwAAQBAJ
Little, R.J., Rubin, D.B.: Statistical analysis with missing data, vol. 793. John Wiley & Sons (2019)
Liu, Y., Lin, S., Clark, R.: Towards consistent variational auto-encoding, pp. 13869–13870 (2020)
Ma, C., Tschiatschek, S., Palla, K., Hernandez-Lobato, J.M., Nowozin, S., Zhang, C.: Eddi: Efficient dynamic discovery of high-value information with partial vae. In: International Conference on Machine Learning (ICML), pp. 4234–4243 (2019)
Ma, C., Zhang, C.: Identifiable Generative Models for Missing Not at Random Data Imputation. 34, 27645–27658 (2021)
Mattei, P.A., Frellsen, J.: MIWAE: Deep generative modelling and imputation of incomplete data sets. In: International Conference on Machine Learning (ICML), pp. 4413–4423 (2019)
Mohan, K., Pearl, J., Tian, J.: Graphical models for inference with missing data. vol. 26 (2013)
Müller, T., McWilliams, B., Rousselle, F., Gross, M., Novák, J.: neural importance sampling. vol. 38, pp. 1–19 (2019)
Nazabal, A., Olmos, P.M., Ghahramani, Z., Valera, I.: Handling incomplete heterogeneous data using vaes. vol. 107, p. 107501. Elsevier (2020)
Newman, D.A.: Missing data: Five practical guidelines. vol. 17, pp. 372–411. Sage Publications Sage CA: Los Angeles, CA (2014)
Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks (2015). https://doi.org/10.48550/ARXIV.1511.06434, arxiv.org/abs/1511.06434
Rezende, D., Mohamed, S.: Variational Inference with Normalizing Flows. In: International Conference on Machine Learning (ICML), pp. 1530–1538 (2015)
Riggi, S., Riggi, D., Riggi, F.: Handling missing data for the identification of charged particles in a multilayer detector: A comparison between different imputation methods. vol. 780, pp. 81–90. Elsevier BV (2015). https://doi.org/10.1016/j.nima.2015.01.063
Rubin, D.B.: Inference and missing data. vol. 63, pp. 581–592. Oxford University Press (1976)
Sinha, S., Dieng, A.B.: Consistency regularization for variational auto-encoders. 34, 12943–12954 (2021)
Smith, D.M.: The cost of lost data. 6, 1–9 (2003)
Stekhoven, D.J., Bühlmann, P.: MissForest-non-parametric missing value imputation for mixed-type data. Bioinformatics 28, 112–118 (2011)
Sun, C., Shrivastava, A., Singh, S., Gupta, A.: Revisiting unreasonable effectiveness of data in deep learning era. In: IEEE International Conference on Computer Vision (ICCV), pp. 843–852 (2017)
Wen, X., Li, Z., Peng, D., Zhou, W., Liu, Y.: Missing data recovery using data fusion of incomplete complementary datasets: A particle image velocimetry application. Phys. Fluids 31, 025105 (2019)
Wu, G., Domke, J., Sanner, S.: Conditional inference in pre-trained variational autoencoders via cross-coding. arXiv preprint arXiv:1805.07785 (2018)
Yoon, J., Jordon, J., Schaar, M.: GAIN: Missing data imputation using generative adversarial nets. In: International Conference on Machine Learning (ICML), pp. 5689–5698 (2018)
Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R.R., Smola, A.J.: Deep sets. vol. 30 (2017)
Zhang, C., Bütepage, J., Kjellström, H., Mandt, S.: Advances in Variational Inference, vol. 41, pp. 2008–2026. IEEE (2018)
Zhu, M., Wang, J., Yan, C.: Non-autoregressive neural machine translation with consistency regularization optimized variational framework. In: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 607–617. Association for Computational Linguistics, Seattle, United States (Jul 2022). https://doi.org/10.18653/v1/2022.naacl-main.45, www.aclanthology.org/2022.naacl-main.45
Zhu, X., Wu, X.: Cost-constrained data acquisition for intelligent data preparation. IEEE17, 1542–1556 (2005)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Sudak, T., Tschiatschek, S. (2023). Posterior Consistency for Missing Data in Variational Autoencoders. In: Koutra, D., Plant, C., Gomez Rodriguez, M., Baralis, E., Bonchi, F. (eds) Machine Learning and Knowledge Discovery in Databases: Research Track. ECML PKDD 2023. Lecture Notes in Computer Science(), vol 14170. Springer, Cham. https://doi.org/10.1007/978-3-031-43415-0_30
Download citation
DOI: https://doi.org/10.1007/978-3-031-43415-0_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43414-3
Online ISBN: 978-3-031-43415-0
eBook Packages: Computer ScienceComputer Science (R0)
