Abstract
Isolating different types of motion in video data is a highly relevant problem in video analysis. Applications can be found, for example, in dynamic medical or biological imaging, where the analysis and further processing of the dynamics of interest is often complicated by additional, unwanted dynamics, such as motion of the measurement subject. In this work, it is empirically shown that a representation of video data via untrained generator networks, together with a specific technique for latent space disentanglement that uses minimal, one-dimensional information on some of the underlying dynamics, allows to efficiently isolate different, highly non-linear motion types. In particular, such a representation allows to freeze any selection of motion types, and to obtain accurate independent representations of other dynamics of interest. Obtaining such a representation does not require any pre-training on a training data set, i.e., all parameters of the generator network are learned directly from a single video.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Data from the ISMRM reconstruction challenge 2014 challenge.ismrm.org.
References
Abdullah, A., Holler, M., Kunisch, K., Landman, M.S.: Source code for: latent-space disentanglement with untrained generator networks for the isolation of different motion types in video data (2023). https://github.com/hollerm/generator_based_motion_isolation
Bustin, A., Fuin, N., Botnar, R.M., Prieto, C.: From compressed-sensing to artificial intelligence-based cardiac MRI reconstruction. Front. Cardiovasc. Med. 7, 17 (2020). https://doi.org/10.3389/fcvm.2020.00017
Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., Abbeel, P.: InfoGAN: interpretable representation learning by information maximizing generative adversarial nets. In: Advances in Neural Information Processing Systems, vol. 29 (2016)
Fortun, D., Bouthemy, P., Kervrann, C.: Optical flow modeling and computation: a survey. Comput. Vis. Image Underst. 134, 1–21 (2015). https://doi.org/10.1016/j.cviu.2015.02.008
Fu, Y., Lei, Y., Wang, T., Curran, W.J., Liu, T., Yang, X.: Deep learning in medical image registration: a review. Phys. Med. Biol. 65(20), 20TR01 (2020). https://doi.org/10.1088/1361-6560/ab843e
Hälvä, H., et al.: Disentangling identifiable features from noisy data with structured nonlinear ICA. In: Advances in Neural Information Processing Systems, vol. 34 (2021)
Hamy, V., et al.: Respiratory motion correction in dynamic MRI using robust data decomposition registration - application to DCE-MRI. Med. Image Anal. 18(2), 301–313 (2014). https://doi.org/10.1016/j.media.2013.10.016
Hyder, R., Asif, M.S.: Generative models for low-dimensional video representation and reconstruction. IEEE Trans. Signal Process. 68, 1688–1701 (2020). https://doi.org/10.1109/TSP.2020.2977256
Hyvärinen, A., Pajunen, P.: Nonlinear independent component analysis: existence and uniqueness results. Neural Netw. 12(3), 429–439 (1999). https://doi.org/10.1016/S0893-6080(98)00140-3
Khemakhem, I., Kingma, D., Monti, R., Hyvarinen, A.: Variational autoencoders and nonlinear ICA: a unifying framework. In: International Conference on Artificial Intelligence and Statistics, pp. 2207–2217 (2020)
Lingala, S.G., DiBella, E., Jacob, M.: Deformation corrected compressed sensing (DC-CS): a novel framework for accelerated dynamic MRI. IEEE Trans. Med. Imaging 34(1), 72–85 (2015). https://doi.org/10.1109/TMI.2014.2343953
Oliveira, F.P., Tavares, J.M.R.: Medical image registration: a review. Comput. Methods Biomech. Biomed. Engin. 17(2), 73–93 (2014). https://doi.org/10.1080/10255842.2012.670855
Paszke, A., et al.: Automatic differentiation in PyTorch. In: Advances in Neural information processing systems (2017)
Pedregosa, F., et al.: Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12, 2825–2830 (2011)
Rahmim, A., Tang, J., Zaidi, H.: Four-dimensional (4D) image reconstruction strategies in dynamic PET: beyond conventional independent frame reconstruction. Med. Phys. 36(8), 3654–3670 (2009). https://doi.org/10.1118/1.3160108
Schloegl, M., Holler, M., Schwarzl, A., Bredies, K., Stollberger, R.: Infimal convolution of total generalized variation functionals for dynamic MRI. Magn. Reson. Med. 78(1), 142–155 (2017). https://doi.org/10.1002/mrm.26352
Tu, Z., et al.: A survey of variational and CNN-based optical flow techniques. Signal Process.: Image Commun. 72, 9–24 (2019). https://doi.org/10.1016/j.image.2018.12.002
Tulyakov, S., Liu, M.Y., Yang, X., Kautz, J.: MoCoGAN: decomposing motion and content for video generation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1526–1535 (2018)
Ulyanov, D., Vedaldi, A., Lempitsky, V.: Deep image prior. Int. J. Comput. Vision 128(7), 1867–1888 (2020). https://doi.org/10.1007/s11263-020-01303-4
Wollny, G., Kellman, P., Santos, A., Ledesma-Carbayo, M.J.: Automatic motion compensation of free breathing acquired myocardial perfusion data by using independent component analysis. Med. Image Anal. 16(5), 1015–1028 (2012). https://doi.org/10.1016/j.media.2012.02.004
Yoo, J., Jin, K.H., Gupta, H., Yerly, J., Stuber, M., Unser, M.: Time-dependent deep image prior for dynamic MRI. IEEE Trans. Med. Imaging 40(12), 3337–3348 (2021). https://doi.org/10.1109/TMI.2021.3084288
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Abdullah, A., Holler, M., Kunisch, K., Landman, M.S. (2023). Latent-Space Disentanglement with Untrained Generator Networks for the Isolation of Different Motion Types in Video Data. In: Calatroni, L., Donatelli, M., Morigi, S., Prato, M., Santacesaria, M. (eds) Scale Space and Variational Methods in Computer Vision. SSVM 2023. Lecture Notes in Computer Science, vol 14009. Springer, Cham. https://doi.org/10.1007/978-3-031-31975-4_25
Download citation
DOI: https://doi.org/10.1007/978-3-031-31975-4_25
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-31974-7
Online ISBN: 978-3-031-31975-4
eBook Packages: Computer ScienceComputer Science (R0)