Abstract
Using optimal transport in image processing tasks has become very popular. However, it still faces difficult computational issues when dealing with high-dimensional distributions. We propose here to use the recently introduced GMM-OT formulation, which consists in restricting the optimal transport problem to the set of Gaussian mixture models. As a proof of concept, we use it to improve the texture model Texto based on optimal transport between distributions of image patches. Using GMM-OT in this texture model allows to deal with larger patches, hence providing results with better geometric details. This new model allows for synthesis, mixing, and style transfer.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
It would be interesting here to have a GMM estimation method that directly minimizes a transport cost between the GMM and the discrete patch distribution.
References
Bonneel, N., Van De Panne, M., Paris, S., Heidrich, W.: Displacement interpolation using Lagrangian mass transport. In: Proceedings of the 2011 SIGGRAPH Asia Conference, pp. 1–12 (2011)
Bonnotte, N.: Unidimensional and evolution methods for optimal transportation. Ph.D. thesis, Paris 11 (2013)
Chizat, L., Roussillon, P., Léger, F., Vialard, F.X., Peyré, G.: Faster Wasserstein distance estimation with the Sinkhorn divergence. Adv. Neural. Inf. Process. Syst. 33, 2257–2269 (2020)
Courty, N., Flamary, R., Habrard, A., Rakotomamonjy, A.: Joint distribution optimal transportation for domain adaptation. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Cuturi, M.: Sinkhorn distances: lightspeed computation of optimal transport. In: Advances in Neural Information Processing Systems, pp. 2292–2300 (2013)
Delon, J., Desolneux, A.: A Wasserstein-type distance in the space of Gaussian mixture models. SIAM J. Imag. Sci. 13(2), 936–970 (2020)
Feydy, J., Roussillon, P., Trouvé, A., Gori, P.: Fast and scalable optimal transport for brain tractograms. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11766, pp. 636–644. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32248-9_71
Galerne, B., Leclaire, A., Rabin, J.: A texture synthesis model based on semi-discrete optimal transport in patch space. SIAM J. Imag. Sci. 11(4), 2456–2493 (2018)
Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2414–2423 (2016)
Genevay, A., Chizat, L., Bach, F., Cuturi, M., Peyré, G.: Sample complexity of Sinkhorn divergences. In: The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1574–1583. PMLR (2019)
Hertrich, J., Houdard, A., Redenbach, C.: Wasserstein patch prior for image superresolution. IEEE Trans. Comput. Imaging 8, 693–704 (2022)
Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4401–4410 (2019)
Leclaire, A., Rabin, J.: A stochastic multi-layer algorithm for semi-discrete optimal transport with applications to texture synthesis and style transfer. J. Math. Imaging Vis. 63(2), 282–308 (2021)
Liang, L., Liu, C., Xu, Y.Q., Guo, B., Shum, H.Y.: Real-time texture synthesis by patch-based sampling. ACM Trans. Graph. 20(3), 127–150 (2001)
Mignon, S., Galerne, B., Hidane, M., Louchet, C., Mille, J.: Semi-unbalanced regularized optimal transport for image restoration. In: Actes du GRETSI (2022)
Ulyanov, D., Lebedev, V., Vedaldi, A., Lempitsky, V.: Texture networks: feed-forward synthesis of textures and stylized images. In: Proceedings of the International Conference on Machine Learning, vol. 48, pp. 1349–1357 (2016)
Weed, J., Bach, F.: Sharp asymptotic and finite-sample rates of convergence of empirical measures in Wasserstein distance. Bernoulli 25(4A), 2620–2648 (2019)
Xia, G., Ferradans, S., Peyré, G., Aujol, J.: Synthesizing and mixing stationary Gaussian texture models. SIAM J. Imag. Sci. 7(1), 476–508 (2014)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Delon, J., Desolneux, A., Facq, L., Leclaire, A. (2023). Optimal Transport Between GMM for Multiscale Texture Synthesis. In: Calatroni, L., Donatelli, M., Morigi, S., Prato, M., Santacesaria, M. (eds) Scale Space and Variational Methods in Computer Vision. SSVM 2023. Lecture Notes in Computer Science, vol 14009. Springer, Cham. https://doi.org/10.1007/978-3-031-31975-4_48
Download citation
DOI: https://doi.org/10.1007/978-3-031-31975-4_48
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-31974-7
Online ISBN: 978-3-031-31975-4
eBook Packages: Computer ScienceComputer Science (R0)