Skip to main content
SpringerLink
Log in

Deep image matting with cross-layer contextual information propagation

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Natural image matting focuses on accurately estimating the opacity of the foreground object in an arbitrary background. Recently, deep learning-based approaches made significant progress in the matting task benefit from their powerful learning ability for semantic features. However, artifacts, blurry structures, and miscalculated pixels still often appear in some difficult regions with background interference and complex details. To address the above issues, we propose a cross-layer contextual information propagation mechanism (CCIP) that can explicitly model the long-range correlations between global and unknown regions. Specifically, we first calculate region affinity at high-level features with rich structure and semantic information; then reconstruct the adjacent low-level features by propagating information from the global region to the unknown region under the guidance of the affinity matrix; finally, transfer the reconstructed information to the corresponding decoder stage to further improve the feature distinctiveness. In addition, we design a simple and effective supervision strategy in a deep-to-shallow manner to gradually optimize the edges and details of the foreground object. We conducted extensive experiments on the common dataset Composition-1k, the alphamatting.com benchmark, and some real-world images. Compared with previous methods, the proposed method achieves competitive performance on the Composition-1k dataset (30.3 on SAD, 6.8 on MSE, 13.3 on Grad, and 26.7 on Con) and alphamatting.com benchmark (17 on average SAD rank and 16.8 on average Grad rank), while simultaneously yielding high-quality matting results on real-world images.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availibility

The alphamattting.com dataset is available in http://www.alphamatting.com/datasets. The composition-1k dataset is available in https://sites.google.com/view/deepimagematting.

Notes

  1. https://pytorch.org.

  2. https://github.com/open-mmlab/mmediting.

References

  1. Zhao H, Li H, Cheng L (2020) Improving retinal vessel segmentation with joint local loss by matting. Patt Recognit 98:107068

    Article  Google Scholar 

  2. Chen J, Li X, Luo L, Ma J (2022) Multi-focus image fusion based on multi-scale gradients and image matting. IEEE Trans Multimed 24:655–667

    Article  ADS  Google Scholar 

  3. Ma Z, Kim D, Shin Y (2020) Shape-from-focus reconstruction using nonlocal matting Laplacian prior followed by MRF-based refinement. Patt Recognit 103:107302

    Article  Google Scholar 

  4. Lin S, Ryabtsev A, Sengupta S, Curless BL, Seitz SM, Kemelmacher-Shlizerman I (2021) Real-time high-resolution background matting. In: Computer vision and pattern recognition, pp 8762–8771

  5. Xu N, Price BL, Cohen S, Huang TS (2017) Deep image matting. In: Computer vision and pattern recognition, pp 311–320

  6. Hou Q, Cheng M, Hu X, Borji A, Tu Z, Torr PHS (2019) Deeply supervised salient object detection with short connections. IEEE Trans Patt Anal Mach Intell 41(4):815–828

    Article  Google Scholar 

  7. Zeng Y, Fu J, Chao H, Guo B (2019) Learning pyramid-context encoder network for high-quality image inpainting. In: Computer vision and pattern recognition, pp 1486–1494

  8. Liu Y, Cheng M, Fan D, Zhang L, Bian J, Tao D (2022) Semantic edge detection with diverse deep supervision. Int J Comput Vis 130(1):179–198

    Article  Google Scholar 

  9. Gastal ESL, Oliveira MM (2010) Shared sampling for real-time alpha matting. Comput Graph Forum 29(2):575–584

    Article  Google Scholar 

  10. Feng X, Liang X, Zhang Z (2016) A cluster sampling method for image matting via sparse coding. In: European conference on computer vision, pp 204–219

  11. Tang J, Aksoy Y, Öztireli C, Gross MH, Aydin TO (2019) Learning-based sampling for natural image matting. In: Computer vision and pattern recognition, pp 3055–3063

  12. Levin A, Lischinski D, Weiss Y (2008) A closed-form solution to natural image matting. IEEE Trans Patt Anal Mach Intell 30(2):228–242

    Article  Google Scholar 

  13. Chen Q, Li D, Tang C (2013) KNN matting. IEEE Trans Patt Anal Mach Intell 35(9):2175–2188

    Article  Google Scholar 

  14. Hu W, Hsu J (2013) Automatic spectral video matting. Patt Recognit 46(4):1183–1194

    Article  ADS  Google Scholar 

  15. Aksoy Y, Aydin TO, Pollefeys M (2017) Designing effective inter-pixel information flow for natural image matting. In: Computer vision and pattern recognition, pp 228–236

  16. Lutz S, Amplianitis K, Smolic A (2018) Alphagan: Generative adversarial networks for natural image matting. In: British machine vision conference, p 259

  17. Lu H, Dai Y, Shen C, Xu S (2019) Indices matter: learning to index for deep image matting. In: International conference on computer vision, pp 3265–3274

  18. Cai S, Zhang X, Fan H, Huang H, Liu J, Liu J, Liu J, Wang J, Sun J (2019) Disentangled image matting. In: International conference on computer vision, pp 8818–8827

  19. Hou Q, Liu F (2019) Context-aware image matting for simultaneous foreground and alpha estimation. In: International conference on computer vision, pp 4129–4138

  20. Yang X, Qiao Y, Chen S, He S, Yin B, Zhang Q, Wei X, Lau RWH (2021) Smart scribbles for image matting. ACM Trans Multimed Comput Commun Appl 16(4):1–21

    Article  CAS  Google Scholar 

  21. Li Y, Lu H (2020) Natural image matting via guided contextual attention. In: AAAI association for the advancement of artificial intelligence, pp 11450–11457

  22. Qiao Y, Liu Y, Yang X, Zhou D, Xu M, Zhang Q, Wei X (2020) Attention-guided hierarchical structure aggregation for image matting. In: Computer vision and pattern recognition, pp 13673–13682

  23. Dai Y, Lu H, Shen C (2021) Learning affinity-aware upsampling for deep image matting. In: Computer vision and pattern recognition, pp 6841–6850

  24. Liu Y, Xie J, Qiao Y, Tang Y, Yang X (2022) Prior-induced information alignment for image matting. IEEE Trans Multimed 24:2727–2738

    Article  Google Scholar 

  25. Yu Q, Zhang J, Zhang H, Wang Y, Lin Z, Xu N, Bai Y, Yuille AL (2021) Mask guided matting via progressive refinement network. In: Computer vision and pattern recognition, pp 1154–1163

  26. Wei T, Chen D, Zhou W, Liao J, Zhao H, Zhang W, Yu N (2021) Improved image matting via real-time user clicks and uncertainty estimation. In: Computer vision and pattern recognition, pp 15374–15383

  27. Ding H, Zhang H, Liu C, Jiang X (2022) Deep interactive image matting with feature propagation. IEEE Trans Image Process 31:2421–2432

    Article  ADS  PubMed  Google Scholar 

  28. Qiao Y, Liu Y, Wei Z, Wang Y, Cai Q, Zhang G, Yang X (2023) Hierarchical and progressive image matting. ACM Trans Multimed Comput Commun Appl 19(2):1–23

    Article  Google Scholar 

  29. Dai Y, Price B, Zhang H, Shen C (2022) Boosting robustness of image matting with context assembling and strong data augmentation. In: Computer vision and pattern recognition, pp 11697–11706

  30. Park G, Son S, Yoo J, Kim S, Kwak N (2022) Matteformer: transformer-based image matting via prior-tokens. In: Computer vision and pattern recognition, pp 11686–11696

  31. Cai H, Xue F, Xu L, Guo L (2022) Transmatting: enhancing transparent objects matting with transformers. In: European conference on computer vision, vol 13689, pp 253–269

  32. Li J, Zhang J, Tao D (2023) Referring image matting. In: Computer vision and pattern recognition, pp 22448–22457

  33. Yi Z, Tang Q, Azizi S, Jang D, Xu Z (2020) Contextual residual aggregation for ultra high-resolution image inpainting. In: Computer vision and pattern recognition, pp 7505–7514

  34. Ding H, Jiang X, Shuai B, Liu AQ, Wang G (2020) Semantic segmentation with context encoding and multi-path decoding. IEEE Trans Image Process 29:3520–3533

    Article  ADS  Google Scholar 

  35. Grady L, Schiwietz T, Aharon S, Westermann R (2005) Random walks for interactive alpha-matting, pp 423–429

  36. Chen L, Zhang H, Xiao J, Nie L, Shao J, Liu W, Chua T (2017) SCA-CNN: spatial and channel-wise attention in convolutional networks for image captioning. In: Computer vision and pattern recognition, pp 6298–6306

  37. Rhemann C, Rother C, Wang J, Gelautz M, Kohli P, Rott P (2009) A perceptually motivated online benchmark for image matting. In: Computer vision and pattern recognition, pp 1826–1833

  38. Lin T, Maire M, Belongie SJ, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Microsoft COCO: common objects in context. In: European conference on computer vision, pp 740–755

  39. Kingma, DP, Ba J (2015) Adam: a method for stochastic optimization. In: Bengio Y, LeCun Y (eds) International conference on learning representations

  40. Everingham M, Gool LV, Williams CKI, Winn JM, Zisserman A (2010) The pascal visual object classes (VOC) challenge. Int J Comput Vis 88(2):303–338

    Article  Google Scholar 

  41. Shahrian E, Rajan D, Price BL, Cohen S (2013) Improving image matting using comprehensive sampling sets. In: Computer vision and pattern recognition, pp 636–643

  42. Cho D, Tai Y, Kweon I (2016) Natural image matting using deep convolutional neural networks. In: European conference on computer vision, pp 626–643

  43. Yu H, Xu N, Huang Z, Zhou Y, Shi H (2021) High-resolution deep image matting. In: AAAI association for the advancement of artificial intelligence, pp 3217–3224

Download references

Acknowledgements

This work has been partially supported by grants from: National Natural Science Foundation of China (No.12071458, 71731009).

Author information

Authors and Affiliations

  1. Beijing Key Laboratory of Information Service Engineering, Beijing Union University, Beijing, 100101, China

    Fenfen Zhou

  2. College of Robotics, Beijing Union University, Beijing, 100027, China

    Fenfen Zhou

  3. School of Economics and Management, UCAS, Beijing, 100190, China

    Yingjie Tian

  4. Research Center on Fictitious Economy and Data Science, CAS, Beijing, 100190, China

    Yingjie Tian

  5. Key Laboratory of Big Data Mining and Knowledge Management, CAS, Beijing, 100190, China

    Yingjie Tian

  6. Computer School, Beijing Information Science and Technology University, Beijing, 100192, China

    Siyu Zhu

Authors
  1. Fenfen Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yingjie Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Siyu Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yingjie Tian.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhou, F., Tian, Y. & Zhu, S. Deep image matting with cross-layer contextual information propagation. Neural Comput & Applic 36, 6809–6825 (2024). https://doi.org/10.1007/s00521-024-09431-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-024-09431-5

Keywords

Search

Navigation