Skip to main content
SpringerLink
Log in

Edge-aware image smoothing via weighted sparse gradient reconstruction

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Edge-ware image smoothing, which aims at removing fine details while respecting salient edges, is a prevalent topic in the field of computational imaging and photography. In this paper, we propose a novel weighted sparse gradient reconstruction model for edge-aware image smoothing. The proposed method first suppresses the low-amplitude gradients with an edge-aware mapping function. The processed gradients are then fed into a weighted \(L_1\) gradient reconstruction model to derive the output. The \(L_1\) regularization enforces the sparsity on the output gradients, thus facilitating the edge-aware property. The weighted scheme further promotes the edge awareness of the filter. In order to solve the proposed model, we propose an efficient solution based on the combination of the augmented Lagrange multipliers and the Fourier domain optimization. The GPU implementation of our filter takes 39ms to process a 720P color image on an NVIDIA RTX 3070. We have applied the proposed filter in various tasks, including edge-aware smoothing, edge extraction, image abstraction, and low-light image enhancement. Both the qualitative and quantitative results validate the superiority of the proposed filter.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data availability

All the datasets explored in this paper are publicly available.

References

  1. Esfandarani, H.T., Milanfar, P.: Fast multilayer Laplacian enhancement. IEEE Trans. Comput. Imaging 2(4), 496–509 (2016)

    Article  MathSciNet  Google Scholar 

  2. Durand, F., Dorsey, J.: Fast bilateral filtering for the display of high-dynamic-range images. ACM Trans. Graph. 21(3), 257–266 (2002)

    Article  Google Scholar 

  3. Guo, X., Li, Y., Ling, H.: LIME: low-light image enhancement via illumination map estimation. IEEE Trans. Image Process. 26(2), 982–993 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  4. Hussein, A.A., Yang, X.: Colorization using edge-preserving smoothing filter. Signal Image Video Process. 8(8), 1681–1689 (2014)

    Article  Google Scholar 

  5. Ye, X., Sang, X., Chen, D., Wang, P., Wang, K., Yan, B., Liu, B., Wang, H., Qi, S.: Superpixel guided network for three-dimensional stereo matching. IEEE Trans. Comput. Imaging 8, 54–68 (2022)

    Article  Google Scholar 

  6. Zhu, H., Sun, X., Zhang, Q., Wang, Q., Robles-Kelly, A., Li, H., You, S.: Full view optical flow estimation leveraged from light field superpixel. IEEE Trans. Comput. Imaging 6, 12–23 (2020)

    Article  MathSciNet  Google Scholar 

  7. He, K., Sun, J., Tang, X.: Guided image filtering. IEEE Trans. Pattern Anal. Mach. Intell. 35(6), 1397–1409 (2013)

    Article  Google Scholar 

  8. Chen, B., Wu, S.: Weighted aggregation for guided image filtering. Signal Image Video Process. 14(3), 491–498 (2020)

    Article  Google Scholar 

  9. Yin, H., Gong, Y., Qiu, G.: Side window filtering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 66, pp. 8758–8766 (2019)

  10. Liu, W., Zhang, P., Lei, Y., Huang, X., Yang, J., Ng, M.: A generalized framework for edge-preserving and structure-preserving image smoothing. IEEE Trans. Pattern Anal. Mach. Intell. 44(10), 6631–6648 (2022)

    Article  Google Scholar 

  11. Yang, Y., Hui, H., Zeng, L., Zhao, Y., Zhan, Y., Yan, T.: Edge-preserving image filtering based on soft clustering. IEEE Trans. Circuits Syst. Video Technol. 32(7), 4150–4162 (2022)

    Article  Google Scholar 

  12. Liu, W., Zhang, P., Huang, X., Yang, J., Shen, C., Reid, I.: Real-time image smoothing via iterative least squares. ACM Trans. Graph. 39(3), 28:1-28:24 (2020)

    Article  Google Scholar 

  13. Chen, Y., Gao, Y.: Adaptive fourth-order diffusion smoothing via bilateral kernel. Signal Image Video Process. 15(6), 1125–1133 (2021)

    Article  Google Scholar 

  14. Feng, Y., Deng, S., Yan, X., Yang, X., Wei, M., Liu, L.: Easy2hard: learning to solve the intractables from a synthetic dataset for structure-preserving image smoothing. IEEE Trans. Neural Netw. Learn. Syst. 33(12), 7223–7236 (2022)

    Article  Google Scholar 

  15. Zhu, F., Liang, Z., Jia, X., Zhang, L., Yu, Y.: A benchmark for edge-preserving image smoothing. IEEE Trans. Image Process. 28(7), 3556–3570 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  16. Fan, Q., Yang, J., Wipf, D.P., Chen, B., Tong, X.: Image smoothing via unsupervised learning. ACM Trans. Graph. 37(6), 259:1-259:14 (2018)

    Article  Google Scholar 

  17. Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images. In: Proceedings of the International Conference on Computer Vision, vol. 66, pp. 839–846 (1998)

  18. Paris, S., Durand, F.: A fast approximation of the bilateral filter using a signal processing approach. Int. J. Comput. Vis. 81(1), 24–52 (2009)

    Article  Google Scholar 

  19. Adams, A., Baek, J., Davis, M.A.: Fast high-dimensional filtering using the permutohedral lattice. Comput. Graph. Forum 29(2), 753–762 (2010)

    Article  Google Scholar 

  20. Yang, Q., Tan, K., Ahuja, N.: Real-time O(1) bilateral filtering. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 557–564 (2009)

  21. Sugimoto, K., Fukushima, N., Kamata, S.: 200 FPS constant-time bilateral filter using SVD and tiling strategy. In: IEEE International Conference on Image Processing, pp. 190–194 (2019)

  22. Chaudhury, K.N., Dabhade, S.D.: Fast and provably accurate bilateral filtering. IEEE Trans. Image Process. 25(6), 2519–2528 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  23. Nair, P., Popli, A., Chaudhury, K.N.: A fast approximation of the bilateral filter using the discrete Fourier transform. Image Process. Line 7, 115–130 (2017)

    Article  Google Scholar 

  24. Levin, A., Lischinski, D., Weiss, Y.: A closed-form solution to natural image matting. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 228–242 (2008)

    Article  Google Scholar 

  25. Li, Z., Zheng, J., Zhu, Z., Yao, W., Wu, S.: Weighted guided image filtering. IEEE Trans. Image Process. 24(1), 120–129 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  26. Sun, Z., Han, B., Li, J., Zhang, J., Gao, X.: Weighted guided image filtering with steering kernel. IEEE Trans. Image Process. 6, 66 (2020)

    MathSciNet  MATH  Google Scholar 

  27. Gastal, E.S.L., Oliveira, M.M.: Domain transform for edge-aware image and video processing. ACM Trans. Graph. 30(4), 69:1-69:11 (2011)

    Article  Google Scholar 

  28. Thévenaz, P., Sage, D., Unser, M.: Bi-exponential edge-preserving smoother. IEEE Trans. Image Process. 21(9), 3924–3936 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  29. Farbman, Z., Fattal, R., Lischinski, D., Szeliski, R.: Edge-preserving decompositions for multi-scale tone and detail manipulation. ACM Trans. Graph. 27(3), 67:1-67:10 (2008)

    Article  Google Scholar 

  30. Min, D., Choi, S., Lu, J., Ham, B., Sohn, K., Do, M.N.: Fast global image smoothing based on weighted least squares. IEEE Trans. Image Process. 23(12), 5638–5653 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  31. Liu, W., Chen, X., Shen, C., Liu, Z., Yang, J.: Semi-global weighted least squares in image filtering. In: Proceedings of the International Conference on Computing Vision, pp. 5862–5870 (2017)

  32. Liu, W., Zhang, P., Chen, X., Shen, C., Huang, X., Yang, J.: Embedding bilateral filter in least squares for efficient edge-preserving image smoothing. IEEE Trans. Circuits Syst. Video Technol. 30(1), 23–35 (2020)

    Article  Google Scholar 

  33. Xu, L., Lu, C., Xu, Y., Jia, J.: Image smoothing via L\({}_{0}\) gradient minimization. ACM Trans. Graph. 30(6), 174:1-174:11 (2011)

    Article  Google Scholar 

  34. Bi, S., Han, X., Yu, Y.: An L\({}_{{1}}\) image transform for edge-preserving smoothing and scene-level intrinsic decomposition. ACM Trans. Graph. 34(4), 78:1-78:12 (2015)

    Article  MATH  Google Scholar 

  35. Badri, H., Yahia, H., Aboutajdine, D.: Fast edge-aware processing via first order proximal approximation. IEEE Trans. Vis. Comput. Graph. 21(6), 743–755 (2015)

    Article  Google Scholar 

  36. Liu, W., Zhang, P., Lei, Y., Huang, X., Yang, J., Ng, M.: A generalized framework for edge-preserving and structure-preserving image smoothing. IEEE Trans. Pattern Anal. Mach. Intell. 44(10), 6631–6648 (2022)

    Article  Google Scholar 

  37. Comaniciu, D., Meer, P.: Mean shift: a robust approach toward feature space analysis. IEEE Trans. Pattern Anal. Mach. Intell. 24(5), 603–619 (2002)

    Article  Google Scholar 

  38. Yang, Q., Wang, S., Ahuja, N.: SVM for edge-preserving filtering. In: Proceedings of the IEEE Conference on Computing Vision Pattern Recognition, pp. 1775–1782 (2010)

  39. Xu, L., Ren, J.S.J., Yan, Q., Liao, R., Jia, J.: Deep edge-aware filters. In: Proceedings of the International Conference Machine Learning, pp. 1669–1678 (2015)

  40. Chen, Q., Xu, J., Koltun, V.: Fast image processing with fully-convolutional networks. In: Proceedings of the International Conference on Computing Vision, pp. 2516–2525 (2017)

  41. Kingma, D.P., Ba, J., Adam: A method for stochastic optimization. In: International Conference on Learning Representations (2015)

  42. Liang, Z., Xu, J., Zhang, D., Cao, Z., Zhang, L.: A hybrid l1–l0 layer decomposition model for tone mapping. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4758–4766 (2018)

  43. Shibata, T., Tanaka, M., Okutomi, M.: Gradient-domain image reconstruction framework with intensity-range and base-structure constraints. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2745–2753 (2016)

  44. Tanaka, M., Shibata, T., Okutomi, M.: Gradient-based low-light image enhancement. In: IEEE International Conference on Consumer Electronics, pp. 1–2 (2019)

  45. Yang, Y., Zheng, H., Zeng, L., Shen, X., Zhan, Y.: L1-regularized reconstruction model for edge-preserving filtering. IEEE Trans Multimed. 66, 1 (2022)

    Google Scholar 

  46. Bhat, P., Curless, B., Cohen, M.F., Zitnick, C.L.: Fourier analysis of the 2d screened poisson equation for gradient domain problems. In: Proceedings of the European Conference on Computer Vision, pp. 114–128 (2008)

  47. Donoho, D.L.: De-noising by soft-thresholding. IEEE Trans. Inf. Theory 41(3), 613–627 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  48. Zhang, Y., Guo, X., Ma, J., Liu, W., Zhang, J.: Beyond brightening low-light images. Int. J. Comput. Vis. 129(4), 1013–1037 (2021)

    Article  Google Scholar 

  49. Liu, R., Ma, L., Zhang, J., Fan, X., Luo, Z.: Retinex-inspired unrolling with cooperative prior architecture search for low-light image enhancement. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 10561–10570 (2021)

  50. Zhang, Y., Di, X., Zhang, B., Wang, C.: Self-supervised image enhancement network: training with low light images only. CoRR arXiv:2002.11300 (2020)

  51. Wang, W., Wei, C., Yang, W., Liu, J.: Gladnet: Low-light enhancement network with global awareness. In: IEEE International Conference on Automatic face & Gesture Recognition, pp. 751–755 (2018)

  52. Lv, F., Lu, F., Wu, J., Lim, C.: MBLLEN: low-light image/video enhancement using cnns. In: British Machine Vision Conference, p. 220 (2018)

  53. Wei, C., Wang, W., Yang, W., Liu, J.: Deep retinex decomposition for low-light enhancement. British Machine Vision Conference p. 155 (2018)

  54. Guo, C.G., Li, C., Guo, J., Loy, C.C., Hou, J., Kwong, S., Cong, R.: Zero-reference deep curve estimation for low-light image enhancement. In: IEEE conference on Computer Vision and Pattern Recognition, pp. 1780–1789 (2020)

  55. Ma, L., Ma, T., Liu, R., Fan, X., Luo, Z.: Toward fast, flexible, and robust low-light image enhancement. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5627–5636 (2022)

  56. Hao, S., Han, X., Guo, Y., Xu, X., Wang, M.: Low-light image enhancement with semi-decoupled decomposition. IEEE Trans. Multimed. 22(12), 3025–3038 (2020)

  57. Fan, M., Wang, W., Yang, W., Liu, J.: Integrating semantic segmentation and retinex model for low-light image enhancement. In: International Conference on Multimedia, pp. 2317–2325 (2020)

  58. Wu, H., Zheng, S., Zhang, J., Huang, K.: Fast end-to-end trainable guided filter. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1838–1847 (2018)

  59. Ham, B., Cho, M., Ponce, J.: Robust guided image filtering using nonconvex potentials. IEEE Trans. Pattern Anal. Mach. Intell. 40(1), 192–207 (2018)

    Article  Google Scholar 

  60. Sun, Z., Han, B., Li, J., Zhang, J., Gao, X.: Weighted guided image filtering with steering kernel. IEEE Trans. Image Process. 29, 500–508 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  61. Zhang, Q., Shen, X., Xu, L., Jia, J.: Rolling guidance filter. In: European Conference on Computer Vision, vol. 8691, pp. 815–830 (2014)

  62. Xu, J., Liu, Z., Hou, Y., Zhen, X., Shao, L., Cheng, M.: Pixel-level non-local image smoothing with objective evaluation. IEEE Trans. Multimed. 23, 4065–4078 (2021)

    Article  Google Scholar 

  63. Deng, G., Galetto, F., Al-nasrawi, M., Waheed, W.: A guided edge-aware smoothing-sharpening filter based on patch interpolation model and generalized gamma distribution. IEEE Open J. Signal Process. 2, 119–135 (2021)

    Article  Google Scholar 

  64. Xu, L., Yan, Q., Xia, Y., Jia, J.: Structure extraction from texture via relative total variation. ACM Trans. Graph. 31(6), 139:1-139:10 (2012)

    Article  Google Scholar 

  65. Yeganeh, H., Wang, Z.: Objective quality assessment of tone-mapped images. IEEE Trans. Image Process. 22(2), 657–667 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  66. Canny, J.F.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8(6), 679–698 (1986)

    Article  Google Scholar 

  67. Winnemöller, H., Olsen, S.C., Gooch, B.: Real-time video abstraction. ACM Trans. Graph. 25(3), 1221–1226 (2006)

    Article  Google Scholar 

  68. Hai, J., Xuan, Z., Yang, R., Hao, Y., Zou, F., Lin, F., Han, S.: R2rnet: low-light image enhancement via real-low to real-normal network. CoRR arXiv:2106.14501 (2021)

  69. Yin, H., Gong, Y., Qiu, G.: Side window filtering. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 8758–8766 (2019)

Download references

Funding

This work is supported by National Natural Science Foundation of China, Grant Nos. 61402205 and 62072150.

Author information

Authors and Affiliations

  1. Department of Computer Science, Jiangsu University, Xuefu, Zhenjiang, 212013, Jiangsu, China

    Lanling Zeng, Yucheng Chen & Yang Yang

  2. School of Artificial Intelligence, Nanjing University of Information Science and Technology, Ningliu, Nanjing, 210044, Jiangsu, China

    Zhigeng Pan

Authors
  1. Lanling Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yucheng Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yang Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhigeng Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

LZ supervised the research; YC conducted the experiments; LZ and YC wrote the main manuscript text; YY and ZP revised the manuscript; all authors reviewed the manuscript.

Corresponding authors

Correspondence to Lanling Zeng or Zhigeng Pan.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (pdf 1143 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zeng, L., Chen, Y., Yang, Y. et al. Edge-aware image smoothing via weighted sparse gradient reconstruction. SIViP 17, 4285–4293 (2023). https://doi.org/10.1007/s11760-023-02661-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-023-02661-5

Keywords

Search

Navigation