Skip to main content
SpringerLink
Log in

Learning to Remove Shadows from a Single Image

  • Manuscript
  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Recent learning-based shadow removal methods have achieved remarkable performance. However, they basically require massive paired shadow and shadow-free images for model training, which limits their generalization capability since these data are often cumbersome to obtain and lack of diversity. To address the problem, we present Self-ShadowGAN, a novel adversarial framework that is able to learn to remove shadows in an image by training solely on the image itself, using the shadow mask as the only supervision. Our approach is built upon the concept of histogram matching, by constraining the deshadowed regions produced by a shadow relighting network share similar histograms to the original shadow-free regions via a histogram-based discriminator. In order to speed up the single image training, we define the shadow relighting network to be lightweight multi-layer perceptions (MLPs) that estimate spatially-varying shadow relighting coefficients, where the parameters of the MLPs are predicted from a low-resolution input by a fast convolutional network and then upsampled back to the original full-resolution. Experimental results show that our method performs favorably against the state-of-the-art shadow removal methods, and is effective to process previously challenging shadow images.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23

Similar content being viewed by others

References

  • Arbel, E., & Hel-Or, H. (2010). Shadow removal using intensity surfaces and texture anchor points. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(6), 1202–1216.

    Article  Google Scholar 

  • Avi-Aharon, M., Arbelle, A., & Raviv, T. R. (2020). DeepHist: Differentiable joint and color histogram layers for image-to-image translation. arXiv preprint arXiv:2005.03995

  • Chen, Z., Long, C., Zhang, L., & Xiao, C. (2021). CANet: A context-aware network for shadow removal. In Proceedings of the international conference on computer vision (pp. 4743–4752).

  • Cun, X., Pun, C.-M., & Shi, C. (2020). Towards ghost-free shadow removal via dual hierarchical aggregation network and shadow matting gan. In Proceedings of the association for the advancement of artificial intelligence (vol. 34, pp. 10680–10687).

  • Ding, B., Long, C., Zhang, L., & Xiao, C. (2019). ARGAN: Attentive recurrent generative adversarial network for shadow detection and removal. In Proceedings of the international conference on computer vision (pp. 10213–10222).

  • Drew, M. S., Finlayson, G. D., & Hordley, S. D. (2003). Recovery of chromaticity image free from shadows via illumination invariance. In Proceedings of The IEEE international conference on computer vision workshops (pp. 32–39).

  • Finlayson, G. D., & Drew, M. S. (2001). 4-Sensor camera calibration for image representation invariant to shading, shadows, lighting, and specularities. In Proceedings of the international conference on computer vision (vol. 2, pp. 473–480).

  • Finlayson, G. D., Drew, M. S., & Lu, C. (2009). Entropy minimization for shadow removal. International Journal of Computer Vision, 85(1), 35–57.

    Article  Google Scholar 

  • Finlayson, G. D., Hordley, S. D., & Drew, M. S. (2002). Removing shadows from images. In Proceedings of the European conference on computer vision (pp. 823–836).

  • Finlayson, G. D., Hordley, S. D., Lu, C., & Drew, M. S. (2005). On the removal of shadows from images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(1), 59–68.

    Article  Google Scholar 

  • Fredembach, C., & Finlayson, G. (2005). Hamiltonian path-based shadow removal. In The British machine vision conference (vol. 2, pp. 502–511).

  • Fu, L., Zhou, C., Guo, Q., Juefei-Xu, F., Yu, H., Feng, W., Liu, Y., & Wang, S. (2021). Auto-exposure fusion for single-image shadow removal. In Proceedings of the IEEE computer vision and pattern recognition (pp. 10571–10580).

  • Gandelsman, Y., Shocher, A., & Irani, M. (2019). Double-DIP: Unsupervised image decomposition via coupled deep-image-priors. In Proceedings of the IEEE computer vision and pattern recognition (pp. 11026–11035).

  • Gharbi, M., Chen, J., Barron, J. T., Hasinoff, S. W., & Durand, F. (2017). Deep bilateral learning for real-time image enhancement. ACM Transactions on Graphics, 36(4), 1–12.

    Article  Google Scholar 

  • Gong, H., & Cosker, D. (2014). Interactive shadow removal and ground truth for variable scene categories. In The British machine vision conference (pp. 1–11).

  • Gryka, M., Terry, M., & Brostow, G. J. (2015). Learning to remove soft shadows. ACM Transactions on Graphics, 34(5), 1–15.

    Article  Google Scholar 

  • Guo, R., Dai, Q., & Hoiem, D. (2011). Single-image shadow detection and removal using paired regions. In Proceedings of the IEEE computer vision and pattern recognition (pp. 2033–2040).

  • Guo, R., Dai, Q., & Hoiem, D. (2012). Paired regions for shadow detection and removal. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(12), 2956–2967.

    Article  Google Scholar 

  • He, K., Gkioxari, G., Dollár, P., & Girshick, R. (2017). Mask R-CNN. In Proceedings of the international conference on computer vision (pp. 2961–2969).

  • He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE computer vision and pattern recognition (pp. 770–778).

  • He, Y., Xing, Y., Zhang, T., & Chen, Q. (2021). Unsupervised portrait shadow removal via generative priors. In Proceedings of the ACM international conference on multimedia (pp. 236–244).

  • Hu, X., Fu, C.-W., Zhu, L., Qin, J., & Heng, P.-A. (2019). Direction-aware spatial context features for shadow detection and removal. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(11), 2795–2808.

    Article  Google Scholar 

  • Hu, X., Jiang, Y., Fu, C.-W., & Heng, P.-A. (2019). Mask-ShadowGAN: Learning to remove shadows from unpaired data. In Proceedings of the international conference on computer vision (pp. 2472–2481).

  • Hu, X., Zhu, L., Fu, C.-W., Qin, J., & Heng, P.-A. (2018). Direction-aware spatial context features for shadow detection. In Proceedings of the IEEE computer vision and pattern recognition (pp. 7454–7462).

  • Inoue, N., & Yamasaki, T. (2020). Learning from synthetic shadows for shadow detection and removal. IEEE Transactions on Circuits and Systems for Video Technology, 6, 66.

    Google Scholar 

  • Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017). Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1125–1134).

  • Jin, Y., Sharma, A., & Tan, R. T. (2021). DC-ShadowNet: Single-image hard and soft shadow removal using unsupervised domain-classifier guided network. In Proceedings of the international conference on computer vision (pp. 5027–5036).

  • Khan, S. H., Bennamoun, M., Sohel, F., & Togneri, R. (2015). Automatic shadow detection and removal from a single image. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(3), 431–446.

    Article  Google Scholar 

  • Le, H., & Samaras, D. (2019). Shadow removal via shadow image decomposition. In Proceedings of the international conference on computer vision (pp. 8578–8587).

  • Le, H., & Samaras, D. (2020). From shadow segmentation to shadow removal. In Proceedings of the European conference on computer vision (pp. 264–281).

  • Le, H., & Samaras, D. (2021). Physics-based shadow image decomposition for shadow removal. IEEE Transactions on Pattern Analysis and Machine Intelligence, 01, 1–1.

    Google Scholar 

  • Levin, A., Lischinski, D., & Weiss, Y. (2007). A closed-form solution to natural image matting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(2), 228–242.

    Article  Google Scholar 

  • Li, Z., & Snavely, N. (2018). Learning intrinsic image decomposition from watching the world. In Proceedings of the IEEE computer vision and pattern recognition (pp. 9039–9048).

  • Lin, Y.-H., Chen, W.-C., & Chuang, Y.-Y. (2020). BEDSR-Net: A deep shadow removal network from a single document image. In Proceedings of the IEEE computer vision and pattern recognition (pp. 12905–12914).

  • Liu, A., Ginosar, S., Zhou, T., Efros, A. A., & Snavely, N. (2020). Learning to factorize and relight a city. In Proceedings of the European conference on computer vision.

  • Liu, F., & Gleicher, M. (2008). Texture-consistent shadow removal. In Proceedings of the European conference on computer vision (pp. 437–450).

  • Liu, Z., Yin, H., Mi, Y., Pu, M., & Wang, S. (2021). Shadow removal by a lightness-guided network with training on unpaired data. IEEE Transactions on Image Processing, 30, 1853–1865.

    Article  Google Scholar 

  • Liu, Z., Yin, H., Wu, X., Wu, Z., Mi, Y., & Wang, S. (2021). From shadow generation to shadow removal. In Proceedings of the IEEE computer vision and pattern recognition (pp. 4927–4936).

  • Ma, L.-Q., Wang, J., Shechtman, E., Sunkavalli, K., & Hu, S.-M. (2016). Appearance harmonization for single image. Computer Graphics Forumshadow removal, 7(35), 189–197.

    Article  Google Scholar 

  • Nestmeyer, T., Lalonde, J.-F., Matthews, I., & Lehrmann, A. (2020). Learning physics-guided face relighting under directional light. In Proceedings of the IEEE computer vision and pattern recognition (pp. 5124–5133) (2020)

  • Nguyen, V., Yago Vicente, T. F., Zhao, M., Hoai, M., & Samaras, D. (2017). Shadow detection with conditional generative adversarial networks. In Proceedings of the international conference on computer vision (pp. 4510–4518).

  • Qu, L., Tian, J., He, S., Tang, Y., & Lau, R. W. (2017). DeshadowNet: A multi-context embedding deep network for shadow removal. In Proceedings of the IEEE computer vision and pattern recognition (pp. 4067–4075).

  • Shaham, T. R., Dekel, T., & Michaeli, T. (2019). SinGAN: Learning a generative model from a single natural image. In Proceedings of the international conference on computer vision (pp. 4570–4580).

  • Shaham, T. R., Gharbi, M., Zhang, R., Shechtman, E., & Michaeli, T. (2021). Spatially-adaptive pixelwise networks for fast image translation. In Proceedings of the IEEE computer vision and pattern recognition (pp. 14882–14891).

  • Shor, Y., & Lischinski, D. (2008). The shadow meets the mask: Pyramid-based shadow removal. Computer Graphics Forum, 27(2), 577–586.

    Article  Google Scholar 

  • Ulyanov, D., Vedaldi, A., & Lempitsky, V. (2018). Deep image prior. In Proceedings of the IEEE computer vision and pattern recognition (pp. 9446–9454).

  • Vasluianu, F.-A., Romero, A., Van Gool, L., & Timofte, R. (2021). Shadow removal with paired and unpaired learning. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp. 826–835).

  • Vicente, T. F. Y., Hou, L., Yu, C.-P., Hoai, M., & Samaras, D. (2016). Large-scale training of shadow detectors with noisily-annotated shadow examples. In Proceedings of the European conference on computer vision (pp. 816–832).

  • Vicente, T. F. Y., & Samaras, D. (2014). Single image shadow removal via neighbor-based region relighting. In proceedings of the European conference on computer vision (pp. 309–320).

  • Wan, J., Yin, H., Wu, Z., Wu, X., Liu, Y., & Wang, S. (2022). Style-guided shadow removal. In Proceedings of the European conference on computer vision (pp. 361–378). Springer.

  • Wang, J., Agrawala, M., & Cohen, M. F. (2007). Soft scissors: An interactive tool for realtime high quality matting. ACM Transactions on Graphics, 26(3), 9.

    Article  Google Scholar 

  • Wang, J., Li, X., & Yang, J. (2018). Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. In Proceedings of the IEEE computer vision and pattern recognition (pp. 1788–1797).

  • Wang, R., Zhang, Q., Fu, C.-W., Shen, X., Zheng, W.-S., & Jia, J. (2019). Underexposed photo enhancement using deep illumination estimation. In Proceedings of The IEEE conference on computer vision and pattern recognition (pp. 6849–6857).

  • Wang, T., Hu, X., Wang, Q., Heng, P.-A., & Fu, C.-W. (2020). Instance shadow detection. In Proceedings of the IEEE computer vision and pattern recognition (pp. 1880–1889).

  • Wu, Q., Zhang, W., Kumar, B. V. (2012). Strong shadow removal via patch-based shadow edge detection. In Proceedings of the IEEE international conference on robotics and automation (pp. 2177–2182).

  • Wu, S., Makadia, A., Wu, J., Snavely, N., Tucker, R., & Kanazawa, A. (2021). De-rendering the world’s revolutionary artefacts. In Proceedings of the IEEE computer vision and pattern recognition (pp. 6338–6347).

  • Wu, T.-P., & Tang, C.-K. (2005). A Bayesian approach for shadow extraction from a single image. In Proceedings of the international conference on computer vision (vol. 1, pp. 480–487).

  • Wu, T.-P., Tang, C.-K., Brown, M. S., & Shum, H.-Y. (2007). Natural shadow matting. ACM Transactions on Graphics, 26(2), 8.

    Article  Google Scholar 

  • Xiao, C., She, R., Xiao, D., & Ma, K.-L. (2013). Fast shadow removal using adaptive multi-scale illumination transfer. Computer Graphics Forum, 32(8), 207–218.

    Article  Google Scholar 

  • Xiao, C., Xiao, D., Zhang, L., & Chen, L. (2013). Efficient shadow removal using subregion matching illumination transfer. Computer Graphics Forum, 32(7), 421–430.

    Article  Google Scholar 

  • Xu, M., Zhu, J., Lv, P., Zhou, B., Tappen, M. F., & Ji, R. (2017). Learning-based shadow recognition and removal from monochromatic natural images. IEEE Transactions on Image Processing, 26(12), 5811–5824.

    Article  MathSciNet  MATH  Google Scholar 

  • Yang, Q., Tan, K.-H., & Ahuja, N. (2012). Shadow removal using bilateral filtering. IEEE Transactions on Image processing, 21(10), 4361-4368.

    Article  MathSciNet  MATH  Google Scholar 

  • Zeng, H., Cai, J., Li, L., Cao, Z., & Zhang, L. (2020). Learning image-adaptive 3d lookup tables for high performance photo enhancement in real-time. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(4), 2058–2073.

    Google Scholar 

  • Zhang, L., Long, C., Zhang, X., & Xiao, C. (2020). RIS-GAN: Explore residual and illumination with generative adversarial networks for shadow removal. In Proceedings of the association for the advancement of artificial intelligence (vol. 34, pp. 12829–12836).

  • Zhang, L., Zhang, Q., & Xiao, C. (2015). Shadow remover: Image shadow removal based on illumination recovering optimization. IEEE Transactions on Image Processing, 24(11), 4623–4636.

    Article  MathSciNet  MATH  Google Scholar 

  • Zhang, Q., Nie, Y., & Zheng, W.-S. (2019). Dual illumination estimation for robust exposure correction. Computer Graphics Forum, 38(7), 243–252.

    Article  Google Scholar 

  • Zhang, Q., Nie, Y., Zhu, L., Xiao, C., & Zheng, W.-S. (2020). Enhancing underexposed photos using perceptually bidirectional similarity. IEEE Transactions on Multimedia, 23, 189–202.

    Article  Google Scholar 

  • Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018). The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 586–595).

  • Zhang, X., Barron, J. T., Tsai, Y.-T., Pandey, R., Zhang, X., Ng, R., & Jacobs, D. E. (2020). Portrait shadow manipulation. ACM Transactions on Graphics, 39(4), 78–1.

    Article  Google Scholar 

  • Zheng, Q., Qiao, X., Cao, Y., & Lau, R. W. (2019). Distraction-aware shadow detection. In Proceedings of the IEEE computer vision and pattern recognition (pp. 5167–5176) (2019)

  • Zhu, J.-Y., Park, T., Isola, P., Efros, A. A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the international conference on computer vision (pp. 2223–2232).

  • Zhu, L., Deng, Z., Hu, X., Fu, C.-W., Xu, X., Qin, J., & Heng, P.-A. (2018). Bidirectional feature pyramid network with recurrent attention residual modules for shadow detection. In Proceedings of the European conference on computer vision (pp. 121–136).

  • Zhu, Y., Huang, J., Fu, X., Zhao, F., Sun, Q., & Zha, Z.-J. (2022). Bijective mapping network for shadow removal. In Proceedings of the IEEE computer vision and pattern recognition (pp. 5627–5636).

  • Zhu, Y., Xiao, Z., Fang, Y., Fu, X., Xiong, Z., & Zha, Z.-J. (2022). Efficient model-driven network for shadow removal. In Proceedings of the AAAI conference on artificial intelligence.

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (U21A20471, 62072191), Guangdong Basic and Applied Basic Research Foundation (2023A1515030002, 2023B1515040025), and Guangzhou Municipal Science and Technology Project (Grant No. 2023A03J0671).

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Sun Yat-Sen University, Guangzhou, 510006, Guangdong, China

    Hao Jiang, Qing Zhang & Wei-Shi Zheng

  2. School of Computer Science and Engineering, South China University of Technology, Guangzhou, 510006, Guangdong, China

    Yongwei Nie

  3. The Hong Kong University of Science and Technology (Guangzhou), Nansha, Guangzhou, 51140, Guangdong, China

    Lei Zhu

  4. The Hong Kong University of Science and Technology, Hong Kong SAR, China

    Lei Zhu

Authors
  1. Hao Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qing Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yongwei Nie
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lei Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wei-Shi Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qing Zhang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jiang, H., Zhang, Q., Nie, Y. et al. Learning to Remove Shadows from a Single Image. Int J Comput Vis 131, 2471–2488 (2023). https://doi.org/10.1007/s11263-023-01823-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-023-01823-9

Keywords

Search

Navigation