Skip to main content
SpringerLink
Log in

Single image rain/snow removal using distortion type information

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Rainy or snowy weather conditions can severely impair the visual quality of images and videos. The rain streaks or snow particles that may vary in shape and size can also affect high-level computer vision system performance. Therefore, pre-processing of these distorted images prior to any other task is necessary. Moreover, due to the lack of temporal information in single images, removal of these artifacts becomes more challenging. In this paper, both the de-raining and de-snowing problems within a single algorithmic framework using a data-driven approach are addressed. In this method, the spatial characteristics of rain streaks and snow particles are investigated and two maps, namely, direction map and intensity map, are generated and exploited in the removal process. Using these two maps, the type of the distortion is classified using a convolutional neural network (CNN) and this information is used in the removal step, where the input image along with the two extracted maps and the information about the distortion type are used to train a deep fully convolutional rain/snow removal network (RSRNet). This network is trained such that it separates the important background scene edges from rain streaks or snow particles and uses the extracted edge map to augment the quality of the output image. Moreover, single images usually suffer from atmospheric haze in the presence of heavy rain or snow. Therefore, a simple dehazing method based on the dark channel prior (DCP) algorithm, which uses the edge map extracted in the RSRNet, is proposed to build a transmission map for the haze removal task. The experimental results on both the real and synthetic single rainy/snowy images demonstrate the superiority of the proposed method compared to the other rain/snow removal methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Notes

  1. https://cloud.google.com/vision/

References

  1. Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M (2016) Tensorflow: A system for large-scale machine learning. Proceedings of the Operating Systems Design and Implementation 16:265–283

    Google Scholar 

  2. Aittala M, Durand F (2018) Burst image deblurring using permutation invariant convolutional neural networks. In: Proceedings of the European conference on computer vision (ECCV), pp 731–747

  3. Bossu J, Hautière N, Tarel J (2011) Rain or snow detection in image sequences through use of a histogram of orientation of streaks. Int J Comput Vis 93 (3):348–367

    Article  Google Scholar 

  4. Brewer N, Liu N (2008) Using the shape characteristics of rain to identify and remove rain from video. Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR), pp 451–458

  5. Cai B, Xu X, Jia K, Qing C, Tao D (2016) Dehazenet: an end-to-end system for single image haze removal. IEEE Trans Image Process (TIP) 25(11):5187–5198

    Article  MathSciNet  Google Scholar 

  6. Canny J (1986) A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), (6): 679–698

  7. Chen D, Chen C, Kang L (2014) Visual depth guided color image rain streaks removal using sparse coding. IEEE Transactions on Circuits and Systems for Video Technology (CSVT) 24(8):1430–1455

    Article  Google Scholar 

  8. Chen Y, Hsu C (2013) A generalized low-rank appearance model for spatio-temporally correlated rain streaks. In: Proceedings of the IEEE international conference on computer vision (ICCV), pp 1968– 1975

  9. Chen L, Papandreou G, Kokkinos I, Murphy K, Yuille A (2017) Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans Pattern Anal Mach Intell (PAMI) 40(4):834–848

    Article  Google Scholar 

  10. Choi L, You J, Bovik A (2015) Referenceless prediction of perceptual fog density and perceptual image defogging. IEEE Trans Image Process (TIP) 24(11):3888–3901

    Article  MathSciNet  Google Scholar 

  11. Dahl G, Sainath T, Hinton G (2013) Improving deep neural networks for LVCSR using rectified linear units and dropout. In: IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 8609–8613

  12. Fu X, Huang J, Ding X, Liao Y, Paisley J (2017) Clearing the skies: a deep network architecture for single-image rain removal. IEEE Trans Image Process (TIP) 26(6):2944–2956

    Article  MathSciNet  Google Scholar 

  13. Fu X, Huang J, Zeng D, Huang Y, Ding X, Paisley J (2017) Removing rain from single images via a deep detail network. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 3855–3863

  14. Fu X, Liang B, Huang Y, Ding X, Paisley J (2019) Lightweight pyramid networks for image deraining. IEEE Transactions on Neural Networks and Learning Systems

  15. Girshick R (2015)

  16. Graps A (1995) An introduction to wavelets. IEEE Comput Sci Eng 2(2):50–61

    Article  Google Scholar 

  17. Gu S, Meng D, Zuo W, Zhang L (2017) Joint convolutional analysis and synthesis sparse representation for single image layer separation. In: Proceedings of the IEEE international conference on computer vision (ICCV), pp 1708–1716

  18. Hao D, Li Q, Li C (2017) Single-image-based rain streak removal using multidimensional variational mode decomposition and bilateral filter. Journal of Electronic Imaging 26(1):13–20

    Article  Google Scholar 

  19. He K, Sun J, Tang X (2010) Single image haze removal using dark channel prior. IEEE Trans Pattern Anal Mach Intell (PAMI) 33(12):2341–2353

    Google Scholar 

  20. He K, Sun J, Tang X (2012) Guided image filtering. IEEE Trans Pattern Anal Mach Intell (PAMI) 35(6):1397–1409

    Article  Google Scholar 

  21. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 770–778

  22. Huang G, Liu Z, Van Der Maaten L, Weinberger K (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 4700– 4708

  23. Jégou S, Drozdzal M, Vazquez D, Romero A, Bengio Y (2017) The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR) workshops, pp 11–19

  24. Kang L, Lin C, Fu Y (2011) Automatic single-image-based rain streaks removal via image decomposition. IEEE Trans Image Process (TIP) 21 (4):1742–1755

    Article  MathSciNet  Google Scholar 

  25. Kim Lee C, Sim J, Kim C (2013) Single-image deraining using an adaptive nonlocal means filter. In: IEEE international conference on image processing (ICIP), pp 914–917

  26. Kingma D, Ba J (2014) Adam: A method for stochastic optimization. In: Proceedings of the international conference on learning representations (ICLR)

  27. Lee S, Yun S, Nam J, Won C, Jung S (2016) A review on dark channel prior based image dehazing algorithms, EURASIP Journal on Image and Video Processing, (1)

  28. Li B, Ren W, Fu D, Tao D, Feng D, Zeng W, Wang Z (2018) Benchmarking single-image dehazing and beyond. IEEE Trans Image Process (TIP) 28(1):492–505

    Article  MathSciNet  Google Scholar 

  29. Li Y, Tan R, Guo X, Lu J, Brown M (2016) Rain streak removal using layer priors. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 2736–2744

  30. Li Y, Tan R, Guo X, Lu J, Brown M (2017) Single image rain streak decomposition using layer priors. IEEE Trans Image Process (TIP) 26 (8):3874–3885

    Article  MathSciNet  Google Scholar 

  31. Li M, Xie Q, Zhao Q, Wei W, Gu S, Tao J, Meng D (2018) Video rain streak removal by multiscale convolutional sparse coding. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 6644–6653

  32. Lin T, Dollár P, Girshick R, He K, Hariharan B, Belongie S (2017) Feature pyramid networks for object detection. In: Proceedings of the IEEE conference on computer vision and pattern R,ecognition (CVPR), pp 2117–2125

  33. Liu Y, Jaw D, Huang S, Hwang J (2018) Desnownet: Context-aware deep network for snow removal. IEEE Trans Image Process (TIP) 27 (6):3064–3073

    Article  MathSciNet  Google Scholar 

  34. Liu J, Yang W, Yang S, Guo Z (2018) Erase or fill? Deep joint recurrent rain removal and reconstruction in videos. In: Proceeding of the IEEE conference on computer vision and pattern recognition (CVPR), pp 3233–3242

  35. Luo Y, Xu Y, Ji H (2015) Removing rain from a single image via discriminative sparse coding. In: Proceedings of the IEEE international conference on computer vision (ICCV), pp 3397–3405

  36. Mannos J, Sakrison D (1974) The effects of a visual fidelity criterion of the encoding of images. IEEE Trans Inform Theory (TIT) 20(4):525–536

    Article  Google Scholar 

  37. McCartney E (1976) Optics of the Atmosphere: Scattering by Molecules and Particles. Wiley, New York

    Google Scholar 

  38. Nayar S, Narasimhan S (1999) Vision in bad weather. Proceedings of the International Conference on Computer Vision (ICCV) 2:820–827

    Google Scholar 

  39. Noroozi M, Chandramouli P, Favaro P (2017) Motion deblurring in the wild. In: German conference on pattern recognition, pp 65–77

  40. Pei S, Tsai Y, Lee C (2014) Removing rain and snow in a single image using saturation and visibility features. In: IEEE international conference on multimedia and expo workshops (ICMEW), pp 1–6

  41. Santhaseelan V, Asari V (2015) Utilizing local phase information to remove rain from video. Int J Comput Vis 112(1):71–89

    Article  Google Scholar 

  42. Shen L, Yue Z, Chen Q, Feng F, Ma J (2018) Deep joint rain and haze removal from a single image. In: IEEE international conference on pattern recognition (ICPR), pp 2821–2826

  43. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 1–9

  44. Tang H, Zhu L, Zhang D, Wang X (2019) Single image rain removal model using pure rain dictionary learning. IET Image Process 13(10):1797–1804

    Article  Google Scholar 

  45. Wang Z, Bovik A, Sheikh H, Simoncelli E (2004) Image quality assessment: From error visibility to structural similarity. IEEE Trans Image Process (TIP) 13(4):600–612

    Article  Google Scholar 

  46. Wang H, Li M, Wu Y, Zhao Q, Meng D (2019) A Survey on Rain Removal from Video and Single Image. arXiv:1909.08326

  47. Wang Y, Liu S, Chen C, Zeng B (2017) A hierarchical approach for rain or snow removing in a single color image. IEEE Trans Image Process (TIP) 26(8):3936–3950

    Article  MathSciNet  Google Scholar 

  48. Wei W, Meng D, Zhao Q, Xu Z, Wu Y (2019) Semi-Supervised Transfer learning for image rain removal. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 3877–3886

  49. Xu L, Ren J, Liu C, Jia J (2014) Deep convolutional neural network for image deconvolution. Advances in Neural Information Processing Systems (NIPS), pp 1790–1798

  50. Xu J, Zhao W, Liu P, Tang X (2012) Removing rain and snow in a single image using guided filter. IEEE International Conference on Computer Science and Automation Engineering (CSAE) 2:304–307

    Article  Google Scholar 

  51. Xu J, Zhao W, Liu P, Tang X (2012) An improved guidance image based method to remove rain and snow in a single image. Comput Inform Sci 5 (3):49

    Google Scholar 

  52. Yang W, Tan R, Feng J, Liu J, Guo Z, Yan S (2017) Deep joint rain detection and removal from a single image. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 1357–1366

  53. Yasarla R, Patel V (2019) Uncertainty Guided Multi-Scale Residual Learning-using a Cycle Spinning CNN for Single Image De-Raining. arXiv:1906.11129

  54. You S, Tan R, Kawakami R, Mukaigawa Y, Ikeuchi K (2015) Adherent raindrop modeling, detection and removal in video. IEEE Trans Pattern Anal Mach Intell (PAMI) 38(9):1721–1733

    Article  Google Scholar 

  55. Zhang H, Patel V (2018) Density-aware single image de-raining using a multi-stream dense network. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 695–704

  56. Zhang H, Patel V (2018) Density-aware single image de-raining using a multi-stream dense network. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 695–704

  57. Zhang H, Sindagi V, Patel V (2019) Image de-raining using a conditional generative adversarial network. IEEE Transactions on Circuits and Systems for Video Technology

Download references

Author information

Authors and Affiliations

  1. ITB A-111, 1280, Main Street west, Hamilton, Ontario, L8S 4K1, Canada

    Hamidreza Fazlali, Shahram Shirani, Michael Bradford & Thia Kirubarajan

Authors
  1. Hamidreza Fazlali
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shahram Shirani
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Michael Bradford
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Thia Kirubarajan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hamidreza Fazlali.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fazlali, H., Shirani, S., Bradford, M. et al. Single image rain/snow removal using distortion type information. Multimed Tools Appl 81, 14105–14131 (2022). https://doi.org/10.1007/s11042-022-12012-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-12012-0

Keywords

Search

Navigation