Elsevier

Neurocomputing

Volume 207, 26 September 2016, Pages 365-373
Neurocomputing

Scene-free multi-class weather classification on single images

https://doi.org/10.1016/j.neucom.2016.05.015Get rights and content

Abstract

Multi-class weather classification is a fundamental and significant technique which has many potential applications, such as video surveillance and intelligent transportation. However, it is a challenging task due to the diversity of weather and lack of discriminate feature. Most existing weather classification methods only consider two-class weather conditions such as sunny-rainy or sunny-cloudy weather. Moreover, they predominantly focus on a fixed scene such as popular tourism and traffic scenario. In this paper, we propose a novel method for scene-free multi-class weather classification from single images based on multiple category-specific dictionary learning and multiple kernel learning. To improve the discrimination of image representation and enhance the performance of multiple weather classification, our approach extracts multiple weather features and learns dictionaries based on these features. To select a good subset of features, we utilize multiple kernel learning algorithm to learn an optimal linear combination of feature kernels. In addition, to evaluate the proposed approach, we collect an outdoor image set that contains 20 K images, called MWI (Multi-class Weather Image) set. Experimental results show the effectiveness of the proposed method.

Introduction

Most of the existing methods in the field of computer vision are based on the assumption that the weather in outdoor images or videos is clear. However, different weather conditions such as rain, snow or haze will decrease the quality of images or videos, as shown in Fig. 1. Such effects may significantly degrade the performances of outdoor vision systems which relies on image/video feature extraction or visual attention modeling. Hence, the applications of weather classification are numerous, such as the detection and observation of weather conditions, image/video analysis, the reliability improvement of video surveillance systems. In this paper, we target at the problem of classifying multiple weather, such as sunny, rainy, snowy, and haze from single images.

Despite its remarkable value, multi-class weather classification has not been thoroughly studied. Some previous researches [1], [2], [3] focused on weather recognition from vehicle camera images for driver assistance. Most of these methods are only able to recognize rainy weather. Furthermore, the applications are limited due to the relatively fixed target scenes. Recently, the authors of [4], [5] focused on two-class weather recognition, include sunny and cloudy. The authors of [4] estimated the weather conditions of popular tourism from images of the same scene. The authors of [5] proposed a collaborative learning framework via analyzing multiple weather cues for two-class weather recognition from single images. The authors of [6] proposed a method to label images of the same scene with three weather conditions including sunny, cloudy, and overcast. The authors of [7] proposed an approach for multi-class weather classification, which could be used for the traffic scene only. However, approaches for the fixed scene weather classification are not able to be applied in the practiced systems due to the following two reasons. First, it needs to learn different classifiers for different scenes. Second, it is hard to collect the training image set in any scene. The darkness in the night is another factor that results in the decrease of image quality. The authors of [8] proposed a Color Estimation Model for night removal from a single input image. They use a guided statistical Dark-to-Day prior to direct optimal performance.

Different from the above works, we propose a new framework for classifying multi-class weather from single images in any scene, which is based on dictionary learning and multiple kernel learning (MKL). Implementation of the kernel idea, however, entails substantial challenges. First, it is difficult to find suitable features to discriminate different weather conditions. Second, the features might be heterogeneous and the feature vectors are high-dimensional. Aiming at solving the above challenges, we first extract multiple features to represent different weather conditions. For example, the sky and shadow features can indicate sunny weather. The dark channel feature can indicate the haze weather. The HOG (Histogram of Oriented Gradients) based template matching feature can indicate rainy weather. The snowflake noise feature can indicate snowy feature. Some global features like contrast and saturation are used to distinguish multi-class weather. To improve the discrimination of image representation and enhance the performance of multiple weather classification, our approach extracts multiple weather features and learns multiple dictionaries based on these features. Then, we use multiple kernel learning algorithm to learn an optimal linear combination of feature kernels for selecting a good subset of features.

The contributions of this paper are as follows:

  • We propose a scene-free multi-class weather classification framework by fusing multiple image features and learning multiple dictionaries. To our knowledge, this work is one of the first attempts towards single image multi-class weather classification.

  • We propose two methods for detecting rain streak and snowflake from single images respectively. First, we propose a Histogram of Orientation Gradients (HOG) based template matching method to detect the rain streaks. Moreover, we regard snowflake as a kind of noise and define several rules to detect and describe snowflakes.

  • We collect an outdoor image set contains 20 K images called MWI (Multi-class Weather Image) set, which provides an extensive testbed for the evaluation of existing methods and development of new approaches.

Section snippets

Related work

In this section, we give a brief review on dictionary learning and multiple kernel learning.

Our approach

In this section, we will propose a general framework for multiple weather classification from single images. The framework is composed by the extraction of multiple features including sky, shadow, rain streak, snowflake, dark channel, contrast and saturation; learning multiple dictionaries; selection and classification features via the learned sparse code and multiple kernel learning algorithm. Next, we will describe each component in detail.

Dataset

We evaluate our approach on our dataset called MWI (Multi-class Weather Image) set. It contains 20 K images obtained from many web albums and films, such as Flicker, Picasa, MojiWeather, Poco, Fengniao. As shown in Fig. 4, most of the images have totally different backgrounds. The distribution statistics on MWI dataset is listed in Table 1. The images are collected by five volunteers, and they chose images with their own common sense. Then, they were asked to label all the images according to

Conclusion

We presented a framework for multi-class weather classification from single images in any scene. Our approach learned multi-feature dictionaries on each feature to improve the discrimination of image representation and enhance the performance of multiple weather classification. We utilized MKL algorithm to learn an optimal linear combination of feature kernels. For training and testing our approach, we collected the MWI (Multi-class Weather Image) set. We evaluated our approach on the dataset,

Acknowledgment

The research reported in this paper is supported by the National Natural Science Foundation of China under Grant Nos. 61332005 and 61402048; The Funds for Creative Research Groups of China under Grant No. 61421061; The Cosponsored Project of Beijing Committee of Education; The Beijing Training Project for the Leading Talents in S&T (ljrc 201502).

Zheng Zhang received her B.S. degree and M.S. degree in 2009 and 2012. She is currently studying on the subject of Computer Science and Technology to pursue her Ph.D. degree in the school of Computer Science, Beijing University of Posts and Telecommunications, China. Her research interests include multimedia computing, deep learning and computer vision.

References (27)

  • H.Y. Fu, H.D. Ma, S.X. Wu, Night removal by color estimation and sparse representation, in: Proceedings of IEEE...
  • J. Mairal, F. Bach, J. Ponce, Task-driven dictionary learning, IEEE Trans. Pattern Anal. Mach. Intell. 34 (2012)...
  • S. Gao, I.W. Tsang, L.-T. Chia, Sparse representation with kernels, IEEE Trans. Image Process. 22 (2013)...
  • Cited by (61)

    • HDNet: A dual-stream network with progressive fusion for image hazing detection

      2022, Journal of Information Security and Applications
    • AWDMC-Net: Classification of Adversarial Weather Degraded Multiclass scenes using a Convolution Neural Network

      2022, Computer Vision and Image Understanding
      Citation Excerpt :

      Then, multiple kernel learning (MKL) is used to learn an adaptive classifier and fuse these local and global features, thereby classifying the weather conditions. Likewise, Zhang et al. (2016b) further extended their previous weather classification method (Zhang and Ma, 2015), where selection and classification of multiple weather features were performed via learned sparse code followed by the MKL algorithm. In Chu et al. (2017), W.T. Chu et al. observed the correlations between the properties of various weather conditions and developed computational models based on a random forest classifier for weather recognition.

    • WCATN: Unsupervised deep learning to classify weather conditions from outdoor images

      2022, Engineering Applications of Artificial Intelligence
    • A CNN-based multi-task framework for weather recognition with multi-scale weather cues

      2022, Expert Systems with Applications
      Citation Excerpt :

      In Song et al. (2014), Song et al. also exploit a series of global features, including image noisy, edge gradient energy, contrast, saturation and so on, to recognize 4 weather conditions. Different from the aforementioned works, both local and global features are explored in Zhang and Ma (2015) and Zhang et al. (2016) to address the problem of multi-class weather classification. Furthermore, a multiple kernel learning method is proposed in Zhang and Ma (2015) to fuse the features.

    View all citing articles on Scopus

    Zheng Zhang received her B.S. degree and M.S. degree in 2009 and 2012. She is currently studying on the subject of Computer Science and Technology to pursue her Ph.D. degree in the school of Computer Science, Beijing University of Posts and Telecommunications, China. Her research interests include multimedia computing, deep learning and computer vision.

    Huadong Ma is a professor and director of Beijing Key Lab of Intelligent Telecommunications Software and Multimedia, executive dean of School of Computer Science, Beijing University of Posts and Telecommunications, China. He received his Ph.D. degree in Computer Science from the Institute of Computing Technology, Chinese Academy of Science in 1995, his M.S. degree in Computer Science from Shenyang Institute of Computing Technology, Chinese Academy of Science in 1990, and his B.S. degree in Mathematics from Henan Normal University in 1984. He visited UNU/IIST as a research fellow in 1998 and 1999, respectively. From 1999 to 2000, he held a visiting position in the Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, Michigan. He was a visiting professor at Hong Kong University of Science and Technology from December 2006 to February 2007. His current research focuses on multimedia system and networking, sensor networks and Internet of Things, and he has published over 180 papers and four books on these fields. He is a member of IEEE and ACM.

    Huiyuan Fu received his B.S. degree in Computer Science from Xi׳an University of Posts and Telecommunications, China, in 2008. He received the Ph.D. degree in Computer Science from Beijing University of Posts and Telecommunications, China, in 2014. Currently, he is a lecturer at the School of Computer Science, Beijing University of Posts and Telecommunications, China. His research interests include big video data, multimedia systems and computer vision.

    Cheng Zhang, received his B.S. degree in 2013. He is currently working toward the Master degree at the School of Computer Science, Beijing University of Posts and Telecommunications, China. His research interests include multimedia computing and computer vision.

    View full text