Abstract
Identifying anomalous activity is a heavy task, and this has led to the progression in the domain of deep learning for video surveillance. With the development of deep learning, anomaly detection techniques have been widely used to improve the performance of various applications, including vision detection systems. However, it is still difficult to apply them directly to practical applications which usually involve the lack of abnormal samples and diversity. This paper proposes a novel Stacked Auto Encoder (SAE) and Extreme Learning Machine (ELM) abnormal detection framework based on multiples features. These features are connected to speed of movement and appearance and fed to a new neural network architecture as temporal and spatiotemporal streams. The use of ELM algorithms with an exceptionally fast learning speed when dealing with abnormal activity localization problems in addition to excellent generalization abilities, a deep learning network achieves a good performance with quick learning speed to further improve the regression performance. The strength of our proposed approaches is demonstrated by experiments with measured abnormal activities’ data. This approach can accurately identify and precisely locate abnormal events.
Similar content being viewed by others
Data availability
All the web access links for the datasets and the machine learning tools used were indicated in ‘References’. Some pseudocodes were presented in this paper. Any questions about implementation details could be addressed to the corresponding author.
References
Abdullah F, Gochoo M, Jalal A (2021) Multi-person tracking and crowd behavior detection via particles gradient motion descriptor and improved entropy classifier. Entropy 23:628
Avenue Dataset. Detection of unusual crowd activity. http://www.cse.cuhk.edu.hk/leojia/projects/detectabnormal/dataset.html (Accessed Sept 28, 2022)
Aziz S, Youssef F (2018) Traffic sign recognition based on multi-feature fusion and ELM classifier. Proc Comput Sci 127:146–153
Berlin SJ, John M (2020) Spiking neural network based on joint entropy of optical flow features for human action recognition. The visual computer 1–15
Bilinski P, Bremond F (2016) Human violence recognition and detection in surveillance videos. In 2016 13th IEEE international conference on advanced video and signal based surveillance (AVSS), pp 30–36. IEEE
Chidananda K, Siva Kumar AP (2022) Human anomaly detection in surveillance videos: a review. Information and Communication Technology for Competitive Strategies (ICTCS 2020), 791–802
Chong YS, Tay YH (2017) Abnormal event detection in videos using spatiotemporal autoencoder. In International symposium on neural networks, pp 189–196. Springer, Cham
Coşar S, Donatiello G, Bogorny V, Garate C, Alvares LO, Brémond F (2017) Toward abnormal trajectory and event detection in video surveillance. IEEE Trans Circ Syst Video Technol 27(3):683–695
Cui X, Liu Q, Gao M, Metaxas DN (2011) Abnormal detection using interaction energy potentials. In: CVPR 2011, pp 3161–3167
Ege Can Ö (2020) Two-stage sparse representation based abnormal crowd event detection in videos
George M, Jose BR, Mathew J, Kokare P (2019) Autoencoder-based abnormal activity detection using parallelepiped spatio-temporal region. IET Comput Vis 13(1):23–30
George M, Jose BR, Mathew J, Kokare P (2019) Autoencoder-based abnormal activity detection using parallelepiped spatiotemporal region. IET Comput Vis 13(1):23–30
Gnouma M, Ejbali R, Zaied M (2017) Human fall detection based on block matching and silhouette area. In ninth international conference on machine vision (ICMV 2016) (Vol 10341 pp 18–22). SPIE
Gnouma M, Ejbali R, Zaied M (2019) Video anomaly detection and localization in crowded scenes. In International joint conference: 12th international conference on computational intelligence in security for information systems (CISIS), pp 87–96. Springer, Cham
Gnouma M, Ladjailia A, Ejbali R, Zaied M (2019) Stacked sparse autoencoder and history of binary motion image for human activity recognition. Multimed Tools Appl 78(2):2157–2179
Gong D, Liu L, Le V, Saha B, Mansour MR, Venkatesh S, Hengel AVD (2019) Memorizing normality to detect anomaly: memory-augmented deep autoencoder for unsupervised anomaly detection. In proceedings of the IEEE/CVF international conference on computer vision, pp 1705–1714
Guangli WU, Liping LIU, Chen Z, Dengtai TAN (2019) Video abnormal event detection based on ELM. In 2019 IEEE 4th International Conference on Signal and Image Processing (ICSIP), pp 367–371. IEEE
Hasan M, Choi J, Neumann J, Roy-Chowdhury AK, Davis LS (2016) Learning temporal regularity in video sequences. In proceedings of the IEEE conference on computer vision and pattern recognition, pp 733–742
Hassner T, Itcher Y, Kliper-Gross O (2012) Violent flows: real-time detection of violent crowd behavior. In 2012 IEEE computer society conference on computer vision and pattern recognition workshops (pp 1–6). IEEE
Huang GB, Wang DH, Lan Y (2011) Extreme learning machines: a survey. Int J Mach Learn Cybern 2(2):107–122
Kantorov V, Laptev I (2014) Efficient feature extraction, encoding and classification for action recognition. In proceedings of the IEEE conference on computer vision and pattern recognition, pp 2593–2600
Labbedi S, Gnouma M, Ejbali R, Zaied M (2021) Violent scenes detection based on connected component analysis. In thirteenth international conference on machine vision (Vol 11605, p 1160527). International Society for Optics and Photonics
Lee S, Kim HG, Ro YM (2019) BMAN: bidirectional multi-scale aggregation networks for abnormal event detection. IEEE Trans Image Process 29:2395–2408
Li N, Chang F (2019) Video anomaly detection and localization via multivariate gaussian fully convolution adversarial autoencoder. Neurocomputing 369:92–105
Li T, Chen X, Zhu F, Zhang Z, Yan H (2021) Two-stream deep spatial-temporal auto-encoder for surveillance video abnormal event detection. Neurocomputing 439:256–270
Li W, Mahadevan V, Vasconcelos N (2013) Anomaly detection and localization in crowded scenes. IEEE Trans Pattern Anal Mach Intell 36(1):18–32
Liu W, Luo W, Lian D, Gao S (2018) Future frame prediction for anomaly detection - a new baseline. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, pp 6536–6545
Lloyd K, Rosin PL, Marshall D, Moore SC (2017) Detecting violent and abnormal crowd activity using temporal analysis of grey level co-occurrence matrix (GLCM)-based texture measures. Mach Vis Appl 28(3):361–371
Lu C, Shi J, Jia J (2013) Abnormal event detection at 150 fps in matlab. In proceedings of the IEEE international conference on computer vision pp 2720–2727
Mahadevan V, Li W, Bhalodia V, Vasconcelos N (2010) Anomaly detection in crowded scenes. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp 1975–1981. IEEE
Mehran R, Oyama A, Shah M (2009) Abnormal crowd behavior detection using social force model. In 2009 IEEE conference on computer vision and pattern recognition (pp 935–942). IEEE
Mirmahboub B, Samavi S, Karimi N, Shirani S (2012) Automatic monocular system for human fall detection based on variations in silhouette area. IEEE Trans Biomed Eng 60(2):427–436
Mishra P, Varadharajan V, Tupakula U, Pilli ES (2018) A detailed investigation and analysis of using machine learning techniques for intrusion detection. IEEE Commun Surv Tutor 21(1):686–728
Nguyen TN, Meunier J (2019) Anomaly detection in video sequence with appearance-motion correspondence. In: 2019 IEEE/CVF international conference on computer vision (ICCV), pp 1273–1283
Primartha R, Tama BA (2017) Anomaly detection using random forest: a performance revisited. In 2017 international conference on data and software engineering (ICoDSE), pp 1–6. IEEE
Rayi VK, Mishra SP, Naik J, Dash PK (2021) Adaptive VMD based optimized deep learning mixed kernel ELM autoencoder for single and multistep wind power forecasting Energy 122585
Sezer ES, Can AB (2018) Anomaly detection in crowded scenes using log-Euclidean covariance matrix. In VISIGRAPP (4: VISAPP), pp 279–286
Sudhakaran S, Lanz O (2017) Learning to detect violent videos using convolutional long short-term memory. In 2017 14th IEEE international conference on advanced video and signal based surveillance (AVSS), pp 1-6. IEEE
Sun K, Zhang J, Zhang C, Hu J (2017) Generalized extreme learning machine autoencoder and a new deep neural network. Neurocomputing 230:374–381
Tudor Ionescu R, Smeureanu S, Alexe B, Popescu M (2017) Unmasking the abnormal events in video. In proceedings of the IEEE international conference on computer vision, pp 2895–2903
Vishwakarma DK, Dhiman C (2019) A unified model for human activity recognition using spatial distribution of gradients and difference of Gaussian kernel. Vis Comput 35(11):1595–1613
Wang L, Qiao Y, Tang X (2015) Action recognition with trajectory-pooled deep-convolutional descriptors. In proceedings of the IEEE conference on computer vision and pattern recognition, pp 4305–4314
Wang J, Xu Z (2015) Crowd anomaly detection for automated video surveillance, IET
Wang S, Zhu E, Yin J, Porikli F (2018) Video anomaly detection and localization by local motion based joint video representation and OCELM. Neurocomputing 277:161–175
Wolpert DH (1992) Stacked generalization. Neural Netw 5(2):241–259
Xu D, Ricci E, Yan Y, Song J, Sebe N (2015) Learning deep representations of appearance and motion for anomalous event detection. arXiv preprint arXiv:1510.01553
Xu D, Yan Y, Ricci E, Sebe N (2017) Detecting anomalous events in videos by learning deep representations of appearance and motion. Comput Vis Image Underst 156:117–127
Yahia S, Said S, Zaied M (2020) A novel classification approach based on extreme learning machine and wavelet neural networks. Multimed Tools Appl 79(19):13869–13890
Yahia S, Said S, Zaied M (2022) Wavelet extreme learning machine and deep learning for data classification. Neurocomputing 470:280–289
Yuan Y, Feng Y, Lu X (2018) Structured dictionary learning for abnormal event detection in crowded scenes. Pattern Recogn 73:99–110
Zhang Y, Dong L, Li S, Li J (2014) Abnormal crowd behavior detection using interest points. In 2014 IEEE international symposium on broadband multimedia systems and broadcasting, pp 1–4. IEEE
Zhang T, Jia W, Yang B, Yang J, He X, Zheng Z (2017) MoWLD: a robust motion image descriptor for violence detection. Multimed Tools Appl 76(1):1419–1438
Zhou JT, Du J, Zhu H, Peng X, Liu Y, Goh RSM (2019) Anomalynet: an anomaly detection network for video surveillance. IEEE Trans Inf Forensics Secur 14(10):2537–2550
Acknowledgements
The authors would like to acknowledge the financial support of this work by grants from General Direction of Scientific Research (DGRST), Tunisia, under the ARUB program.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interests
The authors declare that they have no conflict of interest.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Gnouma, M., Ejbali, R. & Zaied, M. A two-stream abnormal detection using a cascade of extreme learning machines and stacked auto encoder. Multimed Tools Appl 82, 38743–38770 (2023). https://doi.org/10.1007/s11042-023-15060-2
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-023-15060-2