Skip to main content
SpringerLink
Log in

Video anomaly detection based on 3D convolutional auto-encoder

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Video anomaly detection plays a critical role in public safety and security. However, it is hard to perform supervised due to its characteristics such as definition ambiguity, scene dependency, and sample scarcity. This paper proposes an unsupervised video anomaly detection model, called Spatio-Temporal 3D Convolutional Auto-Encoder model (ST-3DCAE) based on the input of the fused features of both motion and appearance. First, to utilize both motion and appearance information in the scene, the optical flow feature map of the scene is extracted with PWCNet and fused with the original video frame as the model input. Then, the 3DConv module and the Convolution Long Short-Term Memory(ConvLSTM) module are then used for extracting the spatio-temporal features, and the 3DSEblock module is used to screen important features. Finally, the reconstruction error between the input and output of the auto-encoder is used to determine whether the video frames are related to abnormal behavior. The proposed model has been validated on publicly available datasets such as UCSD Pedestrian and Avenue datasets. Experimental results analysis, both qualitative and quantitative, demonstrate the effectiveness of the proposed method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Paul, M., Haque, S.M., Chakraborty, S.: Human detection in surveillance videos and its applications-a review. EURASIP J. Adv. Signal Process. 2013(1), 1–16 (2013)

    Article  Google Scholar 

  2. Chandola, V., Banerjee, A., Kumar, V.: Anomaly detection: a survey. ACM Comput. Surveys (CSUR) 41(3), 1–58 (2009)

    Article  Google Scholar 

  3. Liu, W., Luo, W., Lian, D., Gao, S.: Future frame prediction for anomaly detection—a new baseline, pp. 6536–6545 (2018)

  4. Chong, Y.S., Tay, Y.H.: Modeling representation of videos for anomaly detection using deep learning: a review. arXiv preprint arXiv:1505.00523 (2015)

  5. Sun, D., Yang, X., Liu, M. Y., Kautz, J.: Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume, pp. 8934–8943 (2018)

  6. Hu, X., Dai, J., Huang, Y., et al.: A weakly supervised framework for abnormal behavior detection and localization in crowded scenes. Neurocomputing 383, 270–281 (2020)

    Article  Google Scholar 

  7. Tung, F., Zelek, J.S., Clausi, D.A.: Goal-based trajectory analysis for unusual behaviour detection in intelligent surveillance. Image Vis. Comput. 29(4), 230–240 (2011)

    Article  Google Scholar 

  8. Jiang, F., Wu, Y., Katsaggelos, A.K.: A dynamic hierarchical clustering method for trajectory-based unusual video event detection. IEEE Trans. Image Process. 18(4), 907–913 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  9. Li, C., Han, Z., Ye, Q., Jiao, J.: Visual abnormal behavior detection based on trajectory sparse reconstruction analysis. Neurocomputing 119, 94–100 (2013)

    Article  Google Scholar 

  10. Laxhammar, R., Falkman, G.: Online learning and sequential anomaly detection in trajectories. IEEE Trans. Pattern Anal. Mach. Intell. 36(6), 1158–1173 (2013)

    Article  MATH  Google Scholar 

  11. Bera, A., Kim, S., Manocha, D.: Realtime anomaly detection using trajectory-level crowd behavior learning. pp. 50–57 (2016)

  12. Mehran, R., Oyama, A., Shah, M.: Abnormal Crowd Behavior Detection Using Social Force Model, pp. 935–942. IEEE, Manhattan (2009)

    Google Scholar 

  13. Zhao, B., Fei-Fei, L., Xing, E.P.: Online Detection of Unusual Events in Videos Via Dynamic Sparse Coding, pp. 3313–3320. IEEE, Manhattan (2011)

    Google Scholar 

  14. Kaltsa, V., Briassouli, A., Kompatsiaris, I., Hadjileontiadis, L.J., Strintzis, M.G.: Swarm intelligence for detecting interesting events in crowded environments. IEEE Trans. Image Process. 24(7), 2153–2166 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  15. Kratz, L., Nishino, K.: Anomaly Detection in Extremely Crowded Scenes Using Spatio-temporal Motion Pattern Models, pp. 1446–1453. IEEE, Manhattan (2009)

    Google Scholar 

  16. Mahadevan, V., Li, W., Bhalodia, V., Vasconcelos, N.: Anomaly Detection in Crowded Scenes, pp. 1975–1981. IEEE, Manhattan (2010)

    Google Scholar 

  17. Hu, X., Huang, Y., Gao, X., Luo, L., Duan, Q.: Squirrel-cage local binary pattern and its application in video anomaly detection. IEEE Trans. Inf. Forens. Secur. 14(4), 1007–1022 (2018)

    Article  Google Scholar 

  18. Ullah, H., Altamimi, A.B., Uzair, M., Ullah, M.: Anomalous entities detection and localization in pedestrian flows. Neurocomputing 290, 74–86 (2018)

    Article  Google Scholar 

  19. Zhu, X., Liu, J., Wang, J., Li, C., Lu, H.: Sparse representation for robust abnormality detection in crowded scenes. Pattern Recogn. 47(5), 1791–1799 (2014)

    Article  Google Scholar 

  20. Xu, K., Jiang, X., Sun, T.: Anomaly detection based on stacked sparse coding with intraframe classification strategy. IEEE Trans. Multimedia 20(5), 1062–1074 (2018)

    Article  Google Scholar 

  21. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Adv. Neural. Inf. Process. Syst. 25, 1097–1105 (2012)

    Google Scholar 

  22. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  23. Feng, Y., Yuan, Y., Lu, X.: Deep representation for abnormal event detection in crowded scenes, pp. 591–595 (2016)

  24. Lei, Y., Karimi, H.R., Cen, L., Chen, X., Xie, Y.: Processes soft modeling based on stacked autoencoders and wavelet extreme learning machine for aluminum plant-wide application. Control Eng. Practice 108, 104706 (2021)

    Article  Google Scholar 

  25. Xu, D., Yan, Y., Ricci, E., Sebe, N.: Detecting anomalous events in videos by learning deep representations of appearance and motion. Comput. Vis. Image Understand. 156, 117–127 (2017)

    Article  Google Scholar 

  26. Hasan, M., Choi, J., Neumann, J., Roy-Chowdhury, A.K., Davis, L.S.: Learning temporal regularity in video sequences, pp. 733–742 (2016)

  27. Tran, H.T., Hogg, D.: Anomaly Detection Using a Convolutional Winner-take-all Autoencoder. British Machine Vision Association, Durham (2017)

    Book  Google Scholar 

  28. Chong, Y.S., Tay, Y.H.: Abnormal Event Detection in Videos Using Spatiotemporal Autoencoder, pp. 189–196. Springer, Berlin (2017)

    Google Scholar 

  29. Luo, W., Liu, W., Gao, S.: Remembering History with Convolutional lstm for Anomaly Detection, pp. 439–444. IEEE, Manhattan (2017)

    Google Scholar 

  30. Medel, J.R., Savakis, A.: Anomaly detection in video using predictive convolutional long short-term memory networks. arXiv preprintarXiv:1612.00390 (2016)

  31. Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., et al.: Generative adversarial networks. arXiv preprint arXiv:1406.2661 (2014)

  32. Schlegl, T., Seeböck, P., Waldstein, S.M., Schmidt-Erfurth, U., Langs, G.: Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery, pp. 146–157. Springer, Berlin (2017)

    Google Scholar 

  33. Ravanbakhsh, M., Nabi, M., Sangineto, E., Marcenaro, L., Regazzoni, C., Sebe, N.: Abnormal Event Detection in Videos Using Generative Adversarial Nets, pp. 1577–1581. IEEE, Manhattan (2017)

    Google Scholar 

  34. Ravanbakhsh, M., Sangineto, E., Nabi, M., Sebe, N.: Training Adversarial Discriminators for Cross-channel Abnormal Event Detection in Crowds, pp. 1896–1904. IEEE, Manhattan (2019)

    Google Scholar 

  35. Sabokrou, M., Khalooei, M., Fathy, M., Adeli, E.: Adversarially learned one-class classifier for novelty detection, pp. 3379–3388 (2018)

  36. Jamadandi, A., Kotturshettar, S., Mudenagudi, U.: PredGAN: a deep multi-scale video prediction framework for detecting anomalies in videos, pp. 1–8 (2018)

  37. Song, H., Sun, C., Wu, X., Chen, M., Jia, Y.: Learning normal patterns via adversarial attention-based autoencoder for abnormal event detection in videos. IEEE Trans. Multimedia 22(8), 2138–2148 (2019)

    Article  Google Scholar 

  38. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks, pp. 7132–7141

  39. Shi, X., Chen, Z., Wang, H., Yeung, D.Y., Wong, W.K., Woo, W.c.: Convolutional LSTM network: A machine learning approach for precipitation nowcasting. arXiv preprint arXiv:1506.04214 (2015)

  40. Lu, C., Shi, J., Jia, J.: Abnormal event detection at 150 fps in matlab, pp. 2720–2727 (2013)

  41. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  42. Xu, D., Ricci, E., Yan, Y., Song, J., Sebe, N.: Learning deep representations of appearance and motion for anomalous event detection. arXiv preprint arXiv:1510.01553 (2015)

  43. Fan, Y., Wen, G., Li, D., Qiu, S., Levine, M.D., Xiao, F.: Video anomaly detection and localization via gaussian mixture fully convolutional variational autoencoder. Comput. Vis. Image Understand. 195, 102920 (2020)

    Article  Google Scholar 

  44. Li, N., Chang, F.: Video anomaly detection and localization via multivariate gaussian fully convolution adversarial autoencoder. Neurocomputing 369, 92–105 (2019)

    Article  Google Scholar 

  45. Yang, D., Karimi, H.R., Sun, K.: Residual wide-kernel deep convolutional auto-encoder for intelligent rotating machinery fault diagnosis with limited samples. Neural Netw. 141, 133–144 (2021)

    Article  Google Scholar 

  46. Lv, H., Chen, C., Cui, Z., Xu, C., Li, Y.,Yang, J.: Learning normal dynamics in videos with meta prototype network. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 15425–15434 (2021)

  47. Sultani, W., Chen, C., Shah, M.: Real-world anomaly detection in surveillance videos. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 6479–6488 (2018)

Download references

Acknowledgements

This paper is supported by the National Key Research and Development Program of China (2019YFB1705702, 2018YFC1313803). National Natural Science Foundation of China (Fund No. 62175037), and Shanghai Science and Technology Innovation Action Plan (Project No. 20JC1416500).

Author information

Authors and Affiliations

  1. School of Optical-Electrical and Computer Engineering, University of Shanghai For Science and Technology, No.516 Jungong Road, Shanghai, 200093, China

    Xing Hu, Jing Lian, Dawei Zhang & Xiumin Gao

  2. Academy for Engineering and Technology, Fudan University, Shanghai, China

    Linhua Jiang & Wenmin Chen

Authors
  1. Xing Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jing Lian
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dawei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiumin Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Linhua Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Wenmin Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Dawei Zhang, Linhua Jiang or Wenmin Chen.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hu, X., Lian, J., Zhang, D. et al. Video anomaly detection based on 3D convolutional auto-encoder. SIViP 16, 1885–1893 (2022). https://doi.org/10.1007/s11760-022-02148-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-022-02148-9

Keywords

Mathematics Subject Classification

Search

Navigation