Skip to main content
SpringerLink
Log in

Dual-branch network with memory for video anomaly detection

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Anomaly event detection is a video surveillance technology automatically analyzing video sequences without manual intervention by employing machine learning and computer vision technology. In the existing approaches, most of them are utilized to reconstruct or predict the video frame based on an autoencoder (AE). However, impacted by the powerful characterization capabilities of Convolutional Neural Network (CNN), abnormal frames will be improperly reconstructed into normal frames. To solve the above issue, an autoencoder, based on a branch framework of reconstruction and prediction in training, is proposed. A memory module is adopted to reduce the reconstruction error, which is capable of enhancing the robustness of the autoencoder as a prototype memory module. The prediction of high-quality future frames can effectively prevent the reconstruction of abnormal frames, and the two branches can be supplemented with their respective loss functions, thus further elevating the performance of video anomaly detection. The framework for this study is trained from end to end. The methodology put forth in this article is extensively verified on three publicly available data sets, and its robustness to the uncertainty for the common occurrence as well as the efficiency to the sensitivity for the abnormalies are also confirmed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Chandola, V., Banerjee, A., Kumar, V.: Anomaly detection: a survey. ACM Comput. Surv. (CSUR) 41(3), 1–58 (2009)

    Article  Google Scholar 

  2. Khaleghi, A., Mohammad, MS.: Improved anomaly detection in surveillance videos based on a deep learning method. 2018 8th Conference of AI & Robotics and 10th RoboCup Iranopen International Symposium (IRANOPEN). (2018)

  3. Saligrama, V., Konrad, J., Jodoin, P.-M.: Video anomaly identification. IEEE Signal Process. Mag. 27(5), 18–33 (2010). https://doi.org/10.1109/MSP.2010.937393

    Article  Google Scholar 

  4. Suarez, J.J.P., Prospero, C., Naval Jr.: A survey on deep learning techniques for video anomaly detection. arXiv preprint (2020). arXiv:2009.14146.https://doi.org/10.48550/arXiv.2009.14146

  5. Wang, Z., Yang, Z.: A promotion method for generation error-based video anomaly detection. Pattern Recognit. Lett. 140, 88–94 (2020). https://doi.org/10.1016/j.patrec.2020.09.019

    Article  Google Scholar 

  6. Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska-Barwinska, A., Colmenarejo, S.G., Grefenstette, E., Ramalho, T., Agapiou, J.: Hybrid computing using a neural network with dynamic external memory. Nature 538(7626), 471–476 (2016)

    Article  Google Scholar 

  7. Luo, W., Wen, L., Shenghua, G.: Remembering history with convolutional lstm for anomaly detection. 2017 IEEE International Conference on Multimedia and Expo (ICME). (2017). https://doi.org/10.1109/ICME.2017.8019325

  8. Liu, W., Luo, W., Lian, D.: Future frame prediction for anomaly detection–a new baseline. 2018 Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 18–23, 2018, Salt Lake City, UT, USA. New York, 6536–6545. (2018). https:// doi.org/https://doi.org/10.48550/arXiv.1712.09867

  9. Park, H., Jongyoun, N., Bumsub, H.: Learning memory-guided normality for anomaly detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. (2020)

  10. Lai, Y., Yahong, H., Yaowei, W.: Anomaly Detection with Prototype-Guided Discriminative Latent Embeddings. (2021). arXiv preprint arXiv:2104.14945

  11. Zong, B.: Deep autoencoding gaussian mixture model for unsupervised anomaly detection. International conference on learning representations. (2018)

  12. Dong, G., Lingqiao, L., Vuong, L., Budhaditya, S., Moussa, R.M., Svetha, V., Anton van den H.: Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection. Proceedings of the IEEE/CVF International Conference on Computer Vision. (2019)

  13. Li, W., Mahadevan, V., Vasconcelos, N.: Anomaly detection and localization in crowded scenes. IEEE Trans. Pattern anal. Mach. Intell. 36(1), 18–32 (2013). https://doi.org/10.1109/TPAMI.2013.111

    Article  Google Scholar 

  14. Lu, C., Jianping, S., Jiaya, J.: Abnormal event detection at 150 fps in matlab. Proceedings of the IEEE international conference on computer vision. (2013)

  15. Luo, W., Wen, L., Shenghua, G.: A revisit of sparse coding based anomaly detection in stacked RNN framework. Proceedings of the IEEE International Conference on Computer Vision. (2017)

  16. Chen, Y., Xiang, S.Z., Thomas S.H.: One-class SVM for learning in image retrieval. Proceedings 2001 International Conference on Image Processing (Cat. No. 01CH37205). (2001)

  17. Schölkopf, B., Alexander, J.S., Francis, B.: Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT Press (2002)

    Google Scholar 

  18. Lukas, R., Robert, V., Nico, G., Lucas, D., Shoaib A.S., Alexander, B., Emmanuel M., Marius, K.: Deep one-class classification. International conference on machine learning. PMLR. (2018)

  19. Chalapathy, R., Aditya K.M., Sanjay, C.: Anomaly detection using one-class neural networks. (2018) arXiv preprint arXiv:1802.06360. https://doi.org/10.48550/arXiv.1802.06360

  20. Zimek, A., Schubert, E., Kriegel, H.-P.: A survey on unsupervised outlier detection in high-dimensional numerical data. Stat Anal. Data Min 5(5), 363–387 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  21. Zhou, J.T., Du, J., Zhu, H., Peng, X., Liu, Y., Goh, R.S.M.: Anomalynet: an anomaly detection network for video surveillance. IEEE Trans. Inform. Forensic. Secur. 14(10), 2537–2550 (2019)

    Article  Google Scholar 

  22. Ganokratanaa, T., Aramvith, S., Sebe, N.: Video anomaly detection using deep residual-spatiotemporal translation network. Pattern Recognition Letters. Volume 155, March, Pages 143–150 (2022)

  23. Sabzalian, B., Marvi, H., Ahmadyfard, A.: Deep and sparse features for anomaly detection and localization in video. In: 2019 4th International Conference on Pattern Recognition and Image Analysis. IPRIA, 173–178 (2019)

  24. Xu, K., Sun, T., Jiang, X.: Video anomaly detection and localization based on an adaptive intra-frame classification network. IEEE Trans. Multimed. 22(2), 394–406 (2020)

    Article  Google Scholar 

  25. Landi, F., Snoek, C.G.M., Cucchiara, R.: Anomaly locality in video surveillance. ArXiv abs/1901.10364 (2019)

  26. Fan, Y., Wen, G., Li, D., Qiu, S., Levine, M.D., Xiao, F.: Video anomaly detection and localization via Gaussian mixture fully convolutional variational autoencoder. Comput. Vis. Image Underst. 195, 102920 (2020). https://doi.org/10.1016/j.cviu.2020.102920

    Article  Google Scholar 

  27. Mohammadi, B., Mahmood, F., Mohammad, S.: Image/video deep anomaly detection: a survey. (2021) arXiv preprint arXiv:2103.01739

  28. Pang, G., Yan, C., Shen, C., et al.: Self-trained deep ordinal regression for end-to-end video anomaly detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12173–12182 (2020)

  29. Hasan, M., Choi, J., Neumann, J., Amit K.R.-C., Larry S.D.: Learning temporal regularity in video sequences. Proceedings of the IEEE conference on computer vision and pattern recognition. (2016)

  30. Zhao, Y., Deng, B., Shen, C., Liu, Y., Lu, H., Hua, X-S.: Spatio-temporal autoencoder for video anomaly detection. Proceedings of the 25th ACM international conference on Multimedia. (2017)

  31. Ravanbakhsh, M., Nabi, M., Sangineto, E., Marcenaro, L., Regazzoni, C., Sebe, N.: Abnormal event detection in videos using generative adversarial nets. 2017 IEEE International Conference on Image Processing, ICIP. (2017)

  32. Li, D., Nie, X., et al.: Context-related video anomaly detection via generative adversarial network. Pattern Recognit. Lett. 156, 183–189 (2022). https://doi.org/10.1016/j.patrec.2022.03.004

    Article  Google Scholar 

  33. Medel, J.R., Andreas, S.: Anomaly detection in vieo using predictive convolutional long short-term memory networks. (2016). arXiv preprint arXiv:1612.00390

  34. Yu, G., Wang, S., Cai, Z., Zhu, E., Xu, C., Yin, J., Kloft, M.: Cloze test helps: Effective video anomaly detection via learning to complete video events. In Proceedings of the 28th ACM International Conference on Multimedia, 583–591 (2020)

  35. Salisu, A.: Towards a data-driven adaptive anomaly detection system for human activity. Pattern Recognit. Lett. 145, 200–207 (2021)

    Article  Google Scholar 

  36. Yao, Y., Wang, X., Xu, M., Pu, Z., Atkins, E., Crandall, D.: When, where, and what? A new dataset for anomaly detection in driving videos. (2020) arXiv preprint arXiv:2004.03044

  37. Yao, T.: Integrating prediction and reconstruction for anomaly detection. Pattern Recognit. Lett. 129, 123–130 (2020)

    Article  Google Scholar 

  38. Nguyen, T.-N., Jean, M. Anomaly detection in video sequence with appearance-motion correspondence. Proceedings of the IEEE/CVF International Conference on Computer Vision. (2019)

  39. Chang, Y., Tu, Z., Xie, W., et al.: Video anomaly detection with spatio-temporal dissociation[J]. Pattern Recognit. 122, 108213 (2022)

    Article  Google Scholar 

  40. Liu, Z., Nie, Y., Long, C.: A Hybrid Video Anomaly Detection Framework via Memory-Augmented Flow Reconstruction and Flow-Guided Frame Prediction Proceedings of the IEEE/CVF International Conference on Computer Vision. 13588–13597 (2021)

  41. Ronneberger, O., Philipp, F., Thomas, B.: U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical image computing and computer-assisted intervention. Springer, Cham. (2015)

  42. Mathieu, M., Camille, C., Yann, L.: Deep multi-scale video prediction beyond mean square error. (2015) arXiv preprint arXiv:1511.05440.https://doi.org/10.48550/arXiv.1511.05440

  43. Müller, S.G., Frank, H.: Trivialaugment: Tuning-free yet state-of-the-art data augmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision. (2021)

  44. Cubuk, E.D.: Autoaugment: Learning augmentation policies from data. (2018) arXiv preprint arXiv:1805.09501. https://doi.org/10.48550/arXiv.1805.09501

  45. Ho, D.: Population based augmentation: Efficient learning of augmentation policy schedules. International Conference on Machine Learning. PMLR. (2019)

  46. Cubuk, E.D.: Randaugment: Practical automated data augmentation with a reduced search space. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. (2020)

  47. Paszke A, Gross S, Chintala S, Chanan G, Yang E, DeVito Z, Lin Z, Desmaison A, Antiga L, Lerer A.: Automatic differentiation in pytorch. NIPS. 30 (2017)

  48. Kim, J., Kristen, G.: Observe locally, infer globally: a space-time MRF for detecting abnormal activities with incremental updates. 2009 IEEE conference on computer vision and pattern recognition. IEEE. (2009)

  49. Mahadevan, V., Li, W., Bhalodia, V., Vasconcelos, N.: Anomaly detection in crowded scenes. 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE. (2010)

  50. Xu, D., Ricci, E., Yan, Y., Song, J., Sebe, N.: Learning deep representations of appearance and motion for anomalous event detection. (2015) arXiv preprint arXiv:1510.01553. https://doi.org/10.48550/arXiv.1510.01553

  51. Tudor Ionescu, R., Smeureanu, S., Alexe B., Popescu M.: Unmasking the abnormal events in video. Proceedings of the IEEE international conference on computer vision. (2017)

  52. Hinami, R., Tao M., Shin'ichi S.: Joint detection and recounting of abnormal events by learning deep generic knowledge. Proceedings of the IEEE International Conference on Computer Vision. (2017)

  53. Chang, Y., Tu, Z., Xie, W., et al.: Clustering driven deep autoencoder for video anomaly detection[C]//European Conference on Computer Vision. Springer, Cham, 329–345 (2020)

  54. Pang, W., He, Q., Li, Y.: Predicting skeleton trajectories using a Skeleton-Transformer for video anomaly detection. Multimed. Syst. (2022). https://doi.org/10.1007/s00530-022-00915-9

    Article  Google Scholar 

  55. Le, V.T., Kim, Y.G.: Attention-based residual autoencoder for video anomaly detection[J]. Appl. Intell. (2022). https://doi.org/10.1007/s10489-022-03613-1

    Article  Google Scholar 

  56. Chang, Y., Tu, Z., Xie, W., Yuan, J.: Clustering driven deep autoencoder for video anomaly detection, in Proceedings of the European Conference on Computer Vision. Springer, 329–345 (2020)

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant No. 61966022, in part by the Natural Science Foundation of Gansu Province (No.21JR7RA300), Open project of Gansu Dunhuang Cultural Relics Protection and Research Center (No. GDW2021YB15), Excellent graduate innovation in Gansu Project of the Stars (2021CXZX-555).

Author information

Authors and Affiliations

  1. College of Intelligence and Computing, Tianjin University, Tianjin, 300354, China

    Dicong Wang & Qinghua Hu

  2. School of Electronic and Information Engineering, LanZhou JiaoTong University, Lanzhou, 730070, China

    Dicong Wang & Kaijun Wu

Authors
  1. Dicong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qinghua Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kaijun Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, datacollection and analysis were performed by [DW]. The first draft of the manuscript was written by [DW] and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Dicong Wang.

Ethics declarations

Conflict of interest

The authors have declared no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, D., Hu, Q. & Wu, K. Dual-branch network with memory for video anomaly detection. Multimedia Systems 29, 247–259 (2023). https://doi.org/10.1007/s00530-022-00991-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-022-00991-x

Keywords

Search

Navigation