Skip to main content
SpringerLink
Log in

A two-stage adversarial Transformer based approach for multivariate industrial time series anomaly detection

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Sensors in complex industrial systems generate multivariate time series data, frequently leading to diverse abnormal patterns that pose challenges for detection. The existing multivariate abnormal detection methods may encounter difficulties when applied to datasets with low dimensions or sparse relationships between variables. To address these issues, this study proposes a two-stage adversarial Transformer-based anomaly detection method. On the one hand, an autoregressive temporal convolutional network component is embedded before the multi-head attention module to capture features encompassing long-term and local information. Besides, this component utilizes a trainable neural network instead of the vanilla Transformer’s absolute position encoding, resulting in enhanced position information. On the other hand, the proposed two-stage adversarial learning strategy allows the model to effectively learn intricate multivariate data patterns via constraining latent space, thereby enhancing anomaly detection performance. Our method achieves F1 scores of 0.9679, 0.7947, and 0.6452 on a real-world dataset and two public industrial sensor datasets, demonstrating superior overall anomaly detection performance compared to recent advanced works.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Algorithm 1
Fig. 7
Algorithm 2
Fig. 8

Similar content being viewed by others

Data Availability

The public dataset SKAB is available at https://www.kaggle.com/datasets/yuriykatser/skoltech-anomaly-benchmark-skab. The public dataset NAB is available at https://www.kaggle.com/datasets/boltzmannbrain/nab. The real-world dataset SAT is not available for privacy protection.

References

  1. Gültekin O, Cinar E, Özkan K, Yazıcı A (2022) Multisensory data fusion-based deep learning approach for fault diagnosis of an industrial autonomous transfer vehicle. Expert Syst Appl 200:117055

    Article  Google Scholar 

  2. Amini N, Zhu Q (2022) Fault detection and diagnosis with a novel source-aware autoencoder and deep residual neural network. Neurocomputing 488:618–633

    Article  Google Scholar 

  3. Audibert J, Michiardi P, Guyard F, Marti S, Zuluaga MA (2020) USAD: unsupervised anomaly detection on multivariate time series. In: Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. pp 3395–3404

  4. Li G, Jung JJ (2023) Deep learning for anomaly detection in multivariate time series: approaches, applications, and challenges. Information Fusion 91:93–102

    Article  Google Scholar 

  5. Zhou Y, Liang X, Zhang W, Zhang L, Song X (2021) VAE-based deep SVDD for anomaly detection. Neurocomputing 453:131–140

    Article  Google Scholar 

  6. Harrou F, Dairi A, Taghezouit B, Sun Y (2019) An unsupervised monitoring procedure for detecting anomalies in photovoltaic systems using a one-class support vector machine. Sol Energy 179:48–58

    Article  Google Scholar 

  7. Sarmadi H, Karamodin A (2020) A novel anomaly detection method based on adaptive Mahalanobis-squared distance and one-class kNN rule for structural health monitoring under environmental effects. Mech Syst Signal Process 140:106495

    Article  Google Scholar 

  8. Imrana Y, Xiang Y, Ali L, Abdul-Rauf Z (2021) A bidirectional LSTM deep learning approach for intrusion detection. Expert Syst Appl 185:115524

    Article  Google Scholar 

  9. Paula Monteiro R, Lozada MC, Mendieta DRC, Loja RVS, Bastos Filho CJA (2022) A hybrid prototype selection-based deep learning approach for anomaly detection in industrial machines. Expert Systems with Applications 117528

  10. Deng H, Qian G, Luo D, Lv X, Liu H, Li H (2023) MRS-Net: an image inpainting algorithm with multi-scale residual attention fusion. Appl Intell 53(7):7497–7511

    Article  Google Scholar 

  11. He P, Wang L, Cui Y, Wang R, Wu D (2023) Unsupervised feature learning based on autoencoder for epileptic seizures prediction. Applied Intelligence 1–19

  12. Bang J, Park J, Park J (2023) GACaps-HTC: graph attention capsule network for hierarchical text classification. Applied Intelligence 1–18

  13. Li L, Yan J, Wang H, Jin Y (2020) Anomaly detection of time series with smoothness-inducing sequential variational auto-encoder. IEEE Trans Neural Netw Learn Syst 32(3):1177–1191

    Article  Google Scholar 

  14. Maleki S, Maleki S, Jennings NR (2021) Unsupervised anomaly detection with LSTM autoencoders using statistical data-filtering. Appl Soft Comput 108:107443

    Article  Google Scholar 

  15. Thill M, Konen W, Wang H, Bäck T (2021) Temporal convolutional autoencoder for unsupervised anomaly detection in time series. Appl Soft Comput 112:107751

    Article  Google Scholar 

  16. Chen S, Jin G, Ma X (2022) Detection and analysis of real-time anomalies in large-scale complex system. Measurement 184:109929

    Article  Google Scholar 

  17. Jiang B, Chen S, Wang B, Luo B (2022) MGLNN: semi-supervised learning via multiple graph cooperative learning neural networks. Neural Netw 153:204–214

    Article  Google Scholar 

  18. Wu Y, Dai HN, Tang H (2021) Graph neural networks for anomaly detection in industrial internet of things. IEEE Internet Things J 9(12):9214–9231

    Article  Google Scholar 

  19. He Z, Chen P, Li X, Wang Y, Yu G, Chen C, Li X, Zheng Z (2023) A spatiotemporal deep learning approach for unsupervised anomaly detection in cloud systems. IEEE Trans Neural Netw Learn Syst 34(4):1705–1719

    Article  Google Scholar 

  20. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. Adv Neural Inf Process Syst 30:1–11

    Google Scholar 

  21. Wu X, Tang B, Zhao M, Wang J, Guo Y (2023) STR transformer: a cross-domain transformer for scene text recognition. Appl Intell 53(3):3444–3458

    Article  Google Scholar 

  22. Wu H, Xu J, Wang J, Long M (2021) Autoformer: decomposition transformers with auto-correlation for long-term series forecasting. Adv Neural Inf Process Syst 34:22419–22430

    Google Scholar 

  23. Zerveas G, Jayaraman S, Patel D, Bhamidipaty A, Eickhoff C (2021) A transformer-based framework for multivariate time series representation learning. In: Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining. pp 2114–2124

  24. Makhzani A, Shlens J, Jaitly N, Goodfellow I, Frey B (2015) Adversarial autoencoders. arXiv:1511.05644

  25. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2020) Generative adversarial networks. Commun ACM 63(11):139–144

    Article  MathSciNet  Google Scholar 

  26. Jang K, Hong S, Kim M, Na J, Moon I (2021) Adversarial autoencoder based feature learning for fault detection in industrial processes. IEEE Trans Industr Inf 18(2):827–834

    Article  Google Scholar 

  27. Zhou X, Hu Y, Liang W, Ma J, Jin Q (2020) Variational LSTM enhanced anomaly detection for industrial big data. IEEE Trans Industr Inf 17(5):3469–3477

    Article  Google Scholar 

  28. Benidis K, Rangapuram SS, Flunkert V, Wang Y, Maddix D, Turkmen C, Gasthaus J, Bohlke-Schneider M, Salinas D, Stella L et al (2022) Deep learning for time series forecasting: tutorial and literature survey. ACM Comput Surv 55(6):1–36

    Article  Google Scholar 

  29. Xing Y, Zhu J, Li Y, Huang J, Song J (2023) An improved spatial temporal graph convolutional network for robust skeleton-based action recognition. Appl Intell 53(4):4592–4608

    Article  Google Scholar 

  30. Cheng Y, Xu Y, Zhong H, Liu Y (2020) Leveraging semisupervised hierarchical stacking temporal convolutional network for anomaly detection in IoT communication. IEEE Internet Things J 8(1):144–155

    Article  Google Scholar 

  31. Zhan J, Wu C, Ma X, Yang C, Miao Q, Wang S (2022) Abnormal vibration detection of wind turbine based on temporal convolution network and multivariate coefficient of variation. Mech Syst Signal Process 174:109082

    Article  Google Scholar 

  32. Zhou H, Zhang S, Peng J, Zhang S, Li J, Xiong H, Zhang W (2021) Informer: Beyond efficient transformer for long sequence time-series forecasting. In: Proceedings of the AAAI conference on artificial intelligence, vol 35. pp 11106–11115

  33. Wang X, Pi D, Zhang X, Liu H, Guo C (2022) Variational transformer-based anomaly detection approach for multivariate time series. Measurement 191:110791

    Article  Google Scholar 

  34. Cai L, Janowicz K, Mai G, Yan B, Zhu R (2020) Traffic transformer: capturing the continuity and periodicity of time series for traffic forecasting. Trans GIS 24(3):736–755

    Article  Google Scholar 

  35. Lara-Benítez P, Carranza-García M, Riquelme JC (2021) An experimental review on deep learning architectures for time series forecasting. Int J Neural Syst 31(03):2130001

    Article  Google Scholar 

  36. Li S, Jin X, Xuan Y, Zhou X, Chen W, Wang YX, Yan X (2019) Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in neural information processing systems 32

  37. Chen RQ, Shi GH, Zhao WL, Liang CH (2021) A joint model for it operation series prediction and anomaly detection. Neurocomputing 448:130–139

    Article  Google Scholar 

  38. Shao H, Xiao Z, Yao S, Sun D, Zhang A, Liu S, Wang T, Li J, Abdelzaher T (2021) ControlVAE: tuning, analytical properties, and performance analysis. IEEE Trans Pattern Anal Mach Intell 44(12):9285–9297

    Article  Google Scholar 

  39. Creswell A, White T, Dumoulin V, Arulkumaran K, Sengupta B, Bharath AA (2018) Generative adversarial networks: an overview. IEEE Signal Process Mag 35(1):53–65

    Article  Google Scholar 

  40. Wang X, Du Y, Lin S, Cui P, Shen Y, Yang Y (2020) advAE: a self-adversarial variational autoencoder with Gaussian anomaly prior knowledge for anomaly detection. Knowl-Based Syst 190:105187

    Article  Google Scholar 

  41. Huang Y, Tang Y, VanZwieten J (2021) Prognostics with variational autoencoder by generative adversarial learning. IEEE Trans Industr Electron 69(1):856–867

    Article  Google Scholar 

  42. Larsen ABL, Sønderby SK, Larochelle H, Winther O (2016) Autoencoding beyond pixels using a learned similarity metric. In: International conference on machine learning, PMLR. pp 1558–1566

  43. Katser ID, Kozitsin VO (2020) Skoltech Anomaly Benchmark (SKAB). Kaggle. https://doi.org/10.34740/KAGGLE/DSV/1693952

  44. Ahmad S, Lavin A, Purdy S, Agha Z (2017) Unsupervised real-time anomaly detection for streaming data. Neurocomputing 262:134–147

    Article  Google Scholar 

  45. Yin C, Zhang S, Wang J, Xiong NN (2020) Anomaly detection based on convolutional recurrent autoencoder for IoT time series. IEEE Trans Syst Man Cybern Syst 52(1):112–122

    Article  Google Scholar 

  46. Breunig MM, Kriegel HP, Ng RT, Sander J (2000) LOF: identifying density based local outliers. In: Proceedings of the 2000 ACM SIGMOD international conference on management of data. pp 93–104

  47. Zhao Y, Nasrullah Z, Hryniewicki MK, Li Z (2019) LSCP: locally selective combination in parallel outlier ensembles. In: Proceedings of the 2019 SIAM international conference on data mining. SIAM, pp 585–593

  48. Li Z, Xiang Z, Gong W, Wang H (2022) Unified model for collective and point anomaly detection using stacked temporal convolution networks. Appl Intell 52(3):3118–3131

  49. Homayouni H, Ghosh S, Ray I, Gondalia S, Duggan J, Kahn MG (2020) An autocorrelation-based LSTM-autoencoder for anomaly detection on time-series data. In: 2020 IEEE international conference on big data (Big Data). IEEE, pp 5068–5077

  50. Su Y, Zhao Y, Niu C, Liu R, Sun W, Pei D (2019) Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. pp 2828–2837

  51. Zhao H, Wang Y, Duan J, Huang C, Cao D, Tong Y, Xu B, Bai J, Tong J, Zhang Q (2020) Multivariate time-series anomaly detection via graph attention network. In: 2020 IEEE international conference on data mining (ICDM). IEEE, pp 841–850

  52. Deng A, Hooi B (2021) Graph neural network-based anomaly detection in multivariate time series. In: Proceedings of the AAAI conference on artificial intelligence, vol 35. pp 4027–4035

  53. Woolson RF (2007) Wilcoxon signed-rank test. Wiley encyclopedia of clinical trials 1–3

  54. Podobnik B, Stanley HE (2008) Detrended cross-correlation analysis: a new method for analyzing two nonstationary time series. Phys Rev Lett 100(8):084102

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, 211106, China

    Junfu Chen, Dechang Pi & Xixuan Wang

Authors
  1. Junfu Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dechang Pi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xixuan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Junfu Chen carried out the method design, participated in the coding for the experiments, and drafted the manuscript. Dechang Pi provided the GPU service and polished the manuscript. Xixuan Wang participated in reproducing comparison methods.

Corresponding author

Correspondence to Dechang Pi.

Ethics declarations

Ethical and informed consent for data used

Informed consent for publication of this paper and data usage was obtained from the Nanjing University of Aeronautics and Astronautics and all authors.

Conflict of interest

The authors declare that no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, J., Pi, D. & Wang, X. A two-stage adversarial Transformer based approach for multivariate industrial time series anomaly detection. Appl Intell (2024). https://doi.org/10.1007/s10489-024-05395-0

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10489-024-05395-0

Keywords

Search

Navigation