Skip to main content
SpringerLink
Log in

Hilogx: noise-aware log-based anomaly detection with human feedback

  • Regular Paper
  • Published:
The VLDB Journal Aims and scope Submit manuscript

Abstract

Log-based anomaly detection is essential for maintaining system reliability. Although existing log-based anomaly detection approaches perform well in certain experimental systems, they are ineffective in real-world industrial systems with noisy log data. This paper focuses on mitigating the impact of noisy log data. To this aim, we first conduct an empirical study on the system logs of four large-scale industrial software systems. Through the study, we find five typical noise patterns that are the root causes of unsatisfactory results of existing anomaly detection models. Based on the study, we propose HiLogx, a noise-aware log-based anomaly detection approach that integrates human knowledge to identify these noise patterns and further modify the anomaly detection model with human feedback. Experimental results on four large-scale industrial software systems and two open datasets show that our approach improves over 30% precision and 15% recall on average.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. Ai Magazine 35(4), 105–120 (2014)

    Article  Google Scholar 

  2. Brame, C.: Active learning. Vanderbilt University Center for Teaching (2016)

  3. Das, S., Wong, W.K., Dietterich, T., Fern, A., Emmott, A.: Incorporating expert feedback into active anomaly discovery. In: 2016 IEEE 16th International Conference on Data Mining (ICDM), pp. 853–858 (2016). https://doi.org/10.1109/ICDM.2016.0102

  4. Das, S., Wong, W.K., Fern, A., Dietterich, T.G., Siddiqui, M.A.: Incorporating feedback into tree-based anomaly detection. arXiv preprint arXiv:1708.09441 (2017)

  5. Du, M., Chen, Z., Liu, C., Oak, R., Song, D.: Lifelong anomaly detection through unlearning. In: Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, CCS ’19, p. 1283-1297. Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3319535.3363226

  6. Du, M., Li, F., Zheng, G., Srikumar, V.: Deeplog: Anomaly detection and diagnosis from system logs through deep learning. In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, CCS ’17, p. 1285–1298. Association for Computing Machinery, New York, NY, USA (2017). https://doi.org/10.1145/3133956.3134015

  7. Fronza, I., Sillitti, A., Succi, G., Terho, M., Vlasenko, J.: Failure prediction based on log files using random indexing and support vector machines. J. Syst. Softw. 86(1), 2–11 (2013)

    Article  Google Scholar 

  8. Gomez Rodriguez, M., Leskovec, J., Schölkopf, B.: Structure and dynamics of information pathways in online media. In: Proceedings of the Sixth ACM International Conference on Web Search and Data Mining, pp. 23–32 (2013)

  9. Görnitz, N., Kloft, M., Rieck, K., Brefeld, U.: Toward supervised anomaly detection. J. Artif. Intell. Res. 46, 235–262 (2013)

    Article  MathSciNet  Google Scholar 

  10. He, S., Zhu, J., He, P., Lyu, M.R.: Loghub: a large collection of system log datasets towards automated log analytics. arXiv preprint arXiv:2008.06448 (2020)

  11. Jia, T., Chen, P., Yang, L., Li, Y., Meng, F., Xu, J.: An approach for anomaly diagnosis based on hybrid graph model with logs for distributed services. In: 2017 IEEE International Conference on Web Services (ICWS), pp. 25–32 (2017). https://doi.org/10.1109/ICWS.2017.12

  12. Jia, T., Li, Y., Yang, Y., Huang, G., Wu, Z.: Augmenting log-based anomaly detection models to reduce false anomalies with human feedback. In: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 3081–3089 (2022)

  13. Jia, T., Wu, Y., Hou, C., Li, Y.: Logflash: Real-time streaming anomaly detection and diagnosis from system logs for large-scale software systems. In: 2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE), pp. 80–90 (2021). https://doi.org/10.1109/ISSRE52982.2021.00021

  14. Jia, T., Yang, L., Chen, P., Li, Y., Meng, F., Xu, J.: Logsed: Anomaly diagnosis through mining time-weighted control flow graph in logs. In: 2017 IEEE 10th International Conference on Cloud Computing (CLOUD), pp. 447–455 (2017). https://doi.org/10.1109/CLOUD.2017.64

  15. Kamar, E.: Directions in hybrid intelligence: complementing AI systems with human intelligence. In: IJCAI, pp. 4070–4073 (2016)

  16. Kamar, E., Hacker, S., Horvitz, E.: Combining human and machine intelligence in large-scale crowdsourcing. AAMAS 12, 467–474 (2012)

    Google Scholar 

  17. Kim, J., Savchenko, V., Shin, K., Sorokin, K., Jeon, H., Pankratenko, G., Markov, S., Kim, C.J.: Automatic abnormal log detection by analyzing log history for providing debugging insight. In: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Software Engineering in Practice, ICSE-SEIP ’20, p. 71–80. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3377813.3381371

  18. Lim, C., Singh, N., Yajnik, S.: A log mining approach to failure analysis of enterprise telephony systems. In: 2008 IEEE International Conference on Dependable Systems and Networks With FTCS and DCC (DSN), pp. 398–403 (2008). https://doi.org/10.1109/DSN.2008.4630109

  19. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6–12, 2014, Proceedings, Part V 13, pp. 740–755. Springer (2014)

  20. Meng, W., Liu, Y., Zhu, Y., Zhang, S., Pei, D., Liu, Y., Chen, Y., Zhang, R., Tao, S., Sun, P., et al.: Loganomaly: Unsupervised detection of sequential and quantitative anomalies in unstructured logs. In: IJCAI, vol. 19, pp. 4739–4745 (2019)

  21. Moulines, E., Bach, F.: Non-asymptotic analysis of stochastic approximation algorithms for machine learning. Adv. Neural Inf. Process. Syst. 24 (2011)

  22. Nandi, A., Mandal, A., Atreja, S., Dasgupta, G.B., Bhattacharya, S.: Anomaly detection using program control flow graph mining from execution logs. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, p. 215–224. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939712

  23. Pevnỳ, T.: Loda: Lightweight on-line detector of anomalies. Mach. Learn. 102(2), 275–304 (2016)

    Article  MathSciNet  Google Scholar 

  24. Reidemeister, T., Munawar, M.A., Ward, P.A.: Identifying symptoms of recurrent faults in log files of distributed information systems. In: 2010 IEEE Network Operations and Management Symposium—NOMS 2010, pp. 187–194 (2010). https://doi.org/10.1109/NOMS.2010.5488459

  25. Rodriguez, M.G., Balduzzi, D., Schölkopf, B.: Uncovering the temporal dynamics of diffusion networks. arXiv preprint arXiv:1105.0697 (2011)

  26. Siddiqui, M.A., Fern, A., Dietterich, T.G., Wright, R., Theriault, A., Archer, D.W.: Feedback-guided anomaly discovery via online optimization. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’18, p. 2200-2209. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3219819.3220083

  27. Tak, B.C., Tao, S., Yang, L., Zhu, C., Ruan, Y.: Logan: Problem diagnosis in the cloud using log-based reference models. In: 2016 IEEE International Conference on Cloud Engineering (IC2E), pp. 62–67 (2016). https://doi.org/10.1109/IC2E.2016.12

  28. Tong, J., Ying, L., Hongyan, T., Zhonghai, W.: An approach to pinpointing bug-induced failure in logs of open cloud platforms. In: 2016 IEEE 9th International Conference on Cloud Computing (CLOUD), pp. 294–302 (2016). https://doi.org/10.1109/CLOUD.2016.0047

  29. Vaughan, J.W.: Making better use of the crowd: How crowdsourcing can advance machine learning research. J. Mach. Learn. Res. 18(1), 7026–7071 (2017)

    MathSciNet  Google Scholar 

  30. Veeramachaneni, K., Arnaldo, I., Korrapati, V., Bassias, C., Li, K.: Ai\(^{\wedge }\) 2: training a big data machine to defend. In: 2016 IEEE 2nd international conference on big data security on cloud (BigDataSecurity), IEEE international conference on high performance and smart computing (HPSC), and IEEE international conference on intelligent data and security (IDS), pp. 49–54. IEEE (2016)

  31. Xia, B., Bai, Y., Yin, J., Li, Y., Xu, J.: Loggan: a log-level generative adversarial network for anomaly detection using permutation event modeling. Inf. Syst. Front. 23, 285–298 (2021)

    Article  Google Scholar 

  32. Xu, J., Chen, P., Yang, L., Meng, F., Wang, P.: Logdc: Problem diagnosis for declartively-deployed cloud applications with log. In: 2017 IEEE 14th International Conference on e-Business Engineering (ICEBE), pp. 282–287 (2017). https://doi.org/10.1109/ICEBE.2017.52

  33. Xu, W., Huang, L., Fox, A., Patterson, D., Jordan, M.: Online system problem detection by mining patterns of console logs. In: 2009 Ninth IEEE International Conference on Data Mining, pp. 588–597 (2009). https://doi.org/10.1109/ICDM.2009.19

  34. Xu, W., Huang, L., Fox, A., Patterson, D., Jordan, M.I.: Detecting large-scale system problems by mining console logs. In: Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles, SOSP ’09, pp. 117–132. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1629575.1629587

  35. Yang, L., Chen, J., Wang, Z., Wang, W., Jiang, J., Dong, X., Zhang, W.: Plelog: Semi-supervised log-based anomaly detection via probabilistic label estimation. In: 2021 IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), pp. 230–231. IEEE (2021)

  36. Yang, Y., Wu, Y., Pattabiraman, K., Wang, L., Li, Y.: How far have we come in detecting anomalies in distributed systems? an empirical study with a statement-level fault injection method. In: 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE), pp. 59–69 (2020). https://doi.org/10.1109/ISSRE5003.2020.00015

  37. Yen, T.F., Oprea, A., Onarlioglu, K., Leetham, T., Robertson, W., Juels, A., Kirda, E.: Beehive: Large-scale log analysis for detecting suspicious activity in enterprise networks. In: Proceedings of the 29th Annual Computer Security Applications Conference, ACSAC ’13, p. 199–208. Association for Computing Machinery, New York, NY, USA (2013). https://doi.org/10.1145/2523649.2523670

  38. Yin, K., Yan, M., Xu, L., Xu, Z., Li, Z., Yang, D., Zhang, X.: Improving log-based anomaly detection with component-aware analysis. In: 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 667–671 (2020). https://doi.org/10.1109/ICSME46990.2020.00069

  39. Yu, X., Joshi, P., Xu, J., Jin, G., Zhang, H., Jiang, G.: Cloudseer: Workflow monitoring of cloud infrastructures via interleaved logs. SIGARCH Comput. Archit. News 44(2), 489–502 (2016). https://doi.org/10.1145/2980024.2872407

  40. Zhang, C., Peng, X., Sha, C., Zhang, K., Fu, Z., Wu, X., Lin, Q., Zhang, D.: Deeptralog: Trace-log combined microservice anomaly detection through graph-based deep learning. In: Proceedings of the 44th International Conference on Software Engineering, pp. 623–634 (2022)

  41. Zhang, X., Xu, Y., Lin, Q., Qiao, B., Zhang, H., Dang, Y., Xie, C., Yang, X., Cheng, Q., Li, Z., Chen, J., He, X., Yao, R., Lou, J.G., Chintalapati, M., Shen, F., Zhang, D.: Robust log-based anomaly detection on unstable log data. In: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2019, p. 807-817. Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3338906.3338931

Download references

Author information

Authors and Affiliations

  1. Institute for Artificial Intelligence, Peking University, Beijing, China

    Tong Jia & Gang Huang

  2. National Engineering Research Center For Software Engineering, Peking University, Beijing, China

    Ying Li

  3. School of Computer Science, Peking University, Beijing, China

    Yong Yang

  4. National Key Laboratory of Data Space Technology and System, Beijing, China

    Yong Yang & Gang Huang

Authors
  1. Tong Jia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ying Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yong Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gang Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ying Li.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work was supported by the National Key R &D Research Fund of China (2021YFF0704202).

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jia, T., Li, Y., Yang, Y. et al. Hilogx: noise-aware log-based anomaly detection with human feedback. The VLDB Journal 33, 883–900 (2024). https://doi.org/10.1007/s00778-024-00843-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00778-024-00843-2

Keywords

Search

Navigation