Skip to main content
SpringerLink
Log in

Empowering Recommender Systems in ITSM: A Pipeline Reference Model for AI-Based Textual Data Quality Enrichment

  • Conference paper
  • First Online:
Design Science Research for a New Society: Society 5.0 (DESRIST 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13873))

  • 1005 Accesses

Abstract

AI-based recommendation systems to augment working conditions in the field of IT service management (ITSM) have attracted new attention. However, many IT support organizations possess high volumes of tickets but are confronted with low quality, to which they train the underlying models of their AI systems. In particular, support tickets are documented insufficiently due to time pressure and lack of motivation. Following design science research, we design and evaluate an analytics pipeline to address the data quality issue. The pipeline can be applied to assess and extract high-quality support tickets for subsequent model training and operation. Based on a data set of 60.000 real-life support tickets from a manufacturing company, we develop the artifact, instantiate a recommender system and achieve a higher prediction performance in comparison to naïve enrichment methods. In terms of data management literature, we contribute to the understanding of assessing textual ticket data quality. By deriving a pipeline reference model, we move towards a general approach to designing machine learning-driven data quality analytics pipelines for attached recommender systems.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    A = Accuracy; R = Recall; P = Precision; F1 = F1-Score.

  2. 2.

    Cl. = Classifier for predicting a quality score for a given support ticket.

References

  1. Swain, A.K., Garza, V.R.: Key factors in achieving Service Level Agreements (SLA) for Information Technology (IT) incident resolution. Inf. Syst. Front. 1–16 (2022). https://doi.org/10.1007/s10796-022-10266-5

  2. Schmidt, S., Li, M., Peters, C.: Requirements for an IT support system based on hybrid intelligence. In: Proceedings of the Annual Hawaii International Conference on System Sciences. Hawaii International Conference on System Sciences (2022)

    Google Scholar 

  3. Paramesh, S.P., Ramya, C., Shreedhara, K.S.: Classifying the unstructured it service desk tickets using ensemble of classifiers. In: 3rd International Conference on Computational Systems and Information Technology for Sustainable Solutions (CSITSS). IEEE (2018)

    Google Scholar 

  4. Al-Hawari, F., Barham, H.: A machine learning based help desk system for IT service management. J. King Saud Univ. Comput. Inform. Sci. 33(6), 702–718 (2021). https://doi.org/10.1016/j.jksuci.2019.04.001

    Article  Google Scholar 

  5. Fuchs, S., Drieschner, C., Wittges, H.: Proceedings of the 55th Hawaii International Conference on System Sciences (HICSS). University of Hawai'i at Manoa Hamilton Library, Honolulu, Hawai (2022)

    Google Scholar 

  6. Meng, F.J., et al.: Opportunities and Challenges Towards Cognitive IT Service Management in Real World IEEE Symposium on Service-Oriented System Engineering (SOSE). IEEE (2018)

    Google Scholar 

  7. Ali Zaidi, S.S., Fraz, M.M., Shahzad, M., Khan, S.: A multiapproach generalized framework for automated solution suggestion of support tickets. Int. J. Intell. Syst. 37(6), 3654–3681 (2022). https://doi.org/10.1002/int.22701

    Article  Google Scholar 

  8. Zhou, W., et al.: Star: A system for ticket analysis and resolution. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 2181–2190 (2017)

    Google Scholar 

  9. Schmidt, S.L., Li, M.M., Weigel, S., Peters, C.: Knowledge is power: provide your it-support with domain-specific high-quality solution material. In: Chandra Kruse, L., Seidel, S., Hausvik, G.I. (eds.) The Next Wave of Sociotechnical Design. DESRIST 2021. LNCS, vol. 12807. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-82405-1_22

  10. Peters, C., Blohm, I., Leimeister, J.M.: Anatomy of successful business models for complex services: insights from the telemedicine field. J. Manag. Inf. Syst. 32(3), 75–104 (2015). https://doi.org/10.1080/07421222.2015.1095034

    Article  Google Scholar 

  11. Agarwal, S., Sridhara, G., Dasgupta, G.: Automated quality assessment of unstructured resolution text in IT service systems. In: Sheng, Q., Stroulia, E., Tata, S., Bhiri, S. (eds.) Service-Oriented Computing. ICSOC 2016. LNCS, vol. 9936. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46295-0_14

  12. Heinrich, B., Hopf, M., Lohninger, D., Schiller, A., Szubartowicz, M.: Data quality in recommender systems: the impact of completeness of item content data on prediction accuracy of recommender systems. Electron. Mark. 31(2), 389–409 (2019). https://doi.org/10.1007/s12525-019-00366-7

    Article  Google Scholar 

  13. Heinrich, B., Hopf, M., Lohninger, D., Schiller, A., Szubartowicz, M.: Something’s missing? a procedure for extending item content data sets in the context of recommender systems. Inf. Syst. Front. 24(1), 267–286 (2020). https://doi.org/10.1007/s10796-020-10071-y

    Article  Google Scholar 

  14. Wang, Q., Zhou, W., Zeng, C., Li, T., Shwartz, L., Grabarnik, G.Y.: Constructing the knowledge base for cognitive IT service management. In: IEEE International Conference on Services Computing (SCC). IEEE (2017)

    Google Scholar 

  15. Konstan, J.A., Riedl, J.: Recommender systems: from algorithms to user experience. User Model User-Adap. Inter. 22(1–2), 101–123 (2012). https://doi.org/10.1007/s11257-011-9112-x

    Article  Google Scholar 

  16. Picault, J., Ribière, M., Bonnefoy, D., Mercer, K.: How to get the recommender out of the lab? In: Ricci, F., Rokach, L., Shapira, B., Kantor, P.B. (eds.) Recommender Systems Handbook, pp. 333–365. Scholars Portal, Boston, MA, (2011)

    Google Scholar 

  17. Sar Shalom, O., Berkovsky, S., Ronen, R., Ziklik, E., Amihood, A.: Data quality matters in recommender systems. In: Werthner, H., Zanker, M., Golbeck, J., Semeraro, G. (eds.) Proceedings of the 9th ACM Conference on Recommender Systems, pp. 257–260. ACM, New York, NY (2015)

    Google Scholar 

  18. Revina, A., Buza, K., Meister, V.G.: IT ticket classification: the simpler, the better. IEEE Access 8, 193380–193395 (2020). https://doi.org/10.1109/access.2020.3032840

    Article  Google Scholar 

  19. Koehler, J., et al.: Towards Intelligent Process Support for Customer Service Desks: Extracting Problem Descriptions from Noisy and Multi-lingual Texts, S 36–52

    Google Scholar 

  20. Marcuzzo, M., Zangari, A., Schiavinato, M., Giudice, L., Gasparetto, A., Albarelli, A.: A multi-level approach for hierarchical Ticket Classification. In: Proceedings of the Eighth Workshop on Noisy User-generated Text (W-NUT 2022), pp. 201–214 (2022)

    Google Scholar 

  21. Zicari, P., Folino, G., Guarascio, M., Pontieri, L: Discovering accurate deep learning based predictive models for automatic customer support ticket classification. In: Proceedings of the 36th Annual ACM Symposium on Applied Computing. ACM, New York, NY, USA (2021)

    Google Scholar 

  22. Agarwal, S., Aggarwal, V., Akula, A.R., Dasgupta, G.B., Sridhara, G.: Automatic problem extraction and analysis from unstructured text in IT tickets. IBM J. Res. Dev. 61(1):4:41–4:52 (2017). doi:https://doi.org/10.1147/jrd.2016.2629318

  23. Li, M.M., Peters, C., Leimeister, J.M.: Designing a peer-based support system to support shakedown. In: International Conference on Information Systems (ICIS). Seoul, South Korea (2017)

    Google Scholar 

  24. Zschech, P.: Beyond descriptive taxonomies in data analytics: a systematic evaluation approach for data-driven method pipelines. Inf. Syst. E-Bus Manage. 1–35 (2022). https://doi.org/10.1007/s10257-022-00577-0

  25. Batini, C., Barone, D., Mastrella, M., Maurino, A., Ruffini, C.: A framework and a methodology for data quality assessment and monitoring. ICIQ, pp. 333–346 (2007)

    Google Scholar 

  26. Bharati, P., Chaudhury, A.: An empirical investigation of decision-making satisfaction in web-based decision support systems. Decis. Support Syst. 37(2), 187–197 (2004). https://doi.org/10.1016/S0167-9236(03)00006-X

    Article  Google Scholar 

  27. Feldman, M., Even, A., Parmet, Y.: A methodology for quantifying the effect of missing data on decision quality in classification problems. Commun. Statist. Theory Meth. 47(11), 2643–2663 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  28. Woodall, P., Borek, A., Gao, J., Oberhofer, M.A., Koronios, A.: An Investigation of How Data Quality is Affected by Dataset Size in the Context of Big Data Analytics ICIQ (2014)

    Google Scholar 

  29. Zicari, P., Folino, G., Guarascio, M., Pontieri, L.: Combining deep ensemble learning and explanation for intelligent ticket management. Expert Syst. Appl. 206, 117815 (2022). https://doi.org/10.1016/j.eswa.2022.117815

    Article  Google Scholar 

  30. Rizun, N., Revina, A., Meister, V.G.: Assessing business process complexity based on textual data: evidence from ITIL IT ticket processing. BPMJ 27(7), 1966–1998 (2021). https://doi.org/10.1108/BPMJ-04-2021-0217

    Article  Google Scholar 

  31. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv (2019)

    Google Scholar 

  32. Baresi, L., Quattrocchi, G., Tamburri, D.A., den van Heuvel, W.-J.: Automated quality assessment of incident tickets for smart service continuity. In: International Conference on Service-Oriented Computing, pp. 492–499 (2020)

    Google Scholar 

  33. Cavalcanti, Y.C., Da Mota Silveira Neto, P.A., do Carmo Machado I., Vale, T.F., de Almeida, E.S., de Lemos Meira, S.R.: Challenges and opportunities for software change request repositories: a systematic mapping study. J. Softw. Evol. Process 26(7), 620–653 (2014)

    Google Scholar 

  34. Sonnenberg, C., vom Brocke J.: Evaluations in the science of the artificial – reconsidering the build-evaluate pattern in design science research. In: Peffers, K., Rothenberger, M., Kuechler, B. (eds.) Design Science Research in Information Systems. Advances in Theory and Practice. DESRIST 2012. LNCS, vol. 7286. Springer, Berlin, Heidelberg (2012). https://doi.org/10.1007/978-3-642-29863-9_28

  35. Peffers, K., Tuunanen, T., Rothenberger, M.A., Chatterjee, S.: A design science research methodology for information systems research. J. Manag. Inf. Syst. 24(3), 45–77 (2007). https://doi.org/10.2753/MIS0742-1222240302

    Article  Google Scholar 

  36. Brendel, A.B., Lembcke, T.-B., Muntermann, J., Kolbe, L.M.: Toward replication study types for design science research. J. Inf. Technol. 36(3), 198–215 (2021). https://doi.org/10.1177/02683962211006429

    Article  Google Scholar 

  37. Chandra Kruse, L., Nickerson, J.V.: Portraying Design Essence (2018)

    Google Scholar 

  38. Legner, C., Pentek, T., Otto, B.: Accumulating design knowledge with reference models: insights from 12 years’ research into data management. JAIS 21(3), 735–770 (2020). https://doi.org/10.17705/1jais.00618

    Article  Google Scholar 

  39. Das, A.: Knowledge and productivity in technical support work. Manage. Sci. 49(4), 416–431 (2003). https://doi.org/10.1287/mnsc.49.4.416.14419

    Article  MathSciNet  Google Scholar 

  40. Elshan, E., Ebel, P.A., Söllner, M., Leimeister, J.M.: Leveraging low code development of smart personal assistants: an integrated design approach with the SPADE method. J. Manage. Inform. Syst. (JMIS) (2022)

    Google Scholar 

  41. Wambsganß, T., Engel, C.: Using Deep Learning for Extracting User-Generated Knowledge from Web Communities (2021)

    Google Scholar 

  42. Zacharias, J., von Zahn, M., Chen, J., Hinz, O.: Designing a feature selection method based on explainable artificial intelligence. Electron. Mark. 1–26 (2022)

    Google Scholar 

  43. Pitler, E., Nenkova, A.: Revisiting readability: a unified framework for predicting text quality. In: Lapata, M., Tou, N.G.H. (eds.) Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pp. 186–195. Honolulu, Hawai (2008)

    Google Scholar 

  44. Landolt, S., Wambsganss, T., Söllner, M.: A Taxonomy for deep learning in natural language processing. In: Hawaii International Conference on System Sciences, Hawaii (2021)

    Google Scholar 

  45. Cai, L., Zhu, Y.: The challenges of data quality and data quality assessment in the big data era. CODATA 14, 2 (2015). https://doi.org/10.5334/dsj-2015-002

    Article  Google Scholar 

  46. Pipino, L.L., Lee, Y.W., Wang, R.Y.: Data quality assessment. Commun. ACM 45(4), 211–218 (2002). https://doi.org/10.1145/505248.506010

    Article  Google Scholar 

  47. Batini, C., Cappiello, C., Francalanci, C., Maurino, A.: Methodologies for data quality assessment and improvement. ACM Comput. Surv. 41(3), 1–52 (2009). https://doi.org/10.1145/1541880.1541883

    Article  Google Scholar 

  48. Subbarao, M.V., Venkatarao, K., Suresh, C.: Automation of incident response and IT ticket management by ML and NLP mechanisms. J. Theor. Appl. Inf. Technol. 100(12), 3945–3955 (2022)

    Google Scholar 

  49. Vayansky, I., Kumar, S.A.: A review of topic modeling methods. Inf. Syst. 94, 101582 (2020). https://doi.org/10.1016/j.is.2020.101582

    Article  Google Scholar 

  50. Bouguettaya, A., Yu, Q., Liu, X., Zhou, X., Song, A.: Efficient agglomerative hierarchical clustering. Expert Syst. Appl. 42(5), 2785–2797 (2015). https://doi.org/10.1016/j.eswa.2014.09.054

    Article  Google Scholar 

  51. Lee, H.J., Lee, M., Lee, H., Cruz, R.A.: Mining service quality feedback from social media: a computational analytics method. Gov. Inf. Q. 38(2), 101571 (2021)

    Article  Google Scholar 

  52. Liu, J., Zhong, W., Li, R.: A selective overview of feature screening for ultrahigh-dimensional data. Science China Math. 58(10), 1–22 (2015). https://doi.org/10.1007/s11425-015-5062-9

    Article  MathSciNet  MATH  Google Scholar 

  53. Oliveira, D.F., Nogueira, A.S., Brito, M.A.: Performance comparison of machine learning algorithms in classifying information technologies incident tickets. AI 3(3), 601–622 (2022). https://doi.org/10.3390/ai3030035

  54. Frank, U.: Evaluation of reference models Reference modeling for business systems analysis. IGI Global, pp. 118–140 (2007)

    Google Scholar 

  55. Venable, J., Pries-Heje, J., Baskerville, R.: FEDS: a framework for evaluation in design science research. Eur. J. Inf. Syst. 25(1), 77–89 (2016). https://doi.org/10.1057/ejis.2014.36

    Article  Google Scholar 

  56. Shani, G., Gunawardana, A.: Evaluating recommendation systems. In: Ricci, F., Rokach, L., Shapira, B., Kantor, P. (eds.) Recommender Systems Handbook. Springer, Boston, MA (2011). https://doi.org/10.1007/978-0-387-85820-3_8

  57. Gregor, S., Hevner, A.R.: Positioning and presenting design science research for maximum impact. MISQ 37(2), 337–355 (2013). https://doi.org/10.25300/misq/2013/37.2.01

    Article  Google Scholar 

  58. Chua, C., Indulska, M., Lukyanenko, R., Maass, W., Storey, V.C.: MISQ research curation on data management. MISQ Res. Curat. 1–12 (2022)

    Google Scholar 

  59. Schermann, M., Böhmann, T., Krcmar, H.: Explicating design theories with conceptual models: towards a theoretical role of reference models. Wissenschaftstheorie und gestaltungsorientierte Wirtschaftsinformatik, S 175–194. Springer (2009)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Kassel, Kassel, Germany

    Philipp Reinhard, Mahei Manhai Li, Ernestine Dickhaut, Christoph Peters & Jan Marco Leimeister

  2. University of St.Gallen, St.Gallen, Switzerland

    Christoph Peters & Jan Marco Leimeister

Authors
  1. Philipp Reinhard
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mahei Manhai Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ernestine Dickhaut
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Christoph Peters
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jan Marco Leimeister
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Philipp Reinhard .

Editor information

Editors and Affiliations

  1. University of the Western Cape, Cape Town, South Africa

    Aurona Gerber

  2. Georgia State University, Atlanta, GA, USA

    Richard Baskerville

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Reinhard, P., Li, M.M., Dickhaut, E., Peters, C., Leimeister, J.M. (2023). Empowering Recommender Systems in ITSM: A Pipeline Reference Model for AI-Based Textual Data Quality Enrichment. In: Gerber, A., Baskerville, R. (eds) Design Science Research for a New Society: Society 5.0. DESRIST 2023. Lecture Notes in Computer Science, vol 13873. Springer, Cham. https://doi.org/10.1007/978-3-031-32808-4_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-32808-4_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-32807-7

  • Online ISBN: 978-3-031-32808-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation