Skip to main content
SpringerLink
Log in

Automating Systematic Literature Reviews with Natural Language Processing and Text Mining: A Systematic Literature Review

  • Conference paper
  • First Online:
Proceedings of Eighth International Congress on Information and Communication Technology (ICICT 2023)

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 693))

Included in the following conference series:

Abstract

Objectives: An SLR is presented focusing on text mining-based automation of SLR creation. The present review identifies the objectives of the automation studies and the aspects of those steps that were automated. In so doing, the various ML techniques used challenges, limitations, and scope of further research are explained. Methods: Accessible published literature studies primarily focus on automation of study selection, study quality assessment, data extraction, and data synthesis portions of SLR. Twenty-nine studies were analyzed. Results: This review identifies the objectives of the automation studies, steps within the study selection, study quality assessment, data extraction, and data synthesis portions that were automated, and the various ML techniques used challenges, limitations, and scope of further research. Discussion: We describe uses of NLP/TM techniques to support increased automation of systematic literature reviews. This area has attracted increase attention in the last decade due to significant gaps in the applicability of TM to automate steps in the SLR process. There are significant gaps in the application of TM and related automation techniques in the areas of data extraction, monitoring, quality assessment, and data synthesis. There is, thus, a need for continued progress in this area, and this is expected to ultimately significantly facilitate the construction of systematic literature reviews.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 229.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 299.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Systematic reviews. Georgetown University Medical Center. https://guides.dml.georgetown.edu/systematicreviews

  2. Systematic reviews (2001) CRD’s guidance for those carrying out or commissioning reviews. CRD Report Number 4 (2nd edn). NHS Centre for Reviews and Dissemination, University of York

    Google Scholar 

  3. Glover J, Izzo D, Odato K et al (2006) EBM pyramid and EBM page generator. Trustees of Dartmouth College and Yale University

    Google Scholar 

  4. Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. EBSE Technical Report EBSE-2007-01. Keele University. https://docs.edtechhub.org/lib/EDAG684W

  5. Allen IE, Olkin I (1999) Estimating time to conduct a meta-analysis from number of citations retrieved. JAMA 282(7):634–635. https://doi.org/10.1001/jama.282.7.634

    Article  Google Scholar 

  6. Petticrew M, Roberts H (2006) Systematic reviews in the social sciences: a practical guide. Blackwell Publishing Co., Malden

    Book  Google Scholar 

  7. Shojani KG, Sampson M, Ansari MT et al (2007) How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med 147(4):224–233. https://doi.org/10.7326/0003-4819-147-4-200708210-00179

    Article  Google Scholar 

  8. Marshall C, Kitchenham B, Brereton P (2018) Tool features to support systematic reviews in software engineering. E-Informatica Softw Eng J 12(1):79–115. https://doi.org/10.5277/e-Inf180104

    Article  Google Scholar 

  9. Khangura S, Konnyu K, Cushman R et al (2012) Evidence summaries: the evolution of a rapid review approach. Syst Rev 1:10. https://doi.org/10.1186/2046-4053-1-10

    Article  Google Scholar 

  10. Ganann R, Ciliska D, Thomas H (2010) Expediting systematic reviews: methods and implications of rapid reviews. Implementation Sci 5:56. https://doi.org/10.1186/1748-5908-5-56

    Article  Google Scholar 

  11. Featherstone RM, Dryden DM, Foisy M et al (2015) Advancing knowledge of rapid reviews: an analysis of results, conclusions and recommendations from published review articles examining rapid reviews. Syst Rev 4:50. https://doi.org/10.1186/s13643-015-0040-4

    Article  Google Scholar 

  12. Jonnalagadda SR, Goyal P, Huffman MD (2015) Automating data extraction in systematic reviews: a systematic review. Syst Rev 4:78. https://doi.org/10.1186/s13643-015-0066-7

    Article  Google Scholar 

  13. O’Mara-Eves A, Thomas J, McNaught J et al (2015) Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev 4:5. https://doi.org/10.1186/2046-4053-4-5

    Article  Google Scholar 

  14. Feng L, Chiam Y, Lo SK (2017) Text-mining techniques and tools for systematic literature reviews: a systematic literature review. In: 24th Asia-Pacific software engineering conference (APSEC 2017). https://doi.org/10.1109/APSEC.2017.10

  15. Tsafnat G, Glasziou P, Choong MK et al (2014) Systematic review automation technologies. Syst Rev 3(74). https://doi.org/10.1186/2046-4053-3-74

  16. Marshall IJ, Wallace BC (2019) Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. Syst Rev 8:163. https://doi.org/10.1186/s13643-019-1074-9

    Article  Google Scholar 

  17. O’Connor AM, Tsafnat G, Gilbert SB et al (2018) Moving toward the automation of the systematic review process: a summary of discussions at the second meeting of the international collaboration for the automation of systematic reviews (ICASR). Syst Rev 7:3. https://doi.org/10.1186/s13643-017-0667-4

    Article  Google Scholar 

  18. Beller E, Clark J, Tsafnat G et al (2018) Making progress with the automation of systematic reviews: principles of the international collaboration for the automation of systematic reviews (ICASR). Syst Rev 7:77. https://doi.org/10.1186/s13643-018-0740-7

    Article  Google Scholar 

  19. Van Altena AJ, Spijker R, Olabarriaga SD (2019) Usage of automation tools in systematic reviews. Res Syn Meth 10:72–82. https://doi.org/10.1002/jrsm.1335

    Article  Google Scholar 

  20. Olorisade BK, de Quincey E, Brereton OP et al (2016) A critical analysis of studies that address the use of text mining for citation screening in systematic reviews. In: EASE ’16: proceedings of the 20th international conference on evaluation and assessment in software engineering. ACM, Limerick, pp 1–11. https://doi.org/10.1145/2915970.2915982

  21. Higgins J, Green S (2011) Cochrane handbook for systematic reviews of interventions version 5.1.0. The Cochrane Collaboration. http://community.cochrane.org/handbook.

  22. Richardson WS, Wilson MC, Nishikawa J et al (1995) The well-built clinical question: a key to evidence-based decisions. ACP J Club 123(3):A12–A13

    Article  Google Scholar 

  23. Dawes M, Pluye P, Shea L et al (2007) The identification of clinically important elements within medical journal abstracts: patient–population–problem, exposure-intervention, comparison, outcome, duration and results (PECODR). Inform Prim Care 15(1):9–16

    Google Scholar 

  24. Kim S, Martinez D, Cavedon L et al (2011) Automatic classification of sentences to support evidence based medicine. BMC Bioinform 12(Suppl 2):S5

    Article  Google Scholar 

  25. Razavi A, Matwin S, Inkpen D et al (2009) Parameterized contrast in second order soft co-occurrences: a novel text representation technique in text mining and knowledge extraction. In: 2009 IEEE international conference on data mining workshops, pp 71–6

    Google Scholar 

  26. Bragge P, Clavisi O, Turner T et al (2011) The global evidence mapping initiative: scoping research in broad topic areas. BMC Med Res Methodol 11(92). https://doi.org/10.1186/1471-2288-11-92

  27. Snilstveit B, Vojtkova M, Bhavsar A et al (2016) Evidence and gap maps—a tool for promoting evidence informed policy and strategic research agendas. J Clin Epidemiol 79:120–129. https://doi.org/10.1016/j.jclinepi.2016.05.015

    Article  Google Scholar 

  28. Arksey H, O’Malley L (2005) Scoping studies: towards a methodological framework. Int J Soc Res Meth 8:19–32

    Article  Google Scholar 

  29. RTC Collaboration. Review Manager (RevMan) 4.2 for Windows. The Cochrane Collaboration, Oxford (2003)

    Google Scholar 

  30. Tsafnat G, Coiera E (2009) Computational reasoning across multiple models. J Am Med Info Assoc 16(6):768–774

    Article  Google Scholar 

  31. Sim I, Detmer DE (2005) Beyond trial registration: a global trial bank for clinical trial reporting. PLoS Med 2(11):e365

    Article  Google Scholar 

  32. Sim I, Tu SW, Carini S et al (2014) The ontology of clinical research (OCRe): an informatics foundation for the science of clinical research. J Biomed Inf 52:78–91. https://doi.org/10.1016/j.jbi.2013.11.002

    Article  Google Scholar 

  33. Greenhalgh T, Peacock R (2005) Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ 331(7524):1064–1065. https://doi.org/10.1136/bmj.38636.593461.68

    Article  Google Scholar 

  34. Marshal C, Sutton A, O'Keefe H et al (2022) The systematic review toolbox. http://www.systematicreviewtools.com.

  35. Bowes D, Hall T, Beecham S (2012) SLuRp : a tool to help large complex systematic literature reviews deliver valid and rigorous results. In: Proceedings of the 2nd international workshop on evidential assessment of software technologies—EAST ’12, pp 33–36

    Google Scholar 

  36. Hernandes E, Zamboni A, Fabbri S et al (2012) Using GQM and TAM to evaluate StArt—a tool that supports systematic review. CLEI Electr J 15(1):2. http://www.scielo.edu.uy/pdf/cleiej/v15n1/v15n1a03.pdf

  37. Fernández-Sáez AM, Bocco MG, Romero FP (2010) SLR-Tool—a tool for performing systematic literature reviews. In: ICSOFT 2010—proceedings of the 5th international conference on software and data technologies, pp 157–166

    Google Scholar 

  38. Molléri JS, Benitti FBV (2015) SESRA: a web-based automated tool to support the systematic literature review process. In: EASE ’15: proceedings of the 19th international conference on evaluation and assessment in software engineering, pp 1–6. https://doi.org/10.1145/2745802.2745825

  39. Gurbuz HG, Tekinerdogan B (2018) Model-based testing for software safety: a systematic mapping study. Software Qual J 26:1327–1372. https://doi.org/10.1007/s11219-017-9386-2

    Article  Google Scholar 

  40. Van Dinter R, Tekinerdogan B, Cagatay C (2021) Automation of systematic literature reviews: a systematic literature review. Inf and Software Tech 136:106589. https://doi.org/10.1016/j.infsof.2021.106589

    Article  Google Scholar 

  41. Wohlin C (2014) Guidelines for snowballing in systematic literature studies and a replication in software engineering. In: EASE ’14: proceedings of the 18th international conference on evaluation and assessment in software engineering.. ACM, pp 1–10. https://doi.org/10.1145/2601248.2601268

  42. Dybå T, Dingsøyr T (2008) Empirical studies of agile software development: a systematic review. Inf Softw Tech 50(9):833–859

    Article  Google Scholar 

  43. Nguyen-Duc A, Cruzes DS, Conradi R (2015) The impact of global dispersion on coordination, team performance and software quality—a systematic literature review. Inf and Softw Tech 57:277–294

    Article  Google Scholar 

  44. Brownlee J, Tour of evaluation metrics for imbalanced classification. https://machinelearningmastery.com/tour-of-evaluation-metrics-for-imbalanced-classification

  45. Wallace BC, Trikalinos TA, Lau J et al (2010) Semi-automated screening of biomedical citations for systematic reviews. BMC Bioinformatics 11(1):55. https://doi.org/10.1186/1471-2105-11-55

    Article  Google Scholar 

  46. Pham B, Jovanovic J, Bagheri E et al (2021) Text mining to support abstract screening for knowledge syntheses: a semi-automated workflow. Syst Rev 10:156. https://doi.org/10.1186/s13643-021-01700-x

    Article  Google Scholar 

  47. Norman CR, Leeflang M, Porcher R et al (2019) Measuring the impact of screening automation on meta-analyses of diagnostic test accuracy. Syst Rev 8:243. https://doi.org/10.1186/s13643-019-1162-x

    Article  Google Scholar 

  48. Dickson K (2017) Systematic reviews to inform policy: institutional mechanisms and social interactions to support their production. Dissertation. University College London. http://discovery.ucl.ac.uk/id/eprint/10054092/1/KD_PhD_FinalAugust2018_Redacted.pdf

  49. Turing A (1950) Computing machinery and intelligence. Mind LIX (236):433–460. https://doi.org/10.1093/mind/LIX.236.433

  50. Mo Y, Kontonatsios G, Ananiadou S (2015) Supporting systematic reviews using LDA-based document representations. Syst Rev 4:172. https://doi.org/10.1186/s13643-015-0117-0

    Article  Google Scholar 

  51. Cohen AM, Ambert K, McDonagh M (2012) Studying the potential impact of automated document classification on scheduling a systematic review update. BMC Med Inform Decis Mak 12:33. https://doi.org/10.1186/1472-6947-12-33

    Article  Google Scholar 

  52. Callaghan MW, Müller-Hansen F (2020) Statistical stopping criteria for automated screening in systematic reviews. Syst Rev 9:273. https://doi.org/10.1186/s13643-020-01521-4

    Article  Google Scholar 

  53. Miwa M, Thomas J, O’Mara-Eves A et al (2014) Reducing systematic review workload through certainty-based screening. J Biomed Inf 51:242–253. https://doi.org/10.1016/j.jbi.2014.06.005

    Article  Google Scholar 

  54. Basu T, Kumar S, Kalyan A et al (2016) A novel framework to expedite systematic reviews by automatically building information extraction training corpora. arXiv:1606.06424 [cs.IR] (2016). https://arxiv.org/abs/1606.06424

  55. García Adeva JJ, Pikatza Atxa JM, Ubeda CM et al (2014) Automatic text classification to support systematic reviews in medicine. Expert Syst with Appl 41(4):1498–1508. https://doi.org/10.1016/j.eswa.2013.08.047

    Article  Google Scholar 

  56. Ros R, Bjarnason E, Runeson P (2017) A machine learning approach for semi-automated search and selection in literature studies. In: EASE ’17: proceedings of the 21st international conference on evaluation and assessment in software engineering. Association for Computing Machinery, New York, pp 118–127. https://doi.org/10.1145/3084226.3084243

  57. Frunza O, Inkpen D, Matwin S (2010) Building systematic reviews using automatic text classification techniques. In: Proceedings of the 23rd international conference on computational linguistics: poster, vol (COLING ‘10). Association for Computational Linguistics, pp 303–311

    Google Scholar 

  58. Timsina P, Liu J, El-Gayar O (2016) Advanced analytics for the automation of medical systematic reviews. Inf Syst Frontiers 18(2):237–252

    Article  Google Scholar 

  59. El-Gayar OF, Liu J, Timsina P (2015) Active learning for the automation of medical systematic review creation. In: 21st Americas conference on information systems (AMCIS). Puerto Rico Aug 13–15. htttp://aisel.aisnet.org/amcis2015/BizAnalytics/GeneralPresentations/22

    Google Scholar 

  60. Halamoda-Kenzaoui B, Rolland E, Piovesan J et al (2021) Toxic effects of nanomaterials for health applications: how automation can support a systematic review of the literature? J of Appl Tox 42(1):41–51. https://doi.org/10.1002/jat.4204

    Article  Google Scholar 

  61. Olorisade BK, Brereton P, Andras P (2019) The use of bibliography enriched features for automatic citation screening. J of Biomed Inf 94:103202. https://doi.org/10.1016/j.jbi.2019.103202

    Article  Google Scholar 

  62. Bannach-Brown A, Przybyła P, Thomas J et al (2019) Machine learning algorithms for systematic review: reducing workload in a preclinical review of animal studies and reducing human screening error. Syst Rev 8(1):23. https://doi.org/10.1186/s13643-019-0942-7

    Article  Google Scholar 

  63. Bui D, Del Fiol G, Hurdle JF et al (2016) Extractive text summarization system to aid data extraction from full text in systematic review development. J Biomed Inf 64:265–272. https://doi.org/10.1016/j.jbi.2016.10.014

    Article  Google Scholar 

  64. Tsafnat G, Glasziou P, Karystianis G et al (2018) Automated screening of research studies for systematic reviews using study characteristics. Syst Rev 7:64. https://doi.org/10.1186/s13643-018-0724-7

    Article  Google Scholar 

  65. Norman C (2020) Systematic review automation methods. Université Paris-Saclay, Universiteit van Amsterdam https://tel.archives-ouvertes.fr/tel-03060620/document

    Google Scholar 

  66. Norman C, Leeflang M, Zweigenbaum P et al (2018) Automating document discovery in the systematic review process: how to use chaff to extract wheat. In: Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018). European Language Resources Association (ELRA), Miyazaki, Japan. https://aclanthology.org/L18-1582

  67. Karystianis G, Thayer K, Wolfe M et al (2017) Evaluation of a rule-based method for epidemiological document classification towards the automation of systematic reviews. J of Biomed Inf 70:27–34. https://doi.org/10.1016/j.jbi.2017.04.004

    Article  Google Scholar 

  68. Wallace BC, Kuiper J, Sharma A et al (2016) Extracting PICO sentences from clinical trial reports using supervised distant supervision. J Mach Lear Res 17:132

    MathSciNet  Google Scholar 

  69. Marshall IJ, Kuiper J, Wallace BC (2015) Automating risk of bias assessment for clinical trials. J Biomed Health Inf 19(4):1406–1412. https://doi.org/10.1109/JBHI.2015.2431314

    Article  Google Scholar 

  70. Ma Y (2007) Text classification on imbalanced data: application to systematic reviews automation. Dissertation. University of Ottawa

    Google Scholar 

  71. Begert D, Granek J, Irwin B et al (2020) Towards automating systematic reviews on immunization using an advanced natural language processing-based extraction system. Can Commun Dis Rep 46(6):174–179. https://doi.org/10.14745/ccdr.v46i06a04

    Article  Google Scholar 

  72. Scells H, Zuccon G, Koopman B (2019) Automatic boolean query refinement for systematic review literature search. In: The World Wide Web Conference (WWW ‘19). Association for Computing Machinery, New York, pp 1646–1656. https://doi.org/10.1145/3308558.3313544

  73. Khabsa M, Elmagarmid A, Ilyas I et al (2016) Learning to identify relevant studies for systematic reviews using random forest and external information. Mach Learn 102:465–482. https://doi.org/10.1007/s10994-015-5535-7

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

Publication of this work was supported by the National Science Foundation under Award No. OIA-1946391. The content reflects the views of the authors and not necessarily the NSF. The authors are grateful to Deepak Sagaram, MD, for consulting on the list of articles regarding their relevance for inclusion and exclusion.

Author information

Authors and Affiliations

  1. University of Arkansas at Little Rock, Little Rock, AR, 72204, USA

    Girish Sundaram & Daniel Berleant

Authors
  1. Girish Sundaram
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daniel Berleant
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Girish Sundaram .

Editor information

Editors and Affiliations

  1. Department of Design Engineering and Mathematics, Middlesex University London, London, UK

    Xin-She Yang

  2. Department of Biomedical Engineering, University of Reading, England, UK

    R. Simon Sherratt

  3. Department of Computer Science and Engineering, Techno International Newtown, Chakpachuria, West Bengal, India

    Nilanjan Dey

  4. Global Knowledge Research Foundation, Ahmedabad, India

    Amit Joshi

Supplementary Material

Supplementary Material

The supplementary material, including data Tables A1–A4, may be obtained at

https://dberleant.github.io/papers/sundber-supp.pdf.

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Sundaram, G., Berleant, D. (2023). Automating Systematic Literature Reviews with Natural Language Processing and Text Mining: A Systematic Literature Review. In: Yang, XS., Sherratt, R.S., Dey, N., Joshi, A. (eds) Proceedings of Eighth International Congress on Information and Communication Technology. ICICT 2023. Lecture Notes in Networks and Systems, vol 693. Springer, Singapore. https://doi.org/10.1007/978-981-99-3243-6_7

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-3243-6_7

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-3242-9

  • Online ISBN: 978-981-99-3243-6

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation