Abstract
The CLEF eHealth Technology Assisted Reviews (TAR) in Empirical Medicine Tasks focused on evaluating the effectiveness of various technology-assisted review systems in assisting healthcare professionals in retrieving relevant information from vast amounts of medical literature. It ran for three years, from 2017 until 2019, giving the opportunity to research groups to conduct experiments and share results on automatic methods to retrieve relevant studies with high precision and high recall. In this paper, we perform a reproducibility study of one of the top-performing systems of both the years 2018 and 2019 by rerunning the original code that was provided by the authors of the paper. The goals of this paper are 1) to document the pitfalls in the description of the code, 2) to reorganize the code using a better reproducibility approach (R markdown), 3) to propose some minor changes to the code that would improve the performances of the system.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Alharbi, A., Stevenson, M.: Ranking studies for systematic reviews using query adaptation: university of sheffield’s approach to CLEF ehealth 2019 task 2. In: Working Notes of CLEF (2019). https://ceur-ws.org/Vol-2380/paper_185.pdf
Breuer, T., et al.: How to measure the reproducibility of system-oriented IR experiments. In: Proceedings of SIGIR (2020). https://doi.org/10.1145/3397271.3401036
Di Nunzio, G.M.: A distributed effort approach for systematic reviews. IMS unipd at CLEF 2019 ehealth task 2. In: Working Notes of CLEF (2019). https://ceur-ws.org/Vol-2380/paper_205.pdf
Di Nunzio, G.M.: A study of an automatic stopping strategy for technologically assisted medical reviews. In: Proceedings of ECIR (2018). https://doi.org/10.1007/978-3-319-76941-7_61
Di Nunzio, G.M.: A study on a stopping strategy for systematic reviews based on a distributed effort approach. In: Arampatzis, A., et al. (eds.) CLEF 2020. LNCS, vol. 12260, pp. 112–123. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58219-7_10
Di Nunzio, G.: A new decision to take for cost-sensitive naïve bayes classifiers. Inf. Process. Manage. 50(5), 653–674 (2014). https://doi.org/10.1016/j.ipm.2014.04.008
Di Nunzio, G.: An interactive analysis of the costs of technologically assisted medical reviews. In: 12th EAI International Conference on Pervasive Computing Technologies for Healthcare (2018). https://doi.org/10.4108/eai.20-4-2018.2276346
Di Nunzio, G.M., Ciuffreda, G., Vezzani, F.: Interactive sampling for systematic reviews. IMS unipd at CLEF 2018 ehealth task 2. In: Working Notes of CLEF 2018 (2018). https://ceur-ws.org/Vol-2125/paper_186.pdf
Di Nunzio, G., Minzoni, R.: A thorough reproducibility study on sentiment classification: methodology, experimental setting, results. Information 14(2), 76 (2023). https://doi.org/10.3390/info14020076
Goeuriot, L., et al.: CLEF 2017 eHealth evaluation lab overview. In: Jones, G.J.F., et al. (eds.) CLEF 2017. LNCS, vol. 10456, pp. 291–303. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-65813-1_26
Kanoulas, E., Li, D., Azzopardi, L., Spijker, R.: CLEF 2019 technology assisted reviews in empirical medicine overview. In: Working Notes of CLEF (2019). https://ceur-ws.org/Vol-2380/paper_250.pdf
Kelly, L., et al.: Overview of the CLEF eHealth evaluation lab 2019. In: Crestani, F., et al. (eds.) CLEF 2019. LNCS, vol. 11696, pp. 322–339. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-28577-7_26
Knuth, D.E.: Literate programming. Comput. J. 27(2), 97–111 (1984). https://doi.org/10.1093/comjnl/27.2.97
Li, D., Kanoulas, E.: Automatic thresholding by sampling documents and estimating recall. In: Working Notes of CLEF (2019). https://ceur-ws.org/Vol-2380/paper_187.pdf
Suominen, H., et al.: Overview of the CLEF ehealth evaluation lab 2018. In: Bellot, P., et al. (eds.) CLEF 2018. LNCS, vol. 11018, pp. 286–301. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-98932-7_26
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Di Nunzio, G.M., Vezzani, F. (2023). The Best is Yet to Come: A Reproducible Analysis of CLEF eHealth TAR Experiments. In: Arampatzis, A., et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2023. Lecture Notes in Computer Science, vol 14163. Springer, Cham. https://doi.org/10.1007/978-3-031-42448-9_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-42448-9_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-42447-2
Online ISBN: 978-3-031-42448-9
eBook Packages: Computer ScienceComputer Science (R0)