Skip to main content

Advertisement

SpringerLink
Log in

Rectal MRI radiomics inter- and intra-reader reliability: should we worry about that?

  • Hollow Organ GI
  • Published:
Abdominal Radiology Aims and scope Submit manuscript

Abstract

Purpose

The aim of this review paper is to summarize the current literature regarding inter- and intra-reader reliability of radiomics on rectal MRI.

Methods

Original studies examining treatment response prediction in patients with rectal cancer following neoadjuvant therapy using rectal MRI-based radiomics between January 2010 and December 2021 were identified via a PubMed/Medline search. Studies in which intra- and/or inter-reader reliability had been reported were included in this review.

Results

Thirteen studies were selected, with an average number of patients of 145 (range, 20–649). All included studies evaluated T2-weighted imaging (T2WI) and/or diffusion-weighted imaging (DWI) sequences, while 3/13 (23%) also evaluated the contrast-enhanced T1-weighted imaging (T1WI) sequence. Most of the selected studies involved two readers (10/13, 77%), 6/13 (46%) studies used baseline MRI only, 1/13 (8%) study used restaging MRI only, and 6/13 (46%) used both. Segmentation was performed manually in 10/13 (77%) studies, and in a slight majority of studies (7/13, 54%), the entire tumor volume (3D VOI) was segmented, while 4/13 (31%) studies segmented the 2D ROI and 2/13 (15%) segmented both. Intraclass correlation coefficient (ICC) on intra-reader agreement varied from 0.73 to 0.93. ICC to assess inter-reader varied from 0.60 to 0.99. Overall, features obtained from baseline rectal MRI, using 3D VOI and first-order features, had higher agreement.

Conclusion

Based on our qualitative assessment of a small number of non-dedicated studies, there seems to be good reliability, particularly among low-order features extracted from the entire tumor volume using baseline MRI; however, direct evidence remains scarce. More targeted research in this area is required to quantitatively verify reliability, and before these novel radiomic techniques can be clinically adopted.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Habr-Gama A, Perez RO, Nadalin W, Sabbaga J, Ribeiro U, Jr., Silva e Sousa AH, Jr., Campos FG, Kiss DR, Gama-Rodrigues J (2004) Operative versus nonoperative treatment for stage 0 distal rectal cancer following chemoradiation therapy: long-term results. Ann Surg 240 (4):711–717; discussion 717–718

  2. [2] Maas M, Beets-Tan RG, Lambregts DM, Lammering G, Nelemans PJ, Engelen SM, van Dam RM, Jansen RL, Sosef M, Leijtens JW, Hulsewe KW, Buijsen J, Beets GL (2011) Wait-and-see policy for clinical complete responders after chemoradiation for rectal cancer. J Clin Oncol 29 (35):4633-4640. https://doi.org/10.1200/JCO.2011.37.7176

    Article  PubMed  Google Scholar 

  3. Li J, Liu H, Yin J, Liu S, Hu J, Du F, Yuan J, Lv B, Fan J, Leng S, Zhang X (2015) Wait-and-see or radical surgery for rectal cancer patients with a clinical complete response after neoadjuvant chemoradiotherapy: a cohort study. Oncotarget 6 (39):42354–42361. https://doi.org/10.18632/oncotarget.6093

  4. [4] Renehan AG, Malcomson L, Emsley R, Gollins S, Maw A, Myint AS, Rooney PS, Susnerwala S, Blower A, Saunders MP, Wilson MS, Scott N, O'Dwyer ST (2016) Watch-and-wait approach versus surgical resection after chemoradiotherapy for patients with rectal cancer (the OnCoRe project): a propensity-score matched cohort analysis. Lancet Oncol 17 (2):174-183. https://doi.org/10.1016/S1470-2045(15)00467-2

    Article  PubMed  Google Scholar 

  5. [5] Kasi A, Abbasi S, Handa S, Al-Rajabi R, Saeed A, Baranda J, Sun W (2020) Total Neoadjuvant Therapy vs Standard Therapy in Locally Advanced Rectal Cancer: A Systematic Review and Meta-analysis. JAMA Netw Open 3 (12):e2030097. https://doi.org/10.1001/jamanetworkopen.2020.30097

    Article  PubMed  PubMed Central  Google Scholar 

  6. [6] Guillem JG, Chessin DB, Shia J, Moore HG, Mazumdar M, Bernard B, Paty PB, Saltz L, Minsky BD, Weiser MR, Temple LK, Cohen AM, Wong WD (2005) Clinical examination following preoperative chemoradiation for rectal cancer is not a reliable surrogate end point. J Clin Oncol 23 (15):3475-3479. https://doi.org/10.1200/JCO.2005.06.114

    Article  PubMed  Google Scholar 

  7. [7] Kawai K, Ishihara S, Nozawa H, Hata K, Kiyomatsu T, Morikawa T, Fukayama M, Watanabe T (2017) Prediction of Pathological Complete Response Using Endoscopic Findings and Outcomes of Patients Who Underwent Watchful Waiting After Chemoradiotherapy for Rectal Cancer. Dis Colon Rectum 60 (4):368-375. https://doi.org/10.1097/DCR.0000000000000742

    Article  PubMed  Google Scholar 

  8. Park SH, Cho SH, Choi SH, Jang JK, Kim MJ, Kim SH, Lim JS, Moon SK, Park JH, Seo N, Cancer KSoARSGfR (2020) MRI Assessment of Complete Response to Preoperative Chemoradiation Therapy for Rectal Cancer: 2020 Guide for Practice from the Korean Society of Abdominal Radiology. Korean J Radiol 21 (7):812-828. https://doi.org/10.3348/kjr.2020.0483

  9. [9] Kim SH, Lee JM, Park HS, Eun HW, Han JK, Choi BI (2009) Accuracy of MRI for predicting the circumferential resection margin, mesorectal fascia invasion, and tumor response to neoadjuvant chemoradiotherapy for locally advanced rectal cancer. J Magn Reson Imaging 29 (5):1093-1101. https://doi.org/10.1002/jmri.21742

    Article  PubMed  Google Scholar 

  10. [10] Nahas SC, Nahas CSR, Cama GM, de Azambuja RL, Horvat N, Marques CFS, Menezes MR, Junior UR, Cecconello I (2019) Diagnostic performance of magnetic resonance to assess treatment response after neoadjuvant therapy in patients with locally advanced rectal cancer. Abdom Radiol (NY). https://doi.org/10.1007/s00261-019-01894-8

    Article  Google Scholar 

  11. [11] Bates DDB, Golia Pernicka JS, Fuqua JL, Paroder V, Petkovska I, Zheng J, Capanu M, Schilsky J, Gollub MJ (2020) Diagnostic accuracy of b800 and b1500 DWI-MRI of the pelvis to detect residual rectal adenocarcinoma: a multi-reader study. Abdom Radiol (NY) 45 (2):293-300. https://doi.org/10.1007/s00261-019-02283-x

    Article  Google Scholar 

  12. Horvat N, Veeraraghavan H, Khan M, Blazic I, Zheng J, Capanu M, Sala E, Garcia-Aguilar J, Gollub MJ, Petkovska I (2018) MR Imaging of Rectal Cancer: Radiomics Analysis to Assess Treatment Response after Neoadjuvant Therapy. Radiology:172300. https://doi.org/10.1148/radiol.2018172300

  13. [13] Cusumano D, Meijer G, Lenkowicz J, Chiloiro G, Boldrini L, Masciocchi C, Dinapoli N, Gatta R, Casà C, Damiani A, Barbaro B, Gambacorta MA, Azario L, De Spirito M, Intven M, Valentini V (2021) A field strength independent MR radiomics model to predict pathological complete response in locally advanced rectal cancer. Radiol Med 126 (3):421-429. https://doi.org/10.1007/s11547-020-01266-z

    Article  PubMed  Google Scholar 

  14. [14] Petkovska I, Tixier F, Ortiz EJ, Golia Pernicka JS, Paroder V, Bates DD, Horvat N, Fuqua J, Schilsky J, Gollub MJ, Garcia-Aguilar J, Veeraraghavan H (2020) Clinical utility of radiomics at baseline rectal MRI to predict complete response of rectal cancer after chemoradiation therapy. Abdom Radiol (NY). https://doi.org/10.1007/s00261-020-02502-w

    Article  Google Scholar 

  15. [15] Horvat N, Miranda J, El Homsi M, Peoples JJ, Long NM, Simpson AL, Do RKG (2021) A primer on texture analysis in abdominal radiology. Abdom Radiol (NY). https://doi.org/10.1007/s00261-021-03359-3

    Article  PubMed  Google Scholar 

  16. [16] Rizzo S, Botta F, Raimondi S, Origgi D, Fanciullo C, Morganti AG, Bellomi M (2018) Radiomics: the facts and the challenges of image analysis. Eur Radiol Exp 2 (1):36. https://doi.org/10.1186/s41747-018-0068-z

    Article  PubMed  PubMed Central  Google Scholar 

  17. [17] Lubner MG, Smith AD, Sandrasegaran K, Sahani DV, Pickhardt PJ (2017) CT Texture Analysis: Definitions, Applications, Biologic Correlates, and Challenges. Radiographics 37 (5):1483-1503. https://doi.org/10.1148/rg.2017170056

    Article  PubMed  Google Scholar 

  18. [18] Bashir U, Siddique MM, Mclean E, Goh V, Cook GJ (2016) Imaging Heterogeneity in Lung Cancer: Techniques, Applications, and Challenges. AJR Am J Roentgenol 207 (3):534-543. https://doi.org/10.2214/AJR.15.15864

    Article  PubMed  Google Scholar 

  19. Miranda J, Tan GXV, Fernandes MC, Yildirim O, Sims JA, Araujo-Filho JAB, de M Machado FA, Assuncao-Jr AN, Nomura CH, Horvat N (2021) Rectal MRI radiomics for predicting pathological complete response: Where we are. Clin Imaging 82:141–149. https://doi.org/10.1016/j.clinimag.2021.10.005

  20. [20] Li ZY, Wang XD, Li M, Liu XJ, Ye Z, Song B, Yuan F, Yuan Y, Xia CC, Zhang X, Li Q (2020) Multi-modal radiomics model to predict treatment response to neoadjuvant chemotherapy for locally advanced rectal cancer. World J Gastroenterol 26 (19):2388-2402. https://doi.org/10.3748/wjg.v26.i19.2388

    Article  PubMed  PubMed Central  Google Scholar 

  21. [21] Schurink NW, van Kranen SR, Roberti S, van Griethuysen JJM, Bogveradze N, Castagnoli F, Khababi NE, Bakers FCH, de Bie SH, Bosma GPT, Cappendijk VC, Geenen RWF, Neijenhuis PA, Peterson GM, Veeken CJ, Vliegen RFA, Beets-Tan RGH, Lambregts DMJ (2021) Sources of variation in multicenter rectal MRI data and their effect on radiomics feature reproducibility. Eur Radiol. https://doi.org/10.1007/s00330-021-08251-8

    Article  PubMed  PubMed Central  Google Scholar 

  22. [22] Palmisano A, Di Chiara A, Esposito A, Rancoita PMV, Fiorino C, Passoni P, Albarello L, Rosati R, Del Maschio A, De Cobelli F (2020) MRI prediction of pathological response in locally advanced rectal cancer: when apparent diffusion coefficient radiomics meets conventional volumetry. Clin Radiol 75 (10):798.e791-798.e711. https://doi.org/10.1016/j.crad.2020.06.023

    Article  Google Scholar 

  23. Petresc B, Lebovici A, Caraiani C, Feier DS, Graur F, Buruian MM (2020) Pre-Treatment T2-WI Based Radiomics Features for Prediction of Locally Advanced Rectal Cancer Non-Response to Neoadjuvant Chemoradiotherapy: A Preliminary Study. Cancers (Basel) 12 (7). https://doi.org/10.3390/cancers12071894

  24. [24] van Griethuysen JJM, Lambregts DMJ, Trebeschi S, Lahaye MJ, Bakers FCH, Vliegen RFA, Beets GL, Aerts HJWL, Beets-Tan RGH (2020) Radiomics performs comparable to morphologic assessment by expert radiologists for prediction of response to neoadjuvant chemoradiotherapy on baseline staging MRI in rectal cancer. Abdom Radiol (NY) 45 (3):632-643. https://doi.org/10.1007/s00261-019-02321-8

    Article  Google Scholar 

  25. [25] Aker M, Ganeshan B, Afaq A, Wan S, Groves AM, Arulampalam T (2019) Magnetic Resonance Texture Analysis in Identifying Complete Pathological Response to Neoadjuvant Treatment in Locally Advanced Rectal Cancer. Dis Colon Rectum 62 (2):163-170. https://doi.org/10.1097/DCR.0000000000001224

    Article  PubMed  Google Scholar 

  26. [26] Cui Y, Yang X, Shi Z, Yang Z, Du X, Zhao Z, Cheng X (2019) Radiomics analysis of multiparametric MRI for prediction of pathological complete response to neoadjuvant chemoradiotherapy in locally advanced rectal cancer. Eur Radiol 29 (3):1211-1220. https://doi.org/10.1007/s00330-018-5683-9

    Article  PubMed  Google Scholar 

  27. [27] Meng X, Xia W, Xie P, Zhang R, Li W, Wang M, Xiong F, Liu Y, Fan X, Xie Y, Wan X, Zhu K, Shan H, Wang L, Gao X (2019) Preoperative radiomic signature based on multiparametric magnetic resonance imaging for noninvasive evaluation of biological characteristics in rectal cancer. Eur Radiol 29 (6):3200-3209. https://doi.org/10.1007/s00330-018-5763-x

    Article  PubMed  Google Scholar 

  28. [28] Liu Z, Zhang XY, Shi YJ, Wang L, Zhu HT, Tang Z, Wang S, Li XT, Tian J, Sun YS (2017) Radiomics Analysis for Evaluation of Pathological Complete Response to Neoadjuvant Chemoradiotherapy in Locally Advanced Rectal Cancer. Clin Cancer Res 23 (23):7253-7262. https://doi.org/10.1158/1078-0432.CCR-17-1038

    Article  CAS  PubMed  Google Scholar 

  29. [29] Choi MH, Oh SN, Rha SE, Choi JI, Lee SH, Jang HS, Kim JG, Grimm R, Son Y (2016) Diffusion-weighted imaging: Apparent diffusion coefficient histogram analysis for detecting pathologic complete response to chemoradiotherapy in locally advanced rectal cancer. J Magn Reson Imaging 44 (1):212-220. https://doi.org/10.1002/jmri.25117

    Article  PubMed  Google Scholar 

  30. [30] Nougaret S, Vargas HA, Lakhman Y, Sudre R, Do RK, Bibeau F, Azria D, Assenat E, Molinari N, Pierredon MA, Rouanet P, Guiu B (2016) Intravoxel Incoherent Motion-derived Histogram Metrics for Assessment of Response after Combined Chemotherapy and Radiation Therapy in Rectal Cancer: Initial Experience and Comparison between Single-Section and Volumetric Analyses. Radiology 280 (2):446-454. https://doi.org/10.1148/radiol.2016150702

    Article  PubMed  Google Scholar 

  31. [31] Yang C, Jiang ZK, Liu LH, Zeng MS (2020) Pre-treatment ADC image-based random forest classifier for identifying resistant rectal adenocarcinoma to neoadjuvant chemoradiotherapy. Int J Colorectal Dis 35 (1):101-107. https://doi.org/10.1007/s00384-019-03455-3

    Article  PubMed  Google Scholar 

  32. [32] Li Z, Ma X, Shen F, Lu H, Xia Y, Lu J (2021) Evaluating treatment response to neoadjuvant chemoradiotherapy in rectal cancer using various MRI-based radiomics models. BMC Med Imaging 21 (1):30. https://doi.org/10.1186/s12880-021-00560-0

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  33. [33] Cui Y, Yang X, Shi Z, Yang Z, Du X, Zhao Z, Cheng X (2018) Radiomics analysis of multiparametric MRI for prediction of pathological complete response to neoadjuvant chemoradiotherapy in locally advanced rectal cancer. Eur Radiol. https://doi.org/10.1007/s00330-018-5683-9

    Article  PubMed  PubMed Central  Google Scholar 

  34. [34] Meng Y, Zhang Y, Dong D, Li C, Liang X, Zhang C, Wan L, Zhao X, Xu K, Zhou C, Tian J, Zhang H (2018) Novel radiomic signature as a prognostic biomarker for locally advanced rectal cancer. J Magn Reson Imaging. https://doi.org/10.1002/jmri.25968

    Article  PubMed  Google Scholar 

  35. Meng Y, Zhang C, Zou S, Zhao X, Xu K, Zhang H, Zhou C (2018) MRI texture analysis in predicting treatment response to neoadjuvant chemoradiotherapy in rectal cancer. Oncotarget 9 (15):11999–12008. https://doi.org/10.18632/oncotarget.23813

  36. [36] Jha AK, Mithun S, Jaiswar V, Sherkhane UB, Purandare NC, Prabhash K, Rangarajan V, Dekker A, Wee L, Traverso A (2021) Repeatability and reproducibility study of radiomic features on a phantom and human cohort. Scientific Reports 11 (1):2055. https://doi.org/10.1038/s41598-021-81526-8

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  37. [37] Moradmand H, Aghamiri SMR, Ghaderi R (2020) Impact of image preprocessing methods on reproducibility of radiomic features in multimodal magnetic resonance imaging in glioblastoma. 21 (1):179-190. https://doi.org/10.1002/acm2.12795

    Article  Google Scholar 

  38. [38] Traverso A, Wee L, Dekker A, Gillies R (2018) Repeatability and Reproducibility of Radiomic Features: A Systematic Review. International journal of radiation oncology, biology, physics 102 (4):1143-1158. https://doi.org/10.1016/j.ijrobp.2018.05.053

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

The authors thank Joanne Chin, MFA, ELS, for her help in editing this manuscript.

Funding

This research was funded in part through the NIH/NCI Cancer Center Support Grant (P30 CA008748).

Author information

Author notes

  1. Henry C. Kwok and Charlotte Charbel contributed equally to this study.

  2. Jayasree Chakraborty and Natally Horvat contributed equally to this study.

Authors and Affiliations

  1. Department of Radiology, Memorial Sloan Kettering Cancer Center, 1275 York Avenue, New York, NY, 10065, USA

    Henry C. Kwok, Charlotte Charbel, Sofia Danilova, Natalie Gangai, Iva Petkovska & Natally Horvat

  2. Department of Radiology, University of Sao Paulo, Sao Paulo, Brazil

    Joao Miranda

  3. Department of Surgery, Memorial Sloan Kettering Cancer Center, New York, NY, USA

    Jayasree Chakraborty

Authors
  1. Henry C. Kwok
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Charlotte Charbel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sofia Danilova
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Joao Miranda
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Natalie Gangai
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Iva Petkovska
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jayasree Chakraborty
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Natally Horvat
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Natally Horvat.

Ethics declarations

Conflict of interest

The authors declared that they have no conflict of interest.

Ethical approval

For this literature review, no ethical approval is required.

Informed consent

For this type of study formal consent is not required.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

PubMed search terms: (radiomics) AND ((rectal cancer) OR (rectal neoplasm)) AND ((MRI) OR (Magnetic resonance imaging) or (magnetic resonance)) AND (locally advanced) AND ((neoadjuvant) or (chemoradiotherapy) or (CRT) or (TNT) or (total neoadjuvant therapy) or (neoadjuvant chemoradiotherapy)) Filters: from 2010 to 2021.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kwok, H.C., Charbel, C., Danilova, S. et al. Rectal MRI radiomics inter- and intra-reader reliability: should we worry about that?. Abdom Radiol 47, 2004–2013 (2022). https://doi.org/10.1007/s00261-022-03503-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00261-022-03503-7

Keywords

Search

Navigation