Abstract
Existing methods for interpretability of model predictions are largely based on technical insights and are not linked to clinical context. We use the question of predicting response to radiotherapy in colorectal cancer patients as an exemplar for developing prediction models that do provide such contextual information and therefore can effectively support clinical decision making. There is a growing body of evidence that about 30% of colorectal cancer patients do not respond to radiotherapy and will need alternative treatment. The consensus molecular subtypes for colorectal cancer (CMS) provide one such approach to categorising patients based on their disease biology. Here we select the CMS4 subtype as a proxy for stromal infiltration. By jointly predicting a patient’s response to radiotherapy, the presence of CMS4, and the epithelial tissue map from morphological features extracted from standard H &E slides we provide a comprehensive clinically relevant assessment of a biopsy. A graph neural network is trained to achieve this joint prediction task, which subsequently provides novel interpretability maps to aid clinicians in their cancer treatment decision making process. Our model is trained and validated on two private rectal cancer datasets.
Supported by Cancer Research UK.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Süsstrunk, S.: Slic superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2274–2282 (2012)
Alkan, A., Hofving, T., Angenete, E., Yrlid, U.: Biomarkers and cell-based models to predict the outcome of neoadjuvant therapy for rectal cancer patients. Biomark. Res. 9(1) (2021)
Anitei, M.G., et al.: Prognostic and predictive values of the immunoscore in patients with rectal cancer. Clin. Cancer Res. 20(7), 1891–1899 (2014)
Caron, M., et al.: Emerging properties in self-supervised vision transformers. In: Proceedings of the International Conference on Computer Vision (ICCV) (2021)
Chatila, W., Kim, J., Walch, H.: Genomic and transcriptomic determinants of response to neoadjuvant therapy in rectal cancer. Nat. Med. 28, 1646–1655 (2022)
Chen, R.J., Krishnan, R.G.: Self-supervised vision transformers learn visual concepts in histopathology (2022)
Ding, K., Liu, Q., Lee, E., Zhou, M., Lu, A., Zhang, S.: Feature-enhanced graph networks for genetic mutational prediction using histopathological images in colon cancer. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12262, pp. 294–304. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59713-9_29
George, T.J.J., Allegra, C.J., Yothers, G.: Neoadjuvant rectal (NAR) score: a new surrogate endpoint in rectal cancer clinical trials. Curr. Colorect. Cancer Rep. 11(5), 275–280 (2015)
Guinney, J., et al.: The consensus molecular subtypes of colorectal cancer. Nat. Med. 21, 1350–1356 (2015)
Jones, H.J.S., et al.: Stromal composition predicts recurrence of early rectal cancer after local excision. Histopathology 79, 947–956 (2021)
Lee, Y., Park, J., Oh, S.: Derivation of prognostic contextual histopathological features from whole-slide images of tumours via graph deep learning. Nat. Biomed. Eng. (2022)
Lipkova, J., et al.: Artificial intelligence for multimodal data integration in oncology. Cancer Cell 40(10), 1095–1110 (2022)
Lu, W., Toss, M., Dawood, M., Rakha, E., Rajpoot, N., Minhas, F.: Slidegraph+: whole slide image level graphs to predict her2 status in breast cancer. Med. Image Anal. 80, 102486 (2022)
Pina, O., Vilaplana, V.: Self-supervised graph representations of WSIS. In: Bekkers, E., Wolterink, J.M., Aviles-Rivero, A. (eds.) Proceedings of the First International Workshop on Geometric Deep Learning in Medical Image Analysis. Proceedings of Machine Learning Research, vol. 194, pp. 107–117. PMLR (2022)
Qi, L., et al.: Identification of prognostic spatial organization features in colorectal cancer microenvironment using deep learning on histopathology images. Med. Omics 2, 100008 (2021)
Raju, A., Yao, J., Haq, M.M.H., Jonnagaddala, J., Huang, J.: Graph attention multi-instance learning for accurate colorectal cancer staging. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12265, pp. 529–539. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59722-1_51
Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation (2015)
Sirinukunwattana, K., et al.: Image-based consensus molecular subtype (imcms) classification of colorectal cancer using deep learning. Gut 70(3), 544–554 (2021)
Wood, R., et al.: Enhancing local context of histology features in vision transformers. In: Artificial Intelligence over Infrared Images for Medical Applications and Medical Image Assisted Biomarker Discovery. pp. 154–163. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19660-7_15
Xu, J., Luo, X., Wang, G., Gilmore, H., Madabhushi, A.: A deep convolutional neural network for segmenting and classifying epithelial and stromal regions in histopathological images. Neurocomputing 191, 214–223 (2016)
Xu, K., Hu, W., Leskovec, J., Jegelka, S.: How powerful are graph neural networks? (2018)
Zhang, F., et al.: Predicting treatment response to neoadjuvant chemoradiotherapy in local advanced rectal cancer by biopsy digital pathology image features. Clin. Transl. Med. 10(2), e110 (2020)
Zhou, Y., Graham, S., Koohbanani, N.A., Shaban, M., Heng, P.A., Rajpoot, N.M.: CGC-net: cell graph convolutional network for grading of colorectal cancer histology images. In: IEEE/CVF International Conference on Computer Vision Workshops, pp. 388–398. IEEE (2019)
Data Use Declaration and Acknowledgements
The Grampian, Aristotle and FOCUS datasets used in this study were available through the S:CORT consortium. Grampian and Aristotle will be made publicly available shortly. Additional data is available on request to the S:CORT consortium. The S:CORT consortium was reviewed and approved by the South Cambs Research Ethics committee (REC ref 15/EE/0241; IRAS reference 169363). The Stratification in Colorectal Cancer Consortium (S:CORT) was funded by the Medical Research Council and Cancer Research UK (MR/M016587/1). The Aristotle trial was funded by Cancer Research UK (CRUK/08/032). The funders played no role in the analyses performed or the results presented. Financial support: RW - EPSRC Center for Doctoral Training in Health Data Science (EP/S02428X/1), Oxford CRUK Cancer Centre; VHK - Promedica Foundation (F-87701-41-01) and Swiss National Science Foundation (P2SKP3\(\_\)168322/1, P2SKP3\(\_\)168322/2); TSM - S:CORT (see above); JR, KS - Oxford NIHR National Oxford Biomedical Research Centre and the PathLAKE consortium (InnovateUK). The computational aspects of this research were funded from the NIHR Oxford BRC with additional support from the Wellcome Trust Core Award Grant Number 203141/Z/16/Z. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Wood, R. et al. (2023). Joint Prediction of Response to Therapy, Molecular Traits, and Spatial Organisation in Colorectal Cancer Biopsies. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14224. Springer, Cham. https://doi.org/10.1007/978-3-031-43904-9_73
Download citation
DOI: https://doi.org/10.1007/978-3-031-43904-9_73
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43903-2
Online ISBN: 978-3-031-43904-9
eBook Packages: Computer ScienceComputer Science (R0)
