Multiparametric Oncologic Hybrid Imaging: Machine Learning Challenges and Opportunities

 

 Buy Article Permissions and Reprints

Abstract

Background Machine learning (ML) is considered an important technology for future data analysis in health care.

Methods The inherently technology-driven fields of diagnostic radiology and nuclear medicine will both benefit from ML in terms of image acquisition and reconstruction. Within the next few years, this will lead to accelerated image acquisition, improved image quality, a reduction of motion artifacts and – for PET imaging – reduced radiation exposure and new approaches for attenuation correction. Furthermore, ML has the potential to support decision making by a combined analysis of data derived from different modalities, especially in oncology. In this context, we see great potential for ML in multiparametric hybrid imaging and the development of imaging biomarkers.

Results and Conclusion In this review, we will describe the basics of ML, present approaches in hybrid imaging of MRI, CT, and PET, and discuss the specific challenges associated with it and the steps ahead to make ML a diagnostic and clinical tool in the future.

Key Points:

• ML provides a viable clinical solution for the reconstruction, processing, and analysis of hybrid imaging obtained from MRI, CT, and PET.

Zusammenfassung

Hintergrund Maschinelles Lernen (ML) gilt als eine wichtige Technologie für die zukünftige Datenanalyse im Gesundheitswesen.

Methode Die inhärent technologiegetriebene diagnostische Radiologie und Nuklearmedizin werden sowohl bei der Bildaufnahme als auch bei der Bildrekonstruktion von ML profitieren. In den nächsten Jahren wird dies zu einer beschleunigten Bildaufnahme, einer verbesserten Bildqualität, einer Reduzierung von Bewegungsartefakten und – für die PET-Bildgebung – zu einer reduzierten Strahlenexposition und neuen Ansätzen zur Schwächungskorrektur führen. Darüber hinaus hat ML das Potenzial, die Entscheidungsfindung durch eine kombinierte Analyse von Daten aus verschiedenen Modalitäten, insbesondere im Bereich der Onkologie, zu unterstützen. In diesem Zusammenhang sehen wir ein großes Potenzial für ML in der multiparametrischen Hybrid-Bildgebung und der Entwicklung von bildgebenden Biomarkern.

Ergebnisse und Schlussfolgerung In diesem Review werden wir die Grundlagen von ML beschreiben, Ansätze in der hybriden Bildgebung von MRT, CT und PET vorstellen und die damit verbundenen spezifischen Herausforderungen und die kommenden Schritte diskutieren, um ML in Zukunft zu einem diagnostischen und klinischen Werkzeug zu machen.

Kernaussagen:

• ML bietet eine praktikable klinische Lösung für die Rekonstruktion, Verarbeitung und Analyse von Hybrid-Bildgebung der MRT, CT und PET.

Publication History

Received: 22 June 2021

Accepted: 25 November 2021

Article published online:
06 October 2023

© 2022. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

• References

• 1 Annapureddy AR, Angraal S, Caraballo C. et al The National Institutes of Health funding for clinical research applying machine learning techniques in 2017. NPJ digital medicine 2020; 3: 13-13
  CrossrefPubMedGoogle Scholar
• 2 Mildenberger P, Eichelberg M, Martin E. Introduction to the DICOM standard. Eur Radiol 2002; 12: 920-927
  CrossrefPubMedGoogle Scholar
• 3 Willemink MJ, Noël PB. The evolution of image reconstruction for CT-from filtered back projection to artificial intelligence. Eur Radiol 2019; 29: 2185-2195
  CrossrefPubMedGoogle Scholar
• 4 Lee JS. A Review of Deep-Learning-Based Approaches for Attenuation Correction in Positron Emission Tomography. IEEE Transactions on Radiation and Plasma Medical Sciences 2021; 5: 160-184
  CrossrefGoogle Scholar
• 5 Syeda-Mahmood T. Role of Big Data and Machine Learning in Diagnostic Decision Support in Radiology. J Am Coll Radiol 2018; 15: 569-576
  CrossrefPubMedGoogle Scholar
• 6 Hosny A, Parmar C, Quackenbush J. et al Artificial intelligence in radiology. Nat Rev Cancer 2018; 18: 500-510
  CrossrefPubMedGoogle Scholar
• 7 Hofmarcher T, Lindgren P, Wilking N. et al The cost of cancer in Europe 2018. Eur J Cancer 2020; 129: 41-49
  CrossrefPubMedGoogle Scholar
• 8 Ignatiadis M, Sledge GW, Jeffrey SS. Liquid biopsy enters the clinic – implementation issues and future challenges. Nat Rev Clin Oncol 2021; 18: 297-312
  CrossrefPubMedGoogle Scholar
• 9 Gillies RJ, Kinahan PE, Hricak H. Radiomics: Images Are More than Pictures, They Are Data. Radiology 2016; 278: 563-577
  CrossrefPubMedGoogle Scholar
• 10 Bishop C. Pattern recognition and machine learning. Springer; 2006
  Google Scholar
• 11 Kingma DP, Welling M. An introduction to variational autoencoders. In: Now Publishers Inc; 2019: 307–392
  Google Scholar
• 12 Kramer MA. Nonlinear principal component analysis using autoassociative neural networks. AIChE Journal 1991; 37: 233-243
  CrossrefGoogle Scholar
• 13 Hinton GE, Salakhutdinov RR. Reducing the dimensionality of data with neural networks. Science 2006; 313: 504-507
  CrossrefPubMedGoogle Scholar
• 14 Goodfellow IJ, Pouget-Abadie J, Mirza M. et al Generative adversarial nets. In, Advances in Neural Information Processing Systems. January ed; 2014: 2672–2680
  Google Scholar
• 15 Pan SJ, Yang Q. A survey on transfer learning. In; 2010: 1345–1359
  Google Scholar
• 16 Raghu M, Zhang C, Kleinberg J. et al Transfusion: Understanding transfer learning for medical imaging. In; 2019
  Google Scholar
• 17 Tan C, Sun F, Kong T. et al A survey on deep transfer learning. In, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics): Springer Verlag; 2018: 270–279
  Google Scholar
• 18 Brendan McMahanH, Moore E, Ramage D. et al Communication-efficient learning of deep networks from decentralized data. In; 2017
  Google Scholar
• 19 Li T, Sahu AK, Talwalkar A. et al Federated Learning: Challenges, Methods, and Future Directions. IEEE Signal Processing Magazine 2020; 37: 50-60
  CrossrefGoogle Scholar
• 20 Yang Q, Liu Y, Chen T. et al Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology 2019; 10: 1-19
  CrossrefGoogle Scholar
• 21 Rieke N, Hancox J, Li W. et al The future of digital health with federated learning. npj Digital Medicine 2020; 3: 1-7
  CrossrefPubMedGoogle Scholar
• 22 Lambin P, Leijenaar RTH, Deist TM. et al Radiomics: the bridge between medical imaging and personalized medicine. Nature Reviews Clinical Oncology 2017; 14: 749-762
  CrossrefPubMedGoogle Scholar
• 23 Hammernik K, Klatzer T, Kobler E. et al Learning a variational network for reconstruction of accelerated MRI data. Magnetic Resonance in Medicine 2018; 79: 3055-3071
  CrossrefPubMedGoogle Scholar
• 24 Zhu B, Liu JZ, Cauley SF. et al Image reconstruction by domain-transform manifold learning. Nature 2018; 555: 487-492
  CrossrefPubMedGoogle Scholar
• 25 Sandino CM, Cheng JY, Chen F. et al Compressed Sensing: From Research to Clinical Practice with Deep Neural Networks: Shortening Scan Times for Magnetic Resonance Imaging. IEEE Signal Processing Magazine 2020; 37: 117-127
  CrossrefPubMedGoogle Scholar
• 26 Lustig M, Donoho D, Pauly JM. Sparse MRI: The application of compressed sensing for rapid MR imaging. Magnetic Resonance in Medicine 2007; 58: 1182-1195
  CrossrefPubMedGoogle Scholar
• 27 Knoll F, Hammernik K, Zhang C. et al Deep learning methods for parallel magnetic resonance image reconstruction. arXiv preprint arXiv:190401112 2019, DOI:
  Google Scholar
• 28 Hyun CM, Kim HP, Lee SM. et al Deep learning for undersampled MRI reconstruction. Phys Med Biol 2018; 63: 135007
  CrossrefPubMedGoogle Scholar
• 29 Lin DJ, Johnson PM, Knoll F. et al Artificial Intelligence for MR Image Reconstruction: An Overview for Clinicians. J Magn Reson Imag 2020; DOI: 10.1002/jmri.27078.
  CrossrefPubMedGoogle Scholar
• 30 Küstner T, Fuin N, Hammernik K. et al CINENet: deep learning-based 3D cardiac CINE MRI reconstruction with multi-coil complex-valued 4D spatio-temporal convolutions. Scientific Reports 2020; 10: 1-13
  CrossrefPubMedGoogle Scholar
• 31 Sandino CM, Lai P, Vasanawala SS. et al Accelerating cardiac cine MRI using a deep learning-based ESPIRiT reconstruction. Magnetic Resonance in Medicine 2020, DOI: mrm.28420-mrm.28420
  Google Scholar
• 32 Kaplan S, Zhu YM. Full-Dose PET Image Estimation from Low-Dose PET Image Using Deep Learning: a Pilot Study. J Digit Imaging 2019; 32: 773-778
  CrossrefPubMedGoogle Scholar
• 33 Katsari K, Penna D, Arena V. et al Artificial intelligence for reduced dose 18F-FDG PET examinations: a real-world deployment through a standardized framework and business case assessment. EJNMMI Physics 2021; 8: 25
  CrossrefPubMedGoogle Scholar
• 34 Häggström I, Schmidtlein CR, Campanella G. et al DeepPET: A deep encoder–decoder network for directly solving the PET image reconstruction inverse problem. Medical Image Analysis 2019; 54: 253-262
  CrossrefPubMedGoogle Scholar
• 35 Wang YJ, Baratto L, Hawk KE. et al Artificial intelligence enables whole-body positron emission tomography scans with minimal radiation exposure. Eur J Nucl Med Mol Imaging 2021; DOI: 10.1007/s00259-021-05197-3.
  CrossrefPubMedGoogle Scholar
• 36 Knoll F, Holler M, Koesters T. et al Joint MR-PET Reconstruction Using a Multi-Channel Image Regularizer. IEEE Trans Med Imaging 2017; 36: 1-16
  CrossrefPubMedGoogle Scholar
• 37 Liu F, Jang H, Kijowski R. et al Deep Learning MR Imaging-based Attenuation Correction for PET/MR Imaging. Radiology 2018; 286: 676-684
  CrossrefPubMedGoogle Scholar
• 38 Armanious K, Hepp T, Küstner T. et al Independent attenuation correction of whole body [18F]FDG-PET using a deep learning approach with Generative Adversarial Networks. EJNMMI Research 2020; 10: 53
  CrossrefPubMedGoogle Scholar
• 39 Catana C. Motion correction options in PET/MRI. Semin Nucl Med 2015; 45: 212-223
  CrossrefPubMedGoogle Scholar
• 40 Grimm R, Furst S, Souvatzoglou M. et al Self-gated MRI motion modeling for respiratory motion compensation in integrated PET/MRI. Med Image Anal 2015; 19: 110-120
  CrossrefPubMedGoogle Scholar
• 41 Lamare F, Ledesma Carbayo MJ, Cresson T. et al List-mode-based reconstruction for respiratory motion correction in PET using non-rigid body transformations. Phys Med Biol 2007; 52: 5187-5204
  CrossrefPubMedGoogle Scholar
• 42 Manber R, Thielemans K, Hutton BF. et al Practical PET Respiratory Motion Correction in Clinical PET/MR. J Nucl Med 2015; 56: 890-896
  CrossrefPubMedGoogle Scholar
• 43 Küstner T, Schwartz M, Martirosian P. et al MR-based respiratory and cardiac motion correction for PET imaging. Med Image Anal 2017; 42: 129-144
  CrossrefPubMedGoogle Scholar
• 44 Gratz M, Ruhlmann V, Umutlu L. et al Impact of respiratory motion correction on lesion visibility and quantification in thoracic PET/MR imaging. PLOS ONE 2020; 15: e0233209
  CrossrefPubMedGoogle Scholar
• 45 Marin T, Djebra Y, Han PK. et al Motion correction for PET data using subspace-based real-time MR imaging in simultaneous PET/MR. Phys Med Biol 2020; 65: 235022
  CrossrefPubMedGoogle Scholar
• 46 Kolbitsch C, Prieto C, Tsoumpas C. et al A 3D MR-acquisition scheme for nonrigid bulk motion correction in simultaneous PET-MR. Med Phys 2014; 41: 082304
  CrossrefPubMedGoogle Scholar
• 47 Munoz C, Kolbitsch C, Reader AJ. et al MR-Based Cardiac and Respiratory Motion-Compensation Techniques for PET-MR Imaging. PET Clin 2016; 11: 179-191
  CrossrefPubMedGoogle Scholar
• 48 McClelland JR, Hawkes DJ, Schaeffter T. et al Respiratory motion models: a review. Med Image Anal 2013; 17: 19-42
  CrossrefPubMedGoogle Scholar
• 49 Kamnitsas K, Ledig C, Newcombe VFJ. et al Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Medical Image Analysis 2017; 36: 61-78
  CrossrefPubMedGoogle Scholar
• 50 Ghafoorian M, Karssemeijer N, Heskes T. et al Non-uniform patch sampling with deep convolutional neural networks for white matter hyperintensity segmentation. In, 2016 IEEE 13^th International Symposium on Biomedical Imaging (ISBI); 2016: 1414–1417
  Google Scholar
• 51 Brosch T, Tang LYW, Yoo Y. et al Deep 3D Convolutional Encoder Networks With Shortcuts for Multiscale Feature Integration Applied to Multiple Sclerosis Lesion Segmentation. IEEE Transactions on Medical Imaging 2016; 35: 1229-1239
  CrossrefPubMedGoogle Scholar
• 52 Hesamian MH, Jia W, He X. et al Deep Learning Techniques for Medical Image Segmentation: Achievements and Challenges. JDI 2019; 32: 582-596
  Google Scholar
• 53 Morin O, Vallières M, Jochems A. et al A Deep Look Into the Future of Quantitative Imaging in Oncology: A Statement of Working Principles and Proposal for Change. Int J Radiat Oncol Biol Phys 2018; 102: 1074-1082
  CrossrefPubMedGoogle Scholar
• 54 Parmar C, Barry JD, Hosny A. et al Data Analysis Strategies in Medical Imaging. Clin Cancer Res 2018; 24: 3492-3499
  CrossrefPubMedGoogle Scholar
• 55 Xue Y, Chen S, Qin J. et al Application of Deep Learning in Automated Analysis of Molecular Images in Cancer: A Survey. Contrast Media. Molecular Imaging 2017; 2017: 9512370
  Google Scholar
• 56 Setio AAA, Ciompi F, Litjens G. et al Pulmonary Nodule Detection in CT Images: False Positive Reduction Using Multi-View Convolutional Networks. IEEE Transactions on Medical Imaging 2016; 35: 1160-1169
  CrossrefPubMedGoogle Scholar
• 57 Kawahara J, Hamarneh G. Multi-resolution-Tract CNN with Hybrid Pretrained and Skin-Lesion Trained Layers. Cham: Springer International Publishing; 2016: 164-171
  Google Scholar
• 58 Sibille L, Seifert R, Avramovic N. et al (18)F-FDG PET/CT Uptake Classification in Lymphoma and Lung Cancer by Using Deep Convolutional Neural Networks. Radiology 2020; 294: 445-452
  CrossrefPubMedGoogle Scholar
• 59 Li R, Zhang W, Suk HI. et al Deep learning based imaging data completion for improved brain disease diagnosis. Med Image Comput Comput Assist Interv 2014; 17: 305-312
  PubMedGoogle Scholar
• 60 Anthimopoulos M, Christodoulidis S, Ebner L. et al Lung Pattern Classification for Interstitial Lung Diseases Using a Deep Convolutional Neural Network. IEEE Transactions on Medical Imaging 2016; 35: 1207-1216
  CrossrefPubMedGoogle Scholar
• 61 Fakoor R, Ladhak F, Nazi A. et al Using deep learning to enhance cancer diagnosis and classification. In, Proceedings of the international conference on machine learning: ACM New York, USA; 2013
  Google Scholar
• 62 Codella NC, Nguyen Q-B, Pankanti S. et al Deep learning ensembles for melanoma recognition in dermoscopy images. IBM Journal of Research and Development 2017; 61: 1-5
  CrossrefGoogle Scholar
• 63 Yu L, Chen H, Dou Q. et al Automated Melanoma Recognition in Dermoscopy Images via Very Deep Residual Networks. IEEE Transactions on Medical Imaging 2017; 36: 994-1004
  CrossrefPubMedGoogle Scholar
• 64 Esteva A, Kuprel B, Novoa RA. et al Dermatologist-level classification of skin cancer with deep neural networks. Nature 2017; 542: 115-118
  CrossrefPubMedGoogle Scholar
• 65 Jafari MH, Nasr-Esfahani E, Karimi N. et al Extraction of skin lesions from non-dermoscopic images for surgical excision of melanoma. International Journal of Computer Assisted Radiology and Surgery 2017; 12: 1021-1030
  CrossrefPubMedGoogle Scholar
• 66 Nasr-Esfahani E, Samavi S, Karimi N. et al Melanoma detection by analysis of clinical images using convolutional neural network. In, 2016 38^th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC); 2016: 1373–1376
  Google Scholar
• 67 Cireşan DC, Giusti A, Gambardella LM. et al Mitosis Detection in Breast Cancer Histology Images with Deep Neural Networks. Berlin, Heidelberg: Springer Berlin Heidelberg; 2013: 411-418
  Google Scholar
• 68 Sirinukunwattana K, Raza SEA, Tsang Y. et al Locality Sensitive Deep Learning for Detection and Classification of Nuclei in Routine Colon Cancer Histology Images. IEEE Transactions on Medical Imaging 2016; 35: 1196-1206
  CrossrefPubMedGoogle Scholar
• 69 Wang H, Cruz-Roa A, Basavanhally A. et al Mitosis detection in breast cancer pathology images by combining handcrafted and convolutional neural network features. J Med Imaging (Bellingham) 2014; 1: 034003
  CrossrefPubMedGoogle Scholar
• 70 Cruz-Roa A, Basavanhally A, González F. et al Automatic detection of invasive ductal carcinoma in whole slide images with convolutional neural networks. In, Medical Imaging 2014: Digital Pathology: International Society for Optics and Photonics; 2014: 904103
  Google Scholar
• 71 Roth HR, Lu L, Seff A. et al A new 2.5D representation for lymph node detection using random sets of deep convolutional neural network observations. Medical image computing and computer-assisted intervention: MICCAI International Conference on Medical Image Computing and Computer-Assisted Intervention 2014; 17: 520-527
  PubMedGoogle Scholar
• 72 Wang D, Khosla A, Gargeya R. et al Deep learning for identifying metastatic breast cancer. arXiv preprint arXiv:160605718 2016, DOI:
  Google Scholar
• 73 Shin H-C, Roth HR, Gao M. et al Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE transactions on medical imaging 2016; 35: 1285-1298
  CrossrefPubMedGoogle Scholar
• 74 Barbu A, Lu L, Roth H. et al An analysis of robust cost functions for CNN in computer-aided diagnosis. Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization 2018; 6: 253-258
  Google Scholar
• 75 Roth HR, Lu L, Liu J. et al Improving computer-aided detection using convolutional neural networks and random view aggregation. IEEE transactions on medical imaging 2015; 35: 1170-1181
  CrossrefPubMedGoogle Scholar
• 76 Fourcade A, Khonsari RH. Deep learning in medical image analysis: A third eye for doctors. J Stomatol Oral Maxillofac Surg 2019; 120: 279-288
  CrossrefPubMedGoogle Scholar
• 77 Shen D, Wu G, Suk H-I. Deep learning in medical image analysis. Annual review of biomedical engineering 2017; 19: 221-248
  CrossrefPubMedGoogle Scholar
• 78 Litjens G, Kooi T, Bejnordi BE. et al A survey on deep learning in medical image analysis. Medical image analysis 2017; 42: 60-88
  CrossrefPubMedGoogle Scholar
• 79 Lundervold AS, Lundervold A. An overview of deep learning in medical imaging focusing on MRI. ZMedPhys 2019; 29: 102-127
  Google Scholar
• 80 Hosny A, Parmar C, Quackenbush J. et al Artificial intelligence in radiology. Nature Reviews Cancer 2018; 18: 500-510
  CrossrefPubMedGoogle Scholar
• 81 Dayhoff JE, DeLeo JM. Artificial neural networks: opening the black box. Cancer 2001; 91: 1615-1635
  CrossrefPubMedGoogle Scholar
• 82 Chen KT, Schürer M, Ouyang J. et al Generalization of deep learning models for ultra-low-count amyloid PET/MRI using transfer learning. European Journal of Nuclear Medicine and Molecular Imaging 2020; 47: 2998-3007
  CrossrefPubMedGoogle Scholar
• 83 Schölkopf B. Causality for machine learning. arXiv preprint arXiv:191110500 2019, DOI:
  Google Scholar
• 84 Cypko MA, Stoehr M, Kozniewski M. et al Validation workflow for a clinical Bayesian network model in multidisciplinary decision making in head and neck oncology treatment. Int J Comput Assist Radiol Surg 2017; 12: 1959-1970
  CrossrefPubMedGoogle Scholar
• 85 Lucas PJ, van der Gaag LC, Abu-Hanna A. Bayesian networks in biomedicine and health-care. Artif Intell Med 2004; 30: 201-214
  CrossrefPubMedGoogle Scholar
• 86 Maier-Hein L, Eisenmann M, Reinke A. et al Why rankings of biomedical image analysis competitions should be interpreted with care. Nature Communications 2018; 9: 5217
  CrossrefPubMedGoogle Scholar
• 87 Tomaszewski MR, Gillies RJ. The Biological Meaning of Radiomic Features. Radiology 2021; 298: 505-516
  CrossrefPubMedGoogle Scholar
• 88 Elmore JG, Longton GM, Carney PA. et al Diagnostic concordance among pathologists interpreting breast biopsy specimens. Jama 2015; 313: 1122-1132
  CrossrefPubMedGoogle Scholar
• 89 Elton DC. Self-explaining AI as an alternative to interpretable AI. In, International Conference on Artificial General Intelligence: Springer; 2020: 95–106
  Google Scholar
• 90 Shukla-Dave A, Obuchowski NA, Chenevert TL. et al Quantitative imaging biomarkers alliance (QIBA) recommendations for improved precision of DWI and DCE-MRI derived biomarkers in multicenter oncology trials. J Magn Reson Imaging 2019; 49: e101-e121
  CrossrefPubMedGoogle Scholar
• 91 Clark K, Vendt B, Smith K. et al The Cancer Imaging Archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 2013; 26: 1045-1057
  CrossrefPubMedGoogle Scholar
• 92 Bukowski M, Farkas R, Beyan O. et al Implementation of eHealth and AI integrated diagnostics with multidisciplinary digitized data: are we ready from an international perspective?. European Radiology 2020; 30: 5510-5524
  CrossrefPubMedGoogle Scholar
• 93 Commission E. Ethics Guidelines for Trustworthy AI. In; 2019
  Google Scholar