Skip to main content
SpringerLink
Log in

Survey on deep learning in multimodal medical imaging for cancer detection

  • S.I. : Deep Learning in Multimodal Medical Imaging for Cancer
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

The task of multimodal cancer detection is to determine the locations and categories of lesions by using different imaging techniques, which is one of the key research methods for cancer diagnosis. Recently, deep learning-based object detection has made significant developments due to its strength in semantic feature extraction and nonlinear function fitting. However, multimodal cancer detection remains challenging due to morphological differences in lesions, interpatient variability, difficulty in annotation, and imaging artifacts. In this survey, we mainly investigate over 150 papers in recent years with respect to multimodal cancer detection using deep learning, with a focus on datasets and solutions to various challenges such as data annotation, variance between classes, small-scale lesions, and occlusion. We also provide an overview of the advantages and drawbacks of each approach. Finally, we discuss the current scope of work and provide directions for the future development of multimodal cancer detection.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data availability statements

Data derived from public domain resources.

References

  1. Cai D, Sun X, Zhou N, Han X, Yao J (2019) Efficient mitosis detection in breast cancer histology images by rcnn. In: International symposium on biomedical imaging, pp 919–922

  2. Fatima M, Khan MA, Shaheen S, Almujally NA, Wang S-H (2023) B2c3netf2: breast cancer classification using an end-to-end deep learning feature fusion and satin bowerbird optimization controlled newton raphson feature selection. In: CAAI transactions on intelligence technology

  3. Chaudhury S, Sau K, Khan MA, Shabaz M (2023) Deep transfer learning for idc breast cancer detection using fast ai technique and sqeezenet architecture. Math Biosci Eng 20(6):10404–10427

    Article  Google Scholar 

  4. Polat K, Güneş S (2007) Breast cancer diagnosis using least square support vector machine. Digital Signal Process 17(4):694–701

    Article  Google Scholar 

  5. Akay MF (2009) Support vector machines combined with feature selection for breast cancer diagnosis. Expert Syst Appl 36(2):3240–3247

    Article  Google Scholar 

  6. Podolsky MD, Barchuk AA, Kuznetcov VI, Gusarova NF, Gaidukov VS, Tarakanov SA (2016) Evaluation of machine learning algorithm utilization for lung cancer classification based on gene expression levels. Asian Pac J Cancer Prevent 17(2):835–838

    Article  Google Scholar 

  7. Malik S, Akram T, Awais M, Khan MA, Hadjouni M, Elmannai H, Alasiry A, Marzougui M, Tariq U (2023) An improved skin lesion boundary estimation for enhanced-intensity images using hybrid metaheuristics. Diagnostics 13(7):1285

    Article  Google Scholar 

  8. Jabeen K, Khan MA, Balili J, Alhaisoni M, Almujally NA, Alrashidi H, Tariq U, Cha J-H (2023) Bc2netrf: breast cancer classification from mammogram images using enhanced deep learning features and equilibrium-jaya controlled regula falsi-based features selection. Diagnostics 13(7):1238

    Article  Google Scholar 

  9. Jiao L, Zhang F, Liu F, Yang S, Li L, Feng Z, Qu R (2019) A survey of deep learning-based object detection. IEEE Access 7:128837–128868

    Article  Google Scholar 

  10. Rani S, Singh BK, Koundal D, Athavale VA (2022) Localization of stroke lesion in mri images using object detection techniques: a comprehensive review. In: Neuroscience informatics, p 100070

  11. Hu Z, Tang J, Wang Z, Zhang K, Zhang L, Sun Q (2018) Deep learning for image-based cancer detection and diagnosis—a survey. Pattern Recogn 83:134–149

    Article  Google Scholar 

  12. Rathore S, Hussain M, Ali A, Khan A (2013) A recent survey on colon cancer detection techniques. IEEE/ACM Trans Comput Biol Bioinf 10(3):545–563

    Article  Google Scholar 

  13. Baumgartner M, Jäger PF, Isensee F, Maier-Hein KH (2021) Nndetection: a self-configuring method for medical object detection. In: International conference on medical image computing and computer-assisted intervention, pp 530–539

  14. Saba T (2020) Recent advancement in cancer detection using machine learning: systematic survey of decades, comparisons and challenges. J Infect Public Health 13(9):1274–1289

    Article  Google Scholar 

  15. Khan MA, Akram T, Zhang Y-D, Alhaisoni M, Al Hejaili A, Shaban KA, Tariq U, Zayyan MH (2023) Skinnet-endo: multiclass skin lesion recognition using deep neural network and entropy-normal distribution optimization algorithm with elm. Int J Imaging Syst Technol

  16. Painuli D, Bhardwaj S, et al. (2022) Recent advancement in cancer diagnosis using machine learning and deep learning techniques: a comprehensive review. Comput Biol Med 105580

  17. Allugunti VR (2022) Breast cancer detection based on thermographic images using machine learning and deep learning algorithms. Int J Eng Comput Sci 4(1):49–56

    Article  Google Scholar 

  18. Rasool A, Bunterngchit C, Tiejian L, Islam MR, Qu Q, Jiang Q (2022) Improved machine learning-based predictive models for breast cancer diagnosis. Int J Environ Res Public Health 19(6):3211

    Article  Google Scholar 

  19. Prakash NN, Rajesh V, Namakhwa DL, Pande SD, Ahammad SH (2023) A densenet cnn-based liver lesion prediction and classification for future medical diagnosis. Sci Afr 20:e01629

    Google Scholar 

  20. Jiang H, Diao Z, Shi T, Zhou Y, Wang F, Hu W, Zhu X, Luo S, Tong G, Yao Y-D (2023) A review of deep learning-based multiple-lesion recognition from medical images: classification, detection and segmentation. Comput Biol Med 106726

  21. Anter AM, Abualigah L (2023) Deep federated machine learning-based optimization methods for liver tumor diagnosis: a review. Archiv Comput Methods Eng 1–20

  22. Zhang W, Zhu L, Hallinan J, Zhang S, Makmur A, Cai Q, Ooi BC (2022) Boostmis: boosting medical image semi-supervised learning with adaptive pseudo labeling and informative active annotation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 20666–20676

  23. Zhang S, Li Z, Zhou H-Y, Ma J, Yu Y (2023) Advancing 3d medical image analysis with variable dimension transform based supervised 3d pre-training. Neurocomputing

  24. Guan B, Zhang G, Yao J, Wang X, Wang M (2020) Arm fracture detection in x-rays based on improved deep convolutional neural network. Comput Electr Eng 81:106530

    Article  Google Scholar 

  25. Zhao G, Fang C, Li G, Jiao L, Yu Y (2021) Contralaterally enhanced networks for thoracic disease detection. IEEE Trans Med Imaging 40(9):2428–2438

    Article  Google Scholar 

  26. Xie C, Cao S, Wei D, Zhou H, Ma K, Zhang X, Qian B, Wang L, Zheng Y (2021) Recist-net: lesion detection via grouping keypoints on recist-based annotation. In: International symposium on biomedical imaging, pp 921–924

  27. Agarwal R, Díaz O, Yap MH, Lladó X, Martí R (2020) Deep learning for mass detection in full field digital mammograms. Comput Biol Med 121:103774

    Article  Google Scholar 

  28. Cao H, Liu H, Song E, Ma G, Xu X, Jin R, Liu T, Hung C-C (2020) A two-stage convolutional neural networks for lung nodule detection. IEEE J Biomed Health Inform 24(7):2006–2015

    Google Scholar 

  29. Xu X, Wang C, Guo J, Yang L, Bai H, Li W, Yi Z (2020) Deepln: a framework for automatic lung nodule detection using multi-resolution ct screening images. Knowl Based Syst 189:105128

    Article  Google Scholar 

  30. Jiang C, Wang S, Liang X, Xu H, Xiao N (2020) Elixirnet: relation-aware network architecture adaptation for medical lesion detection. Proc AAAI Conf Artif Intell 34:11093–11100

    Google Scholar 

  31. Wang S, Zhu Y, Lee S, Elton DC, Shen TC, Tang Y, Peng Y, Lu Z, Summers RM (2022) Global-local attention network with multi-task uncertainty loss for abnormal lymph node detection in mr images. Med Image Anal 77:102345

    Article  Google Scholar 

  32. Tang Y-B, Yan K, Tang Y-X, Liu J, Xiao J, Summers RM (2019) Uldor: a universal lesion detector for ct scans with pseudo masks and hard negative example mining. In: International symposium on biomedical imaging, pp 833–836

  33. Li C, Wang X, Liu W, Latecki LJ, Wang B, Huang J (2019) weakly supervised mitosis detection in breast histopathology images using concentric loss. Med Image Anal 53:165–178

    Article  Google Scholar 

  34. Sun Y, Huang X, Molina EGL, Dong L, Zhang Q (2020) Signet ring cells detection in histology images with similarity learning. In: International symposium on biomedical imaging, pp 490–494

  35. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 779–788

  36. Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C-Y, Berg AC (2016) Ssd: Single shot multibox detector. In: European conference on computer vision, pp 21–37

  37. Ren S, He K, Girshick R, Sun J (2015) Faster r-cnn: towards real-time object detection with region proposal networks. In: Advances in neural information processing systems, vol 28

  38. Rezatofighi H, Tsoi N, Gwak J, Sadeghian A, Reid I, Savarese S (2019) Generalized intersection over union: a metric and a loss for bounding box regression. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 658–666

  39. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3431–3440

  40. Lin H, Chen H, Graham S, Dou Q, Rajpoot N, Heng P-A (2019) Fast scannet: fast and dense analysis of multi-gigapixel whole-slide images for cancer metastasis detection. IEEE Trans Med Imaging 38(8):1948–1958

    Article  Google Scholar 

  41. Tao Q, Ge Z, Cai J, Yin J, See S (2019) Improving deep lesion detection using 3d contextual and spatial attention. In: International conference on medical image computing and computer-assisted intervention, pp 185–193

  42. Wu J, Xu H, Zhang S, Li X, Chen J, Zheng J, Gao Y, Tian Y, Liang Y, Ji R (2021) Joint segmentation and detection of covid-19 via a sequential region generation network. Pattern Recogn 118:108006

    Article  Google Scholar 

  43. Zhang D, Chen B, Li S (2021) Sequential conditional reinforcement learning for simultaneous vertebral body detection and segmentation with modeling the spine anatomy. Med Image Anal 67:101861

    Article  Google Scholar 

  44. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 770–778

  45. Xie H, Yang D, Sun N, Chen Z, Zhang Y (2019) Automated pulmonary nodule detection in ct images using deep convolutional neural networks. Pattern Recogn 85:109–119

    Article  Google Scholar 

  46. Zhu Y, Wang Q-C, Xu M-D, Zhang Z, Cheng J, Zhong Y-S, Zhang Y-Q, Chen W-F, Yao L-Q, Zhou P-H et al (2019) Application of convolutional neural network in the diagnosis of the invasion depth of gastric cancer based on conventional endoscopy. Gastrointest Endosc 89(4):806–815

    Article  Google Scholar 

  47. Li B, Li Y, Eliceiri KW (2021) Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 14318–14328

  48. Lotter W, Diab AR, Haslam B, Kim JG, Grisot G, Wu E, Wu K, Onieva JO, Boyer Y, Boxerman JL et al (2021) Robust breast cancer detection in mammography and digital breast tomosynthesis using an annotation-efficient deep learning approach. Nat Med 27(2):244–249

    Article  Google Scholar 

  49. Aly GH, Marey M, El-Sayed SA, Tolba MF (2021) Yolo based breast masses detection and classification in full-field digital mammograms. Comput Methods Programs Biomed 200:105823

    Article  Google Scholar 

  50. Zhang X, Chen F, Yu T, An J, Huang Z, Liu J, Hu W, Wang L, Duan H, Si J (2019) Real-time gastric polyp detection using convolutional neural networks. PLoS ONE 14(3):e0214133

    Article  Google Scholar 

  51. Zlocha M, Dou Q, Glocker B (2019) Improving retinanet for ct lesion detection with dense masks from weak recist labels. In: International conference on medical image computing and computer-assisted intervention, pp 402–410

  52. Sun Y, Huang X, Zhou H, Zhang Q (2021) Srpn: similarity-based region proposal networks for nuclei and cells detection in histology images. Med Image Anal 72:102142

    Article  Google Scholar 

  53. Qiu Y, Wang R, Hou F, Singh S, Ma Z, Jia X (2022) Adversarial multi-task learning with inverse mapping for speech enhancement. Appl Soft Comput 120:108568

    Article  Google Scholar 

  54. Han Z, Huang H, Fan Q, Li Y, Li Y, Chen X (2022) Smd-yolo: an efficient and lightweight detection method for mask wearing status during the covid-19 pandemic. In: Computer methods and programs in biomedicine, p 106888

  55. Ashraf R, Afzal S, Rehman AU, Gul S, Baber J, Bakhtyar M, Mehmood I, Song O-Y, Maqsood M (2020) Region-of-interest based transfer learning assisted framework for skin cancer detection. IEEE Access 8:147858–147871

    Article  Google Scholar 

  56. Zhang S, Li X, Zong M, Zhu X, Wang R (2017) Efficient knn classification with different numbers of nearest neighbors. IEEE Trans Neural Netw Learn Syst 29(5):1774–1785

    Article  MathSciNet  Google Scholar 

  57. Bazai SU, Jang-Jaccard J, Wang R (2017) Anonymizing k-nn classification on mapreduce. In: International conference on mobile networks and management, pp 364–377

  58. Li Z, Dong M, Wen S, Hu X, Zhou P, Zeng Z (2019) Clu-cnns: object detection for medical images. Neurocomputing 350:53–59

    Article  Google Scholar 

  59. Yang S, Zhou X, Wang J, Xie G, Lv C, Gao P, Lv B (2020) Unsupervised domain adaptation for cross-device oct lesion detection via learning adaptive features. In: International symposium on biomedical imaging, pp 1570–1573

  60. Figueira G, Wang Y, Sun L, Zhou H, Zhang Q (2020) “Adversarial-based domain adaptation networks for unsupervised tumour detection in histopathology. In: International symposium on biomedical imaging, pp 1284–1288

  61. Li Y, Zhang L, Chen H, Cheng L (2020) Mass detection in mammograms by bilateral analysis using convolution neural network. Comput Methods Programs Biomed 195:105518

    Article  Google Scholar 

  62. Zhang L, Li Y, Chen H, Cheng L (2019) Mammographic mass detection by bilateral analysis based on convolution neural network. In: International conference on image processing, pp 784–788

  63. Zhou H-Y, Wang C, Li H, Wang G, Zhang S, Li W, Yu Y (2021) Ssmd: semi-supervised medical image detection with adaptive consistency and heterogeneous perturbation. Med Image Anal 72:102117

    Article  Google Scholar 

  64. Li H, Han X, Kang Y, Shi X, Yan M, Tong Z, Bu Q, Cui L, Feng J, Yang L (2020) A novel loss calibration strategy for object detection networks training on sparsely annotated pathological datasets. In: International conference on medical image computing and computer-assisted intervention, pp 320–329

  65. Wang Y, Lu L, Cheng C-T, Jin D, Harrison AP, Xiao J, Liao C-H, Miao S (2019) Weakly supervised universal fracture detection in pelvic x-rays. In: International conference on medical image computing and computer-assisted intervention, pp 459–467

  66. Yang H-H, Wang F-E, Sun C, Huang K-C, Chen H-W, Chen Y, Chen H-C, Liao C-Y, Kao S-H, Wang Y-CF, et al (2021) Leveraging auxiliary information from emr for weakly supervised pulmonary nodule detection. In: International conference on medical image computing and computer-assisted intervention, pp 251–261

  67. Bijoy M, Akondi SM, Abdul Fathaah S, Raut A, Pournami P, Jayaraj P (2022) Cervix type detection using a self-supervision boosted object detection technique. Int J Imaging Syst Technol

  68. Zhou D, Tian Y, Chen W-G, Huang G (2021) Self-supervised saliency estimation for pixel embedding in road detection. IEEE Signal Process Lett 28:1325–1329

    Article  Google Scholar 

  69. Qiu Y, Wang R, Singh S, Ma Z, Hou F (2021) Self-supervised learning based phone-fortified speech enhancement

  70. Li Z, Cui Z, Wang S, Qi Y, Ouyang X, Chen Q, Yang Y, Xue Z, Shen D, Cheng J-Z (2021) Domain generalization for mammography detection via multi-style and multi-view contrastive learning. In: International conference on medical image computing and computer-assisted intervention, pp 98–108

  71. Shou Y, Meng T, Ai W, Xie C, Liu H, Wang Y (2022) Object detection in medical images based on hierarchical transformer and mask mechanism. Comput Intell Neurosci 2022

  72. Cranmer M, Sanchez Gonzalez A, Battaglia P, Xu R, Cranmer K, Spergel D, Ho S (2020) Discovering symbolic models from deep learning with inductive biases. In: Advances in neural information processing systems, vol 33, pp 17429–17442

  73. Mao Y, Wang N, Zhou W, Li H (2021) Joint inductive and transductive learning for video object segmentation. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 9670–9679

  74. Wang D, Zhang Y, Zhang K, Wang L (2020) Focalmix: semi-supervised learning for 3d medical image detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3951–3960

  75. Liu J, Yao J, Bagheri M, Sandfort V, Summers RM (2019) A semi-supervised cnn learning method with pseudo-class labels for atherosclerotic vascular calcification detection. In: International symposium on biomedical imaging, pp 780–783

  76. Li J, Yang S, Huang X, Da Q, Yang X, Hu Z, Duan Q, Wang C, Li H (2019) Signet ring cell detection with a semi-supervised learning framework. In: International conference on information processing in medical imaging, pp 842–854

  77. Zhao Z, Pang F, Liu Z, Ye C (2021) Positive-unlabeled learning for cell detection in histopathology images with incomplete annotations. In: International conference on medical image computing and computer-assisted intervention, pp 509–518

  78. Settles B (2009) Active learning literature survey

  79. Tian Y, Cheng G, Gelernter J, Yu S, Song C, Yang B (2020) Joint temporal context exploitation and active learning for video segmentation. Pattern Recogn 100:107158

    Article  Google Scholar 

  80. Liu J, Cao L, Tian Y (2020) Deep active learning for effective pulmonary nodule detection. In: International conference on medical image computing and computer-assisted intervention, pp 609–618

  81. Guo Z, Zhang R, Li Q, Liu X, Nemoto D, Togashi K, Niroshana SI, Shi Y, Zhu X (2020) Reduce false-positive rate by active learning for automatic polyp detection in colonoscopy videos. In: International symposium on biomedical imaging, pp 1655–1658

  82. Elakkiya R, Subramaniyaswamy V, Vijayakumar V, Mahanti A (2021) Cervical cancer diagnostics healthcare system using hybrid object detection adversarial networks. IEEE J Biomed Health Inform 26(4):1464–1471

    Article  Google Scholar 

  83. Elakkiya R, Teja KSS, Jegatha Deborah L, Bisogni C, Medaglia C (2022) Imaging based cervical cancer diagnostics using small object detection-generative adversarial networks. Multimedia Tools Appl 81(1):191–207

    Article  Google Scholar 

  84. Zhao J, Li D, Xiao X, Accorsi F, Marshall H, Cossetto T, Kim D, McCarthy D, Dawson C, Knezevic S et al (2021) United adversarial learning for liver tumor segmentation and detection of multi-modality non-contrast mri. Med Image Anal 73:102154

    Article  Google Scholar 

  85. Bisla D, Choromanska A, Berman RS, Stein JA, Polsky D (2019) Towards automated melanoma detection with deep learning: data purification and augmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops

  86. Shamsolmoali P, Zareapoor M, Wang R, Jain DK, Yang J (2019) G-ganisr: gradual generative adversarial network for image super resolution. Neurocomputing 366:140–153

    Article  Google Scholar 

  87. Shamsolmoali P, Zareapoor M, Granger E, Zhou H, Wang R, Celebi ME, Yang J (2021) Image synthesis with adversarial networks: a comprehensive survey and case studies. Inf Fusion 72:126–146

    Article  Google Scholar 

  88. Liu L, Muelly M, Deng J, Pfister T, Li L-J (2019) Generative modeling for small-data object detection. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 6073–6081

  89. Han C, Kitamura Y, Kudo A, Ichinose A, Rundo L, Furukawa Y, Umemoto K, Li Y, Nakayama H (2019) Synthesizing diverse lung nodules wherever massively: 3d multi-conditional gan-based ct image augmentation for object detection. In: International conference on 3D vision, pp 729–737

  90. Hammami M, Friboulet D, Kéchichian R (2020) Cycle gan-based data augmentation for multi-organ detection in ct images via yolo. In: International conference on image processing, pp 390–393

  91. Wang Z, Dong N, Rosario SD, Xu M, Xie P, Xing EP (2019) Ellipse detection of optic discand-cup boundary in fundus images. In: International symposium on biomedical imaging, pp 601–604

  92. Han Y, Chen C, Tewfik A, Ding Y, Peng Y (2021) Pneumonia detection on chest x-ray using radiomic features and contrastive learning. In: International symposium on biomedical imaging, pp 247–251

  93. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems, vol 30

  94. Tian Yan, Gelernter Judith, Wang Xun, Li Jianyuan, Yu Yizhou (2019) Traffic sign detection using a multi-scale recurrent attention network. IEEE Trans Intell Transport Syst 20(12):4466–4475. https://doi.org/10.1109/TITS.2018.2886283

    Article  Google Scholar 

  95. Li S, Zhang Z, Lu Y (2020) Efficient detection of emvi in rectal cancer via richer context information and feature fusion. In: International symposium on biomedical imaging, pp 1464–1468

  96. Cao L, Yang J, Rong Z, Li L, Xia B, You C, Lou G, Jiang L, Du C, Meng H et al (2021) A novel attention-guided convolutional network for the detection of abnormal cervical cells in cervical cancer screening. Med Image Anal 73:102197

    Article  Google Scholar 

  97. Shao Q, Gong L, Ma K, Liu H, Zheng Y (2019) Attentive ct lesion detection using deep pyramid inference with multi-scale booster. In: International conference on medical image computing and computer-assisted intervention, pp 301–309

  98. Tian Y, Zhang Y, Zhou D, Cheng G, Chen W-G, Wang R (2020) Triple attention network for video segmentation. Neurocomputing 417:202–211

    Article  Google Scholar 

  99. Yu X, Lou B, Zhang D, Winkel D, Arrahmane N, Diallo M, Meng T, Busch Hv, Grimm R, Kiefer B, et al (2020) Deep attentive panoptic model for prostate cancer detection using biparametric mri scans. In: International conference on medical image computing and computer-assisted intervention, pp 594–604

  100. Shahroudnejad A, Qin X, Balachandran S, Dehghan M, Zonoobi D, Jaremko J, Kapur J, Jagersand M, Noga M, Punithakumar K (2021) Tun-det: a novel network for thyroid ultrasound nodule detection. In: International conference on medical image computing and computer-assisted intervention, pp 656–667

  101. Zhang Z, Li S, Wang Z, Lu Y (2020) A novel and efficient tumor detection framework for pancreatic cancer via ct images. In: International conference of the IEEE engineering in medicine and biology society, pp 1160–1164

  102. Zhang Y, Khan MA, Zhu Z, Wang S (2023) Snelm: squeezenet-guided elm for covid-19 recognition. Comput Syst Sci Eng 46(1):13–26

    Article  Google Scholar 

  103. Liu Y, Ji S (2021) Cleftnet: augmented deep learning for synaptic cleft detection from brain electron microscopy. IEEE Trans Med Imaging 40(12):3507–3518

    Article  Google Scholar 

  104. Luo L, Chen H, Zhou Y, Lin H, Heng P-A (2021) Oxnet: deep omni-supervised thoracic disease detection from chest x-rays. In: International conference on medical image computing and computer-assisted intervention, pp 537–548

  105. Wang Y, Zhang Y, Liu L, Zhong C, Tian J, Zhang Y, Shi Z, He Z (2020) Fgb: feature guidance branch for organ detection in medical images. In: International symposium on biomedical imaging, pp 349–353

  106. Ma J, Li X, Li H, Wang R, Menze B, Zheng W-S (2021) Cross-view relation networks for mammogram mass detection. In: International conference on pattern recognition, pp 8632–8638

  107. Li Z, Zhang S, Zhang J, Huang K, Wang Y, Yu Y (2019) Mvp-net: multi-view fpn with position-aware attention for deep universal lesion detection. In: International conference on medical image computing and computer-assisted intervention, pp 13–21

  108. Tian Y, Cao Y, Wu J, Hu W, Song C, Yang T (2019) Multi-cue combination network for action-based video classification. IET Comput Vis 13(6):542–548

    Article  Google Scholar 

  109. Ren Y, Lu J, Liang Z, Grimm LJ, Kim C, Taylor-Cho M, Yoon S, Marks JR, Lo JY (2021)Retina-match: ipsilateral mammography lesion matching in a single shot detection pipeline. In: International conference on medical image computing and computer-assisted intervention, pp 345–354

  110. Liu Y, Zhou Z, Zhang S, Luo L, Zhang Q, Zhang F, Li X, Wang Y, Yu Y (2019) From unilateral to bilateral learning: Detecting mammogram masses with contrasted bilateral network. In: International conference on medical image computing and computer-assisted intervention, pp 477–485

  111. Liu Y, Zhang F, Zhang Q, Wang S, Wang Y Yu Y (2020) Cross-view correspondence reasoning based on bipartite graph convolutional network for mammogram mass detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3812–3822

  112. Chen J, Zhang Y, Wang J, Zhou X, He Y, Zhang T (2021) Ellipsenet: anchor-free ellipse detection for automatic cardiac biometrics in fetal echocardiography. In: International conference on medical image computing and computer-assisted intervention, pp 218–227

  113. Gräbel P, Özkan Ö, Crysandt M, Herwartz R, Baumann M, Klinkhammer BM, Boor P, Brümmendorf TH, Merhof D (2020) Circular anchors for the detection of hematopoietic cells using retinanet. In: International symposium on biomedical imaging, pp 249–253

  114. Nguyen EH, Yang H, Deng R, Lu Y, Zhu Z, Roland JT, Lu L, Landman BA, Fogo AB, Huo Y (2021) Circle representation for medical object detection. IEEE Trans Med Imaging 41(3):746–754

    Article  Google Scholar 

  115. Luo X, Song T, Wang G, Chen J, Chen Y, Li K, Metaxas DN, Zhang S (2022) Scpm-net: an anchor-free 3d lung nodule detection network using sphere representation and center points matching. Med Image Anal 75:102287

    Article  Google Scholar 

  116. Hu H, Liu A, Zhou Q, Guan Q, Li X, Chen Q (2021) An adaptive learning method of anchor shape priors for biological cells detection and segmentation. Comput Methods Programs Biomed 208:106260

    Article  Google Scholar 

  117. Tian Y, Zhang Y, Chen W-G, Liu D, Wang H, Xu H, Han J, Ge Y (2022) 3d tooth instance segmentation learning objectness and affinity in point cloud. ACM Trans Multimed Comput Commun Appl 18(4):1–16

    Article  Google Scholar 

  118. Xia K, Yin H (2019) Liver detection algorithm based on an improved deep network combined with edge perception. IEEE Access 7:175135–175142

    Article  Google Scholar 

  119. Ouyang X, Che J, Chen Q, Li Z, Zhan Y, Xue Z, Wang Q, Cheng J-Z, Shen D (2021) Self-adversarial learning for detection of clustered microcalcifications in mammograms. In: International conference on medical image computing and computer-assisted intervention, pp 78–87

  120. Xiao L, Zhu C, Liu J, Luo C, Liu P, Zhao Y (2019) “Learning from suspected target: Bootstrapping performance for breast cancer detection in mammography,” in International Conference on Medical Image Computing and Computer-assisted Intervention, pp. 468–476

  121. Yu J, Chen C, Yang X, Wang Y, Yan D, Zhang J, Ni D (2020) Computer-aided tumor diagnosis in automated breast ultrasound using 3d detection network. In: International conference on medical image computing and computer-assisted intervention, pp 181–189

  122. Huang D, Cheng J, Fan R, Su Z, Ma Q Li J (2021) Bone marrow cell recognition: training deep object detection with a new loss function. In: International conference on imaging systems and techniques, pp 1–6

  123. Willoughby J, Sallo F, Zouache M, Cilkova M, Dubis A, Lilaonitkul W (2022) Object detection on medical images with the aid of contrastive gated attention. Investig Ophthalmol Visual Sci 63(7):F0268-2998

    Google Scholar 

  124. Suykens JA, Vandewalle J (1999) Least squares support vector machine classifiers. Neural Process Lett 9(3):293–300

    Article  Google Scholar 

  125. Chen K-B, Xuan Y, Lin A-J, Guo S-H (2021) Esophageal cancer detection based on classification of gastrointestinal ct images using improved faster rcnn. Comput Methods Prog Biomed 207:106172

    Article  Google Scholar 

  126. Harsono IW, Liawatimena S, Cenggoro TW (2020) Lung nodule detection and classification from thorax ct-scan using retinanet with transfer learning. In: Journal of King Saud University-computer and information sciences

  127. Bria A, Marrocco C, Tortorella F (2020) Addressing class imbalance in deep learning for small lesion detection on medical images. Comput Biol Med 120:103735

    Article  Google Scholar 

  128. Lin Y, Su J, Wang X, Li X, Liu J, Cheng K-T, Yang X (2019) Automated pulmonary embolism detection from ctpa images using an end-to-end convolutional neural network. In: International conference on medical image computing and computer-assisted intervention, pp 280–288

  129. Li Y, Fan Y (2020) Deepseed: 3d squeeze-and-excitation encoder-decoder convolutional neural networks for pulmonary nodule detection. In: 2020 IEEE 17th international symposium on biomedical imaging, pp 1866–1869

  130. Wollmann T, Rohr K (2021) Deep consensus network: aggregating predictions to improve object detection in microscopy images. Med Image Anal 70:102019

    Article  Google Scholar 

  131. Chen Z, Wang R, Zhang Z, Wang H, Xu L (2019) Background-foreground interaction for moving object detection in dynamic scenes. Inf Sci 483:65–81

    Article  Google Scholar 

  132. Lin C, Wu H, Wen Z, Qin J (2021) Automated malaria cells detection from blood smears under severe class imbalance via importance-aware balanced group softmax. In: International conference on medical image computing and computer-assisted intervention, pp 455–465

  133. Li H, Han H, Zhou SK (2020) Bounding maps for universal lesion detection. In: International conference on medical image computing and computer-assisted intervention, pp 417–428

  134. Li H, Chen L, Han H, Chi Y, Zhou SK (2021) “Conditional training with bounding map for universal lesion detection,” in International Conference on Medical Image Computing and Computer-assisted Intervention, pp. 141–152

  135. Liu J, Cao L, Akin O, Tian Y (2019) 3dfpn-hs\(^{2}\): 3d feature pyramid network based high sensitivity and specificity pulmonary nodule detection. In: International conference on medical image computing and computer-assisted intervention, pp 513–521

  136. Tian Y, Hu W, Jiang H, Wu J (2019) Densely connected attentional pyramid residual network for human pose estimation. Neurocomputing 347:13–23

    Article  Google Scholar 

  137. Tian Y, Wang X, Wu J, Wang R, Yang B (2019) Multi-scale hierarchical residual network for dense captioning. J Artifi Intell Res 64:181–196

    Article  MathSciNet  Google Scholar 

  138. Zhang F, Luo L, Sun X, Zhou Z, Li X, Yu Y, Wang Y (2019) Cascaded generative and discriminative learning for microcalcification detection in breast mammograms. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 12578–12586

  139. Dass JMA, Kumar SM (2022) A novel approach for small object detection in medical images through deep ensemble convolution neural network. Int J Adv Comput Sci Appl 13(3)

  140. Qadir HA, Shin Y, Solhusvik J, Bergsland J, Aabakken L, Balasingham I (2021) Toward real-time polyp detection using fully cnns for 2d gaussian shapes prediction. Med Image Anal 68:101897

    Article  Google Scholar 

  141. Yu X, Lou B, Shi B, Winkel D, Arrahmane N, Diallo M, Meng T, von Busch H, Grimm R, Kiefer B, et al (2020) False positive reduction using multiscale contextual features for prostate cancer detection in multi-parametric mri scans. In: International symposium on biomedical imaging, pp 1355–1359

  142. Xue L, Yan W, Luo P, Zhang X, Chaikovska T, Liu K, Gao W, Yang K (2021) Detection and localization of hand fractures based on ga_faster r-cnn. Alex Eng J 60(5):4555–4562

    Article  Google Scholar 

  143. Li F, Huang H, Wu Y, Cai C, Huang Y, Ding X (2019) Lung nodule detection with a 3d convnet via iou self-normalization and maxout unit. In: International conference on acoustics, speech and signal processing, pp 1214–1218

  144. Xu Q, Zhong Y, Neumann U (2022) Behind the curtain: learning occluded shapes for 3d object detection. Proc AAAI Conf Artif Intell 36:2893–2901

    Google Scholar 

  145. Nawaz M, Nazir T, Baili J, Khan MA, Kim YJ, Cha J-H (2023) Cxray-effdet: chest disease detection and classification from x-ray images using the efficientdet model. Diagnostics 13(2):248

    Article  Google Scholar 

  146. Chakraborty J, Midya A, Mukhopadhyay S, Rangayyan RM, Sadhu A, Singla V, Khandelwal N (2019) Computer-aided detection of mammographic masses using hybrid region growing controlled by multilevel thresholding. J Medical Biol Eng 39(3):352–366

    Article  Google Scholar 

  147. Wang A, Sun Y, Kortylewski A, Yuille AL (2020) Robust object detection under occlusion with context-aware compositionalnets. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 12645–12654

  148. Xiao L, Luo C, Yu T, Luo Y, Wang M, Yu F, Li Y, Tian C, Qiao J (2020) Deepacev2: automated chromosome enumeration in metaphase cell images using deep convolutional neural networks. IEEE Trans Med Imaging 39(12):3920–3932

    Article  Google Scholar 

  149. Jaeger PF, Kohl SA, Bickelhaupt S, Isensee F, Kuder TA, Schlemmer H-P, Maier-Hein KH (2020) Retina u-net: embarrassingly simple exploitation of segmentation supervision for medical object detection. In: Machine learning for health workshop, pp 171–183

  150. Yan K, Tang Y, Peng Y, Sandfort V, Bagheri M, Lu Z, Summers RM (2019) Mulan: multitask universal lesion analysis network for joint lesion detection, tagging, and segmentation. In: International conference on medical image computing and computer-assisted intervention, pp 194–202

  151. Sanchez-Matilla R, Robu M, Luengo I, Stoyanov D (2021) Scalable joint detection and segmentation of surgical instruments with weak supervision. In: International conference on medical image computing and computer-assisted intervention, pp 501–511

  152. Zhang X, Zhu X, Tang K, Zhao Y, Lu Z, Feng Q (2022) Ddtnet: a dense dual-task network for tumor-infiltrating lymphocyte detection and segmentation in histopathological images of breast cancer. Med Image Anal 78:102415

    Article  Google Scholar 

  153. Zhao X, Xie P, Wang M, Li W, Pickhardt PJ, Xia W, Xiong F, Zhang R, Xie Y, Jian J et al (2020) Deep learning-based fully automated detection and segmentation of lymph nodes on multiparametric-mri for rectal cancer: a multicentre study. EBioMedicine 56:102780

    Article  Google Scholar 

  154. Yu D, Zhang K, Huang L, Zhao B, Zhang X, Guo X, Li M, Gu Z, Fu G, Hu M et al (2020) Detection of peripherally inserted central catheter (picc) in chest x-ray images: a multi-task deep learning model. Comput Methods Prog Biomed 197:105674

    Article  Google Scholar 

  155. Chen Y, Chen J, Xiao B, Wu Z, Chi Y, Xie X, Hua X (2019) Volume r-cnn: unified framework for ct object detection and instance segmentation. In: International symposium on biomedical imaging, pp 872–876

  156. Kale A, Jawade I, Kakade P, Jadhav R, Kulkarni N (2022) Pairnet: a deep learning-based object detection and segmentation system. In: Machine intelligence and smart systems, pp 423–436

  157. Boot T, Irshad H (2020) Diagnostic assessment of deep learning algorithms for detection and segmentation of lesion in mammographic images. In: International conference on medical image computing and computer-assisted intervention, pp 56–65

  158. Zhang W, Li G, Wang F, Yu Y, Lin L, Liang H, et al. (2019) Simultaneous lung field detection and segmentation for pediatric chest radiographs. In: International conference on medical image computing and computer-assisted intervention, pp 594–602

  159. Feng X, Duan L, Chen J (2021) An automated method with anchor-free detection and u-shaped segmentation for nuclei instance segmentation. In: Proceedings of the ACM international conference on multimedia in Asia, pp 1–6

  160. Le N, Yamazaki K, Quach KG, Truong D, Savvides M (2021) A multi-task contextual atrous residual network for brain tumor detection and segmentation. In: International conference on pattern recognition, pp 5943–5950

  161. Cheng G, Cheng J, Luo M, He L, Tian Y, Wang R (2020) Effective and efficient multitask learning for brain tumor segmentation. J Real-Time Image Proc 17(6):1951–1960

    Article  Google Scholar 

  162. Tian Y, Wang H, Wang X (2017) Object localization via evaluation multi-task learning. Neurocomputing 253:34–41

    Article  Google Scholar 

  163. Zheng H, Wang R, Ji W, Zong M, Wong WK, Lai Z, Lv H (2020) Discriminative deep multi-task learning for facial expression recognition. Inf Sci 533:60–71

    Article  Google Scholar 

  164. Chen T, Dai B, Wang R, Liu D (2014) Gaussian-process-based real-time ground segmentation for autonomous land vehicles. J Intell Robot Syst 76(3):563–582

    Article  Google Scholar 

  165. Al-Antari MA, Han S-M, Kim T-S (2020) Evaluation of deep learning detection and classification towards computer-aided diagnosis of breast lesions in digital x-ray mammograms. Comput Methods Prog Biomed 196:105584

    Article  Google Scholar 

  166. Sui D, Zhang K, Liu W, Chen J, Ma X, Tian Z (2021) Cst: a multitask learning framework for colorectal cancer region mining based on transformer. BioMed Res Int 2021

  167. Sainz de Cea MV, Diedrich K, Bakalo R, Ness L, Richmond D (2020) Multi-task learning for detection and classification of cancer in screening mammography. In: International conference on medical image computing and computer-assisted intervention, pp 241–250

  168. Liu Y, Wu Y-H, Ban Y, Wang H, Cheng M-M (2020) Rethinking computer-aided tuberculosis diagnosis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 2646–2655

  169. Tang H, Zhang C, Xie X (2019) Nodulenet: decoupled false positive reduction for pulmonary nodule detection and segmentation. In: International conference on medical image computing and computer-assisted intervention, pp 266–274

  170. Liang Y, Pan C, Sun W, Liu Q, Du Y (2021) Global context-aware cervical cell detection with soft scale anchor matching. Comput Methods Prog Biomed 204:106061

    Article  Google Scholar 

  171. Moon WK, Huang Y-S, Hsu C-H, Chien T-YC, Chang JM, Lee SH, Huang C-S, Chang R-F (2020) Computer-aided tumor detection in automated breast ultrasound using a 3-d convolutional neural network. Comput Methods Prog Biomed 190:105360

    Article  Google Scholar 

  172. Capia ER, Sousa AM, Falcão AX (2020) Improving lung nodule detection with learnable non-maximum suppression. In: International symposium on biomedical imaging, pp 1861–1865

  173. Tang H, Liu X, Xie X (2019) An end-to-end framework for integrated pulmonary nodule detection and false positive reduction. In: 2019 IEEE 16th international symposium on biomedical imaging, pp 859–862

  174. Su R, van der Sluijs M, Cornelissen SA, Lycklama G, Hofmeijer J, Majoie CB, van Doormaal PJ, van Es AC, Ruijters D, Niessen WJ et al (2022) Spatio-temporal deep learning for automatic detection of intracranial vessel perforation in digital subtraction angiography during endovascular thrombectomy. Med Image Anal 77:102377

    Article  Google Scholar 

  175. Zhang Z, Wang Y, Zhang J, Mu X (2019) Comparison of multiple feature extractors on faster rcnn for breast tumor detection. In: International symposium on next generation electronics, pp 1–4

  176. Hou F, Wang R, Zhou Y (2021) Transfer learning for fine-grained entity typing. Knowl Inf Syst 63(4):845–866

    Article  Google Scholar 

  177. Wang X, Cai Z, Gao D, Vasconcelos N (2019) Towards universal object detection by domain attention. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 7289–7298

  178. Ajmal M, Khan MA, Akram T, Alqahtani A, Alhaisoni M, Armghan A, Althubiti SA, Alenezi F (2022) Bf2sknet: best deep learning features fusion-assisted framework for multiclass skin lesion classification. Neural Comput Appl 1–17

  179. Liu X, Guo X, Liu Y, Yuan Y (2021) Consolidated domain adaptive detection and localization framework for cross-device colonoscopic images. Med Image Anal 71:102052

    Article  Google Scholar 

  180. Cho H, Nishimura K, Watanabe K, Bise R (2021) Cell detection in domain shift problem using pseudo-cell-position heatmap. In: International conference on medical image computing and computer-assisted intervention, pp 384–394

  181. Lin T-Y, Goyal P, Girshick R, He K, Dollár P (2017) Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision, pp 2980–2988

Download references

Author information

Author notes

  1. Yan Tian and Zhaocheng Xu have contributed equally to this work.

Authors and Affiliations

  1. School of Computer Science and Technology, Zhejiang Gongshang University, Hangzhou, 310018, China

    Yan Tian & Xuran Zhao

  2. School of Mathematical and Computational Sciences, Massey University, Auckland, 0632, New Zealand

    Zhaocheng Xu & Yujun Ma

  3. School of Information Science and Technology, Nantong University, Nantong, 226019, China

    Weiping Ding

  4. School of Data Science and Artificial Intelligence, Wenzhou University of Technology, Wenzhou, 325000, China

    Ruili Wang

  5. Department of Big Data in Health Science, The First Affiliated Hospital of Wenzhou Medical University, Wenzhou, 325000, China

    Zhihong Gao

  6. Zhejiang Engineering Research Center of Intelligent Medicine, Wenzhou, 325000, China

    Zhihong Gao

  7. Jianpei Technology Co. Ltd, Hangzhou, 311200, China

    Guohua Cheng & Linyang He

Authors
  1. Yan Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhaocheng Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yujun Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Weiping Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ruili Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Zhihong Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Guohua Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Linyang He
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Xuran Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Weiping Ding, Ruili Wang or Zhihong Gao.

Ethics declarations

Conflict of interest

All authors declare that they have no conflicts of interest regarding the publication of this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work was supported in part by the National Natural Science Foundation of China under Grant 61972351 and 62111530300; in part by the Special Project for Basic Business Expenses of Zhejiang Provincial Colleges and Universities under Grant JRK22003 and in part by Zhejiang Engineering Research Center of Intelligent Medicine under Grant 2016E10011.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tian, Y., Xu, Z., Ma, Y. et al. Survey on deep learning in multimodal medical imaging for cancer detection. Neural Comput & Applic (2023). https://doi.org/10.1007/s00521-023-09214-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00521-023-09214-4

Keywords

Search

Navigation