Skip to main content

Advertisement

SpringerLink
Log in

Gray-to-color image conversion in the classification of breast lesions on ultrasound using pre-trained deep neural networks

  • Original Article
  • Published:
Medical & Biological Engineering & Computing Aims and scope Submit manuscript

Abstract

Breast ultrasound (BUS) image classification in benign and malignant classes is often based on pre-trained convolutional neural networks (CNNs) to cope with small-sized training data. Nevertheless, BUS images are single-channel gray-level images, whereas pre-trained CNNs learned from color images with red, green, and blue (RGB) components. Thus, a gray-to-color conversion method is applied to fit the BUS image to the CNN’s input layer size. This paper evaluates 13 gray-to-color conversion methods proposed in the literature that follow three strategies: replicating the gray-level image to all RGB channels, decomposing the image to enhance inherent information like the lesion’s texture and morphology, and learning a matching layer. Besides, we introduce an image decomposition method based on the lesion’s structural information to describe its inner and outer complexity. These gray-to-color conversion methods are evaluated under the same experimental framework using a pre-trained CNN architecture named ResNet-18 and a BUS dataset with more than 3000 images. In addition, the Matthews correlation coefficient (MCC), sensitivity (SEN), and specificity (SPE) measure the classification performance. The experimental results show that decomposition methods outperform replication and learning-based methods when using information from the lesion’s binary mask (obtained from a segmentation method), reaching an MCC value greater than 0.70 and specificity up to 0.92, although the sensitivity is about 0.80. On the other hand, regarding the proposed method, the trade-off between sensitivity and specificity is better balanced, obtaining about 0.88 for both indices and an MCC of 0.73. This study contributes to the objective assessment of different gray-to-color conversion approaches in classifying breast lesions, revealing that mask-based decomposition methods improve classification performance. Besides, the proposed method based on structural information improves the sensitivity, obtaining more reliable classification results on malignant cases and potentially benefiting clinical practice.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. H. Sung, J. Ferlay, R.L. Siegel, M. Laversanne, I. Soerjomataram, A. Jemal, F. Bray, Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: A Cancer Journal for Clinicians 71(3), 209–249 (2021)

  2. Checka CM, Chun JE, Schnabel FR, Lee J, Toth H (2012) The relationship of mammographic density and age: implications for breast cancer screening. American Journal of Roentgenology 198(3):292–295

    Article  Google Scholar 

  3. Hadadi I, Rae W, Clarke J, McEntee M, Ekpo E (2021) Diagnostic performance of adjunctive imaging modalities compared to mammography alone in women with non-dense and dense breasts: a systematic review and meta-analysis. Clinical Breast Cancer 21(4):278–291

    Article  PubMed  Google Scholar 

  4. Brem RF, Lenihan MJ, Lieberman J, Torrente J (2015) Screening breast ultrasound: past, present, and future. American Journal of Roentgenology 204(2):234–240

    Article  PubMed  Google Scholar 

  5. Bassett LW, Kimme-Smith C (1991) Breast sonography. American Journal of Roentgenology 156(3):449–455

    Article  CAS  PubMed  Google Scholar 

  6. Chen DR, Hsiao YH (2008) Computer-aided diagnosis in breast ultrasound. Journal of Medical Ultrasound 16(1):46–56

    Article  Google Scholar 

  7. D’Orsi C, Sickles E, Mendelson E, Morris E (2013) ACR BI-RADS atlas, breast imaging reporting and data system, 5th edn. American College of Radiology, Reston, VA

    Google Scholar 

  8. Q. Wei, Y.J. Yan, X.R.Y. Ge-Ge Wu and, F. Jiang, J. Liu, G. Wang, Y. Wang, J. Song, Zhi-Ping, C.F. Dietrich, X.W. Cui, The diagnostic performance of ultrasound computer-aided diagnosis system for distinguishing breast masses: a prospective multicenter study. European Radiology 32, 4046–4055 (2022)

  9. Gao Y, Geras KJ, Lewin AA, Moy L (2019) New frontiers: an update on computer-aided diagnosis for breast imaging in the age of artificial intelligence. American Journal of Roentgenology 212(2):300–307

    Article  PubMed  Google Scholar 

  10. Chan HP, Samala RK, Hadjiiski LM (2020) CAD and AI for breast cancer-recent development and challenges. The British Journal of Radiology 93(1108):20190580

    Article  PubMed  Google Scholar 

  11. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436–444

    Article  CAS  PubMed  Google Scholar 

  12. Dong S, Wang P, Abbas K (2021) A survey on deep learning and its applications. Computer Science Review 40:100379

    Article  Google Scholar 

  13. Li Z, Liu F, Yang W, Peng S, Zhou J (2022) A survey of convolutional neural networks: analysis, applications, and prospects. IEEE Transactions on Neural Networks and Learning Systems 33(12):6999–7019

    Article  PubMed  Google Scholar 

  14. Yadav SS, Jadhav SM (2019) Deep convolutional neural network based medical image classification for disease diagnosis. Journal of Big Data 6:113

    Article  Google Scholar 

  15. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC, Fei-Fei L (2015) ImageNet large scale visual recognition challenge. International Journal of Computer Vision 115(3):211–252

    Article  Google Scholar 

  16. Ayana G, Dese K, Choe SW (2021) Transfer learning in breast cancer diagnoses via ultrasound imaging. Cancers 13(4):738

    Article  PubMed  PubMed Central  Google Scholar 

  17. B. Zeimarani, M.G.F. Costa, N.Z. Nurani, S.R. Bianco, W.C. De Albuquerque Pereira, C.F.F.C. Filho, Breast lesion classification in ultrasound images using deep convolutional neural network. IEEE Access 8, 133349–133359 (2020)

  18. Kim S, Park J, Yi J, Kim H (2022) End-to-end convolutional neural network framework for breast ultrasound analysis using multiple parametric images generated from radiofrequency signals. Applied Sciences 12(10):4942

    Article  CAS  Google Scholar 

  19. Kriti J, Virmani R (2020) Agarwal, deep feature extraction and classification of breast ultrasound images. Multimedia Tools and Applications 79:27257–27292

    Article  Google Scholar 

  20. Zhang E, Seiler S, Chen M, Lu W, Gu X (2020) BIRADS features-oriented semi-supervised deep learning for breast ultrasound computer-aided diagnosis. Physics in Medicine & Biology 65(12):125005

    Article  Google Scholar 

  21. Byra M (2021) Breast mass classification with transfer learning based on scaling of deep representations. Biomedical Signal Processing and Control 69:102828

    Article  Google Scholar 

  22. Ç. Cabıoğlu, H. Oğul, Computer-aided breast cancer diagnosis from thermal images using transfer learning, in Bioinformatics and Biomedical Engineering, ed. by I. Rojas, O. Valenzuela, F. Rojas, L.J. Herrera, F. Ortuño (Springer, 2020), pp. 716–726

  23. Moon WK, Lee YW, Ke HH, Lee SH, Huang CS, Chang RF (2020) Computer-aided diagnosis of breast ultrasound images using ensemble learning from convolutional neural networks. Computer Methods and Programs in Biomedicine 190:105361

    Article  PubMed  Google Scholar 

  24. Z. Zhuang, Y. Kang, A.N. Joseph Raj, Y. Yuan, W. Ding, S. Qiu, Breast ultrasound lesion classification based on image decomposition and transfer learning. Medical Physics 47(12), 6257–6269 (2020)

  25. Yap MH, Goyal M, Osman F, Martí R, Denton E, Juette A, Zwiggelaar R (2020) Breast ultrasound region of interest detection and lesion localisation. Artificial Intelligence in Medicine 107:101880

    Article  PubMed  Google Scholar 

  26. Zhuang Z, Yang Z, Raj ANJ, Wei C, Jin P, Zhuang S (2021) Breast ultrasound tumor image classification using image decomposition and fusion based on adaptive multi-model spatial feature fusion. Computer Methods and Programs in Biomedicine 208:106221

    Article  PubMed  Google Scholar 

  27. Huang K, Zhang Y, Cheng H, Xing P, Zhang B (2021) Semantic segmentation of breast ultrasound image with fuzzy deep learning network and breast anatomy constraints. Neurocomputing 450:319–335

    Article  Google Scholar 

  28. Daoud MI, Al-Ali A, Alazrai R, Al-Najar MS, Alsaify BA, Ali MZ, Alouneh S (2022) An edge-based selection method for improving regions-of-interest localizations obtained using multiple deep learning object-detection models in breast ultrasound images. Sensors 22(18):6721

    Article  PubMed  PubMed Central  Google Scholar 

  29. S. Cai, Y. Zhu, J. Zhang, T. Liu, A study on the combination of image preprocessing method based on texture feature and segmentation algorithm for breast ultrasound images, in 2022 2nd International Conference on Consumer Electronics and Computer Engineering (ICCECE) (2022), pp. 760–764

  30. Byra M, Galperin M, Ojeda-Fournier H, Olson L, O’Boyle M, Comstock C, Andre M (2019) Breast mass classification in sonography with transfer learning using a deep convolutional neural network and color conversion. Medical Physics 46(2):746–755

    Article  PubMed  PubMed Central  Google Scholar 

  31. Chen CM, Chou YH, Han KC, Hung GS, Tiu CM, Chiou HJ, Chiou SY (2003) Breast lesions on sonograms: computer-aided diagnosis with nearly setting-independent features and artificial neural networks. Radiology 226(2):504–514

    Article  PubMed  Google Scholar 

  32. Gonzalez RC, Woods RE (2018) Digital image processing, 4th edn. Prentice-Hall Inc, Upper Saddle River, U.S.A

    Google Scholar 

  33. Liao YY, Tsui PH, Li CH, Chang KJ, Kuo WH, Chang CC, Yeh CK (2011) Classification of scattering media within benign and malignant breast tumors based on ultrasound texture-feature-based and Nakagami-parameter images. Medical Physics 38(4):2198–2207

    Article  PubMed  Google Scholar 

  34. A. Telea, J.J.v. Wijk, An augmented fast marching method for computing skeletons and centerlines, in Eurographics / IEEE VGTC Symposium on Visualization, ed. by D. Ebert, P. Brunet, I. Navazo (The Eurographics Association, 2002), pp. 251–259

  35. Yap MH, Pons G, Martí J, Ganau S, Sentís M, Zwiggelaar R, Davison AK, Martí R (2018) Automated breast ultrasound lesions detection using convolutional neural networks. IEEE Journal of Biomedical and Health Informatics 22(4):1218–1226

    Article  PubMed  Google Scholar 

  36. Al-Dhabyani W, Gomaa M, Khaled H, Fahmy A (2020) Dataset of breast ultrasound images. Data in Brief 28:104863

    Article  PubMed  Google Scholar 

  37. Piotrzkowska-Wróblewska H, Dobruch-Sobczak K, Byra M, Nowicki A (2017) Open access database of raw ultrasonic signals acquired from malignant and benign breast lesions. Medical Physics 44(11):6105–6109

    Article  PubMed  Google Scholar 

  38. Gómez-Flores W, Pereira WCA (2020) A comparative study of pre-trained convolutional neural networks for semantic segmentation of breast tumors in ultrasound. Computers in Biology and Medicine 126:104036

    Article  PubMed  Google Scholar 

  39. K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016), pp. 770–778

  40. Masud M, Hossain MS, Alhumyani H, Alshamrani SS, Cheikhrouhou O, Ibrahim S, Muhammad G, Rashed AEE, Gupta BB (2021) Pre-trained convolutional neural networks for breast cancer detection using ultrasound images. ACM Transactions on Internet Technology 21(4):85

    Article  Google Scholar 

  41. J. Deng, W. Dong, R. Socher, L. Li, Kai Li, Li Fei-Fei, ImageNet: a large-scale hierarchical image database, in 2009 IEEE Conference on Computer Vision and Pattern Recognition (2009), pp. 248–255

  42. L. Wan, M. Zeiler, S. Zhang, Y. Le Cun, R. Fergus, Regularization of neural networks using dropconnect, in Proceedings of the 30th International Conference on Machine Learning, vol. 28, ed. by S. Dasgupta, D. McAllester (2013), pp. 1058–1066

  43. Shorten C, Khoshgoftaar TM (2016) A survey on image data augmentation for deep learning. Journal of Big Data 6:60

    Article  Google Scholar 

  44. Geiping J, Somepalli G, Shwartz-Ziv R, Wilson AG, Goldstein T, Goldblum M, How much data is augmentation worth?, in ICML, (2022) Workshop on Spurious Correlations. Invariance and Stability 2022:1–8

  45. Afrin H, Larson NB, Fatemi M, Alizad A (2023) Deep learning in different ultrasound methods for breast cancer, from diagnosis to prognosis: current trends, challenges, and an analysis. Cancers 15(12):3139

    Article  PubMed  PubMed Central  Google Scholar 

  46. Bengio Y (2012) Practical recommendations for gradient-based training of deep architectures. In: Montavon G, Orr GB, Müller KR (eds) Neural networks: tricks of the trade, 2nd edn. Springer, Berlin Heidelberg, Berlin, Heidelberg, pp 437–478

    Chapter  Google Scholar 

  47. Aurelio YS, de Almeida GM, de Castro CL, Braga AP (2019) Learning from imbalanced data sets with weighted cross-entropy function. Neural Processing Letters 50:1937–1949

    Article  Google Scholar 

  48. Chicco D, Jurman G (2020) The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics 21:6

    Article  PubMed  PubMed Central  Google Scholar 

  49. Sokolova M, Lapalme G (2009) A systematic analysis of performance measures for classification tasks. Information Processing & Management 45(4):427–437

    Article  Google Scholar 

  50. V. García, R.A. Mollineda, J.S. Sánchez, Index of balanced accuracy: a performance measure for skewed class distributions, in Pattern Recognition and Image Analysis, ed. by H. Araujo, A.M. Mendonça, A.J. Pinho, M.I. Torres (Springer, 2009), pp. 441–448

  51. Hand DJ, Till RJ (2001) A simple generalisation of the area under the ROC curve for multiple class classification problems. Machine Learning 45(2):171–186

    Article  Google Scholar 

  52. Dietterich TG (1998) Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation 10(7):1895–1923

    Article  CAS  PubMed  Google Scholar 

  53. Holm S (1979) A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6(2):65–70

    Google Scholar 

  54. Liu Z, Yang C, Huang J, Liu S, Zhuo Y, Lu X (2021) Deep learning framework based on integration of S-Mask R-CNN and Inception-v3 for ultrasound image-aided diagnosis of prostate cancer. Future Generation Computer Systems 114:358–367

    Article  Google Scholar 

  55. Wu J, Zeng P, Liu P, Lv G (2022) Automatic classification method of liver ultrasound standard plane images using pre-trained convolutional neural network. Connection Science 34(1):975–989

    Article  Google Scholar 

  56. H. Zhou, Y. Jin, L. Dai, M. Zhang, Y. Qiu, K. wang, J. Tian, J. Zheng, Differential diagnosis of benign and malignant thyroid nodules using deep learning radiomics of thyroid ultrasound images. European Journal of Radiology 127, 108992 (2020)

  57. Hsu ST, Su YJ, Hung CH, Chen MJ, Lu CH, Kuo CE (2022) Automatic ovarian tumors recognition system based on ensemble convolutional neural network with ultrasound imaging. BMC Medical Informatics and Decision Making 22:298

Download references

Author information

Authors and Affiliations

  1. Centro de Investigación y de Estudios Avanzados del IPN, Unidad Tamaulipas, Ciudad Victoria, 87138, Tamaulipas, Mexico

    Wilfrido Gómez-Flores

  2. Federal University of Rio de Janeiro, COPPE/PEB, Rio de Janeiro, 21941-914, RJ, Brazil

    Wagner Coelho de Albuquerque Pereira

Authors
  1. Wilfrido Gómez-Flores
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wagner Coelho de Albuquerque Pereira
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wilfrido Gómez-Flores.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gómez-Flores, W., Pereira, W.C.d.A. Gray-to-color image conversion in the classification of breast lesions on ultrasound using pre-trained deep neural networks. Med Biol Eng Comput 61, 3193–3207 (2023). https://doi.org/10.1007/s11517-023-02928-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11517-023-02928-6

Keywords

Search

Navigation