Skip to main content
Log in

A two-stage approach solo_GAN for overlapping cervical cell segmentation based on single-cell identification and boundary generation

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Accurate cell segmentation is a pivotal step throughout the cervical cancer treatment continuum, encompassing early screening, guiding treatment decisions, and assessing long-term prognosis. Currently, in clinical practice, pathologists rely on microscopic examination of cell characteristics followed by manual annotation, leveraging their expert knowledge. Nonetheless, this approach is labor-intensive, time-consuming, and subject to subjectivity. While existing segmentation methods successfully delineate cell clusters, they struggle with single-cell segmentation in overlapping scenarios, often relying on pixel classification methods. This approach tends to produce discontinuous boundaries and incomplete contours. In this paper, we introduce a two-phase framework, Solo_GAN, capable of generating complete boundaries for single cells in overlapping cell clusters and complex backgrounds. In the first phase of our framework, we propose a target detection model based on YOLOv3 for identifying single-cell regions of interest (ROI) in Pap cervical images. In the second stage, the ROIs are fed into a novel network called the dual-domain mapping segmentation network, which is used to generate complete single-cell boundary maps. The process involves conversion between cervical cell images in different states, achieving the generation of single-cell boundaries while retaining the original features of the image. Our method has been extensively evaluated on a Hybrid cervical cell dataset (public and private sets) for its effectiveness. The results demonstrate that our approach consistently outperforms the state-of-the-art methods and proves highly effective. We provide visualizations of the output at each processing stage and compare them with mainstream methodologies. The cell segmentation method proposed in this paper holds considerable significance for clinical research on cells in different stages of cervical cancer.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

Data availability

The data used in this study are available upon request from the corresponding author. Due to privacy and ethical considerations, some data may be subject to restrictions. Requests for data access should be directed to [22115032@bjtu.edu.cn].

References

  1. Schiffman M, Castle PE, Jeronimo J, Rodriguez AC, Wacholder S (2007) Human papillomavirus and cervical cancer. The lancet 370(9590):890–907. https://doi.org/10.1016/S0140-6736(07)61416-0

    Article  Google Scholar 

  2. Buskwofie A, David-West G, Clare CA (2020) A review of cervical cancer: incidence and disparities. J Natl Med Assoc 112(2):229–232. https://doi.org/10.1016/j.jnma.2020.03.002

    Article  Google Scholar 

  3. Arbyn M, Weiderpass E, Bruni L et al (2020) Estimates of incidence and mortality of cervical cancer in 2018: a worldwide analysis. Lancet Glob Health 8(2):e191–e203. https://doi.org/10.1016/S2214-109X(19)30482-6

    Article  Google Scholar 

  4. Guimarães YM, Godoy LR, Longatto-Filho A et al (2022) Management of early-stage cervical cancer: a literature review. Cancers 14(3):575. https://doi.org/10.3390/cancers14030575

    Article  Google Scholar 

  5. Jiang P, Li X, Shen H et al (2023) A Survey on Deep Learning-based Cervical Cytology Screening: from Cell Identification to Whole Slide Image Analysis. Research Square 19(3):665. https://doi.org/10.21203/rs.3.rs-2680912/v1

    Article  Google Scholar 

  6. Liao J, Li X, Gan Y et al (2023) Artificial intelligence assists precision medicine in cancer treatment. Front Oncol 12:998222. https://doi.org/10.3389/fonc.2022.998222

    Article  Google Scholar 

  7. Cao L, Yang J, Rong Z et al (2021) A novel attention-guided convolutional network for the detection of abnormal cervical cells in cervical cancer screening. Med Image Anal 73:102197. https://doi.org/10.1016/j.media.2021.102197

    Article  Google Scholar 

  8. Hou X, Shen G, Zhou L et al (2022) Artificial intelligence in cervical cancer screening and diagnosis. Front Oncol 12:851367. https://doi.org/10.3389/fonc.2022.851367

    Article  Google Scholar 

  9. Conceição T, Braga C, Rosado L et al (2019) A review of computational methods for cervical cells segmentation and abnormality classification. Int J Mol Sci 20(20):5114. https://doi.org/10.3390/ijms20205114

    Article  Google Scholar 

  10. Zhao M, Wang H, Han Y et al (2021) Seens: Nuclei segmentation in pap smear images with selective edge enhancement. Futur Gener Comput Syst 114:185–194. https://doi.org/10.1016/j.future.2020.07.045

    Article  Google Scholar 

  11. Ke J, Jiang Z, Liu C et al (2019) Selective detection and segmentation of cervical cells. Proceedings of the 2019 11th international conference on bioinformatics and biomedical technology 123(5):55–61. https://doi.org/10.1007/s11042-020-09206-9

    Article  Google Scholar 

  12. Martinez-Mas J, Bueno-Crespo A, Martinez-Espana R et al (2020) Classifying Papanicolaou cervical smears through a cell merger approach by deep learning technique. Expert Syst Appl 160:113707. https://doi.org/10.1016/j.eswa.2020.113707

    Article  Google Scholar 

  13. Wang Z et al (2019) Cell segmentation for image cytometry: advances, insufficiencies, and challenges. Cytometry A 95(7):708–711. https://doi.org/10.1002/cyto.a.23686

    Article  Google Scholar 

  14. Devi NL, Thirumurugan P et al (2022) A literature survey of automated detection of cervical cancer cell in Pap smear images. World Review of Science, Technology and Sustainable Development 18(1):74–82. https://doi.org/10.1504/WRSTSD.2022.119330

    Article  Google Scholar 

  15. Lugagne JB, Lin H, Dunlop MJ (2020) DeLTA: Automated cell segmentation, tracking, and lineage reconstruction using deep learning. PLoS Comput Biol 16(4):e1007673

    Article  Google Scholar 

  16. Jiang H, Zhou Y, Lin Y et al (2022) Deep learning for computational cytology: A survey. Med Image Anal 35(11):102691. https://doi.org/10.1016/j.media.2022.102691

    Article  Google Scholar 

  17. Zhao Y, Fu C, Zhang W et al (2022) Automatic Segmentation of Cervical Cells Based on Star-Convex Polygons in Pap Smear Images. Bioengineering 10(1):47. https://doi.org/10.3390/bioengineering10010047

    Article  Google Scholar 

  18. Chen T, Zheng W, Ying H et al (2022) A task decomposing and cell comparing method for cervical lesion cell detection. IEEE Trans Med Imaging 41(9):2432–2442. https://doi.org/10.1109/TMI.2022.3163171

    Article  Google Scholar 

  19. Anaya-Isaza A, Mera-Jiménez L, Zequera-Diaz M (2021) An overview of deep learning in medical imaging. Informatics in medicine unlocked 26:100723

    Article  Google Scholar 

  20. Bohlender S, Oksuz I, Mukhopadhyay A (2021) A survey on shape-constraint deep learning for medical image segmentation. IEEE Rev Biomed Eng 45(10):2421–2564. https://doi.org/10.1109/RBME.2021.3136343

    Article  Google Scholar 

  21. Win KP, Kitjaidure Y, Hamamoto K et al (2020) Computer-assisted screening for cervical cancer using digital image processing of pap smear images. Appl Sci 10(5):1800. https://doi.org/10.3390/app10051800

    Article  Google Scholar 

  22. Arya M, Mittal N, Singh G (2020) Three segmentation techniques to predict the dysplasia in cervical cells in the presence of debris. Multimedia Tools and Applications 79:24157–24172. https://doi.org/10.1007/s11042-020-09206-9

    Article  Google Scholar 

  23. Sarwar A, Sheikh AA, Manhas J et al (2020) Segmentation of cervical cells for automated screening of cervical cancer: a review. Artif Intell Rev 53:2341–2379. https://doi.org/10.1007/s10462-019-09735-2

    Article  Google Scholar 

  24. Song Y, Zhu L, Qin J et al (2019) Segmentation of overlapping cytoplasm in cervical smear images via adaptive shape priors extracted from contour fragment. IEEE Trans Med Imaging 38(12):2849–2862. https://doi.org/10.1109/TMI.2019.2915633

    Article  Google Scholar 

  25. Tareef A, Song Y, Huang H et al (2018) Multi-pass fast watershed for accurate segmentation of overlapping cervical cells. IEEE Trans Med Imaging 37(9):2044–2059. https://doi.org/10.1109/TMI.2018.2815013

    Article  Google Scholar 

  26. Kale A, Aksoy S (2010) Segmentation of cervical cell images. In: 2010 20th international conference on pattern recognition. IEEE 45(6):239–2402

    Chapter  Google Scholar 

  27. Wan T, Xu S, Sang C et al (2019) Accurate segmentation of overlapping cells in cervical cytology with deep convolutional neural networks. Neurocomputing 365:157–170. https://doi.org/10.1016/j.neucom.2019.06.086

    Article  Google Scholar 

  28. Li C, Chen H, Li X et al (2020) A review for cervical histopathology image analysis using machine vision approaches. Artif Intell Rev 53:4821–4862. https://doi.org/10.1007/s10462-020-09808-7

    Article  Google Scholar 

  29. Alyafeai Z, Ghouti L et al (2020) A fully-automated deep learning pipeline for cervical cancer classification. Expert Syst Appl 141:112951. https://doi.org/10.1016/j.eswa.2019.112951

    Article  Google Scholar 

  30. Song Y, Tan EL, Jiang X et al (2016) Accurate cervical cell segmentation from overlapping clumps in pap smear images. IEEE Trans Med Imaging 36(1):288–300. https://doi.org/10.1109/TMI.2019.2913056

    Article  Google Scholar 

  31. Allehaibi KHS, Nugroho LE, Lazuardi L et al (2019) Segmentation and classification of cervical cells using deep learning. IEEE Access 7:116925–116941. https://doi.org/10.1109/ACCESS.2019.2936017

    Article  Google Scholar 

  32. Zhao Y, Fu C, Xu S et al (2022) LFANet: Lightweight feature attention network for abnormal cell segmentation in cervical cytology images. Comput Biol Med 145:105500. https://doi.org/10.1016/j.compbiomed.2022.105500

    Article  Google Scholar 

  33. Wang CW, Liou YA, Lin YJ et al (2021) Artificial intelligence-assisted fast screening cervical high grade squamous intraepithelial lesion and squamous cell carcinoma diagnosis and treatment planning. Sci Rep 11(1):16244. https://doi.org/10.1038/s41598-021-95545-y

    Article  Google Scholar 

  34. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. Springer International Publishing 12(8):234–241. https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  35. Khadangi A, Boudier T, Rajagopal V. (2021) EM-net: Deep learning for electron microscopy image segmentation. In: 2020 25th international conference on pattern recognition (ICPR). IEEE 45(2):31–38. https://doi.org/10.1109/ICPR48806.2021.9413098

    Chapter  Google Scholar 

  36. Ting G, Weixing W, Wei L et al (2017) Rock particle image segmentation based on improved normalized cut. International Journal of Control and Automation 10(4):271–286

    Article  Google Scholar 

  37. Xun S, Li D, Zhu H et al (2022) Generative adversarial networks in medical image segmentation: A review. Comput Biol Med 140:105063. https://doi.org/10.1016/j.compbiomed.2021.105063

    Article  Google Scholar 

  38. Pathak D, Krahenbuhl P, Donahue J et al (2016) Context encoders: Feature learning by inpainting. InProceedings of the IEE CVF Conference on Computer Vision and Pattern Recognition 2(3):4

    Google Scholar 

  39. Zhu JY, Park T, Isola P et al (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. Proc IEEE Int Conf Comput Vis 98(1):2223–2232

    Google Scholar 

  40. Ledig C, Theis L, Huszár F et al (2017) Photo-realistic single image super-resolution using a generative adversarial network. Proc IEEE Conf Comput Vis Pattern Recognit 11(13):4681–4690

    Google Scholar 

  41. Reed S, Akata Z, Yan X et al (2016) Generative adversarial text to image synthesis. In: International conference on machine learning, vol 29. PMLR, pp 1060–1069

    Google Scholar 

  42. Huang J, Yang G, Li B et al (2021) Segmentation of cervical cell images based on generative adversarial networks. IEEE Access 9:115415–115428. https://doi.org/10.1109/ACCESS.2021.3104609

    Article  Google Scholar 

  43. Elemento O, Leslie C, Lundin J et al (2021) Artificial intelligence in cancer research, diagnosis and therapy. Nat Rev Cancer 21(12):747–752. https://doi.org/10.1038/s41568-021-00399-1

    Article  Google Scholar 

  44. Song Y, Zhang L, Chen S et al (2014) A deep learning based framework for accurate segmentation of cervical cytoplasm and nuclei. In: 2014 36th annual international conference of the IEEE engineering in medicine and biology society. IEEE 66(12):2903–2906. https://doi.org/10.1109/EMBC.2014.6944230

    Chapter  Google Scholar 

  45. Araújo FHD, Silva RRV, Ushizima DM et al (2019) Deep learning for cell image segmentation and ranking. Comput Med Imaging Graph 72:13–21. https://doi.org/10.1016/j.compmedimag.2019.01.003

    Article  Google Scholar 

  46. Chen L, Shen C, Li S et al (2018) Automatic PET cervical tumor segmentation by deep learning with prior information. Medical Imaging 2018: Image Processing. SPIE 10574:834–839. https://doi.org/10.1117/12.2293926

    Article  Google Scholar 

  47. Yu S, Feng X, Wang B, Dun H, Zhang S, Zhang R, Huang X (2021) Automatic classification of cervical cells using deep learning method. IEEE Access 9:32559–32568

    Article  Google Scholar 

  48. SheelaShiney TS, Rose RJ (2023) Deep auto encoder based extreme learning system for automatic segmentation of cervical cells. IETE J Res 69(7):4066–4086. https://doi.org/10.1080/03772063.2021.1958075

    Article  Google Scholar 

  49. Cheng S, Liu S, Yu J et al (2021) Robust whole slide image analysis for cervical cancer screening using deep learning. Nat Commun 12(1):5639

    Article  Google Scholar 

  50. Han Z, Wei B, Mercado A et al (2018) Spine-GAN: Semantic segmentation of multiple spinal structures. Med Image Anal 50:23–35. https://doi.org/10.1016/j.media.2018.08.005

    Article  Google Scholar 

  51. AlexKP VMS, Chennamsetty SS et al (2017) Generative adversarial networks for brain lesion detection. Medical Imaging 2017: Image Processing. SPIE 10133:113–121. https://doi.org/10.1117/12.2254487

    Article  Google Scholar 

  52. Quan TM, Nguyen-Duc T, Jeong WK (2018) Compressed sensing MRI reconstruction using a generative adversarial network with a cyclic loss. IEEE Trans Med Imaging 37(6):1488–1497. https://doi.org/10.1109/TMI.2018.2820120

    Article  Google Scholar 

  53. Mahmood F, Chen R, Durr NJ (2018) Unsupervised reverse domain adaptation for synthetic medical images via adversarial training. IEEE Trans Med Imaging 37(12):2572–2581. https://doi.org/10.1109/TMI.2018.2842767

    Article  Google Scholar 

  54. Sandfort V, Yan K, Pickhardt PJ et al (2019) Data augmentation using generative adversarial networks (CycleGAN) to improve generalizability in CT segmentation tasks. Sci Rep 9(1):16884

    Article  Google Scholar 

  55. Elakkiya R, Subramaniyaswamy V, Vijayakumar V et al (2021) Cervical cancer diagnostics healthcare system using hybrid object detection adversarial networks. IEEE J Biomed Health Inform 26(4):1464–1471. https://doi.org/10.1109/JBHI.2021.3094311

    Article  Google Scholar 

  56. Chen S, Gao D, Wang L et al (2020) Cervical cancer single cell image data augmentation using residual condition generative adversarial networks. In: 2020 3rd international conference on artificial intelligence and big data (ICAIBD). IEEE 23(12):237–241. https://doi.org/10.1109/ICAIBD49809.2020.9137494

    Chapter  Google Scholar 

  57. Ganesan P, Xue Z, Singh S et al (2019) Performance evaluation of a generative adversarial network for deblurring mobile-phone cervical images. In: The 41st annual international conference of the IEEE engineering in medicine and biology society (EMBC). IEEE 41(3):4487–4490. https://doi.org/10.1109/EMBC.2019.8857124

    Chapter  Google Scholar 

  58. Zhang H, Goodfellow I, Metaxas D et al (2019) Self-attention generative adversarial networks. In: International conference on machine learning. PMLR 24(12):7354–7363. https://proceedings.mlr.press/v97/zhang19d.html

    Google Scholar 

  59. Bnouni N, Rekik I, Rhim MS et al (2020) Context-aware synergetic multiplex network for multi-organ segmentation of cervical cancer MRI. International workshop on predictive intelligence in medicine. Cham: Springer International Publishing 22(12):1–11. https://doi.org/10.1007/978-3-030-59354-4_1

    Chapter  Google Scholar 

  60. Almahairi A, Rajeshwar S, Sordoni A et al (2018) Augmented cyclegan: Learning many-to-many mappings from unpaired data. In: International conference on machine learning. PMLR 6(7):195–204. https://proceedings.mlr.press/v80/almahairi18a.html

    Google Scholar 

  61. Jia D, He Z, Zhang C et al (2022) Detection of cervical cancer cells in complex situation based on improved YOLOv3 network. Multimedia Tools and Applications 81(6):8939–8961. https://doi.org/10.1007/s11042-022-11954-9

    Article  Google Scholar 

  62. Long F (2020) Microscopy cell nuclei segmentation with enhanced U-Net. BMC Bioinformatics 21:1–12. https://doi.org/10.1186/s12859-019-3332

    Article  Google Scholar 

  63. Commowick O, Cervenansky F, Cotton F et al (2021) MSSEG-2 challenge proceedings: Multiple sclerosis new lesions segmentation challenge using a data management and processing infrastructure. MICCAI 2021-24th international conference on medical image computing and computer assisted intervention 12(68):120–126. https://inria.hal.science/hal-03358968v3

    Google Scholar 

  64. Wang Q, Zhou X, Wang C et al (2019) WGAN-based synthetic minority over-sampling technique: improving semantic fine-grained classification for lung nodules in CT images. IEEE Access 7:18450–18463. https://doi.org/10.1109/ACCESS.2019.2896409

    Article  Google Scholar 

  65. Jia D, Zhou J, Zhang C (2022) Detection of cervical cells based on improved SSD network. Multimedia Tools and Applications 81(10):13371–13387. https://doi.org/10.1007/s11042-021-11015-7

    Article  Google Scholar 

  66. Shi C, Pan Q, Rehman M (2022) Cervical cancer cell image detection method based on improved YOLOv4. IEEE 2022 7th international conference on intelligent computing and signal processing (ICSP) 78(46):1996–2000. https://doi.org/10.1109/ICSP54964.2022.9778577

    Article  Google Scholar 

  67. Wu N, Jia D, Zhang C et al (2023) Cervical cell extraction network based on optimized yolo. Math Biosci Eng 20(2):2364–2381. https://doi.org/10.3934/mbe.2023111

    Article  Google Scholar 

  68. Kaldera H, Gunasekara SR, Dissanayake MB (2019) Brain tumor classification and segmentation using faster R-CNN. IEEE 2019 advances in science and engineering technology international conferences (ASET) 45(12):1–6. https://doi.org/10.1109/ICASET.2019.8714263

    Article  Google Scholar 

  69. Chen J, Li P, Xu T et al (2022) Detection of cervical lesions in colposcopic images based on the RetinaNet method. Biomed Signal Process Control 75:103589. https://doi.org/10.1016/j.bspc.2022.103589

    Article  Google Scholar 

  70. Meng Z, Zhao Z, Su F et al (2021) Hierarchical spatial pyramid network for cervical precancerous segmentation by reconstructing deep segmentation networks. Proc IEEE/CVF Conf Comput Vis Pattern Recognit 75(16):3738–3745. https://doi.org/10.1109/CVPRW53098.2021.00414

    Article  Google Scholar 

  71. Jia AD, Li BZ, Zhang CC (2020) Detection of cervical cancer cells based on strong feature CNN-SVM network. Neurocomputing 411:112–127. https://doi.org/10.1016/j.neucom.2020.06.006

    Article  Google Scholar 

  72. Su J, Xu X, He Y et al (2016) (2016) Automatic detection of cervical cancer cells by a two-level cascade classification system. Anal Cell Pathol 30(5):3245–3605. https://doi.org/10.1155/2016/9535027

    Article  Google Scholar 

  73. Li X, Xu Z, Shen X et al (2021) Detection of cervical cancer cells in whole slide images using deformable and global context aware faster RCNN-FPN. Curr Oncol 28(5):3585–3601. https://doi.org/10.3390/curroncol28050307

    Article  Google Scholar 

  74. Hussain E, Mahanta LB, Das CR et al (2020) A shape context fully convolutional neural network for segmentation and classification of cervical nuclei in Pap smear images. Artif Intell Med 107:101897. https://doi.org/10.1016/j.artmed.2020.101897

    Article  Google Scholar 

  75. Chen J, Zhang B (2021) Segmentation of overlapping cervical cells with mask region convolutional neural network. Comput Math Methods Med 105:52106. https://doi.org/10.1155/2021/3890988

    Article  Google Scholar 

  76. El Jurdi R, Petitjean C, Honeine P et al (2021) High-level prior-based loss functions for medical image segmentation: A survey. Comput Vis Image Underst 210:103248. https://doi.org/10.1016/j.cviu.2021.103248

    Article  Google Scholar 

  77. Chaki J, Woźniak M (2023) A deep learning based four-fold approach to classify brain MRI: BTSCNet. Biomed Signal Process Control 85:104902. https://doi.org/10.1016/j.bspc.2023.104902

    Article  Google Scholar 

  78. Chaki J, Woźniak M (2023) Deep learning for neurodegenerative disorder (2016 to 2022): A systematic review. Biomed Signal Process Control 80:104223. https://doi.org/10.1016/j.bspc.2022.104223

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Contributions

Dongyao Jia and Zihao He conceived and designed the study. Chuanwang Zhang and Ziqi Li. conducted data collection and analysis. Zihao He drafted the manuscript. Dongyao Jia critically reviewed and revised the manuscript. All authors approved the final version for publication.

Corresponding author

Correspondence to Dongyao Jia.

Ethics declarations

Competing interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Ethical considerations and informed consent

This study was conducted in accordance with the ethical guidelines and regulations set forth by Guangdong Provincial People’s Hospital Ethics Committee. Prior to data collection, written informed consent was obtained from all participants involved in the study.

The data used in this study were de-identified and stored securely to maintain confidentiality. Only authorized researchers had access to the data, and all data handling and analysis procedures complied with applicable privacy laws and regulations.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The project code URL is: (https://gitee.com/affreden/solo_GAN).

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

He, Z., Jia, D., Zhang, C. et al. A two-stage approach solo_GAN for overlapping cervical cell segmentation based on single-cell identification and boundary generation. Appl Intell 54, 4621–4645 (2024). https://doi.org/10.1007/s10489-024-05378-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-024-05378-1

Keywords

Navigation