Skip to main content

Advertisement

SpringerLink
Log in

FABnet: feature attention-based network for simultaneous segmentation of microvessels and nerves in routine histology images of oral cancer

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Perineural invasion (PNI), lymphovascular invasion (LVI) and tumor angiogenesis have strong correlation with cancer recurrence, metastasis and poor patient survival. The accurate segmentations of nerves and microvessels can be considered as the preliminary step in objective identification of PNI, LVI and tumor angiogenic analysis in histology images. We proposed a deep network for simultaneous segmentation of microvessel and nerves in routinely used H&E-stained histology images. The network is designed as an encoder–decoder architecture with embedded feature attention blocks and an uncertainty prediction. The proposed network uses Xception residual blocks, followed by atrous spatial pyramid pooling for feature extraction at multiple scales. Feature attention blocks are used in the skip connections from encoder to decoder as well as in the decoder up-sampling, which enables the network in focusing on more salient features while making prediction for segmentation. The method is evaluated using 7780 images of size 512 × 512 pixels, extracted from 20 WSIs of oral squamous cell carcinoma tissue at 20× magnification. The ensemble of network outputs at test time is used to obtain a better segmentation result and simultaneous generation of network prediction uncertainty map. The proposed network achieves state-of-the-art results compared to currently used deep neural networks for semantic segmentation (FCN-8, U-Net, Segnet and DeepLabV3+). The proposed network also gives robust segmentation performance when applied to the full digital whole slide image.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Nishida N, Yano H, Nishida T, Kamura T, Kojiro M (2006) Angiogenesis in cancer. Vasc Health Risk Manag 2(3):213

    Article  Google Scholar 

  2. Liebig C, Ayala G, Wilks JA, Berger DH, Albo D (2009) Perineural invasion in cancer: a review of the literature. Cancer Interdiscip Int J Am Cancer Soc 115(15):3379

    Google Scholar 

  3. Al-Sukhni E, Attwood K, Gabriel EM, LeVea CM, Kanehira K, Nurkin SJ (2017) Lymphovascular and perineural invasion are associated with poor prognostic features and outcomes in colorectal cancer: a retrospective cohort study. Int J Surg 37:42. https://doi.org/10.1016/j.ijsu.2016.08.528

    Article  Google Scholar 

  4. Noma D, Inamura K, Matsuura Y, Hirata Y, Nakajima T, Yamazaki H, Hirai Y, Ichinose J, Nakao M, Ninomiya H, Mun M, Nakagawa K, Masuda M, Ishikawa Y, Okumura S (2018) Prognostic effect of lymphovascular invasion on TNM staging in stage I non-small-cell lung cancer. Clinical Lung Cancer 19(1):e109. https://doi.org/10.1016/j.cllc.2017.06.001

    Article  Google Scholar 

  5. Liebig C, Ayala G, Wilks J, Verstovsek G, Liu H, Agarwal N, Berger DH, Albo D (2009) Perineural invasion is an independent predictor of outcome in colorectal cancer. J Clin Oncol 27(31):5131

    Article  Google Scholar 

  6. Kurtz KA, Hoffman HT, Zimmerman MB, Robinson RA (2005) Perineural and vascular invasion in oral cavity squamous carcinoma: increased incidence on re-review of slides and by using immunohistochemical enhancement. Arch Pathol Lab Med 129(3):354

    Google Scholar 

  7. Leon SP, Folkerth RD, Black PM (1996) Microvessel density is a prognostic indicator for patients with astroglial brain tumors. Cancer Interdiscip Int J Am Cancer Soc 77(2):362

    Google Scholar 

  8. Long J, Shelhamer E, Darrell T. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431–3440

  9. Ronneberger O, Fischer P, Brox T. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 234–241

  10. Badrinarayanan V, Kendall A, Cipolla R (2017) Segnet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans Pattern Anal Mach Intell 39(12):2481

    Article  Google Scholar 

  11. Chen LC, Zhu Y, Papandreou G, Schroff F, Adam H (2018) Computer vision—ECCV. Springer, NewYork, pp 833–851

    Google Scholar 

  12. Janowczyk A, Madabhushi A (2016) Deep learning for digital pathology image analysis: a comprehensive tutorial with selected use cases. J Pathol Inform 7(1):29

    Article  Google Scholar 

  13. Jin Z, Zhou G, Gao D, Zhang Y (2018) EEG classification using sparse Bayesian extreme learning machine for brain–computer interface. Neural Comput Appl. https://doi.org/10.1007/s00521-018-3735-3

    Article  Google Scholar 

  14. Sirinukunwattana K, Raza SEA, Tsang YW, Snead DRJ, Cree IA, Rajpoot NM (2016) Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images. IEEE Trans Med Imaging 35(5):1196

    Article  Google Scholar 

  15. Xu J, Xiang L, Liu Q, Gilmore H, Wu J, Tang J, Madabhushi A (2016) Stacked sparse autoencoder (SSAE) for nuclei detection on breast cancer histopathology images. IEEE Trans Med Imaging 35(1):119

    Article  Google Scholar 

  16. Saha M, Chakraborty C, Racoceanu D (2018) Efficient deep learning model for mitosis detection using breast histopathology images. Comput Med Imaging Graph 64:29

    Article  Google Scholar 

  17. Öztürk Ş, Akdemir B (2019) A convolutional neural network model for semantic segmentation of mitotic events in microscopy images. Neural Comput Appl 31:3719–3728

    Article  Google Scholar 

  18. Saltz J, Gupta R, Hou L, Thorsson V (2018) Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images. Cell Rep. 23(1):181. https://doi.org/10.1016/j.celrep.2018.03.086

    Article  Google Scholar 

  19. Xu J, Luo X, Wang G, Gilmore H, Madabhushi A (2016) Adeep convolutional neural network for segmenting and classifying epithelial and stromal regions in histopathological images. Neurocomputing 191:214

    Article  Google Scholar 

  20. Wang X, Guo Y, Wang Y, Yu J (2019) Automatic breast tumor detection in ABVS images based on convolutional neural network and superpixel patterns. Neural Comput Appl 31(4):1069–1081

    Article  Google Scholar 

  21. Graham S, Chen H, Gamper J, Dou Q, Heng PA, Snead D, Tsang YW, Rajpoot N (2018) MILD-Net: minimal information loss dilated network for gland instance segmentation in colon histology images. Med Image Anal 52:199–211

    Article  Google Scholar 

  22. Chen H, Qi X, Yu L, Dou Q, Qin J, Heng PA (2017) DCAN: deep contour-aware networks for object instance segmentation from histology images. Med Image Anal 36:135. https://doi.org/10.1016/j.media.2016.11.004

    Article  Google Scholar 

  23. Xu Y, Li Y, Wang Y, Liu M, Fan Y, Lai M, Chang EI (2017) Gland instance segmentation using deep multichannel neural networks. IEEE Trans Biomed Eng 64(12):2901. https://doi.org/10.1109/TBME.2017.2686418

    Article  Google Scholar 

  24. Selvanambi R, Natarajan J, Karuppiah M, Islam SH, Hassan MM, Fortino G (2018) Lung cancer prediction using higher-order recurrent neural network based on glowworm swarm optimization. Neural Comput Appl. https://doi.org/10.1007/s00521-018-3824-3

    Article  Google Scholar 

  25. Kather JN, Marx A, Reyes-Aldasoro CC, Schad LR, Zöllner FG, Weis CA (2015) Continuous representation of tumor microvessel density and detection of angiogenic hotspots in histological whole-slide images. Oncotarget 6(22):19163

    Article  Google Scholar 

  26. Yi F, Yang L, Wang S, Guo L, Huang C, Xie Y, Xiao G (2018) Microvessel prediction in H&E stained pathology images using fully convolutional neural networks. BMC Bioinform 19(1):64

    Article  Google Scholar 

  27. Fraz MM, Shaban M, Graham S, Khurram SA, Rajpoot NM (2018) Uncertainty driven pooling network for microvessel segmentation in routine histology images. In: Stoyanov D et al (eds) Computational pathology and ophthalmic medical image analysis. Springer, Cham, pp 156–164

    Chapter  Google Scholar 

  28. Chen LC, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2018) Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFS. IEEE Trans Pattern Anal Mach Intell 40(4):834

    Article  Google Scholar 

  29. Lin G, Milan A, Shen C, Reid I. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 5168–5177. https://doi.org/10.1109/CVPR.2017.549

  30. Romera E, Álvarez JM, Bergasa LM, Arroyo R (2018) ERFNet: efficient residual factorized ConvNet for real-time semantic segmentation. IEEE Trans Intell Transp Syst 19(1):263

    Article  Google Scholar 

  31. Mnih V, Heess N, Graves A et al (2014) Recurrent models of visual attention. In: Advances in neural information processing systems, pp 2204–2212

  32. Xu K, Ba J, Kiros R, Cho K, Courville A, Salakhudinov R, Zemel R, Bengio Y (2015) International conference on machine learning, pp 2048–2057

  33. Du W, Wang Y, Qiao Y (2018) Recurrent spatial-temporal attention network for action recognition in videos. IEEE Trans Image Process 27(3):1347. https://doi.org/10.1109/TIP.2017.2778563

    Article  MathSciNet  MATH  Google Scholar 

  34. Chen L, Yang Y, Wang J, Xu W, Yuille AL. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 3640–3649. https://doi.org/10.1109/CVPR.2016.396

  35. Beck JL, Katafygiotis LS (1998) Updating models and their uncertainties I: bayesian statistical framework. J Eng Mech 124(4):455

    Article  Google Scholar 

  36. Kendall A, Badrinarayanan V, Cipolla R (2015). arXiv preprint arXiv:1511.02680

  37. Gal Y, Ghahramani Z. In: International conference on machine learning, pp 1050–1059

  38. Kendall A, Gal Y. In: Advances in neural information processing systems, pp 5580–5590

  39. Chen LC, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2014). arXiv preprint arXiv:1412.7062

  40. Szegedy C, Ioffe S, Vanhoucke V, Alemi AA. In: AAAI, vol 4, p 12

  41. Xie S, Girshick R, Dollár P, Tu Z, He K. In: IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 5987–5995

  42. Chollet F. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 1800–1807. https://doi.org/10.1109/CVPR.2017.195

  43. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017). arXiv preprint arXiv:1704.04861

  44. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115(3):211

    Article  MathSciNet  Google Scholar 

  45. Goodfellow I, Bengio Y, Courville A, Bengio Y (2016) Deep learning, vol 1. MIT Press, Cambridge

    MATH  Google Scholar 

  46. Bahdanau D, Cho K, Bengio Y (2014). arXiv preprint arXiv:1409.0473

  47. He K, Gkioxari G, Dollár P, Girshick R. In: IEEE international conference on computer vision (ICCV), pp 2980–2988. https://doi.org/10.1109/ICCV.2017.322

  48. BenTaieb A, Hamarneh G. Intravascular imaging and computer assisted stenting, and large-scale annotation of biomedical data and expert label synthesis. Springer, pp 155–163

  49. Graham S, Rajpoot NM. In: IEEE 15th international symposium on biomedical imaging (ISBI 2018), pp 590–594. https://doi.org/10.1109/ISBI.2018.8363645

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Warwick, Coventry, UK

    M. M. Fraz, S. Graham, M. Shaban & N. M. Rajpoot

  2. School of Clinical Dentistry, University of Sheffield, Sheffield, UK

    S. A. Khurram

  3. Shaukat Khanum Memorial Cancer Hospital Research Centre, Lahore, Pakistan

    M. Hassan & A. Loya

  4. National University of Sciences and Technology (NUST), Islamabad, Pakistan

    M. M. Fraz

  5. The Alan Turing Institute, London, UK

    M. M. Fraz & N. M. Rajpoot

Authors
  1. M. M. Fraz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. S. A. Khurram
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. S. Graham
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. M. Shaban
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. M. Hassan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. A. Loya
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. N. M. Rajpoot
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to M. M. Fraz.

Ethics declarations

Conflict of interest

The authors declared that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fraz, M.M., Khurram, S.A., Graham, S. et al. FABnet: feature attention-based network for simultaneous segmentation of microvessels and nerves in routine histology images of oral cancer. Neural Comput & Applic 32, 9915–9928 (2020). https://doi.org/10.1007/s00521-019-04516-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-019-04516-y

Keywords

Search

Navigation