Skip to main content
SpringerLink
Log in

Multitask Learning with Multiscale Residual Attention for Brain Tumor Segmentation and Classification

  • Research Article
  • Published:
Machine Intelligence Research Aims and scope Submit manuscript

Abstract

Automatic segmentation and classification of brain tumors are of great importance to clinical treatment. However, they are challenging due to the varied and small morphology of the tumors. In this paper, we propose a multitask multiscale residual attention network (MMRAN) to simultaneously solve the problem of accurately segmenting and classifying brain tumors. The proposed MMRAN is based on U-Net, and a parallel branch is added at the end of the encoder as the classification network. First, we propose a novel multiscale residual attention module (MRAM) that can aggregate contextual features and combine channel attention and spatial attention better and add it to the shared parameter layer of MMRAN. Second, we propose a method of dynamic weight training that can improve model performance while minimizing the need for multiple experiments to determine the optimal weights for each task. Finally, prior knowledge of brain tumors is added to the postprocessing of segmented images to further improve the segmentation accuracy. We evaluated MMRAN on a brain tumor data set containing meningioma, glioma, and pituitary tumors. In terms of segmentation performance, our method achieves Dice, Hausdorff distance (HD), mean intersection over union (MIoU), and mean pixel accuracy (MPA) values of 80.03%, 6.649 mm, 84.38%, and 89.41%, respectively. In terms of classification performance, our method achieves accuracy, recall, precision, and F1-score of 89.87%, 90.44%, 88.56%, and 89.49%, respectively. Compared with other networks, MMRAN performs better in segmentation and classification, which significantly aids medical professionals in brain tumor management. The code and data set are available at https://github.com/linkenfaqiu/MMRAN.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. J. R. McFaline-Figueroa, E. Q. Lee. Brain tumors. The American journal of medicine, vol. 131, no. 8, pp. 874–882, 2018. DOI: https://doi.org/10.1016/j.amjmed.2017.12.039.

    Article  Google Scholar 

  2. C. Chen, Y. Hu, L. Lyu, S. Yin, Y. Yu, S. Jiang, P. Zhou. Incidence, demographics survival of patients with primary pituitary tumors: A SEER database study in 2004–2016. Scientific Reports, vol. 11, no. 1, pp. 1–9, 2021. DOI: https://doi.org/10.1038/s41598-020-79139-8.

    Google Scholar 

  3. S. Bakas, H. Akbari, A. Sotiras, M. Bilello, M. Rozycki, J. S. Kirby, J. B. Freymann, K. Farahani, C. Davatzikos. Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Scientific Data, vol. 4, no. 1, pp. 1–13, 2017.

    Article  Google Scholar 

  4. L. Hou, D. Samaras, T. M. Kurc, Y. Gao, J. E. Davis J. H. Saltz. Patch-based convolutional neural network for whole slide tissue image classification. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, pp. 2424–2433, 2016.

  5. K. He, X. Zhang, S. Ren J. Sun. Deep residual learning for image recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, pp. 770–778, 2016.

  6. G. Huang, Z. Liu, L. Van Der Maaten K. Q. Weinberger. Densely connected convolutional networks. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Hawaii, USA, pp. 4700–4708, 2017.

  7. J. Long, E. Shelhamer T. Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA, pp. 3431–3440, 2015.

  8. O. Ronneberger, P. Fischer, T. Brox. U-Net: Convolutional networks for biomedical image segmentation. In Proceedings of International Conference on Medical Image Computing and Computer-assisted Intervention, Springer, Munich, Germany, pp. 234–241, 2015.

  9. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville Y. Bengio. Generative adversarial nets. Advances in Neural Information Processing Systems, Montreal, Canada, Article number 27, 2014.

  10. S. Ruder. An overview of multi-task learning in deep neural networks, [Online], Available: https://arxiv.org/abs/1706.05098, 2017.

  11. S. Chaudhari, V. Mithal, G. Polatkan, R. Ramanath. An attentive survey of attention models. ACM Transactions on Intelligent Systems and Technology, vol. 12, pp. 1–32, 2021.

    Article  Google Scholar 

  12. S. Vandenhende, S. Georgoulis, W. Van Gansbeke, M. Proesmans, D. Dai L. Van Gool. Multi-task learning for dense prediction tasks: A survey, [Online], Available: https://arxiv.org/abs/2004.13379, 2021.

  13. X. Chen, B. M. Williams, S. R. Vallabhaneni, G. Czanner, R. Williams, Y. Zheng. Learning active contour models for medical image segmentation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Los Angeles, USA, pp. 11632–11640, 2019.

  14. M. H. Guo, T. X. Xu, J. J. Liu. Attention mechanisms in computer vision: A survey. Computational Visual Media, vol. 8, pp. 331–368, 2022. DOI: https://doi.org/10.1007/s41095-022-0271-y.

    Article  Google Scholar 

  15. J. Hu, L. S. Sun. Squeeze-and-excitation networks. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, pp. 7132–7141, 2018.

  16. A. G. Roy, N. Navab, C. Wachinger. Concurrent spatial and channel “squeeze & excitation” in fully convolutional networks. In Proceedings of International Conference on Medical Image Computing and Computer-assisted Intervention, Springer, Granada, Spain, pp. 421–429, 2018.

  17. S. Woo, J. Park, J. Y. Lee, I. S. Kweon. Cbam: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision, Springer, Munich, Germany, pp. 3–19, 2018.

  18. F. Wang, M. Jiang, C. Qian, S. Yang, C. Li, H. Zhang, X. Wang X. Tang. Residual attention network for image classification. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, pp. 3156–3164, 2017.

  19. C. Z. Wu, J. Sun, J. Wang, L. F. Xu, S. Zhan. Encoding-decoding network with pyramid self-attention module for retinal vessel segmentation. International Journal of Automation and Computing, vol. 18, no. 6, pp. 973–980, 2021. DOI: https://doi.org/10.1007/s11633-020-1277-0.

    Article  Google Scholar 

  20. S. Mehta, E. Mercan, J. Bartlett, D. Weaver, J. G. Elmore, L. Shapiro. Y-Net: Joint segmentation and classification for diagnosis of breast biopsy images. In Proceedings of International Conference on Medical Image Computing and Computer-assisted Intervention, Springer, Granada, Spain, pp. 893–901, 2018.

  21. E. Z. Chen, X. Dong, X. Li, H. Jiang, R. Rong J. Wu. Lesion attributes segmentation for melanoma detection with multi-task U-Net. In Proceedings of the 16th IEEE International Symposium on Biomedical Imaging, Venezia, Italy, pp. 485–488, 2019.

  22. T. He, J. Hu, Y. Song, J. Guo Z. Yi. Multi-task learning for the segmentation of organs at risk with label dependence. Medical Image Analysis, vol. 61, Article number 101666, 2020. DOI: https://doi.org/10.1016/j.media.2020.101666.

  23. K. He, X. Zhang, S. Ren, J. Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, pp. 1904–1916, 2015. DOI: https://doi.org/10.1109/TPAMI.2015.2389824.

    Article  Google Scholar 

  24. T. Y. Lin, P. Goyal, R. Girshick, K. He P. Dollar. Focal loss for dense object detection. In Proceedings of IEEE International Conference on Computer Vision, Venezia, Italy, pp. 2980–2988, 2017.

  25. J Božič, D. Tabernik, D Skočaj. End-to-end training of a two-stage neural network for defect detection. In Proceedings of the 25th International Conference on Pattern Recognition, IEEE, Milan, Italy, pp. 5619–5626, 2021.

  26. Q. T. Ostrom, M. Adel Fahmideh, D. J. Cote, I. S. Muskens, J. M. Schraw, M. E. Scheurer, M. L. Bondy. Risk factors for childhood and adult primary brain tumors. Neuro-oncology, vol. 21, pp. 1357–1375, 2019. DOI: https://doi.org/10.1093/neuonc/noz123.

    Article  Google Scholar 

  27. C. H. Wu, Y. J. Liao, T. Y. Lin, Y. C. Chen, S. S. Sun, Y. W. H. Liu, S. M. Hsu. A volume-equivalent spherical necrosis-tumor-normal liver model for estimating absorbed dose in yttrium-90 microsphere therapy. Medical Physics, vol. 43, pp. 6082–6088, 2016. DOI: https://doi.org/10.1118/1.4965044.

    Article  Google Scholar 

  28. Yang W, Feng Q J, Yu M. Content-based retrieval of brain tumor in contrast-enhanced MRI images using tumor margin information and learned distance metric. Medical physics, vol. 39, no. 11, pp. 6929–6942, 2012.

    Article  Google Scholar 

  29. X. Xiao, S. Lian, Z. Luo, S. Li. Weighted Res-UNet for high-quality retina vessel segmentation. In Proceedings of the 9th International Conference on Information Technology in Medicine and Education. IEEE, Hangzhou, China, pp. 327–331, 2018.

    Google Scholar 

  30. S. A. Kamran, A. Sharif, A. Tavakkoli, S. L. Zuckerbrod, K. M. Sanders, S. A. Baker. RV-GAN: Segmenting retinal vascular structure in fundus photographs using a novel multi-scale generative adversarial network. In Proceedings of International Conference on Medical Image Computing and Computer-assisted Intervention, Springer, Strasbourg, France, pp. 34–44, 2021.

  31. S. K. Datta, M. A. Shaikh, S. N. Srihari. Soft Attention Improves Skin Cancer Classification Performance. In Proceedings of Interpretability of Machine Intelligence in Medical Image Computing, and Topological Data Analysis and Its Applications for Medical Data, Springer, Strasbourg, France, pp. 13–23, 2021.

  32. Y. Xie, J. Zhang, Y. Xia. A mutual bootstrapping model for automated skin lesion segmentation and classification. IEEE Transactions on Medical Imaging, vol. 39, no. 7, pp. 2482–2493, 2020. DOI: https://doi.org/10.1109/TMI.2020.2972964.

    Article  Google Scholar 

Download references

Acknowledgements

This paper was supported by National Natural Science Foundation of China (No. 61977063 and 61872020). The authors thank all the patients for providing their MRI images and School of Biomedical Engineering at Southern Medical University, China for providing the brain tumor data set. We appreciate Dr. Fenfen Li, Wenzhou Eye Hospital, Wenzhou Medical University, China, for her support with clinical consulting and language editing.

Author information

Authors and Affiliations

  1. School of Artificial Intelligence, Beijing Normal University, Beijing, 100875, China

    Gaoxiang Li, Xiao Hui, Wenjing Li & Yanlin Luo

Authors
  1. Gaoxiang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiao Hui
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wenjing Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yanlin Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yanlin Luo.

Ethics declarations

The authors declared that they have no conflicts of interest to this work.

Additional information

Colored figures are available in the online version at https://link.springer.com/journal/11633

Gaoxiang Li received the B. Sc. degree in information and computing science from College of Science, Zhejiang University of Technology, China in 2020. He is currently a master student in computer application technology at School of Artificial Intelligence, Beijing Normal University, China.

His research interest is medical image processing.

Xiao Hui received the B.Sc. degree in computer science and technology from School of Artificial Intelligence, Beijing Normal University, China in 2020. She is currently a master student in computer application technology at School of Artificial Intelligence, Beijing Normal University, China.

Her research interests include medical image processing and virtual surgery.

Wenjing Li received the B. Sc. degree in computer science and technology from School of Artificial Intelligence, Beijing Normal University, China in 2021. She is currently a master student in computer application technology at School of Artificial Intelligence, Beijing Normal University, China.

Her research interest is medical image processing.

Yanlin Luo received the B. Sc. degree in mathematics and the M. Sc. degree in computer software and theory from School of Mathematics and Statistics, Lanzhou University, China in 1990 and 1993, respectively, and the Ph. D. degree in applied mathematics from School of Mathematical Sciences, Zhejiang University, China in 1997. She is currently an associate professor at School of Artificial Intelligence, Beijing Normal University, China.

Her research interests include medical image processing and visualization and virtual reality.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, G., Hui, X., Li, W. et al. Multitask Learning with Multiscale Residual Attention for Brain Tumor Segmentation and Classification. Mach. Intell. Res. 20, 897–908 (2023). https://doi.org/10.1007/s11633-022-1392-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11633-022-1392-6

Keywords

Search

Navigation