Skip to main content
SpringerLink
Log in

HEU-Net: hybrid attention residual block-based network with external skip connections for metal corrosion semantic segmentation

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Regularly detect the corrosion of metal structures and take countermeasures according to the degree of corrosion, which can reduce the potential safety hazards and avoid unnecessary economic losses. This paper presents a deep learning-based metal corrosion detection method, which is used to accurately segment the corrosion regions. This model incorporates Hybrid Attention Residual Block (HARB) and External Skip Connections (ESC) in the basic architecture of U-Net. HARB can reweight the features so that the network pays more attention to the corrosion regions and ignores other irrelevant regions. Shallow features and deep features are fused using ESC to make the features of network learning more effective and comprehensive, which is conducive to improving the segmentation accuracy. In order to further improve the semantic segmentation performance, the Attention Gate Residual Block (AGRB) is proposed, which can optimize segmentation by strengthening the regions to be segmented and reducing the activation value of the background, while overcoming the semantic and scale inconsistencies between input features. In this paper, comprehensive comparative experiments are conducted on the Metal Corrosion dataset. The experimental results show that the F1 and mIoU of the proposed method are 91.21% and 89.56%, respectively. Compared with U-Net, HEU-Net achieves a 7.91% improvement in F1 and a 10.16% improvement in mIoU. These results show that our method outperforms state-of-the-art models and can achieve better semantic segmentation results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability statement

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. Michailidis, N., Castaneda, H.: Corrosion. In the International Academy for Production CIRP Encyclopedia of Production Engineering. Berlin, Springer (2018). https://doi.org/10.1007/978-3-642-35950-7_16862-1

    Book  Google Scholar 

  2. Trujillo, M., Sadki, M.: Sensitivity analysis for texture models applied to rust steel classification. Electron. Imaging Sci. Technol. San Jose, Calif, USA Proc. SPIE 5303, 161–169 (2004)

    Google Scholar 

  3. Zhang, J.: Study on Corrosion Status and Control Strategies in Energy Field in China. In: Hou, B. (ed.) The Cost of Corrosion in China. Springer, Singapore (2019). https://doi.org/10.1007/978-981-32-9354-0_4

    Chapter  Google Scholar 

  4. Hansson, C.M.: The impact of corrosion on society. Metall. Mater. Trans. A 42, 2952–2962 (2011). https://doi.org/10.1007/s11661-011-0703-2

    Article  Google Scholar 

  5. Hou, B.: Introduction to a Study on Corrosion Status and Control Strategies in China. The Cost of Corrosion in China. Singapore, Springer (2019). https://doi.org/10.1007/978-981-32-9354-0_1

    Book  Google Scholar 

  6. Wang, D., Peng, B., Pan, Y., Chen, A.: Segmentation and quantitative analysis of corrosion images based on deep neural networks. J. South China Univ. Technol. (Nat. Sci. Ed.) (2018). https://doi.org/10.3969/j.issn.1000-565X.2018.12.015

    Article  Google Scholar 

  7. Boutros, F., Damer, N., Kirchbuchner, F., Kuijper, A.: Elasticface Elastic margin loss for deep face recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1578–1587) (2022). doi:https://doi.org/10.48550/arXiv.2109.09416

  8. Yan, C., Meng, L., Li, L., Zhang, J., Wang, Z., Yin, J., Zhang, J., Sun, Y., Zheng, B.: Age invariant face recognition by multi-feature fusion and decomposition with self-attention. ACM Trans. Multimed. Comput. Commun. Appl. (TOMM). 18(1), 1–18 (2022). https://doi.org/10.1145/3472810

    Article  Google Scholar 

  9. Qiu, H., Gong, D., Li, Z., Liu, W., Tao, D.: End2End occluded face recognition by masking corrupted features. IEEE Trans. Pattern Anal. Mach. Intell. (2021). https://doi.org/10.1109/TPAMI.2021.3098962

    Article  Google Scholar 

  10. Chen, W., Huang, H., Peng, S., Zhou, C., Zhang, C.: YOLO-face: a real-time face detector. Vis Comput (2021). https://doi.org/10.1007/s00371-020-01831-7

    Article  Google Scholar 

  11. Yang, S., Luo, P., Loy, C.C., Tang, X.: Faceness-net: face detection through deep facial part responses. IEEE Trans. Pattern Anal. Mach. Intell. (2018). https://doi.org/10.1109/TPAMI.2017.2738644

    Article  Google Scholar 

  12. Lv, Z., Li, Y., Feng, H., Lv, H.: Deep learning for security in digital twins of cooperative intelligent transportation systems. IEEE Trans. Intell. Transp. Syst. (2021). https://doi.org/10.1109/TITS.2021.3113779

    Article  Google Scholar 

  13. Cai, Y., Dai, L., Wang, H., Chen, L., Li, Y., Sotelo, M.A., Li, Z.: Pedestrian motion trajectory prediction in intelligent driving from far shot first-person perspective video. IEEE Trans. Intell. Transp. Syst. (2021). https://doi.org/10.1109/TITS.2021.3052908

    Article  Google Scholar 

  14. Yang, T., Liang, R., Huang, L.: Vehicle counting method based on attention mechanism SSD and state detection. Vis. Comput. (2021). https://doi.org/10.1007/s00371-021-02161-y

    Article  Google Scholar 

  15. Hu, J., Liu, R., Chen, Z., Wang, D., Zhang, Y., Xie, B.: Octave convolution-based vehicle detection using frame-difference as network input. Vis. Comput. (2022). https://doi.org/10.1007/s00371-022-02425-1

    Article  Google Scholar 

  16. Syed-Ab-Rahman, S.F., Hesamian, M.H., Prasad, M.: Citrus disease detection and classification using end-to-end anchor-based deep learning model. Appl. Intell. 52(1), 927–938 (2022)

    Article  Google Scholar 

  17. Ashwinkumar, S., Rajagopal, S., Manimaran, V., Jegajothi, B.: Automated plant leaf disease detection and classification using optimal MobileNet based convolutional neural networks. Mater. Today: Proc. 51, 480–487 (2022). https://doi.org/10.1016/j.matpr.2021.05.584

    Article  Google Scholar 

  18. Vallabhajosyula, S., Sistla, V., Kolli, V.K.K.: Transfer learning-based deep ensemble neural network for plant leaf disease detection. J. Plant Dis. Prot. 129(3), 545–558 (2022). https://doi.org/10.1007/s41348-021-00465-8

    Article  Google Scholar 

  19. Shah, D., Trivedi, V., Sheth, V., Shah, A., Chauhan, U.: ResTS: Residual deep interpretable architecture for plant disease detection. Inf. Process. Agric. 9(2), 212–223 (2022). https://doi.org/10.1016/j.inpa.2021.06.001

    Article  Google Scholar 

  20. Gajjar, R., Gajjar, N., Thakor, V.J., Patel, N.P., Ruparelia, S.: Real- time detection and identification of plant leaf diseases using convolutional neural networks on an embedded platform. Vis. Comput. (2021). https://doi.org/10.1007/s00371-021-02164-9

    Article  Google Scholar 

  21. Fuentes, A., Yoon, S., Kim, S.C., Park, D.S.: A robust deep-learning-based detector for real-time tomato plant diseases and pests recognition. Sensors (2017). https://doi.org/10.3390/s17092022

    Article  Google Scholar 

  22. Su, B., Chen, H., Zhou, Z.: BAF-detector: an efficient CNN-based detector for photovoltaic cell defect detection. IEEE Trans. Industr. Electron. 69(3), 3161–3171 (2021). https://doi.org/10.1109/TIE.2021.3070507

    Article  Google Scholar 

  23. Ha, H., Jeong, J.: CNN-based defect inspection for injection molding using edge computing and industrial IoT systems. Appl. Sci. 11(14), 6378 (2021). https://doi.org/10.3390/app11146378

    Article  Google Scholar 

  24. Gao, H., Zhang, Y., Lv, W., Yin, J., Qasim, T., Wang, D.: A deep convolutional generative adversarial networks-based method for defect detection in small sample industrial parts images. Appl. Sci. 12(13), 6569 (2022). https://doi.org/10.3390/app12136569

    Article  Google Scholar 

  25. Üzen, H., Turkoglu, M., Aslan, M., Hanbay, D.: Depth-wise squeeze and excitation block-based efficient-Unet model for surface defect detection. Vis Comput (2022). https://doi.org/10.1007/s00371-022-02442-0

    Article  Google Scholar 

  26. Huang, Y., Qiu, C., Yuan, K.: Surface defect saliency of magnetic tile. Vis. Comput. 36, 85–96 (2020). https://doi.org/10.1007/s00371-018-1588-5

    Article  Google Scholar 

  27. Jonathan, L., Evan, S., Trevor, D.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3431–3440. (2015)

  28. Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional Networks for Biomedical Image Segmentation. Springer International Publishing, Berlin (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Book  Google Scholar 

  29. Zhang, Z., Liu, Q., Wang, Y.: Road extraction by deep residual U-Net. IEEE Geosci. Remote Sens. Lett. 15(5), 749–753 (2018). https://doi.org/10.1109/LGRS.2018.2802944

    Article  Google Scholar 

  30. Jie, H., Li, S., Gang, S.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7132–7141. (2018). doi:https://doi.org/10.1109/TPAMI.2019.2913372

  31. Oktay, O., Schlemper, J., Folgoc, L. le, Lee, M., Heinrich, M., Misawa, K., Mori, K., McDonagh, S., Hammerla, N.Y., Kainz, B., Glocker, B., Rueckert, D.: Attention U-Net: Learning Where to Look for the Pancreas. arXiv preprint arXiv:1804.03999v3 (2018)

  32. Zhou, Z., Siddiquee, M.M.R., Tajbakhsh, N., Liang, J.: Unet++ A nested u-net architecture for medical image segmentation deep learning in medical image analysis and multimodal learning for clinical decision support, pp. 3–11. Springer, Toronto (2018)

    Google Scholar 

  33. Medeiros, F.N., Ramalho, G.L., Bento, M.P., et al.: On the evaluation of texture and color features for nondestructive corrosion detection. EURASIP J. Adv. Signal Process. 2010, 817473 (2010). https://doi.org/10.1155/2010/817473

    Article  Google Scholar 

  34. Khayatazad, M., Pue, L.D., Waele, W.D.: Detection of corrosion on steel structures using automated image processing. Dev. Built Environ. 3, 100022 (2020)

    Article  Google Scholar 

  35. Hoang, N.: Image processing-based pitting corrosion detection using metaheuristic optimized multilevel image thresholding and machine-learning approaches. Math. Probl. Eng. (2020). https://doi.org/10.1155/2020/6765274

    Article  Google Scholar 

  36. Fei, Z., Yang, E., Yang, B., Yu, L.: Image enhancement and corrosion detection for UAV visual inspection of pressure vessels. In: Intelligent Life System Modelling, Image Processing and Analysis. pp. 145–154. Springer, Singapore (2021). doi:https://doi.org/10.1007/978-981-16-7207-1_15

  37. Yao, Y., Yang, Y., Wang, Y., Zhao, X.: Artificial intelligence-based hull structural plate corrosion damage detection and recognition using convolutional neural network. Appl. Ocean Res. (2019). https://doi.org/10.1016/j.apor.2019.05.008

    Article  Google Scholar 

  38. Bastian, B.T., Jaspreeth, N., Ranjith, S.K., Jiji, C.V.: Visual inspection and characterization of external corrosion in pipelines using deep neural network. NDT E Int.: Indep. Nondestruct. Test. Eval. (2019). https://doi.org/10.1016/j.ndteint.2019.102134

    Article  Google Scholar 

  39. Zhou, Q., Ding, S., Feng, Y., Qing, G., Hu, J.: Corrosion inspection and evaluation of crane metal structure based on UAV vision. SIViP (2022). https://doi.org/10.1007/s11760-021-02126-7

    Article  Google Scholar 

  40. Chen, Q., Wen, X., Lu, S., Sun, D.: Corrosion detection for large steel structure base on UAV integrated with image processing system. IOP Conf. Ser.: Mater. Sci. Eng. (2019). https://doi.org/10.1088/1757-899X/608/1/012020

    Article  Google Scholar 

  41. Liu L., Tan E., Yin X.J., Zhen, Y., Cai, Z.Q.: Deep learning for coating condition assessment with active perception. In: Proceedings of the 2019 3rd High Performance Computing and Cluster Technologies Conference (2019). doi:https://doi.org/10.1145/3341069.3342966

  42. Qian, C.: Evaluation of deep learning-based semantic segmentation approaches for autonomous corrosion detection on metallic surfaces. Purdue Univ. Grad. Sch. (2019). https://doi.org/10.25394/PGS.9959090.v1

    Article  Google Scholar 

  43. Ma, Y., Yang, Y., Yao, Y., Li, S., Zhao, X.: Image-based corrosion recognition for ship steel structures. Smart Struct. NDE Ind. (2018). https://doi.org/10.1117/12.2296540

    Article  Google Scholar 

  44. Papamarkou, T., Guy, H., Kroencke, B., Miller, J., Robinette, P., Schultz, D., Hinklle, J., Pullum, L., Schuman, C., Renshaw, J., Chatzidakis, S.: Automated detection of corrosion in used nuclear fuel dry storage canisters using residual neural networks. Nucl. Eng. Technol. (2020). https://doi.org/10.1016/j.net.2020.07.020

    Article  Google Scholar 

  45. Katsamenis, I., Protopapadakis, E., Doulamis, A., Doulamis, N., Voulodimos, A.: Pixel-level corrosion detection on metal constructions by fusion of deep learning semantic and contour segmentation. In: International Symposium on Visual Computing (pp. 160–169). Springer, Cham (2020). doi:https://doi.org/10.1007/978-3-030-64556-4_13

  46. Pirie, C., Moreno-Garcia, C.F.: Image Pre-processing and Segmentation for Real-Time Subsea Corrosion Inspection. In: Iliadis, L., Macintyre, J., Jayne, C., Pimenidis, E. (eds) Proceedings of the 22nd Engineering Applications of Neural Networks Conference. EANN 2021. Proceedings of the International Neural Networks Society (2021). Vol. 3. Springer, Cham. doi:https://doi.org/10.1007/978-3-030-80568-5_19

  47. Shi, J., Dang, J., Cui, M., Zuo, R., Shimizu, K., Tsunoda, A., Suzuki, Y.: Improvement of damage segmentation based on pixel-level data balance using vgg-unet. Appl. Sci. 11(2), 518 (2021). https://doi.org/10.3390/app11020518

    Article  Google Scholar 

  48. Han, Q., Zhao, N., Xu, J.: Recognition and location of steel structure surface corrosion based on unmanned aerial vehicle images. J. Civ. Struct. Heal. Monit. 11(5), 1375–1392 (2021). https://doi.org/10.1007/s13349-021-00515-7

    Article  Google Scholar 

  49. Stoean, R., Bacanin, N., Ionescu, L., Boicea, M., Gărău, A.M., Ghiţescu, C.C.: Semantic Segmentation for Corrosion Detection in Archaeological Artefacts before Restoration. In: 2021 23rd International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC) (pp. 246–251) (2021). doi:https://doi.org/10.1109/SYNASC54541.2021.00049

  50. Nash, W., Zheng, L., Birbilis, N.: Deep learning corrosion detection with confidence. npj. Mater. Degrad. (2022). https://doi.org/10.1038/s41529-022-00232-6

    Article  Google Scholar 

  51. Nash, W., Drummond, T., Birbilis, N.: Deep learning AI for corrosion detection. In: CORROSION (2019)

  52. Sanghyun, W., Jongchan, P., Joon-Young, L., In, S.K.: Cbam: Convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19. (2018)

  53. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016) doi:https://doi.org/10.1109/CVPR.2016.90

  54. Torralba, A., Efros, A. A.: Unbiased look at dataset bias. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1521–1528. (2011)

  55. Torralba, A., Russell, B.C., Yuen, J.: Labelme: Online image annotation and applications. Proc. IEEE 98(8), 1467–1484 (2010)

    Article  Google Scholar 

  56. Zhong, Z., Zheng, L., Kang, G., Li, S., Yang, Y.: Random erasing data augmentation. In: Proceedings of the AAAI conference on artificial intelligence (2020). doi:https://doi.org/10.1609/aaai.v34i07.7000

  57. Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2881–2890. (2017). doi:https://doi.org/10.1109/CVPR.2017.660

  58. Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P.: SegFormer: simple and efficient design for semantic segmentation with transformers. Adv. Neural Inf. Process. Syst., NeurIPS 2021(34), 12077–12090 (2021). https://doi.org/10.48550/arXiv.2105.15203

    Article  Google Scholar 

  59. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B: Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10012–10022 (2021). doi:https://doi.org/10.1088/1757-899X/608/1/012020

Download references

Acknowledgements

This work is supported by open foundation of “Pioneer” and “Leading Goose” R&D Program of Zhejiang; 2022C01130.

Author information

Authors and Affiliations

  1. Ocean College, Zhejiang University, Zhoushan, China

    Tiancheng Zhu & Shiqiang Zhu

  2. Zhejiang Lab, Hangzhou, China

    Tiancheng Zhu, Shiqiang Zhu, Tao Zheng & Wei Song

  3. Zhejiang Construction Engineering Group Co., Ltd, Hangzhou, China

    Hongliang Ding

  4. Zhoushan Institute of Calibration and Testing for Qualitative and Technical Supervision, Zhoushan, China

    Cunjun Li

Authors
  1. Tiancheng Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shiqiang Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tao Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hongliang Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wei Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Cunjun Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wei Song.

Ethics declarations

Conflict of interest

There is no conflict of interest in the submission.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhu, T., Zhu, S., Zheng, T. et al. HEU-Net: hybrid attention residual block-based network with external skip connections for metal corrosion semantic segmentation. Vis Comput 40, 1273–1287 (2024). https://doi.org/10.1007/s00371-023-02846-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-023-02846-6

Keywords

Search

Navigation