Abstract
The United Nations aims to preserve, evaluate, and conserve cultural heritage (CH) structures as part of sustainable development. The design life expectancy of many CH structures is slowly approaching its end. It is thus imperative to conduct frequent visual inspections of CH structures following conservation guidelines to ensure their structural integrity. This study implements a custom defect detection, and localization supervised deep learning model based on the you only look once (YOLO) v5 real-time object detection algorithm by implementing a case study of the Dadi-Poti tombs in Hauz Khas Village, New Delhi. The custom YOLOv5 model is trained to automatically detect four defects, namely, discoloration, exposed bricks, cracks, and spalling, and tested on a dataset comprising 10291 images. The validity and performance of the custom YOLOv5 model are compared with a ResNet 101 architecture-based faster region-based convolutional neural network (R-CNN), and conventional manual visual inspection methods are used to convey the significance of the developed artificial intelligence-based model. The maximum average precision (mAP) of the custom YOLOv5 model and faster R-CNN is 93.7% and 85.1%, respectively.
Similar content being viewed by others
Data availability
The Mendeley database (https://data.mendeley.com/datasets/gnyzwrz4gt/1) has some of the image data collected by authors, Marco B and Michela C, that was used for the YOLOv5 models. The author can provide the Python code for replication of results upon reasonable request.
References
Wang N, Zhao X, Zhao P, Zhang Y, Zou Z, Jinping O (2019) Automatic damage detection of historic masonry buildings based on mobile deep learning. Autom Constr 103:53–66
Agdas D, Rice JA, Martinez JR, Lasa IR (2016) Comparison of visual inspection and structural-health monitoring as bridge condition assessment methods. J Perform Constr Facil 30(3):04015049
Georgopoulos GD, Telioni EC, Tsontzou A (2016) The contribution of laser scanning technology in the estimation of ancient greek monuments’ deformations. Surv Rev 48(349):303–308
Costanzo A, Minasi M, Casula G, Musacchio M, Buongiorno MF (2014) Combined use of terrestrial laser scanning and ir thermography applied to a historical building. Sensors 15(1):194–213
Armesto-González J, Riveiro-Rodríguez B, González-Aguilera D, Rivas-Brea MT (2010) Terrestrial laser scanning intensity data applied to damage detection for historical buildings. J Archaeol Sci 37(12):3037–3047
Yuan L, Guo J, Wang Q (2020) Automatic classification of common building materials from 3d terrestrial laser scan data. Autom Constr 110:103017
Mishra M, Lourenço PB, Ramana GV (2022) Structural health monitoring of civil engineering structures by using the internet of things: a review. J Build Eng 48:103954. https://doi.org/10.1016/j.jobe.2021.103954
Ramos LF, Miranda T, Mishra M, Fernandes FM, Manning E (2015) A bayesian approach for ndt data fusion: the saint torcato church case study. Eng Struct 84:120–129
Nghiem H-L, Heib M, Emeriault F (2015) Method based on digital image correlation for damage assessment in masonry structures. Eng Struct 86:1–15
Torres B, Varona FB, Baeza FJ, Bru D, Ivorra S (2020) Study on retrofitted masonry elements under shear using digital image correlation. Sensors 20(7):2122
Rezaie A, Achanta R, Godio M, Beyer K (2020) Comparison of crack segmentation using digital image correlation measurements and deep learning. Construct Build Mater 261:120474
Galantucci RA, Fatiguso F (2019) Advanced damage detection techniques in historical buildings using digital photogrammetry and 3d surface anlysis. J Cult Herit 36:51–62
Kim B, Cho S (2019) Image-based concrete crack assessment using mask and region-based convolutional neural network. Struct Control Health Monit 26(8):e2381
Perumal R, Venkatachalam SB (2021) Non invasive detection of moss and crack in monuments using image processing techniques. J Ambient Intell Humaniz Comput 12(5):5277–5285
Adamopoulos E (2021) Learning-based classification of multispectral images for deterioration mapping of historic structures. J Build Pathol Rehabilitation 6(1):1–15
Newman C, Edwards D, Martek I, Lai J, Thwala WD, Rillie I (2021) Industry 4.0 deployment in the construction industry: a bibliometric literature review and UK-based case study. Smart Sustain Built Environ 10(4):557–580. https://doi.org/10.1108/SASBE-02-2020-0016
Rahimian FP, Goulding JS, Abrishami S, Seyedzadeh S, Elghaish F (2021) Industry 4.0 solutions for building design and construction: a paradigm of new opportunities. Routledge, p 420. eBook ISBN 9781003106944. https://doi.org/10.1201/9781003106944
Prieto AJ, Ortiz R, Macías-Bernal JM, Chávez MJ, Ortiz Pi (2019) Artificial intelligence applied to the preventive conservation of heritage buildings. In Science and Digital Technology for Cultural Heritage. CRC Press pages 245–249
Bienvenido-Huertas D, Nieto-Julián JE, Moyano JJ, Macías-Bernal JM, Castro J (2019) Implementing artificial intelligence in h-bim using the J48 algorithm to manage historic buildings. Int J Archit Herit 14(8):1148–1160. https://doi.org/10.1080/15583058.2019.1589602
Sánchez-Aparicio LJ, Masciotta M-G, García-Alvarez J, Ramos LF, Oliveira DV, Martín-Jiménez JAn, González-Aguilera D, Monteiro P (2020) Web-gis approach to preventive conservation of heritage buildings. Autom Construct 118:103304
Sony S, Dunphy K, Sadhu A, Capretz M (2021) A systematic review of convolutional neural network-based structural condition assessment techniques. Eng Struct 226:111347
Nazarian E, Taylor T, Weifeng T, Ansari F (2018) Machine-learning-based approach for post event assessment of damage in a turn-of-the-century building structure. J Civ Struct Health Monit 8(2):237–251
Mishra M (2021) Machine learning techniques for structural health monitoring of heritage buildings: a state-of-the-art review and case studies. J Cult Herit 47:227–245
Zou Z, Zhao P, Zhao X (2021) Automatic segmentation, inpainting, and classification of defective patterns on ancient architecture using multiple deep learning algorithms. Struct Control Health Monit 28(7):e2742
Trier ØD, Cowley DC, Waldeland AU (2019) Using deep neural networks on airborne laser scanning data: results from a case study of semi-automatic mapping of archaeological topography on arran, scotland. Archaeol Prospect 26(2):165–175
Mishra M, Bhatia AS, Maity D (2020) Predicting the compressive strength of unreinforced brick masonry using machine learning techniques validated on a case study of a museum through nondestructive testing. J Civil Struct Health Monit 10(3):389–403
Mansuri LE, Patel DA (2022) Development of automated web-based condition survey system for heritage monuments using deep learning. In: Belayutham S, Che Ibrahim CKI, Alisibramulisi A, Mansor H, Billah M (eds) Proceedings of the 5th International Conference on Sustainable Civil Engineering Structures and Construction Materials. Lecture Notes in Civil Engineering, vol 215. Springer, Singapore. https://doi.org/10.1007/978-981-16-7924-7_76
Yao G, Sun Y, Wong M, Lv X (2021) A real-time detection method for concrete surface cracks based on improved yolov4. Symmetry 13(9):1716
Narazaki Yasutaka, Hoskere Vedhus, Yoshida Koji, Spencer Billie F, Fujino Yozo (2021) Synthetic environments for vision-based structural condition assessment of japanese high-speed railway viaducts. Mech Syst Signal Process 160:107850
Hoskere V, Narazaki Y, Spencer BF Jr (2022) Physics-based graphics models in 3d synthetic environments as autonomous vision-based inspection testbeds. Sensors 22(2):532
Mansuri LE, Patel DA (2021) Artificial intelligence-based automatic visual inspection system for built heritage. Smart Sustain Built Environ. https://doi.org/10.1108/SASBE-09-2020-0139
Mansuri LE, Patel DA (2022) Artificial intelligence for heritage conservation: a case study of automatic visual inspection system. In: Li RYM, Chau KW, Ho DCW (eds) Current state of art in artificial intelligence and ubiquitous cities. Springer, Singapore. https://doi.org/10.1007/978-981-19-0737-1_1
LabelImg Tzutalin (2015) Git code. https://github.com/tzutalin/labelImg. Accessed 10 Mar 2022
Chaiyasarn K, Sharma M, Ali L, Khan W, Poovarodom N (2018) Crack detection in historical structures based on convolutional neural network. GEOMATE J 15(51):240–251
Dais D, Bal IE, Smyrou E, Sarhosis V (2021) Automatic crack classification and segmentation on masonry surfaces using convolutional neural networks and transfer learning. Autom Construct 125:103606
Wang N, Zhao Q, Li S, Zhao X, Zhao P (2018) Damage classification for masonry historic structures using convolutional neural networks based on still images. Comput-Aided Civil Infrastruct Eng 33(12):1073–1089
Wang N, Zhao X, Zou Z, Zhao P, Qi F (2020) Autonomous damage segmentation and measurement of glazed tiles in historic buildings via deep learning. Comput-Aided Civil Infrastruct Eng 35(3):277–291
Guo J, Wang Q, Li Y (2021) Evaluation-oriented façade defects detection using rule-based deep learning method. Autom Construct 131:103910
Monna F, Rolland T, Denaire A, Navarro N, Granjon L, Barbé R, Chateau-Smith C (2021) Deep learning to detect built cultural heritage from satellite imagery.-spatial distribution and size of vernacular houses in sumba, indonesia. J Cult Herit 52:171–183
Mondal TG, Jahanshahi MR, R-TW, Zheng YW (2020) Deep learning-based multi-class damage detection for autonomous post-disaster reconnaissance. Struct Control Health Monit 27(4):e2507
Sharma E, Agrawal P, Verma NK (2019) Detection of dust deposition using convolutional neural network for heritage images. In Computational Intelligence: Theories, Applications and Future Directions-Volume II, pages 347–359. Springer
Masrour T, El Hassani I, Bouchama MS (2020) Deep convolutional neural networks with transfer learning for old buildings pathologies automatic detection. In: Ezziyyani M (eds) Advanced Intelligent Systems for Sustainable Development (AI2SD’2019). AI2SD 2019. Advances in Intelligent Systems and Computing, vol 1104. Springer, Cham. https://doi.org/10.1007/978-3-030-36671-1_18
Zou Z, Zhao X, Zhao P, Qi F, Wang N (2019) Cnn-based statistics and location estimation of missing components in routine inspection of historic buildings. J Cult Herit 38:221–230
Masrour T, Hassani IE, Bouchama MS (2019) Deep convolutional neural networks with transfer learning for old buildings pathologies automatic detection. In International Conference on Advanced Intelligent Systems for Sustainable Development, pages 204–216. Springer
Dung CV et al (2019) Autonomous concrete crack detection using deep fully convolutional neural network. Autom Construct 99:52–58
Yang X, Li H, Yantao Y, Luo X, Huang T, Yang X (2018) Automatic pixel-level crack detection and measurement using fully convolutional network. Comput-Aided Civil Infrastruct Eng 33(12):1090–1109
Azimi M, Eslamlou AD, Pekcan G (2020) Data-driven structural health monitoring and damage detection through deep learning: state-of-the-art review. Sensors 20(10):2778
Han Q, Pan Y, Yang D, Ying X (2022) CNN-based bolt loosening identification framework for prefabricated large-span spatial structures. J Civil Struct Health Monitor 12:517–536. https://doi.org/10.1007/s13349-022-00561-9
Zhou Q, Ding S, Qing G, Jingbo H (2022) Uav vision detection method for crane surface cracks based on faster r-cnn and image segmentation. J Civil Struct Health Monitor 1–11
Kung R-Y, Pan N-H, Wang CCN, Lee P-C (2021) Application of deep learning and unmanned aerial vehicle on building maintenance. Adv Civil Eng 2021:5598690. https://doi.org/10.1155/2021/5598690
Chen K, Yadav A, Khan A, Meng Y, Zhu K (2019) Improved crack detection and recognition based on convolutional neural network. Model Simul Eng 2019:8796743. https://doi.org/10.1155/2019/8796743
Cha Y-J, Choi W, Suh G, Mahmoudkhani S, Büyüköztürk O (2018) Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types. Comput-Aided Civil Infrastruct Eng 33(9):731–747
Liu Z, Cao Y, Wang Y, Wang W (2019) Computer vision-based concrete crack detection using u-net fully convolutional networks. Autom Construct 104:129–139
Deng J, Ye L, Lee VC-S (2021) Imaging-based crack detection on concrete surfaces using you only look once network. Struct Health Monit 20(2):484–499
Feng C, Zhang H, Li Y, Wang S, Wang H (2021) Efficient real-time defect detection for spillway tunnel using deep learning. J Real-Time Image Process 18(6):2377–2387
Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition 779–788
Redmon J, Farhadi A (2017) Yolo9000: better, faster, stronger. In Proceedings of the IEEE conference on computer vision and pattern recognition 7263–7271
Park SE, Eem S-H, Jeon H (2020) Concrete crack detection and quantification using deep learning and structured light. Construct Build Mater 252:119096
Jingwei LX, Yang SL, Wang X, Luo S, Lee VC-S, Ding L (2020) Automated pavement crack detection and segmentation based on two-step convolutional neural network. Comput-Aided Civil Infrastruct Eng 35(11):1291–1305
Jiang S, Zhang J (2020) Real-time crack assessment using deep neural networks with wall-climbing unmanned aerial system. Comput-Aided Civil Infrastruct Eng 35(6):549–564
Teng S, Liu Z, Chen G, Cheng L (2021) Concrete crack detection based on well-known feature extractor model and the yolo_v2 network. Appl Sci 11(2):813
Li S, Xingyu G, Xiangrong X, Dawei X, Zhang T, Liu Z, Dong Q (2021) Detection of concealed cracks from ground penetrating radar images based on deep learning algorithm. Construct Build Mater 273:121949
Mishra M, Jain V, Singh SK, Maity D (2022) Two-stage method based on the you only look once framework and image segmentation for crack detection in concrete structures. Archit Struct Construct 2(1):1–18
Maeda H, Sekimoto Y, Seto T, Kashiyama T, Omata H (2018) Road damage detection and classification using deep neural networks with smartphone images. Comput-Aided Civil Infrastruct Eng 33(12):1127–1141
Yuchuan D, Pan N, Zihao X, Fuwen DY, Shen, Hua K (2020) Pavement distress detection and classification based on YOLO network. Int J Pavement Eng 22(13):1659–1672
Fu-Jun D, Jiao S-J (2022) Improvement of lightweight convolutional neural network model based on yolo algorithm and its research in pavement defect detection. Sensors 22(9):3537
Liu Z, Wenxiu W, Xingyu G, Li S, Wang L, Zhang T (2021) Application of combining yolo models and 3d gpr images in road detection and maintenance. Remote Sens 13(6):1081
Zhou S, Yufeng BX, Wei JL, Ye Z, Li F, Yuchuan D (2021) Automated detection and classification of spilled loads on freeways based on improved yolo network. Mach Vis Appl 32(2):1–12
Hou X, Zeng Y, Xue J (2020) Detecting structural components of building engineering based on deep-learning method. J Construct Eng Manag 146(2):04019097
Ukhwah EN, Yuniarno Eko M, Suprapto YK (2019) Asphalt pavement pothole detection using deep learning method based on yolo neural network. In 2019 International Seminar on Intelligent Technology and Its Applications (ISITIA), pages 35–40. IEEE
Park S-S, Tran V-T, Lee D-E (2021) Application of various yolo models for computer vision-based real-time pothole detection. Appl Sci 11(23):11229
Ge Liangfu, Dan Danhui, Li Hui (2020) An accurate and robust monitoring method of full-bridge traffic load distribution based on yolo-v3 machine vision. Struct Control Health Monit 27(12):e2636
RongXin W et al. (2019) Research on safety helmet wearing yolo-v3 detection technology improvement in mine environment. In Journal of Physics: Conference Series, volume 1345(4), page 042045. IOP Publishing
Hu J, Gao X, Wu H, Gao S (2019) Detection of workers without the helments in videos based on yolo v3. In 2019 12th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), pages 1–4. IEEE
Li Yang, Chen Jun (2022) Computer vision-based counting model for dense steel pipe on construction sites. J Construct Eng Manag 148(1):04021178
Park Sehwan, Kim Jinpyung, Jeon Kyoyoung, Kim Junkyeong, Park Seunghee (2011) Improvement of gpr-based rebar diameter estimation using yolo-v3. Remote Sens 13(10):2021b
Yang Xinyue, Gao Yuqing, Fang Cheng, Zheng Yue, Wang Wei (2022) Deep learning-based bolt loosening detection for wind turbine towers. Struct Control Health Monit 29(6):e2943
Zhao Y, Deng X, Lai H (2020) A yolo-based method to recognize structural components from 2d drawings. In Construction Research Congress 2020: Computer Applications, pages 753–762. American Society of Civil Engineers Reston, VA
Idjaton K, Desquesnes X, Treuillet S, Brunetaud X (2022) Transformers with yolo network for damage detection in limestone wall images. In International Conference on Image Analysis and Processing, pages 302–313. Springer
Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980
Malta A, Mendes M, Farinha T (2021) Augmented reality maintenance assistant using yolov5. Appl Sci 11(11):4758
Ren S, He K, Girshick R, Sun J (2015) Faster r-cnn: towards real-time object detection with region proposal networks. Adv Neural Inf Process Syst 28:91–99
Girshick R (2015) Fast r-cnn. In Proceedings of the IEEE international conference on computer vision 1440–1448
Cha Y-J, Choi W, Büyüköztürk O(2017) Deep learning-based crack damage detection using convolutional neural networks. Comput-Aided Civil Infrastruct Eng 32(5):361–378
Kumar SS, Wang M, Abraham DM, Jahanshahi MR, Iseley T, Cheng JCP (2020) Deep learning-based automated detection of sewer defects in cctv videos. J Comput Civil Eng 34(1):04019047
Lin T-Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Microsoft coco: common objects in context. In European conference on computer vision, pages 740–755. Springer
Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M et al. (2016) \(\{\)TensorFlow\(\}\): A system for \(\{\)Large-Scale\(\}\) machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16), pages 265–283
Chollet F (2015) keras, GitHub. https://github.com/fchollet/keras
Everingham M, Gool L Van, Williams CKI, Winn J, Zisserman A (2010) The pascal visual object classes (voc) challenge. Int J Comput Vis 88(2):303–338
Huang J, Rathod V, Sun C, Zhu M, Korattikara A, Fathi A, Fischer I et al (2017) Speed/accuracy trade-offs for modern convolutional object detectors. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7310–7311. Tensorflow object detection api. Code: https://github.com/tensorflow/models/tree/master/research/object_detection
Ćosović M, Janković R (2020) Cnn classification of the cultural heritage images. In 2020 19th International Symposium INFOTEH-JAHORINA (INFOTEH), pages 1–6. IEEE
Trier ØD, Reksten JH, Løseth K (2021) Automated mapping of cultural heritage in norway from airborne lidar data using faster r-cnn. Int J Appl Earth Obs Geoinform 95:102241
Wang N, Zhao X, Wang L, Zou Z (2019) Novel system for rapid investigation and damage detection in cultural heritage conservation based on deep learning. J Infrastruct Syst 25(3):04019020
Kwon D, Jeongmin Y (2019) Automatic damage detection of stone cultural property based on deep learning algorithm. Int Arch Photogramm Remote Sens Spat Inf Sci 42:639–643
Samhouri M, Al-Arabiat L, Al-Atrash F (2022) Prediction and measurement of damage to architectural heritages facades using convolutional neural networks. Neural Comput Appl 34:18125–18141. https://doi.org/10.1007/s00521-022-07461-5
Mishra M, Ramana GV (2022) Data for Dadi-Poti (Cultural Heritage) - Sample Image Dataset used for Automatic Visual Inspection System, Mendeley Data, V2. https://doi.org/10.17632/gnyzwrz4gt.2
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material containing 20 second video for CH damage detection model.
Supplementary file1 (MP4 1672 kb)
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Mishra, M., Barman, T. & Ramana, G.V. Artificial intelligence-based visual inspection system for structural health monitoring of cultural heritage. J Civil Struct Health Monit 14, 103–120 (2024). https://doi.org/10.1007/s13349-022-00643-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13349-022-00643-8