Automated Detection of Surface Cracks and Numerical Correlation with Thermal-Structural Behaviors of Fire Damaged Concrete Beams

There are two specific aims in this study; first is to develop and validate an automated crack detection technique for the fire damaged beam. Second is to investigate whether the detected crack information and thermal-structural behaviors can be numerically related. To fulfill the aims, fire tests and residual strength tests are conducted on RC beams having different fire exposure time periods and sustained load levels. To detect the automated cracks, surface images of the fire damaged beam surfaces are taken with digital cameras and an automatic crack detection method is developed using a convolutional neural network (CNN) which is a deep learning technique primarily used for analyzing intricate structures of high-dimensional data [such as high definition (HD) images and videos]. The quantity of cracks detected using the proposed CNN changes depending on the test variables, and the changing trends are similar to those of the crack lengths obtained from the optical observation. Additionally, it is found that the quantity of the automatically detected cracks is numerically related to the temperatures inside the beams as well as the stiffnesses obtained from the residual strength tests.


Introduction
Although concrete is known as a thermal resistant material, concrete structures are damaged when exposed to fire. To repair or reuse the structures after the fire, it is important to estimate and quantify damages accurately and promptly. One of the common investigation methods is optical observation of crack and deformation from the fire damaged structures. It would be cost effective if such optical observation is done quantitatively without requiring expensive testing machines or man power. Moreover, it would be very powerful if the crack information can be used as a guide to evaluate the performance of fire damaged concrete structures.
The visual assessment methods of fire damaged concrete have been focused on investigations of color change, deflection, and cracking/spalling of the surface. In the studies by Short et al. (2001) and Guise (1997), color image analysis is used to investigate changes in color for concrete subjected to elevated temperatures. Hager (2014) also investigates color change of ordinary and high-performance concretes upon heating using an image analysis software package (Scion image v. 4.0.3). Toumi and Resheidat (2010) analyze quantification of surface cracking of concrete heated to different temperatures ranging from 105 °C to 1250 °C using an image scanning technique. On the other hands, there is a study about concrete specimens with pre-made cracks to investigate the effect of cracks on temperature distributions Page 2 of 12 Ryu et al. Int J Concr Struct Mater (2020) 14:12 after earthquake fire (Wu et al. 2014). Xu et al. (2003) focus on the effect of binder (w/b) ratios and PFA dosages on cracking of concretes. Kim et al. (2013) investigate evolution of temperature-induced discontinuities at the sub-millimeter level and the cause of physical changes in heated mortar specimens (such as pore structure and density prevail) using high-performance X-ray computed tomography (CT), X-ray diffraction (XRD), and scanning electron microscopy (SEM). Regarding crack detection, various image processing and analysis techniques have been applied to detect edge information in fire damaged concrete structures. Li et al. (2018b) develop an edge detection algorithm using a local binary pattern (LBP) algorithm. They then analyze including the lengths and widths of the cracks. Other edge detection algorithms based on generic programming (Nishikawa et al. 2012) and Gabor filters (Zalama et al. 2014) have been used for identifying major cracks in surface images of concretes. However, hand-crafted image features are limited in the evaluation of fire damaged reinforced concrete (RC) beams as the structures have smoke particles on the surfaces and a large amount of noise similar to thin cracks. Furthermore, illumination differences in the acquisition significantly impacts reliable image feature extraction.
The convolutional neural network (CNN) is a deep learning technique that is primarily used for analyzing intricate structures of high-dimensional data such as high definition (HD) images and videos (LeCun et al. 2015). CNNs have a common structure that consists of cascaded convolutional layers and pooling layers ending with fully connected layers. Operating units called neurons are arranged to have feature maps and are connected to local patches to convolve the neurons of the previous layers. The learned image features from CNNs are known to be more robust and reliable to noise than hand-crafted features when a large database is available for training the models. Accordingly, advanced deep learning techniques are actively studied in damage assessments using image analysis of RC beams. Cha et al. (2017) use a CNN that can train image features to detect concrete cracks in an end-to-end manner. Lin et al. (2017) demonstrate significantly improved accuracies for localizing damaged areas in a structure when using automatically extracted image features from the CNN model. However, researches on the detection and utilization of cracks for fire damaged structural members have not been comprehensively studied. One of the reasons may be because it takes too long time and manpower to measure cracks quantitatively, especially when there are too many cracks on the concrete surface due to fire. Therefore, this study proposes a machine learning technique to detect cracks from the fire damaged concrete.
In addition, numerical correlations between the detected cracks and thermal-structural behaviors of the fire damaged beams are presented. For the study, fire tests are performed on the RC beams having different fire exposure time periods and sustained load levels. The automatic crack detection method is developed and validated with the crack length obtained using optical observation. Residual strength tests are also conducted to measure maximum load bearing capacity and stiffness of the fire damaged beams.

Details of Test Specimens
Five RC beams are fabricated with dimensions of 250 × 400 × 5000 mm (width × depth × length) as listed in Table 1. Three steel bars (diameter = 19 mm) are located in the bottom of the beams and stirrups with 150 mm spacing are used to prevent shear failure as shown in Fig. 1. Variables of this study are fire exposure time and sustained load level. During the fire tests, specimens are subjected to sustained loads of 4.82 tonf, 7.23 tonf, or 9.65 tonf, which correspond to 40, 60, or 80% of the nominal moment of the specimen, respectively. All the specimens are cured for 4 months at room temperature and then preheated at 200-300 °C for 2 h to prevent moisture effects during the fire test. After the fire test, the fire damaged beams are subjected to residual strength tests to measure maximum load bearing capacities and stiffnesses. Table 2 shows the mixing proportion of concrete and the mechanical properties of the concrete and steel bars used in the RC beams are obtained from material tests; the compressive strength of concrete is found to be 25 MPa and the tensile strength is measured as 3 MPa after 28 days of curing at room temperature (see Table 3). The strength and elastic modulus of the reinforcing bar are 448 MPa and 205 GPa, respectively.

Test Setup and Data Measurements
During fire tests, beams are loaded with a four-point loading system. Effective span is 4700 mm and distance Page 3 of 12 Ryu et al. Int J Concr Struct Mater (2020) 14:12 between the two loading points is 1200 mm. After the certain level of load is applied, the beams are heated on three surfaces of the specimens (one bottom and two side surfaces) according to the time-temperature heating curve developed by the International Standard Organization (ISO), as illustrated in Fig. 2. Temperatures inside the beam are obtained from thermocouples located at a bottom corner of a cross section (which is 40 mm away from the both bottom and side heated surfaces). In order to see if the temperature varies along the beam length, two thermocouples are located at mid span and 1/4 span (as shown in Fig. 3). After the fire tests, maximum load bearing capacities as well as stiffnesses of the fire damaged beams are obtained from the residual strength tests.
Loading points and support conditions for the residual strength test are the same as those used in the fire tests.    Page 4 of 12 Ryu et al. Int J Concr Struct Mater (2020) 14:12

Image Data Acquisition and Pre-processing
Immediately after the fire test, surface images of the fire damaged beams are taken by digital camera. First, the side surfaces of fire damaged beams are divided into 10 zones along the length. Then, the camera takes pictures of each zone while moving in a sideways direction on a tripod which is kept at a distance of 2 m from the beam surface. Because the CNNs process the concrete images patch by patch, each zone is divided into a number of image patches (as shown in Fig. 4). The resolution of the image patch is set to 540 × 540 pixels and corresponds to an area of 50 × 50 mm 2 on the concrete beam. In other words, a size of one pixel becomes 0.09 × 0.09 mm 2 . It is noted that the crack detection algorithm can analyze an image patch at high resolution of about 0.6 mm pixel size and the presented algorithm uses higher precision in the edge analysis. Some outliers are removed if the image patch contained undesired artifacts such as paint. To confirm that the proposed CNNs detect cracks properly, surface cracks are also drawn manually from the imported surface images through CAD and the total lengths of the surface cracks are obtained.

Proposed CNN Method
A two-streamed CNN architecture is introduced in the authors' previous work ) and developed to detect the edges with high-fidelity in the fire damaged RC beams considering threshold values. Since the pixels of the image have wide range of intensity values from 0 to 255, threshold is used to set criteria of pixel intensity to be recognized as cracks. After the input concrete image passes through the CNN model, the output image, called the fire-damage-prediction-image (FDPI), is generated at the end of the network. The FDPI is supposed to include important cracks used for safety evaluation. The proposed CNN uses Convolution-Deconvolution architecture as shown in Fig. 5. Specifically, the series of convolution layers colored blue in Fig. 5 are used to extract the trained image features reflecting the length, thickness, or overall shapes used for distinguishing real cracks from noise. The convolution layers find whether the primitive image features that actively respond to cracks reside in the concrete image. The pooling layers colored with light blue perform down-sampling of the input feature maps so that the same filter kernel can see larger scopes and reduce the complexity. The convolution and the pooling layers provide output latent vectors. Meanwhile, the deconvolution layers (colored orange) reconstruct the region-of-interest from the feature maps and  Page 5 of 12 Ryu et al. Int J Concr Struct Mater (2020) 14:12 the latent vectors. Un-pooling layers increase the size of the feature maps to that of the original image (as opposed to the pooling layers). The model presented here differs from previous CNNbased edge detection algorithms (Cha et al. 2017;Xie and Tu 2017) in that the proposed network uses twostreamed CNNs consisting of so-called a weighting feature network and a low-level feature network. The motivation for these architectural changes is to provide both abstract and primitive information in crack detection. In CNNs, the convolution layers train hierarchical features with respect to the different levels. The layers closer to the input layer provide more primitive image representations such as lines and blobs. In contrast, the layers closer to the output layer provide more abstract representations. The low-level features can be efficiently used for edge detection algorithms (Cha et al. 2017;Xie and Tu 2017) because the cracks are formed with composites of lines and other low-level features. However, the features can be too sensitive to noise or some color changes (especially when the RC beams suffer from firedamage). To alleviate these problems, our model incorporates a weighting feature network to find a wider range of cracks using the convolution-deconvolution structure. The weighting feature network produces high pixel intensities in an image to indicate an important region of cracks. At the end of the network, element-wise multiplication is used to combine the two feature maps from both the weighting feature network and the low-level feature network. This is so that the FDPI can capture only the critical cracks that can play a key role in the safety evaluation. It is also shown in Fig. 5 that the low-level feature network uses a skipping path with elementwise addition to deliver the low-level features to the deconvolution layers so that the information is not lost in the end of the network. The method is motivated by previous studies to achieve precise localization with CNNs (Ronneberger et al. 2015).

Training and Testing
The proposed network model is implemented using Caffe software framework based on C++ (Jia et al. 2014). Training and testing are performed with a NVIDIA Geforce GTX 1080 graphic processing unit (GPU), Intel@3.5GHzX8 CPU, and 32 GB memory. Data sets of ~ 88,000 natural images and ~ 20,000 RC beam images are used for training the parameters (Li et al. 2014;Yan et al. 2013). Our model is first initialized using pre-trained parameters from Imagenet datasets (Russakovsky et al. 2015) and further trained with other natural images such as VOC (Everingham et al. 2015) and BSDS (Martin et al. 2001) datasets, which are widely used for edge detection and saliency detection. Afterwards, the CNN model is fine-tuned using crack images. Especially, the images of RC beams are prepared by dividing surface images of various fire damaged beams into small patches with size of 50 × 50mm 2 . However, the annotation per-pixel of all the training images requires a lot of human labors and costs. Therefore, this study uses semisupervised learning with pseudo-labels that have been effectively used in a saliency detection problem (Li et al. 2018a). The training procedure is shown in detail; in the first round, input of the training samples are fed into the saliency detection network and some of the output samples are chosen if they display results close to the actual cracks. Page 6 of 12 Ryu et al. Int J Concr Struct Mater (2020)  For training, a cost function J is defined as the mean squared error function between the predicted edge image and the original edge (ground truth) as follows: where O is the ground truth of FDPI, H w is the network function, w is the network parameter set, and I is the input concrete RC image. The loptimal network parameter set w * are trained to minimize the cost function, given as: The optimization in (2)  where ξ is the learning rate. The learning rate is set to 10 −2 , which is kept constant during the training. The training is stopped after 10K iterations. In the finetuning, the learning rate is the same, but the training is stopped after 5K iterations.
(2) w * = arg min J (w) (3) w n = w n−1 + ξ ∂ ∂w J (w) Fig. 6 Examples of input crack images in the first column and the output images from weighting feature network, low-level feature network, and the final output in the second, third, and fourth column, respectively ).

Fig. 7
Result of the proposed CNN method for specimen P1-60.  Figure 6 reveals the representative results from the weighting feature network and the low-level feature network when the two different types of input images are given. The first column represents the original crack images. The second, third, and fourth columns represent the corresponding outputs of the weighting feature network, the low-level feature network, and the final combined network, respectively. The weighting feature network captures the region where the important cracks exist, which can be seen in the second column. As compared to the first row, the second row rejects most of the regions whose pixel intensities in the weighting feature network are nearly zero. This observation confirms that there are no distinct cracks in the second row. For the outputs shown in the third column, the low-level feature network detects the fine edge information that has been not captured in the weighting feature network. Combining both images in the last column, the FDPI shows the important crack information in final output. Figure 7 is a black-white inverted image of P1-60 specimen obtained from the proposed CNN model, showing detected cracks and edges of pore in black lines. In order to validate the crack detection method, surface cracks on RC beams are drawn using AutoCAD (Autodesk Inc, San Rafael, CA) from optical observation as illustrated in Fig. 8a-e. The surface cracks are dispersed over the heated surfaces and observed more  Page 8 of 12 Ryu et al. Int J Concr Struct Mater (2020) 14:12 in the lower part than in the upper part. Compared to Fig. 8a, crack lines detected from Fig. 7 are similar to those obtained from optical observation. For the more quantitative comparison, total lengths of the cracks obtained from the optical observations are calculated using AutoCAD as listed in Table 4. As shown in the table, the number of pixels of the crack parts obtained from the proposed CNN model is compared with the total crack lengths obtained from optical observation. Since the CNN method gives output in a form of number of pixels, direct comparison between the results from the CNN model and the optical observation is not possible. One may insist that the area of cracked part can be calculated by multiplying the number of pixels with the area of one pixel. Then the crack length obtained from the optical observation also need to be transferred into the area of cracked part by multiplying with the crack width. From an image patch of 50 × 50 mm 2 of the fire damaged beam surface (Fig. 9), it is found that the widths of the cracks are widely varied from 0.1 to 0.9 mm. Due to its wide variation and large number of cracks, it would be time consuming to measure width of each crack manually and is difficult to average them properly. Instead, this study investigates the ratio of the number of pixels obtained from the CNN model to the crack length obtained from the optical observation, in order to see if consistency of the ratios can be found. It is interesting to note that the ratios in Table 4 are almost same among the specimens having different variables. This tells that the proposed CNN method recognizes cracks of the fire damaged concrete beams from the surface images and follows similar changing tendencies of total crack lengths obtained from the optical observations. Considering that the number of pixels for the cracked part corresponds to approximately 15-17% of the heated surfaces, the proposed CNN method likely recognizes the more area as cracked than the actual cracks. This is because the proposed CNN still detects pore edges and recognizes as cracks regardless of using weighting network, when the pores are relatively large and clear as shown in Fig. 10a, b. Since threshold is often used to remove such noises, Table 5 includes test results of the CNN model with various threshold values of 0, 100,   Page 9 of 12 Ryu et al. Int J Concr Struct Mater (2020) 14:12 180, and 200 in order to see if the threshold is capable of excluding detection of pore edges. Implementation of threshold drastically reduces the number of pixels as shown in Table 5. Specifically, the number of pixels decreases by almost 60% as the threshold value increases from 100 to 200, which shows that the implementation of threshold is an effective method to reduce the number of pixels by excluding detection of pore edges. However, Table 5 also shows that orders of specimens based on the number of pixels change, when the threshold is used. This is because the implementation of threshold excludes detection of some micro cracks dispersed on the damaged beam surfaces as well as detection of pore edges. Implementation of threshold can rather be suggested when the concrete surface has relatively large major cracks with minor number of noises. Instead, this study finds number of pixels for pore edges by using the proposed CNN model from the images of unheated concrete surfaces. Then, number of pixels for pore edges is subtracted from the number of pixels recognized as cracks from the proposed CNN model. This method allows excluding detection of initial defects such as noises and pores without using thresholds improves accuracy of the crack detection results of the proposed CNN model. Table 6 lists numbers of pixels excluding pore edges, which show similar trends as the total crack lengths obtained from the optical observations. Moreover, the area of detected cracks from the proposed CNN model reduces as 4.7-7.4% of the heated surfaces. The proposed CNN model needs about 6.3 min of computational loads to detect cracks and store the results as shown in Fig. 7, while it takes days for one person to manually draw crack lines of the fire damaged beam surface (i.e. Fig. 8a) from optical observation. Even considering time to take photos of the fire damaged beam surfaces for image processing, the proposed CNN model is able to save time significantly to detect cracks of the fire damaged concrete beams. Other advantages of using the proposed CNN model are; the model gives objective results, and does not require expensive inspection tools or man power but commonly used digital camera and a computer.

Discussion
This section discusses the numerical correlation between the cracks and thermal-structural behaviors of the fire damaged beams in order to see if the detected crack information can be used as one of the fire damage evaluation indices. The hypothesis of choosing crack as fire damage evaluation index is that even if both loading and fire cause cracks in the beam, cracks occurred by loading accelerate heat propagation through the cracks and the accelerated heat propagation eventually causes increase of temperature as well as thermally induced cracks. First, the relationship between the crack lengths of localized areas and the concrete temperature of the corresponding locations is illustrated in Fig. 11. As shown, the crack length and the concrete temperature at localized area are not directly matched because the cracks in the localized area cannot represent overall behaviors of fire damaged beams, whereas the overall behaviors are highly influenced by temperature exposure. Instead, Fig. 12 shows that total crack lengths of fire damaged beams have much better correlation as R 2 = 0.96 with the concrete temperatures measured at mid span. Therefore, it can be concluded that total crack lengths of the beam surfaces are highly influenced by the concrete temperatures. Likely, the number of pixels detected using the proposed  Page 10 of 12 Ryu et al. Int J Concr Struct Mater (2020) 14:12 CNN has a strong correlation to the concrete temperature as R 2 = 0.91, as shown in Fig. 13. It is found that R 2 value for temperature-total crack length curve (Fig. 12) is slightly higher than R 2 value for temperature-the number of pixels curve (Fig. 13). The difference of R 2 values may be because of the proposed CNN model tends to recognize all edges of micro cracks dispersed on concrete surfaces as fire induced cracks, while only major cracks are counted from the optical observation. Nonetheless, it can be said that the numbers of pixels obtained from the proposed CNN model are correlated with temperatures as good as obtained from the optical observation, because both R 2 values are higher than 0.9. From the residual strength tests, maximum load bearing capacities and initial stiffness values (Sullivan et al. 2004;Ryu et al. 2018) of the fire damaged RC beams are also obtained and compared with the number of pixels as listed in Table 7. It is interesting to note that the differences in maximum loads between the fire damaged specimens are small. For example, specimen P1-120 shows the lowest maximum load among the specimens but the maximum difference is only about 5% compared to the P2-60 specimen. This is because the maximum load bearing capacities of the beams are highly governed by the reinforcing steel bars and the strength of the steel bars does not show significant decrease at the temperatures under 500 °C. Rather than the maximum loads, the differences of the initial stiffness of the fire damaged RC beams having different fire exposure time and sustained load level are relatively large due to damages of the concrete. Therefore, this study investigates the relationship between the number of pixels and stiffness of fire damaged beams from Fig. 14. The figure shows that there is an obvious decrease of stiffness with increase of number of pixels with high correlation as R 2 = 0.91, and the number of pixels can also be related to structural behaviors of fire damaged beams.
The detailed results depending on the test variables can be found from Table 7. Specifically, the number of crack pixels of fire damaged beams tends to increase with the fire exposure time. The number of crack pixels is 11,683,021 for the P1 specimen heated for 1 h, while the number of crack pixels is 17,638,559 for P1 specimen heated for 2 h. In the case of specimens loaded with 60% of their nominal moments, the numbers of crack pixels are 13,619,863 and 18,303,122 for P2-60