An Overview of Challenges Associated with Automatic Detection of Concrete Cracks in the Presence of Shadows

Abstract

Detection and assessment of cracks in civil engineering structures such as roads, bridges, dams and pipelines are crucial tasks for maintaining the safety and cost-effectiveness of those concrete structures. With the recent advances in machine learning, the development of ANN- and CNN-based algorithms has become a popular approach for the automated detection and identification of concrete cracks. However, most of the proposed models are trained on images taken in ideal conditions and are only capable of achieving high accuracy when applied to the concrete images devoid of irregular illumination conditions, shadows, shading, blemishes, etc. An overview of challenges related to the automatic detection of concrete cracks in the presence of shadows is presented in this paper. In particular, difficulties associated with the application of deep learning-based methods for the classification of concrete images with shadows are demonstrated. Moreover, the limitations of the shadow removal techniques for the improvement of the crack detection accuracy are discussed.

1. Introduction

Concrete structures such as roads, bridges, beams and pillars are often subjected to high levels of stress and strain both in onshore and offshore environments. The stress in the concrete structures is caused by continuous loading and unloading; changes in temperature; and effects of weathering, which could result in the propagation of cracks [1,2]. Early detection and assessment of such failures are crucial for ensuring the safety and cost-effective maintenance of these concrete structures.

Crack detection could be done through invasive and non-invasive techniques. Invasive techniques are often time-consuming and expensive and usually involve specialized tools and methods such as infrared light, thermal, ultrasonic testing and testing of concrete samples in the laboratory environment. All the invasive and non-invasive methods require structural experts to analyze and interpret the data [3]. The findings of such methodologies are often subjected to human interpretation and knowledge. With the advent of improved imaging capabilities and increased computational power, non-invasive techniques that utilize image analysis of the concrete structures to investigate the presence and propagation of cracks have gained a lot of momentum. In the last few decades, more than 50 papers tried to address these problems using contemporary image analysis algorithms coupled with different pre-processing and post-processing techniques. A comprehensive report of these methods focused on advantages and disadvantages has been published in [4].

Some recent techniques use conventional pre-processing schemes for crack identification in rough concrete surfaces with a complex pattern of cracks. The crack identification technique proposed in [5] is based on an adaptive PDE-based pre-processing algorithm for concrete crack images. The implementation of this technique is combined with a local maximum-based thresholding scheme. The proposed approach based on automated operation, selective sharpening, and smoothing leads to better crack detection. Some images containing shadows in [5] Figure 8 indicate that illumination correction by the illumination-reflectance model to normalize non-uniform lightning is capable of identifying smooth shadows (images 5 and 6 on the last row) but has problems with high intensity shadows because the outlines of high intensity shadows are identified as cracks (images 2 and 3 on the last row). Another recent study [6] tries to solve the crack identification problem by proposing effective pre-processing algorithms for the enhancement and analysis of digital images of concrete surfaces. This technique works well with nonuniform lightning; however, it would be interesting to further explore if this approach is robust to shadows of variable intensity and shape. It is also shown that the proposed CIEAD-2 technique might not perform well when surfaces are very rough—the segmented image is contaminated with noise. However, this problem is solved with the improved CIEAD-2 technique. Yet another study [7] proposes automated crack segmentation via saturation channel thresholding, area classification, and a fusion of a modified-level set segmentation with Canny edge detection. It is demonstrated that this approach is able to identify cracks with nonuniform lightning (Figure 16, 3rd row, [7]). However, its applicability for images contaminated by high intensity optical shadows is not investigated. Moreover, the segmentation of images with rough surfaces and uniform illumination might not provide robust results in some cases—rough regions in the background sometimes can be identified as micro-cracks (Figure 18, part g, 2nd row, [7]). However, this problem might be solved by fine tuning parameters of the proposed concrete crack processing algorithm. This technique as well as techniques in [5,6] has a benefit that it is not as computationally and structurally complex as methods based on artificial neural networks. However, the discussed approaches are not robust enough in some special cases when images are extremely noisy, or when the concrete surfaces are very rough and contain complex crack patterns intertwined with high-intensity shadows.

More recently, deep-learning-based methods, using ANN- and CNN-type neural networks, have been applied to automatically process the images for crack/failure identification in concrete onshore installations [8,9,10,11,12]. With the development of artificial intelligence and deep learning technologies, several applications, based on the use of CNN-type neural networks, for the identification of crack on concrete surfaces, have emerged. A detailed summary of such studies is presented in Table 1. It can be noted that many of these methods could be characterized by a high classification accuracy. Application of the deep learning methodology is a relatively new approach as it could be seen from the list of publications provided in Table 1. Most of the published data spans from a period ranging from 2011 to 2021. A chronological summary of such studies is presented further.

In [13], the authors use the back-propagation neural network that is trained on 105 images of various concrete structures, and the trained network is tested on 120 new images. The recognition rate of the images with and without cracks is $90\%$ and $92\%$, respectively. A methodology for crack detection and identification using the object detection method as well as fuzzy logic and ANN-based models is published in [14]. In this study, the authors utilize 205 images of 256 × 256 resolution and claim a very high crack detection accuracy of $96\%$. It is important to note, however, that a very limited number of images is used in both of the previously mentioned studies. An image segmentation method for the automatic peak detection enabling crack identification is presented in [15]. A review of over 50 publications related to the application of image analysis for crack detection is published in [4].

First extensive dataset of over 40,000 images of 256 × 256 resolution is utilized in [9]. The authors apply a CNN-based deep-learning method for crack detection and achieve very high accuracy of over $98\%$. In [16,17], CNN-based methods are applied to the same set of 40,000 images of 227 × 227 resolution, both claiming good prediction accuracy of over $99\%$ and $90\%$ respectively. Differently from the previous examples, in [18], the authors have used 60,000 images of a very high resolution (3120 × 4160), for training and testing of the applied CNN-based crack identification method with a very high prediction accuracy of about $99\%$.

The authors of [8] use the same dataset presented in [9] and further improve the accuracy of the CNN to $99.9\%$. A comparison of different deep-learning networks is performed in [19,20] and uses over 20,000 images of higher resolution of about 1024 × 1024. The authors of [21,22] also apply CNN for crack detection and present a comparison of the new CNN-based method with R-CNN and U-Net learning architectures. However, the achieved accuracy is rather low (approximately $82.5\%$). A novel approach for crack detection and identification utilizing long and short-term-memory-based deep-learning convolutional neural networks (CNN-LSTM) is presented in [23].

Table 1. The overview of image processing methods and application of neural networks for crack identification in concrete structures.

Paper	Feature Enhancement/ Pre-Processing	Classification of Fracture Dimensions	Type of Neural Network	Open-Source	Image Resolution	Filter Size	Filter Type	Images in the Dataset	Accuracy, %
[24]	N	N	CNN	VGG16	256 × 256	5 × 5		3500	92.3
[8]	Y	Y	CNN	N/A	227 × 227			40,000	99.9
[14]	Y	Y	ANN vs. Fuzzy	N/A	256 × 256		Object Detection	205	96.1
[16]			CNN vs. SVM vs. Boost	ConvNet	227 × 227			40,000	99.7
[9]	Y		CNN	N/A	256 × 256			40,000	98
[15]	Y	Image Segmentation Approach	No	N/A			Automatic Peak Detection	N/A	N/A
[19]	N	N	KNN vs. Decsion Tree vs. SVM vs. Ranorest		227 × 227			20,000
[4]	Y	Y	No (Review Paper)	N/A	N/A		N/A	N/A	N/A
[20]	Y	N	CNN	AlexNet, LeNet-5, CIFAR10	1024 × 1024	32 × 32			94
[21]	Y	N	CNN	N/A	N/A	N/A	ASPP	N/A	96.7
[23]	N/A	N/A	CNN-LSTM	N/A	N/A	N/A	N/A	N/A
[22]	N/A	N/A	New CNN vs. R-CNN vs. U-Net	N/A	576 × 576	N/A	N/A	N/A	82.5
[17]	N	N	FCN	VGG16	227 × 227			40,000	90
[18]	N/A	N/A	CNN	AlexNet	3120 × 4160	N/A		60,000	99.06
[13]	Y	N	Backpropagation NN	N/A	N/A	N/A	N/A	225	92
[25]	N	Y	R-CNN					376
[11]	N/A	N	U-Net	N/A	N/A	N/A	N/A	N/A	78.12
[26]			AlexNet, VGGNet13, and ResNet18		227 × 227			10,000
[27]	N	N	CNN	N/A	16 × 16	N/A	N/A	64,000	93
[28]	Y	N	SVM	N/A	N/A	N/A	N/A	N/A	94
[29]	N	N	ANN	N/A	N/A	N/A	N/A	N/A	N/A

Table 1. The overview of image processing methods and application of neural networks for crack identification in concrete structures.

Paper	Feature Enhancement/ Pre-Processing	Classification of Fracture Dimensions	Type of Neural Network	Open-Source	Image Resolution	Filter Size	Filter Type	Images in the Dataset	Accuracy, %
[24]	N	N	CNN	VGG16	256 × 256	5 × 5		3500	92.3
[8]	Y	Y	CNN	N/A	227 × 227			40,000	99.9
[14]	Y	Y	ANN vs. Fuzzy	N/A	256 × 256		Object Detection	205	96.1
[16]			CNN vs. SVM vs. Boost	ConvNet	227 × 227			40,000	99.7
[9]	Y		CNN	N/A	256 × 256			40,000	98
[15]	Y	Image Segmentation Approach	No	N/A			Automatic Peak Detection	N/A	N/A
[19]	N	N	KNN vs. Decsion Tree vs. SVM vs. Ranorest		227 × 227			20,000
[4]	Y	Y	No (Review Paper)	N/A	N/A		N/A	N/A	N/A
[20]	Y	N	CNN	AlexNet, LeNet-5, CIFAR10	1024 × 1024	32 × 32			94
[21]	Y	N	CNN	N/A	N/A	N/A	ASPP	N/A	96.7
[23]	N/A	N/A	CNN-LSTM	N/A	N/A	N/A	N/A	N/A
[22]	N/A	N/A	New CNN vs. R-CNN vs. U-Net	N/A	576 × 576	N/A	N/A	N/A	82.5
[17]	N	N	FCN	VGG16	227 × 227			40,000	90
[18]	N/A	N/A	CNN	AlexNet	3120 × 4160	N/A		60,000	99.06
[13]	Y	N	Backpropagation NN	N/A	N/A	N/A	N/A	225	92
[25]	N	Y	R-CNN					376
[11]	N/A	N	U-Net	N/A	N/A	N/A	N/A	N/A	78.12
[26]			AlexNet, VGGNet13, and ResNet18		227 × 227			10,000
[27]	N	N	CNN	N/A	16 × 16	N/A	N/A	64,000	93
[28]	Y	N	SVM	N/A	N/A	N/A	N/A	N/A	94
[29]	N	N	ANN	N/A	N/A	N/A	N/A	N/A	N/A

Thermal image-based crack identification and detection is performed for the first time using a U-Net type learning network in [11]. The prediction accuracy of U-Net using thermal images is rather low—approximately $78\%$. In [26], the authors show a comparison of three neural networks, namely, AlexNet [30], VGGNet13 and ResNet18 for the detection and classification of crack images. The tests indicate that the ResNet18 model generates the most satisfactory results. Moreover, it is shown that the trained YOLOv3 model detects the crack area with a satisfactory accuracy.

Complete turn-key solutions and frameworks for crack detection from images of various scenes have been developed recently. One such solution is DeepCrack, an end-to-end trainable deep convolutional neural network for automatic crack detection by learning high-level features for crack representation [31,32]. This architecture comprises extended fully convolutional networks (FCN) and deeply supervised nets (DSN). The model learns and aggregates multi-scale and multi-level features from the low-level convolutional layers to the high-level convolutional layers during the training of the model. Direct supervision for features of each convolutional stage is provided by DSN. On one hand, the results of DeepCrack are very promising—the network is able to identify cracks in different scenes. However, some testing samples such as Figure 10 in [31] indicate that in some cases, this architecture has problems with misidentifying shadows as cracks.

It is clear that precise image acquisition techniques are crucial for successful further processing of these images. Employment of different lighting and illumination conditions such as shown in [33] might increase the success of further identification of cracks in images. However, in many cases this kind of laboratory-based experimental setup is not feasible and fine-tuning of illumination conditions is not possible. Moreover, in some cases acquired images might contain shadows. The detection and identification of concrete cracks in the presence of shadows on the analyzed images remains a challenging task.

Appearance of shadows results in various challenges in the detection of objects. Thus, multiple methods for shadow detection and removal were developed over the years. Early papers on shadow detection and removal have been presented in [34,35]. A comprehensive survey of various shadow detection and removal methods published in literature could be found in [36]. More recently, artificial-neural-network-based deep-learning methods have also been deployed for the detection and removal of shadows [37,38,39]. Although several researchers have tried to solve the problem of shadow removal, it remains a complex topic with a moderate to good success rate [36]. The shadow removal process might have a severe impact on the quality of the analyzed image. That can make further object detection even more challenging than before the removal of the shadow. This paper discusses challenges related to the identification of cracks on concrete surfaces in presence of shadows.

This paper is organized as follows: the introduction and a detailed literature review of the different concrete-crack detection methods and shadow detection and removal approaches is presented in Section 1. concrete-crack detection challenges associated with real-time images are discussed in Section 2. Difficulties related to the detection and removal of shadows from real-time images are outlined in Section 3. Finally, discussion and conclusion are presented in Section 4.

2. The Difficulties of Automatic Concrete-Crack Detection in the Presence of Shadow Effects

It is important to note that real-time concrete images are often impacted by illumination conditions, randomness, irregular sizes of crack shapes and various types of noise such as shadows, shading, blemishes, concrete spall, etc. Thus, the application of the published automatic concrete-crack detection methodologies (see Table 1) on real-world images of concrete structures in the presence of the aforementioned effects often yields poor results. In this section, several of the published deep-learning methods (see Table 1) for crack detection are applied in order to demonstrate the difficulties associated with the presence of shadow effects in the concrete images.

Firstly, the training and testing datasets are selected as follows:

• Training dataset—the dataset published in [40], consisting of 40,000 concrete images (227 × 227 pixels with RGB channels) with and without cracks. As mentioned above, this database has already been used by several researchers for the development of deep-learning methods for concrete-crack detection (see Table 1). Sample images of each type are displayed in the Figure 1. Note that no shadow effects are present in the images included in this dataset.
• Testing dataset—the dataset comprising 500 concrete images (227 × 227 pixels with RGB channels) with and without cracks with the shadow effects present in all images. Samples images of each type are depicted in the Figure 2.

Next, three different classification neural networks, namely, AlexNet [30], SqueezeNet [41] and VGG16 [42], are selected for further testing. These networks are chosen based on their performance in terms of speed versus prediction accuracy (see Figure 3). It can be seen from Figure 3 that AlexNet and SqueezeNet can generally be characterized by high prediction speed and lower accuracy, whereas VGG16 is usually slower in prediction but can achieve higher accuracy. In order to further benchmark the speed of those networks on the data analyzed in this paper, all three of the above are trained and tested on the training dataset, yielding the results displayed in

Table 2. Results of the prediction speed testing for the three neural networks: AlexNet, SqueezeNet and VGG16 trained and tested on the training dataset using the computer with $50\%$ GPU and $50\%$ CPU distribution.

Neural Network	Prediction Time (in min)
AlexNet	1490
SqueezeNet	315
VGG16	10,800

.

Finally, the three classification neural networks are trained on the training dataset using the transfer learning approach presented by [43] and tested on the concrete images with shadows. A sample workflow is shown in the schematic diagram in Figure 4.

Naturally, all three classification networks trained on concrete images devoid of any information on shadow shapes yield poor prediction accuracy of approximately 50%. This is in stark contrast to the accuracy of approximately 100%, which can be achieved when both training and testing are performed on images without shadows (see Table 1). Several examples of the wrong classification of concrete images with shadows are displayed in the Figure 5 and Figure 6. From the Grad-CAM map displayed in the Figure 6b, it can be observed that the neural network trained on concrete images without shadows is unable to detect cracks in the presence of shadows.

It is clear that training the network on datasets without shadows and testing them by using concrete crack images with shadows is not fair. One solution would be to train the network using concrete crack images with shadows; however, publicly available databases such as Mendeley Concrete Crack Images for Classification lack images with shadows and manual creation of such databases takes a lot of effort. An approach that could potentially improve the prediction accuracy for the images with shadows could be based on pre-processing the concrete images via shadow removal techniques. However, the application of such techniques is also associated with certain challenges, which are discussed in the next section.

3. The Challenges Associated with the Application of Shadow Removal Techniques

As mentioned earlier, using shadow removal methods before the application of deep-learning techniques as a means to improve the classification of concrete images is not straightforward. The main difficulty of such approach is related to the fact that shadow removal can often cause serious deterioration of the image quality. The original concrete images with and without cracks are shown in Figure 7. All images are captured in the presence of different shadow pattern.

Firstly, a methodology proposed in [34] is applied to two concrete images from the testing dataset described earlier. The main idea of this technique is as follows: an illumination invariant shadow-free image is derived and utilized to detect the edges of the shadow, which are then used to remove the shadow from the original image. The results of this procedure are displayed in the Figure 7b. It can be seen that the quality of the obtained images is severely compromised. Further image processing related to the detection and/or identification of concrete cracks on such an image could be very challenging. The results are not that bad in images 2, 3 and 7; however, a method completely fails with ghost-free shadow removal in images 1, 4–6, and 8. A similar approach—the method of shadow detection and removal using mean and Gaussian filtering outlined in [34,36]—is applied with the results depicted in Figure 7c. Again, it can be seen that the quality of the produced images is very poor and similar to Figure 7b. Furthermore, on one of obtained images the concrete crack is barely visible which would doubtlessly have a negative impact on the accuracy of the concrete image classification methods. Another color-model based shadow detection and removal technique, presented in [44], is applied as follows. Original RGB images are converted into a LAB color model, and the shadows are detected on the basis of the obtained values in the A and B planes. The shadows are then removed by multiplying the R, G and B values corresponding to each shadow region by appropriate constants. The resulting images are displayed in the Figure 7d. Once more, the obtained images are far from an accurate representation of the original concrete images—images are not ghost-free and cracks are removed in places.

Various conventional schemes such as homomorphic filtering and retinex, as well as other variational approaches, have been successfully applied to illumination correction for images with shadows, where pre-processing is preferred to extensive shadow removal using highly complex methods. As shown in [49], homomorphic filter algorithm has been developed for removing shadows through the analysis of the relationship between the shadow and the brightness of the image. Another approach, a methodology for shadow removal via the application of the non-local retinex model, is demonstrated in [45,46]. The main assumption of the retinex models is that the input image is a product of the illumination and the true reflectance of the object. As shown in [45,46], retinex formulations based on the application of non-local differential operators are particularly suitable for texture-preserving shadow removal. Application of this method for images with concrete cracks is shown in Figure 7e, and the results are not satisfactory. Shadows are not removed in images 2, 3 and 7; information of concrete texture is lost in images 4–8; and image 1 is almost black. A user-centric technique proposed in [47] is also tested for concrete crack images. The authors of [47] attempt to achieve a fast, robust and high-quality shadow removal by adopting an interactive approach where the user has to provide rough strokes to indicate the shadows and lit image areas. The penumbra is then registered onto a normalized frame in order to estimate the changes of non-uniform shadow illumination, which allows for efficient shadow removal. The results of this technique are shown in Figure 7f and are quite promising. There are still some shadow ghosts in images 2 and 8, as well as some crack features lost in 4 and 5, but overall performance is quite good compared to other techniques.

Recent advances in deep learning resulted in new shadow removal methods and techniques. A new solution to shadow removal task has been proposed in [50]. This approach is designed using the shadow-aware FusionNet that takes the shadow image as input to generate fusion weight maps across all the over-exposure images as well as boundary-aware RefineNet to further eliminate the remaining shadow trace. Another approach that also employs deep neural networks is proposed in [51,52]—the direction-aware attention mechanism in a spatial recurrent neural network (RNN) is formulated by introducing attention weights when aggregating spatial context features in the RNN. This enables the recovery of direction-aware spatial context (DSC) for the detection and removal of shadows. This design is developed into the DSC module and embedded in a convolutional neural network to learn the DSC features at different levels. The recent development of Generative Adversarial Networks (GANs) has also resulted in a promising shadow removal applications such as Mask-ShadowGAN, a new deep framework that automatically learns to produce a shadow mask from the input shadow image and then takes the mask to guide the shadow generation via re-formulated cycle-consistency constraints. This framework learns to produce shadow masks and to remove shadows at the same time, which results in increased overall performance. Another example of successful GAN application for shadow removal is based on techniques proposed in [48]. In order to achieve the ghost-free shadow removal, the authors of [48] apply the following techniques: (a) a novel network structure—dual hierarchical aggregation network—is proposed for shadow detection and removal, which hierarchically aggregates multicontext features for attention and prediction; (b) a shadow matting generative adversarial network is designed in order to produce realistic shadows mattings using shadow-free images and shadow masks, and the obtained images are utilized to enhance the training datasets. We tested this method for images with concrete cracks. This technique should enable ghost-free shadow removal; however, the results in Figure 7g show that all images do contain shadow ghosts. The results of images 1–6 are more or less acceptable; however, images 7–8 show that this technique fails completely.

All the above examples help to showcase that some shadow removal techniques can be successfully applied to certain images (as demonstrated in the publications addressed above). However, in some cases obtaining a shadow-free image is not robust and the produced images are not suitable for the further crack detection. This difficulty occurs due to the fact that in the concrete images, the shadow shapes are often overlaying the crack. That makes it difficult to differentiate between shadows and cracks and, in turn, to remove shadows without any impact on the cracks. Even with introduction of novel techniques based on artificial neural networks, shadow removal is still a challenging task due to its inherent background-dependent and spatial-variant properties. Even the state-of-the-art techniques based on deep neural networks sometimes might have difficulties when recovering traceless shadow-removed concrete images with and without cracks.

4. Discussion

In this paper, a brief review of deep-learning-based concrete-crack detection studies published in literature is presented. Difficulties associated with automatic concrete-crack detection in presence of shadows effects are highlighted, with help of a series of experiments, involving the application of three different classification neural networks (AlexNet, SqueezeNet and VGG16). The experiments demonstrate that the deep-learning methods trained on images acquired in laboratory conditions are incapable of accurate detection of cracks in the concrete images with shadows. Furthermore, it is shown that even the application of pre-processing techniques for shadow removal does not always improve the detection accuracy since it can lead to deterioration in image quality (which makes the resulting concrete images unsuitable for further image processing).

This paper highlights difficulties associated with the detection and identification of cracks in the real-life concrete images. This paper also highlights the limitations of the published studies, which are only utilizing the images taken in ideal conditions of onshore concrete structures in controlled illumination (laboratory) conditions. Results presented in this paper show that artificial neural networks trained on ideal laboratory images are only good enough to classify ideal images, and accuracy of classification fails significantly in presence of noises, particularly shadows. Both, conventional as well as deep-learning-based methods have some issues related to robust crack identification in extreme conditions. Some of these issues might be solved by fine-tuning the parameters of these methods in order to adapt them to a specific situation. The literature review and computational tests performed in this article show that one single technique capable of identifying concrete cracks when digital images are contaminated by extreme noise, or when concrete surfaces are rough and contain complex crack patterns with high intensity shadows, does not exist. This task gets even more challenging when the noise, the roughness of the surface, and the shadow intensity and its shape are variable in the processed dataset. In such a case, fine-tuning of the parameters of the existing techniques becomes a very challenging task.

In order to achieve full automation of the crack detection via the deep-learning approaches, with the use of unmanned aerial vehicles (UAV) or drones, the methodology must be robust enough to deal with real-life imaging affected by various illumination conditions, randomness, and irregular sizes, of crack shapes, visual noises and shadows. Challenges presented in this paper highlight the need for the development of a deep-learning framework that could perform seamless classification of concrete surface crack in presence of shadows and random visual noises. The challenges could be mitigated and the real time classification of concrete surfaces could be achieved by development of a deep learning network, which is trained using an exhaustive data base of more realistic concrete crack images.