Autonomous damage recognition in visual inspection of laminated composite structures using deep learning

This study proposes the exploitation of deep learning for quantitative assessment of visual detectability of different types of in ‐ service damage in laminated composite structures such as aircraft and wind turbine blades. A comprehensive image ‐ based data set is collected from the literature containing common microscale damage mechanisms (matrix cracking and ﬁ bre breakage) and macroscale damage mechanisms (impact and erosion). Then, automated classi ﬁ cation of the damage type and severity was done by pre ‐ trained version of AlexNet that is a stable convolutional neural network for image processing. Pre ‐ trained ResNet ‐ 50 and 5 other user ‐ de ﬁ ned convolutional neural networks were also used to evaluate the performance of AlexNet. The results demonstrated that employing AlexNet network, using the relatively small image dataset, provided the highest accuracy level (87% – 96%) for identifying the damage severity and types in a reasonable computational time. The generated knowledge and the collected image data in this paper will facilitate further research and development in the ﬁ eld of autonomous visual inspection of composite structures with the potential to signi ﬁ cantly reduce the costs, health & safety risks and downtime associated with integrity assessment.


Introduction
Composite materials have the advantages of high strength to weight ratio, good vibration damping ability, and high wear, creep, corrosion, fatigue and temperature resistances [1]. Due to these excellent properties, composite materials are wildly used in different sectors such as civil, aerospace, wind energy, oil & gas, automotive, etc. Despite the advantages, an important problem for composites is their susceptibility to damage that can result in fatigue life reduction or catastrophic failure if unseen [2][3][4]. Most polymeric composite materials have brittle and laminated nature, making them susceptible and sensitive to damage. As a result, in safety critical applications, engineers are forced to apply conservative design approaches based on low allowable strains. For example, maximum allowable design strains can be as low as 0.1% for carbon fibre composites, despite maximum fibre failure strains of up to 2% [5,6]. Different damage mechanisms can happen in composite components, ranging from microscopic matrix cracking and fibre breakage to large, and critical impact damage [7,8]. These damage mechanisms can be induced by operational loadings during service or unwanted events during manufacturing and assembly. Fig. 1 shows examples of in-service surface damage in composite structures. Among these surface damage mechanisms, impact damage is very common for the aerospace industry, whereas erosion of the leading edge is observed frequently in composite wind turbine blades. These in-service damage mechanisms are likely to contain different microscale damage mechanisms such as fibre breakage, matrix cracking, and delamination [9,10]. Fig. 2 shows a schematic of these microscale damage mechanisms for a laminated composite under low velocity impact.
If a composite material component is damaged, the size, shape, depth, type, and extent of the damage and its restitution approach should be determined. A typical repair procedure and an example of barely visible impact damage repair in a laminated composite is shown in Fig. 3.
Of immediate importance for the composite integrity and serviceability evaluation is the ability to identify the damage and measure its extent by an appropriate non-destructive inspection (NDI) technique. Several NDI techniques are used in the composite field, including visual testing or visual inspection [12], optical testing [13], ultrasonic testing [14], acoustic emission testing [15], thermographic testing [16], infrared thermography testing [17], radiographic testing [18], acousto-ultrasonic [19], shearography testing [20], electromagnetic testing [21], etc. Most, if not all, of these NDI techniques require high levels of operator experience to successfully apply and interpret the results. These NDI techniques are usually expensive, timeconsuming, and sophisticated, and the component has to be out of service for the inspection; thus causing further inconvenience. These precautions reduce the inherent performance advantages of composites and even make them unsuitable for many applications in which catastrophic failure cannot be tolerated. As a result, there is a need for cost-effective and reliable inspection solutions to ensure safety, reliability, and longer service life of composite structures.
Visual inspection is the main method of routine inspection for different composite structures in aircraft, wind turbine blades, and many other sectors [12]. It is considered the quickest, cheapest, and most common method to find cracks or surface dents, and it can reduce the need for a full scan by other expensive and complicated NDI techniques, or in some cases, it can reduce the need for other types of NDI if no critical damage is revealed. If a visual inspection reveals critical damage to a composite structure, inspectors may request nondestructive testing such as Ultrasonic to determine the extent of the associated subsurface damage to determine the need for repair or replacement. For instance, over 80 percent of inspections on large aircraft are visual inspections, rope access visual inspection of composite wind turbine blades is also the most common inspection practice [22]. Therefore, visual inspection is the most used and least expensive and quickest method for assessing the condition of safety-related failures on critical composite structures. Consequently, reliable, and accurate visual inspection is vital to the continued safe operation of composite structures. Currently, visual inspection is mainly done by skilled operators, so the accuracy depends on the operator and there are health and safety risks. There are plenty of factors such as lighting, inspection time, inspector tiredness and experience, and environmental conditions which influence the reliability of visual inspection and probability of detection [23,24].   Advances in automation [25], data analytics [26,27], image acquisition techniques [28,29], artificial intelligence technologies [30,31], and computationally efficient smartphones, inexpensive highresolution cameras and drones [32], have recently enabled the capacity to build automated visual inspection systems that can surpass human accuracy. A schematic of recent advances that enables the potential of next-generation autonomous visual inspection systems in composite structures is shown in Fig. 4.
High-quality algorithms and high-quality data for training those algorithms are essential factors that need to be established to develop autonomous visual inspection systems in composite structures. In this paper, the efficiency of artificial neural network (ANN) algorithms is evaluated for identifying the damage types and severities in visual inspection of composite structures. ANNs are a subclass of semisupervised machine learning techniques, and they have been successfully used in several studies for damage classification of composite materials. Among various ANNs, convolutional neural networks (CNNs) have attracted high attention in effectively handling imagebased data due to their ability in extracting deep patterns. The ANNs also take the advantages of dataset augmentation and transfer learning that make it possible to train accurate models when limited data is available. Several CNN architectures have been proposed for image classification including AlexNet [33], FuseNet [34], ZF Net [35], and ResNet [36]. Saeed et al. [37] applied AlexNet CNN for thermography defect detection and depth estimation of 3D printed Carbon Fibre Reinforced Plastics (CFRPs), based on the pulsed thermography images taken from the samples with embedded air pockets. The proposed method identified embedded defects without any human interventions with high accuracy above 88%. Bang et al. [38] also employed CNN and transfer learning for the classification of thermographic images of carbon/epoxy composite specimens. They used Inception V2 architecture to identify the presence of the defects as well as their shapes (i.e. spheroidal, circular and irregularly shaped defects). Gong et al. [39] applied CNN for inclusion defect detection of aeronautics composite materials based on the X-ray images. According to the obtained results, the proposed CNN could accurately extract X-ray images features and detect the presence of the inclusion. There are some other studies towards the application of CNN in damage detection of composite materials which are based on non-image data generated from various inspection techniques, such as ultrasonic signals [40], structural vibration responses [41], lamb waves [42], distributed strains [43] and PZT sensors [44]. For example, Meng et al. [40] successfully used CNN for the classification of ultrasonic signals from CFRP samples to classify the voids and delamination defects.
In previous applications of CNNs on image-based data, the images were obtained through NDT techniques such as thermography and Xray [37][38][39] from defects that occur during the manufacturing of composite materials. These studies were only focused on single class detection models (i.e., the presence of defects or not), without consideration of damage types and severity. A thorough search of the relevant literature yielded that machine learning-based image processing has not been exploited in identification and classification of visually inspected in-service damage mechanisms in composite structures. Despite many research publications on in-service induced damage in composite structures, there is no comprehensive publication summarising different visible damage mechanisms on composite structures.
To address the aforementioned challenges, this paper introduces a novel exploitation of CNNs for quantitative assessment of image-based data taken from visual observation of different types of in-service damage in laminated composite structures. A comprehensive image-based data set of common in-service damage mechanisms (matrix cracking, fibre breakage, impact, and erosion) were collected from the literature. The data set was successfully used to train the CNNs to evaluate their accuracy and robustness in identifying the various in-service damage mechanisms and their severity (for example high energy or low energy impacts). Given the CNNs ability to detect different damage mechanisms on diverse material combinations, the introduced system can be implemented for a wide range of industries such as aerospace, wind, civil and oil & gas.

Convolutional neural network
Deep learning is a subset of machine learning that mimics the behaviour of the human brain in processing data by learning tasks directly from sound, text, and images. CNN is a type of deep learning, developed to automatically and adaptively process structured arrays of data [37,45]. CNN consists of an input layer, several hidden layers, and an output layer. The hidden layers themselves include convolutional layers, pooling layers, activation layers, fully connected layer, and Softmax classification layer. The convolution layers consist of a set of filters (with learnable weights), which are exerted on the input image to extract its main features. An example of convolution operation with a 3×3 filter, stride size of two and padding size of one is illustrated in Fig. 5. As depicted, the convolution operation convolves the input lay- Fig. 4. A schematic of recent advances that enables the potential of nextgeneration autonomous visual inspection systems. ers by sliding the filter through the input data horizontally and vertically, calculates the dot product of the weights and the input, and then adds a bias term. The step size of the filter movement is determined by the stride size. As shown in Fig. 5, the convolution operation is accompanied by a padding operation, which inserts additional layers to the image border. This operation leads to more accurate image analysis since it prevents data shrinkage and information loss in the image borders. Without padding, the input data progressively shrinks every time after the convolution operation. Also, the pixels in the image borders get covered (by the filters) only one time, while the filters continuously cover the middle pixels. This leads to the loss of information in the image borders. To overcome these problems, the application of padding operation is of great importance for accurate image classification. After convolution and padding operations, the activation layer adds some non-linearity to the network, since most of the realworld problems are non-linear [39]. For example, rectified linear unit (ReLU) activation function applies a threshold operation to each element, where any input value less than zero is set to zero. The pooling layer is then applied to progressively decrease the size of the layers (by performing the down-sampling operation), which leads to the reduction in the number of iterations, weights and consequently the computation cost. An example of pooling operation with a 2×2 filter and stride size of two is demonstrated in Fig. 6. Through these steps, the input image is converted to a high-level feature map which is further processed by the fully connected layer that connects every neuron in one layer to every neuron in another layer, as shown in Fig. 7. Finally, the Softmax layer is employed to classify the input images. In a typical CNN, high precision image classification requires a very large labelled dataset, with a massive amount of training data with different possible variations in size, orientation, number of objects, etc [46]. Hence, application of pre-trained models (transfer learning) is of great importance for efficient classification purposes. In transfer learning, the network has already been trained by a large dataset that includes various classes of objects (not essentially relevant to the specific target task). By fine tuning this pre-trained network, it can be employed as a starting point to learn a new task, in accordance with the classification goal. Fig. 8 shows flowchart of the damage classification process applied in this paper.

AlexNet
The superiority of AlexNet CNN architecture over others in exploitation of transfer learning for the classification of defects in CFRP thermograms [37] and satellite image data [47] has been demonstrated. AlexNet network is one of the most widely used CNN architectures that has been successfully trained on more than a million images [33]. AlexNet network can learn rich feature representations for various types of images, which eliminates the need for timeconsuming training of the network from the scratch. Furthermore, stable implementation of pre-trained version of AlexNet is developed in Matlab [48] that is used in this study. AlexNet architecture is illus-trated in Fig. 9, which contains five convolutional layers, three pooling layers, three fully connected layers and one Softmax layer. Fig. 10 summarises the collected image data for this study including un-damaged, impact damage, erosion, matrix cracking and fibre breakage. A comprehensive set of images were collected from the literature from laminated composite materials with different thicknesses, materials, layups, texture, etc. The low and high impact damage types were distinguished from each other visually, where the images with a significant visible fibre breakage were categorised as high energy. A dataset containing 20, 24, 16, 52, 25, 39, 28 and 24 images was collected for matrix cracking (Fig. 11), fibre breakage (Fig. 12), undamaged (Fig. 13), low energy impacted face (Fig. 14), high energy impacted face (Fig. 15), low energy back face (Fig. 16), high energy back face (Fig. 17), and erosion (Fig. 18), respectively. The impact and erosion related images were collected from the literature

Classification of microscale damage mechanisms
The performance of AlexNet network is assessed in the classification of microscale damage mechanisms such as matrix cracking and       fibre breakage. 75% of the dataset images were randomly selected for training purposes, and 25% were selected for validation purposes. First of all, the images were resized to meet the Alexnet input layer condition (i.e. image sizes of 227 × 227 × 3) using an augmented image datastore algorithm. Then, these images were used for training the deep learning network, using an initial learning rate of 0.0002. After training the network, its performance was evaluated by classification of the validation images. In this case, the best validation accuracy was 91%, obtained in the case of a learning rate of 5e-5, as shown in Fig. 19. Some samples of the validation images classified by the network are depicted in Fig. 20. Finally, the network was implemented for the classification of unseen images. As illustrated in Fig. 21, the network has successfully classified the unseen images with a high accuracy level, where all the six unseen images have been accurately classified.
In order to evaluate the performance of AlexNet network, it is compared with five other user-defined neural networks and Resnet-50 [77] that is an established image processing CNN and was pre-trained with the ImageNet database [78]. The architecture of the user-defined networks (i.e. the number of convolutional layers, number of pooling layers, number and size of filters, etc.) was determined based on the data available in the literature [37][38][39]41], as summarised in Table 1. The obtained results (i.e. the validation accuracy and CPU evaluation time) from classification of microscale damage mechanisms are listed in Table 2. It should be mentioned that all the computations were performed by MATLAB on Intel Core i7 CPU @ 1.6 GHz and RAM 4 GB. As illustrated in Table 2

Classification of macroscopic damage mechanisms
In this section, the performance of AlexNet network is illustrated in the classification of impact damage, erosion, and un-damaged samples. The severity of the impact damage (i.e. high energy and low energy impact damage) was also distinguished by the network. Some sample classification results of the damage severity for the impacted face are depicted in Fig. 25, and the obtained accuracy is shown in Fig. 26. As shown, the validation accuracy is 96%, achieved in the case of     the learning rate of 0.0001. It should be mentioned that the item marked by a red circle is wrongly classified by the network. This object belongs to low impact energy category, but it is classified as high impact energy, which is due to its similar features to high energy damage. Followed by the training and validation processes, AlexNet network was implemented for the classification of some unseen images, as illustrated in Fig. 27. The obtained results indicate the promising performance of the network for accurate classification of the damage severity for the impacted face. In the following, the network was used for the classification of the back face images. As illustrated in Figs. 28-30, yet again, the network could successfully classify the damage severity for the back face impact, with an accuracy level of 87%.
AlexNet network was also used to distinguish the various macroscopic damage mechanisms, i.e. impact, erosion, and un-damaged. As illustrated in Fig. 31, the best validation accuracy is 93%, achieved in the case of a learning rate of 0.0001. Some sample classification results of validation images and unseen images are depicted in Figs. 32 and 33. Again, the network demonstrated a promising performance and could accurately classify the various macroscopic damage mechanisms, as well as the un-damaged case.
Finally, AlexNet network was used to discriminate the impacted face and back face images. The obtained validation accuracies for low and high energy impact cases of the impacted and back faces are respectively 78% and 73%, as depicted in Figs. 34 and 35. In this example, the classification accuracies are much less than those of the previous examples, which is due to the similar features between the impacted face images and the back face images. In other words, there are not many obvious discrepancies between the images, so that the features extracted by the network are not distinct enough, even by a naked eye, to yield accurate training and consequently reliable classification. This fact is better illustrated in Figs. 36 and 37, which demonstrate the classification of the impacted face and back face images. As shown, some items (marked by a red circle) are wrongly classified by the network. In such cases, it is required to train the network with a much larger and comprehensive dataset containing enough variations in extractable features.

Future research
This paper illustrated the potential of deep learning techniques in autonomous damage detection of impact and erosion in composite structures. However, there is a diverse range of damage types in composite structures, and different parameters such as environmental conditions, illumination, cleanliness, geometry, inspection angle and colour / finish that may influence the damage detectability using image processing. Therefore, further experimental and modelling research is required to develop a comprehensive and high-quality dataset for different damage types in composite structures and their affecting parameters for a reliable machine learning based autonomous inspection. More research needs to be done on measuring the surface damage size, correlating the visible damage on the surface to the extent of potential invisible damage, and to predict residual life of the structures considering the damage content. The probability of surface damage detection, and its relationship with surface damage size for visual inspection needs to be established to be used in design calculations of structural strength and durability.

Conclusion
In this study convolutional neural network (CNN) in conjunction with transfer learning was used for the classification of composite materials damage types and damage severity. For this purpose, the pre-trained AlexNet network, as one of the most accurate transfer learning methods, was implemented. The network was used for the      -AlexNet network outperformed Resnet-50 and the user-defined deep neural networks regarding the accuracy level for identifying the damage type in a reasonable computational time. -The validation accuracy of the network strongly depends on the learning rate, where its optimum value was achieved using the trial and error method. -The obtained accuracy in the classification of microscopic damage mechanisms (i.e. matrix cracking and fibre breakage) was 91%, achieved in the case of a learning rate of 5e-5. -For damage severity classification (i.e. low energy or high energy impact), the validation accuracy was 96% for the impacted side, and 86% for the back face. For the case of macroscopic damage type indentification (i.e. erosion, impact and un-damaged), the best validation accuracy was 93%, achieved in the case of a learning rate of 0.0001.
-In spite of AlexNet network's high accuracy in the classification of various damage types and damage severity, it couldn't accurately classify the damage side (i.e. impacted face or back face). The obtained accuracies for low and high energy impact cases were 78% and 73%, respectively. This can be related to the similar fea-        tures between the impacted face images and the back face images; where, there were not much obvious discrepancies between the images, so that the features extracted by the network were not distinct enough to yield accurate training and consequently reliable classification. -The obtained results indicate the promising performance of deep learning to automate visual inspection, however it is highlighting the need for an improved dataset library, and customised classifiers for deep learning training. -Future works could focus on developing comprehensive and highquality datasets for different damage types in composite structures, and correlating the damage extent to the residual lifetime of the structure, making it possible to accurately train advanced deep learning algorithms for autonomous visual inspection purposes.