Quality Assessment of Tire Shearography Images via Ensemble Hybrid Faster Region-Based ConvNets

: In recent times, the application of enabling technologies such as digital shearography combined with deep learning approaches in the smart quality assessment of tires, which leads to intelligent tire manufacturing practices with automated defects detection. Digital shearography is a prominent approach that can be employed for identifying the defects in tires, usually not visible to human eyes. In this research, the bubble defects in tire shearography images are detected using a unique ensemble hybrid amalgamation of the convolutional neural networks / ConvNets with high-performance Faster Region-based convolutional neural networks. It can be noticed that the routine of region-proposal generation along with object detection is accomplished using the ConvNets. Primarily, the sliding window based ConvNets are utilized in the proposed model for dividing the input shearography images into regions, in order to identify the bubble defects. Subsequently, this is followed by implementing the Faster Region-based ConvNets for identifying the bubble defects in the tire shearography images and further, it also helps to minimize the false-positive ratio (sometimes referred to as the false alarm ratio). Moreover, it is evident from the experimental results that the proposed hybrid model o ﬀ ers a cent percent detection of bubble defects in the tire shearography images. Also, it can be witnessed that the false-positive ratio gets minimized to 18 percent.


Introduction
Industry 4.0 is the novel digital technology meant for industries, and this paradigm enables the communication, collection, and analysis of data through machines, thereby allowing quicker, more agile, and efficient processes for making superior quality goods with minimal expenditure. Moreover, this digital industrial technology will assist in enhancing productivity, enabling industrial development, and revamping the profile of the personnel involved, thereby nurturing changes in the competence of business organizations and states. Further, this paradigm will foster superior efficiencies and will modify the conventional production associations between the suppliers, manufacturers, and clients and also the communication amongst humans and machines. Besides, due to the phenomenal growth in technology and agile expansion of Industry 4.0, several manufacturing firms have embraced automation, thereby replacing labor-intensive tasks in conventional production units [1]. Also, it can Since 2012, there has been rapid progress in the convolutional neural networks-based research on image, visual, and computer-vision-based tasks [11,12]. The approach presented in the preliminary study [13] establishes two CNN architectures for tire bubble-defects diagnosis. Although, the scheme presented in [13] provides a precise identification of the bubble defects, however, the false alarm ratio, also known as the false positive ratio-more than twenty percent-is significant. The Faster R-CNN comprises two networks, primarily for generating the region proposals it makes use of a region proposal network (RPN) and secondly, a network that utilizes these region proposals for detecting the bubble defects [14].
The key contributions of this work are summarized as follows: • The substantial contribution of this work lies in improving the architecture established earlier in [13] for effectively realizing intelligent tire manufacturing with automated defects detection. • A Faster Region-based convolutional neural networks (R-CNN) is combined along with the architecture described in [13] for minimizing the false positive ratio. • Further, this significant modification to the CNN architecture helps in minimizing the labor cost involved in the tire manufacturing industry. • The results of the proposed hybrid model indicate that this approach asserts a hundred percent detection of bubble defects in the tire shearography images. • From the results, it can be perceived that the false alarm ratio can be minimized to 18 percent.  Since 2012, there has been rapid progress in the convolutional neural networks-based research on image, visual, and computer-vision-based tasks [11,12]. The approach presented in the preliminary study [13] establishes two CNN architectures for tire bubble-defects diagnosis. Although, the scheme presented in [13] provides a precise identification of the bubble defects, however, the false alarm ratio, also known as the false positive ratio-more than twenty percent-is significant. The Faster R-CNN comprises two networks, primarily for generating the region proposals it makes use of a region proposal network (RPN) and secondly, a network that utilizes these region proposals for detecting the bubble defects [14].
The key contributions of this work are summarized as follows: • The substantial contribution of this work lies in improving the architecture established earlier in [13] for effectively realizing intelligent tire manufacturing with automated defects detection. • A Faster Region-based convolutional neural networks (R-CNN) is combined along with the architecture described in [13] for minimizing the false positive ratio. • Further, this significant modification to the CNN architecture helps in minimizing the labor cost involved in the tire manufacturing industry.

•
The results of the proposed hybrid model indicate that this approach asserts a hundred percent detection of bubble defects in the tire shearography images.

•
From the results, it can be perceived that the false alarm ratio can be minimized to 18 percent. Since 2012, there has been rapid progress in the convolutional neural networks-based research on image, visual, and computer-vision-based tasks [11,12]. The approach presented in the preliminary study [13] establishes two CNN architectures for tire bubble-defects diagnosis. Although, the scheme presented in [13] provides a precise identification of the bubble defects, however, the false alarm ratio, also known as the false positive ratio-more than twenty percent-is significant. The Faster R-CNN comprises two networks, primarily for generating the region proposals it makes use of a region proposal network (RPN) and secondly, a network that utilizes these region proposals for detecting the bubble defects [14].
The key contributions of this work are summarized as follows: • The substantial contribution of this work lies in improving the architecture established earlier in [13] for effectively realizing intelligent tire manufacturing with automated defects detection. • A Faster Region-based convolutional neural networks (R-CNN) is combined along with the architecture described in [13] for minimizing the false positive ratio. • Further, this significant modification to the CNN architecture helps in minimizing the labor cost involved in the tire manufacturing industry. • The results of the proposed hybrid model indicate that this approach asserts a hundred percent detection of bubble defects in the tire shearography images. • From the results, it can be perceived that the false alarm ratio can be minimized to 18 percent.

Materials and Methods
A two-stage hybrid model for detecting bubble defects in tires is proposed in this work. The primary stage includes a CNN architecture for diagnosing tire bubble-defects, and the second stage makes use of a Faster Region-based ConvNets architecture for minimizing the false positive or the false alarm ratio (FAR). A flow diagram of the proposed two-stage ensemble hybrid model is portrayed in Figure 3.

Faster Region-Based Convolutional Neural Networks
A model referred to as Regions with CNN features (R-CNN), which is a scalable object detection approach that enhances the mean average precision, was established by the research in [15]. In the research desscribed in [16], another improved version of the R-CNN model known as the Fast R-CNN was deployed with various novelties for enhancing the training and testing speed at the same time augmenting the accuracy of detection. Further, the work presented in [17] established a Faster R-CNN that introduced an RPN which shares the convolutional features of the full image with the network responsible for detection; hence, this ensures that the region proposals are achieved at a low cost. It can be observed that the RPN approach is deployed instead of the Selective Search (SS) technique [18] in the case of Faster R-CNN/ Faster Region-based ConvNets. Further, this method considerably reduces the time-period necessary for extracting the candidate regions and also for increasing the overall efficiency. Figure 4 illustrates the architectural model of the Faster R-CNN network.
The Faster R-CNN network with a ZF-net exhibits the detection results with an accuracy of 59.9% for the PASCAL VOC 2007 test set [19][20][21]. Besides, for the same test set, the Faster R-CNN network with VGG16 architecture accomplishes the detection results with 73.2% accuracy [19][20][21]. Henceforth, it can be observed that the Faster R-CNN with VGG16 architecture achieves superior accuracy, which makes it the most sought after approach. Moreover, this technique is utilized in this research to enhance the detection accuracy of the tire bubble defects. In Figure 5, the architectural model of the fully convolutional region proposal network [22] is depicted. Figure 5 portrays the fact that the fully convolutional region proposal network applies a 3 × 3 window over the feature maps received from the ConvNets. Subsequently, for assessing the candidate regions, we make use of the anchors with various areas and ratios. Additionally, the chosen candidate expanses are placed into the 256-dimensional trajectory, and further, they are passed on as the inputs to the box regression layer (reg) and a box-class layer (cls). For each proposal, the outcome of the box-class layer approximates the target object or the non-target object probabilities. Consequently, a positive label will be allocated for an anchor with an Intersection-over Union (IoU) overlay proportion more significant than the value 0.7 in comparison to some ground truth box. Besides, the negative label will be allocated for the non-positive anchor with an Intersection-over Union proportion lesser than 0.3 for the remaining ground truth boxes. It can be clearly noted that the anchors which are neither positive nor negative have no role in the training for

Materials and Methods
A two-stage hybrid model for detecting bubble defects in tires is proposed in this work. The primary stage includes a CNN architecture for diagnosing tire bubble-defects, and the second stage makes use of a Faster Region-based ConvNets architecture for minimizing the false positive or the false alarm ratio (FAR). A flow diagram of the proposed two-stage ensemble hybrid model is portrayed in Figure 3.

Faster Region-Based Convolutional Neural Networks
A model referred to as Regions with CNN features (R-CNN), which is a scalable object detection approach that enhances the mean average precision, was established by the research in [15]. In the research desscribed in [16], another improved version of the R-CNN model known as the Fast R-CNN was deployed with various novelties for enhancing the training and testing speed at the same time augmenting the accuracy of detection. Further, the work presented in [17] established a Faster R-CNN that introduced an RPN which shares the convolutional features of the full image with the network responsible for detection; hence, this ensures that the region proposals are achieved at a low cost. It can be observed that the RPN approach is deployed instead of the Selective Search (SS) technique [18] in the case of Faster R-CNN/ Faster Region-based ConvNets. Further, this method considerably reduces the time-period necessary for extracting the candidate regions and also for increasing the overall efficiency. Figure 4 illustrates the architectural model of the Faster R-CNN network.
The Faster R-CNN network with a ZF-net exhibits the detection results with an accuracy of 59.9% for the PASCAL VOC 2007 test set [19][20][21]. Besides, for the same test set, the Faster R-CNN network with VGG16 architecture accomplishes the detection results with 73.2% accuracy [19][20][21]. Henceforth, it can be observed that the Faster R-CNN with VGG16 architecture achieves superior accuracy, which makes it the most sought after approach. Moreover, this technique is utilized in this research to enhance the detection accuracy of the tire bubble defects. In Figure 5, the architectural model of the fully convolutional region proposal network [22] is depicted. Figure 5 portrays the fact that the fully convolutional region proposal network applies a 3 × 3 window over the feature maps received from the ConvNets. Subsequently, for assessing the candidate regions, we make use of the anchors with various areas and ratios. Additionally, the chosen candidate expanses are placed into the 256-dimensional trajectory, and further, they are passed on as the inputs to the box regression layer (reg) and a box-class layer (cls). For each proposal, the outcome of the box-class layer approximates the target object or the non-target object probabilities. Consequently, a positive label will be allocated for an anchor with an Intersection-over Union (IoU) overlay proportion more significant than the value 0.7 in comparison to some ground truth box. Besides, the negative label will be allocated for the non-positive anchor with an Intersection-over Union proportion lesser than 0.3 for the remaining ground truth boxes. It can be clearly noted that the anchors which are neither positive nor negative have no role in the training for accomplishing the target. In the box regression layer, the positive sample co-ordinates achieved by the box-class layer are modified to suit the ground truth's bounding box aptly.

Image Enhancement
The classification capability and competence of the convolutional neural networks rely heavily on the two vital parameters, namely, the quality and quantity of the training samples. Nevertheless, the arduous task for this research is the identification of speckle patterns encompassing the bubble defects. In order to overcome this issue; hence, the blocks from the speckle patters encompassing the bubble defects were randomly chosen. Also, the chosen data were rotated horizontally and vertically, and then the resultant dataset helps in achieving the essential dataset required for the training process. The imperfect bubble blocks were detached physically. In this way, this research could achieve about ten times the training data. Hence, this approach makes sure that the patterns of the tire bubble defects were adequate for the training process. accomplishing the target. In the box regression layer, the positive sample co-ordinates achieved by the box-class layer are modified to suit the ground truth's bounding box aptly.

Image Enhancement
The classification capability and competence of the convolutional neural networks rely heavily on the two vital parameters, namely, the quality and quantity of the training samples. Nevertheless, the arduous task for this research is the identification of speckle patterns encompassing the bubble defects. In order to overcome this issue; hence, the blocks from the speckle patters encompassing the bubble defects were randomly chosen. Also, the chosen data were rotated horizontally and vertically, and then the resultant dataset helps in achieving the essential dataset required for the training process. The imperfect bubble blocks were detached physically. In this way, this research could achieve about ten times the training data. Hence, this approach makes sure that the patterns of the tire bubble defects were adequate for the training process.

Classification of the Bubble Defects in Tires
It can be noticed from [13] that two convolutional neural network architectures were established for diagnosing the bubble-defects available in the treads and sidewalls of the tires. Though this approach accurately classifies the tire bubble-defects, nevertheless, the FAR seems marginally more significant than 20 percentage. Thus, our work enhances the approach in [13] by incorporating a Faster-RCNN network for reducing the false alarm ratio. The modified hybrid Faster Region-Based Convolutional Neural Networks architecture is illustrated in Figure 6. The various components of the proposed hybrid model are presented in Table 1. In the CNN, the hyper-parameters settings for tire tread are learning rate = 0.01, epoch = 30000, batch size = 40,

Classification of the Bubble Defects in Tires
It can be noticed from [13] that two convolutional neural network architectures were established for diagnosing the bubble-defects available in the treads and sidewalls of the tires. Though this approach accurately classifies the tire bubble-defects, nevertheless, the FAR seems marginally more significant than 20 percentage. Thus, our work enhances the approach in [13] by incorporating a Faster-RCNN network for reducing the false alarm ratio. The modified hybrid Faster Region-Based Convolutional Neural Networks architecture is illustrated in Figure 6. The various components of the proposed hybrid model are presented in Table 1. In the CNN, the hyper-parameters settings for tire tread are learning rate = 0.01, epoch = 30000, batch size = 40,

Classification of the Bubble Defects in Tires
It can be noticed from [13] that two convolutional neural network architectures were established for diagnosing the bubble-defects available in the treads and sidewalls of the tires. Though this approach accurately classifies the tire bubble-defects, nevertheless, the FAR seems marginally more significant than 20 percentage. Thus, our work enhances the approach in [13] by incorporating a Faster-RCNN network for reducing the false alarm ratio. The modified hybrid Faster Region-Based Convolutional Neural Networks architecture is illustrated in Figure 6. The various components of the proposed hybrid model are presented in Table 1. In the CNN, the hyper-parameters settings for tire tread are learning rate = 0.01, epoch = 30,000, batch size = 40, gamma = 0.001, power = 0.75, and momentum = 0.9. The hyper-parameters settings for tire sidewall are learning rate = 0.00001, epoch = 18,000, batch size = 18, gamma = 0.001, power = 0.75, and momentum = 0.9. In the Faster-RCNN, the learning rate, step size, and momentum are set as 0.00001, 50,000, and 0.9, respectively, for both tire tread and tire sidewall.

The Sliding Window Phase
In this work, the original shearography image had a size of 1360 × 1024 pixels. In order to facilitate bubble defect detection, the shearography tire images are fragmented into a variety of blocks via the sliding window phase. Subsequently, it is evident that the location of the tire bubble defects is not known; consecutive sliding windows with 50% overlapping regions for the extraction of speckle patterns are used to avoid fragmenting the bubble defects and causing erroneous results.
The overlapping threshold has been selected to poise the time-period required for processing and also for the efficient detection of bubble defects. The abstract depiction of the sliding window indicating the overlap is presented in Figure 7.

The Sliding Window Phase
In this work, the original shearography image had a size of 1360 × 1024 pixels. In order to facilitate bubble defect detection, the shearography tire images are fragmented into a variety of blocks via the sliding window phase. Subsequently, it is evident that the location of the tire bubble defects is not known; consecutive sliding windows with 50% overlapping regions for the extraction of speckle patterns are used to avoid fragmenting the bubble defects and causing erroneous results.
The overlapping threshold has been selected to poise the time-period required for processing and also for the efficient detection of bubble defects. The abstract depiction of the sliding window indicating the overlap is presented in Figure 7.  Further, this research establishes a classifier that performs the process of detection of bubble defects in treads and sidewalls of tires, which is presented in Section 2.3. Moreover, the sliding window described in Section 2.4 is used to check the segmented block images sequentially for bubble defects. If the classifier classifies a block as containing bubbles, the Faster R-CNN is used to determine if the result is a false positive. If the resultant image is not false positive, then in the original image, the respective location of the block is encircled. As a result, this image is passed on to the professional operators for assessing the quality of the tires and also for removing the defective piece. Furthermore, this devised semi-automated assessment process offers cost-leadership when compared with the traditional manual inspection and also improves the reliability of the inspection process.

Results
In this work, the diagnosis of bubble defects in tires established in [13] and the Faster Region-based Convolutional Neural Networks approach is amalgamated for obtaining 100% detection of defects and also aiding in reducing the false alarm ratio. The evaluation metrics, such as the accuracy, sensitivity, and specificity, are used for assessing the performance of the proposed hybrid model. These metrics are computed using the following expressions: where TP stands for true positive, it represents the amount of diagnosed bubble patterns, which really possesses the bubble defects, and FP stands for false-positive and, it indicates the amount of not bubble patterns, which are wrongly diagnosed as bubble defects. True negative (TN) illustrates the amount of not bubble patterns, which are diagnosed as not bubble defects. Positives (P) represents the real bubble defects and negatives (N) denotes the not bubble defects. Among the evaluation metrics, sensitivity is the most necessary measure for achieving the complete detection of bubble defects. Moreover, a tire company provided the shearography images deployed in this research. Usually, the tire bubble defects were physically delineated with the assistance of experienced professionals. The amount of training images and blocks are clearly organized in Table 2. Further, it is evident that for the training process, the tire manufacturer supplied the 325 tire shearography images with bubble defects. Subsequently, the image enhancement approach is deployed for imitating 8596 and 5052 blocks from 223 tire tread images and 102 tire sidewall images containing bubble defects. Additionally, Table 3 indicates the test dataset, it comprises of 541 tire shearography images deprived of bubble defects and 256 tire shearography images having bubble defects. An area with bubble defects is expected to be smaller than the area of the default anchor of the Faster R-CNN. Therefore, in this work, the anchor's ratio and scale are adjusted according to the area of the bubble defects. Table 4 shows the ratio and scale adjustment of the anchors. Twelve anchor configurations are used for candidate regions in the marking of bubbles. The proposed hybrid model has been compared with various classifiers including the Support Vector Machine (SVM) [23], Random Forest Model [24], Haar-like AdaBoost Method [25], Chang's method [13], and the integrated model comprising of SVM, Random Forest Model, AdaBoost method. Besides, the proposed model was compared with these methods for verifying its performance. Table 5 illustrates the diagnosis of bubble-defects in treads of tire shearography images for several existing methods in comparison with the proposed ensemble hybrid model in terms of the evaluation metrics such as accuracy, sensitivity, and specificity. Additionally, Table 6 depicts the diagnosis of bubble-defects in sidewalls of tire shearography images for numerous prevailing approaches in comparison with the proposed ensemble hybrid model in terms of the assessment metrics such as accuracy, sensitivity, and specificity. Further, it can be witnessed from these tables that the work in [13] and the presented ensemble hybrid approach achieve 100 percent sensitivity, by successfully identifying each and every bubble-defect. Also, it can be observed that the presented ensemble hybrid approach surpasses all other existing approaches in terms of specificity. However, the presented ensemble hybrid approach requires a processing time of approximately 7 seconds/image, whereas the approach established in [13] takes only a processing-time of roughly 6 seconds/image. Nevertheless, the presented ensemble hybrid model is superior in other means and also in terms of specificity, when compared with the other existing approaches. Figure 8a-d illustrate the shearography images or the speckle patterns acquired using digital shearography, and Figure 8e-h depict the detection of bubble defects in tires using the proposed hybrid Faster Region-based convolutional neural networks model. Figure 8e-h indicate the fact that all bubble defects in tires have been detected successfully. Figure 9a-d depict the false positive or the false alarm inspection results in [13], where the shearography images do not have bubble defects; however, they get misrepresented as possessing the bubble defects. Figure 9e-h illustrate the assessment results of the hybrid Faster Region-based convolutional neural networks model using the same set of input images. It can be witnessed in Figure 9e-h that the shearography images have no bubble defects. Besides, it reveals the fact that the proposed hybrid Faster Region-based convolutional neural networks model effectively reduces the false-positive ratio or the false alarm rate.

Conclusions
In the tire manufacturing process, the diagnosis of bubble-defects in the treads and sidewalls of shearography tire images represents a significant task. Therefore, enabling smart tire quality assessment seems to be an essential way of realizing intelligent tire manufacturing practices that can ensure automated detection of defects. Further, an ensemble hybrid combination of the CNN with a high-performance Faster Region-based ConvNets for classifying and diagnosing the bubble-defects present in the tire shearography images. The proposed hybrid Faster Region-based convolutional neural networks model reduces misjudgments caused by human errors and achieves high consistency in the quality of bubble-defect detection. It is clearly evident from the results that in addition to thoroughly diagnosing the bubble-defects in tires, the hybrid Faster Region-based convolutional neural networks model decreases the false alarm ratio of not-bubble defects in tires from 20% to a rate of 18%. Also, it has to be noted that this hybrid system model was deployed in a tire manufacturing unit, and it produced efficient results in automatically diagnosing the bubble-defects in treads and sidewalls of tires. In the future work, more advanced CNN enabled approaches can be implemented for automated detection of defects [26][27][28][29][30], thus ensuring and realizing a sustainable tire manufacturing process.