Fault detection of electrolyzer plate based on improved Mask R-CNN and infrared images

The Architecture of Mask R-CNN, shown in figure 3, is mainly composed of feature extraction network, region proposal network (RPN), RoI Align layer, and classification and regression layer.

Zoom In Zoom Out Reset image size

The process is: (a) obtain the feature map by ResNet50 and FPN, the size is m × n × d; (b) divide the input image corresponding to the feature map into m × n regions and generate k proposals in each region; (c) filter proposals according to the NMS algorithm; (d) project the proposals on the feature map to obtain the corresponding feature matrix; and (e) scale each feature matrix to a uniform size through the RoIAlign layer. Then, the prediction results are obtained through a series of fully connected layers.

Extracting image depth features is an important step in the entire model training. This paper uses ResNet50 combined with FPN as a feature extraction network. Using multi-layer convolution can extract image features more efficiently, and the residual structure solves the problem of gradient disappearance. Compared with VGG [24] and ZF [25], the overall network performance has great advantages. FPN generates a new set of feature layers based on the RestNet50, and each feature layer is the result of the fusion of different stages of convolutional layers in RestNet50.

FPN combines deep and shallow feature fusion with multi-resolution prediction by connecting high-resolution, low-semantic shallow features to low-resolution, high-semantic deep features to achieve good detection results for multi-scale targets. Its structure is shown in figure 4. The output of the residual block of levels 3–5 of ResNet50 on the left is taken as the feature of FPN, denoted as (C3, C4, C5). (M5, M4, M3) are the intermediate layers obtained by upsampling from the deepest layer, and then connecting them to (C5, C4, C3) to fuse multi-scale features. In order to reduce the impact of upsampling, the layer must perform a 3 × 3 convolution operation. Finally, a multi-scale feature map is obtained, denoted as (P3, P4, P5).

Zoom In Zoom Out Reset image size

In the early two-stage object detection algorithm, the selective search algorithm [26] was used to generate predicted boxes by clustering similar regions. This process is very time-consuming and generates a lot of redundant boxes, which seriously affects the detection efficiency. The RPN proposed in Faster-RCNN effectively reduces the generation time and number of proposals, and there is no need to repeatedly calculate the feature map corresponding to each predicted box, which greatly improves the recognition efficiency. As shown in figure 5, on the feature map, each pixel is used as an anchor point to generate k proposals. In the infrared image of the electrode plate, the size and shape of the faulty target are relatively fixed. In order to make the proposal have a higher initial matching degree with the target fault and reduce the amount of subsequent bounding box regression calculations, the initial scale of the proposal is set to (32 × 32, 64 × 64, 128 × 128) and the initial aspect ratio is set to (1:4, 1:8), so the k in this paper is 6. Because FPN is used to fuse the deep/shallow features, the three different initial scales directly correspond to (P3, P4, P5).

Zoom In Zoom Out Reset image size

In figure 5, a k proposals are generated in total at the corresponding pixel points of each feature layer, and then the proposals are input into the classification layer and regression layer; 2k class scores and 4k bounding box regression parameters are obtained. Based on the class scores, the NMS algorithm is used to eliminate the redundant proposals, and a part of the remaining proposals is sampled for RPN training.

The NMS algorithm is the most commonly used post-processing algorithm in object detection, and is mainly used to eliminate redundant proposals. The NMS algorithm flow is shown in figure 6. When the initial proposals generated by the RPN layer are screened, the proposal with the highest score is retained, the other proposals are sorted by score, and the IoU of the proposal and the highest score proposal, respectively, are judged. If the IoU is greater than the threshold, it will be eliminated until all the proposals are filtered.

Zoom In Zoom Out Reset image size

The traditional NMS algorithm uses a greedy strategy to eliminate the proposal, which may cause missed detection. As shown in figure 7, when multiple plates fail, the predicted box of adjacent targets may be eliminated as a redundant box. The reserved predicted box is not the most appropriate.

Zoom In Zoom Out Reset image size

To solve this problem, we propose a multi-stage Gaussian penalty NMS algorithm. When the IoU of the current box and the highest score box exceeds the upper threshold (${T_{\text{u}}}$), it can be considered as a complete overlap and the current box is eliminated. The remaining proposals are kept, but the Gaussian penalty is given according to the degree of overlap. When IoU is less than the lower threshold (${T_{\text{l}}}$), the score is not adjusted (different target). The improved NMS algorithm is as follows:

$$\begin{align}s_i^*\left\{ {\begin{array}{*{20}{l}} {{s_i}{\text{ }}} &{{\text{iou}}(M,{b_i}) \le {T_{\text{l}}}} \\ {{s_i}{e^{ - \frac{{{\text{iou}}{{(M,{b_i})}^2}}}{\sigma }{\text{ }}}}}&{{T_{\text{l}}} < {\text{iou}}(M,{b_i}) \le {T_{\text{u}}}} \\ {} \\ {0{\text{ }}} & {{\text{iou}}(M,{b_i}) > {T_{\text{u}}}} \end{array}} \right.\end{align} \tag{ 1 }$$

where $M$ is the proposal with the highest score, ${b_i}$ is the current proposal to be screened, ${s_i}$ is the original score of the current box, and $s^{*}_i$ is the score after reassignment, set $\sigma$ to 0.65. When ${T_{\text{l}}}$ is 0.4 and ${T_{\text{u}}}$ is 0.85, the precision and recall rate can reach a high level, and the redundant boxes can be efficiently eliminated.

The original loss function of the Mask R-CNN model is shown in equation (2). It consists of three parts, namely class loss, bounding box regression loss, and mask loss. Since there is no need to perform semantic segmentation on the target in this paper, mask segmentation is not trained to reduce the amount of calculation

$$\begin{align}L(\,p,u,b,{b^{{\text{gt}}}}) = {L_{{\text{cls}}}}(\,p,u) + {L_{{\text{loc}}}}(b,{b^{{\text{gt}}}}) + {L_{{\text{msk}}}},\end{align} \tag{ 2 }$$

where ${L_{\text{cls}}}\,$is class loss, and it can be computed by:

$$\begin{align}{L_{{\text{cls}}}}(\,p,u) = - \log {p_{\text{u}}}\end{align} \tag{ 3 }$$

where $p$ is the class probability distribution predicted by the classifier and $u\,$ is the corresponding real class label.

${L_{\text{loc}}}\,$ is the bounding box regression loss, and is computed by smooth_{L 1} norm. The following formulas are used to compute ${L_{{\text{loc}}}}$

$$\begin{align}{L_{{\text{loc}}}}(b,{b^{{\text{gt}}}}) = \sum\limits_{i = x,y,w,h} {\text{smoot}}{{\text{h}}_{L1}}({b_i} - b_i^{{\text{gt}}}),\end{align} \tag{ 4 }$$

$$\begin{align}{\text{smoot}}{{\text{h}}_{L1}}(t) = \left\{ {\begin{array}{*{20}{c}} {0.5{t^2}} \\ {\left| t \right| - 0.5} \end{array}\begin{array}{*{20}{c}} {} \\ {} \end{array}\begin{array}{*{20}{c}} {\left| t \right| < 1} \\ {{\text{otherwise}}} \end{array}} \right.\end{align} \tag{ 5 }$$

where $b$ is the regression parameter $\left( {{b_x},{b_y},{b_w},{b_h}} \right)$ of the corresponding class predicted by the boundary regressor, and ${b^{{\text{gt}}}}$ is the bounding box regression parameter $(b_x^{{\text{gt}}},b_y^{{\text{gt}}},b_w^{{\text{gt}}},b_h^{{\text{gt}}})$ of the real target.

It should be noted that when the smooth_{L 1} norm is used as the loss of the bounding box regression, the loss is smaller, but the IoU value is not necessarily higher. As shown in figure 8, when the smooth_{L 1} norm is all 3, the IoU values are quite different.

Zoom In Zoom Out Reset image size

IoU loss directly performs bounding box regression according to the evaluation criteria and accurately describes the distance and scale relationship between the two boxes. Compared with smooth_{L 1}, it has excellent performance, but there are also many problems. When the predicted box and the target box do not overlap or are fully contained, the relationship between the two boxes cannot be correctly reflected, resulting in no gradient backhaul and thereby weakening network performance. Based on this, we propose the G2-IoU loss mechanism, namely the Globally Generalized IoU loss. G2-IoU considers the coincidence rate of the target box and the predicted box, and considers the distance and scale factors between the two boxes. G2-IoU is as shown in equation (6). In G2-IoU, penalty terms based on the Euclidean distance between the center points of the two boxes and the diagonal distance of the smallest bounding box [27], and on the scale of the two boxes, are added to the algorithm. Thus, the sensitivity of the loss function to distance and scale is increased, which is more conducive to the accurate positioning of the predicted box

$$\begin{align}{\text{G2-IoU}} = {\text{IoU}} - {L_{\text{d}}}(b,{b^{{\text{gt}}}}) - {L_{\text{s}}}(b,{b^{{\text{gt}}}}).\end{align} \tag{ 6 }$$

The second term, ${L_{\text{d}}}\left( {b,{b^{{\text{gt}}}}} \right)$, is the normalized Euclidean distance penalty term, and its formula is as follows:

$$\begin{align}{L_{\text{d}}}(b,{b^{{\text{gt}}}}) = \frac{{{\rho ^2}(b,{b^{{\text{gt}}}})}}{{{c^2}}}\end{align} \tag{ 7 }$$

where $\rho$ is the Euclidean distance of the coordinates of the center point between the target box and the predicted box, while $c$ is the diagonal distance of the smallest bounding box of the two boxes.

Figure 9 shows that no matter what state the two boxes are in, the bounding box loss can be quickly regressed through ${L_{\text{d}}}\left( {b,{b^{{\text{gt}}}}} \right)$. Even if the two boxes are completely contained and the $c$ is fixed, the loss can be calculated by $d$.

Zoom In Zoom Out Reset image size

The third term, ${L_{\text{s}}}\left( {b,{b^{{\text{gt}}}}} \right){ }$, is the penalty term based on the scale of the predicted box and the target box (scale includes shape and area). Assuming that the main diagonal coordinate of the predicted box is $\left( {{b_{x1}},{b_{y1}},{b_{x2}},{b_{y2}}} \right)$ and that of the target box is $(b_{x1}^{{\text{gt}}},b_{y1}^{{\text{gt}}},b_{x2}^{{\text{gt}}},b_{y2}^{{\text{gt}}})$, the following formulas are used to compute ${L_{\text{s}}}\left( {b,{b^{{\text{gt}}}}} \right)$

$$\begin{align}{L_{\text{s}}}(b,{b^{{\text{gt}}}}) = \left\{ {\begin{array}{*{20}{c}} {\lambda (1 - {\text{IoU}})\ln (\tau )\ln (\delta )} \\ { - \lambda (1 - {\text{IoU}})\ln (\delta )} \end{array}\begin{array}{*{20}{c}} {}&{\begin{array}{*{20}{c}} {\delta \ge 0.5} \\ {{\text{otherwise}}} \end{array}} \end{array}} \right.\end{align} \tag{ 8 }$$

$$\begin{align}\delta = \min \left( {\frac{{({b_{x1}} - {b_{x2}})(b_{y1}^{{\text{gt}}} - b_{y2}^{{\text{gt}}})}}{{({b_{y1}} - {b_{y2}})(b_{x1}^{{\text{gt}}} - b_{x2}^{{\text{gt}}})}},\frac{{({b_{y1}} - {b_{y2}})(b_{x1}^{{\text{gt}}} - b_{x2}^{{\text{gt}}})}}{{({b_{x1}} - {b_{x2}})(b_{y1}^{{\text{gt}}} - b_{y2}^{{\text{gt}}})}}} \right)\end{align} \tag{ 9 }$$

$$\begin{align}\tau = \min \left( {\frac{{({b_{x1}} - {b_{x2}})({b_{y1}} - {b_{y2}})}}{{(b_{x1}^{{\text{gt}}} - b_{x2}^{{\text{gt}}})(b_{y1}^{{\text{gt}}} - b_{y2}^{{\text{gt}}})}},\frac{{(b_{x1}^{{\text{gt}}} - b_{x2}^{{\text{gt}}})(b_{y1}^{{\text{gt}}} - b_{y2}^{{\text{gt}}})}}{{({b_{x1}} - {b_{x2}})({b_{y1}} - {b_{y2}})}}} \right)\end{align} \tag{ 10 }$$

where λ is the penalty equilibrium coefficient, δ is the shape similarity coefficient of the two boxes, and τ is the area similarity coefficient of the two boxes. When the difference in shape is great, the priority is to calculate the loss based on IoU, Euclidean distance, and shape similarity, so τ is effective only when δ is greater than 0.5.

When the centers of the two boxes overlap, ${\rho ^2}\left( {b,{b^{gt}}} \right)\,$and $\,c$ become fixed values, and the punishment based on the distance between the two boxes becomes invalid. At this time, the loss can also be calculated according to the scale of the two boxes.

In summary, the final bounding box regression loss function G2-IoU loss is:

$$\begin{align}{L_{{\text{G2-IoU}}}} = 1 - {\text{IoU}} + {L_{\text{d}}}(b,{b^{{\text{gt}}}}) + {L_{\text{s}}}(b,{b^{{\text{gt}}}}{\text{)}}{\text{}}.\end{align} \tag{ 11 }$$

When the two boxes coincide with each other, the loss is 0. When the two boxes do not coincide at all, the loss is non-negative and bounded.

The data was collected from the electrolysis workshop of a smelter in Hunan Province, China. After rectifying and segmenting the original infrared images, 2000 electrolytic cell images with a size of 155 × 548 × 3 were obtained. Figure 10 shows the segmented single-cell image.

Zoom In Zoom Out Reset image size

The dataset is divided into the training set, validation set, and test set, among which 1280 are training set, 320 are validation set, and 400 are test set.

As shown in figure 11, the faults are divided into three levels: first-level (F1), second-level (F2), and third-level (F3). In order to enrich the target types and facilitate fault location, the electrolyte inlet (In) and outlet (Out) are also marked in the dataset.

Zoom In Zoom Out Reset image size

In order to verify the fault detection effect of Mask R-CNN and the effectiveness of the proposed improved algorithm, this experiment uses three algorithms for comparison: (a) original Mask R-CNN; (b) original Mask R-CNN algorithm but the bounding box loss function uses G2-IoU loss, denoted as Mask R-CNN + G2-IoU; (c) improved Mask R-CNN. ResNet50 and FPN are used to extract feature maps for all three models. The SGD optimizer is used for training. The learning rate is 0.005, momentum factor is 0.9, and weight decay value is 0.0005. The batch size is 4, epoch is 25, and number of iterations is 8000. In order to facilitate training, the image size will be adjusted to 800 × 800 during input.

In addition, the improved model proposed in this paper is compared with other popular object detection models on the same test set, such as Faster R-CNN and YOLOv3, so as to show the detection capability of the improved model more comprehensively.

The evaluation index of this experiment adopts the evaluation index of the COCO data set commonly used in the target detection field, that is, the average precision (AP) and the mean average precision (mAP) under different IoU thresholds.

AP is an important indicator for evaluating the detection effect of a single class. It can be calculated through precision (P) and recall (R). P refers to the percentage of the number of targets correctly detected by the model in the total number of detected targets, and R refers to the percentage of the number of targets correctly detected by the model in the total number of targets of this type. mAP is the average AP of all target categories. The calculation formula of each indicator is as follows:

$$\begin{align}P = \frac{\text{TP}}{{\text{TP}} + {\text{FP}}}\end{align} \tag{ 12 }$$

$$\begin{align}R = \frac{\text{TP}}{{\text{TP}} + {\text{FN}}}\end{align} \tag{ 13 }$$

$$\begin{align}{\text{AP}} = \int\limits_0^1 P(r){\text{d}}r\end{align} \tag{ 14 }$$

$$\begin{align}{\text{mAP}} = \frac{1}{5}\sum\limits_{n = 1}^5 {\text{AP}}(n)\end{align} \tag{ 15 }$$

where ${\text{TP}}$ is the number of target categories detected correctly, $\,{\text{FP}}$ is the number of non-target categories detected incorrectly, $\,{\text{FN}}$ is the number of target categories not detected, and $r$ is $\,R$.

The loss convergence curves of the three models are shown in figure 12. It can be found that the improved model starts to stabilize after the 600 iterations and finally converges at about 0.08. The loss converges faster than the original model and the loss value is smaller, which proves the effectiveness of the G2-IoU loss function of the improved model.

Zoom In Zoom Out Reset image size

The mAP curves are shown in figure 13. The mAP of the improved model increased rapidly in the first seven epochs; then there was a slight increase as the number of iterations increased and eventually stabilized at around 0.86. The improved model is significantly improved compared with the original models.

Zoom In Zoom Out Reset image size

Table 1 shows the detection results of the three models used in the experiment on the test set, including the average precision (AP) of each class when the IoU threshold is 0.5 (AP_0.5) and the mAP under different IoU thresholds (mAP_0.5:0.95, mAP_0.75, mAP_0.5).

As shown from table 1 the detection accuracy of the Improved Mask R-CNN is greatly improved compared with the Original Mask R-CNN. When the IoU threshold is 0.5, the model accuracy reaches 86.8%, an increase of 10.4%. In the case of only improving the loss function, the accuracy rate also reached 83.3%, an increase of 6.9%. Among them, the detection accuracy of the F1 and F3 has been improved the most.

The detection results based on different object detection models are shown in table 2. Compared with Faster R-CNN, a common two-stage model, the mAP and detection speed of the Improved Mask R-CNN are improved. Although YOLOv3 has the best real-time performance and the fastest detection speed, its accuracy and recall rate are very low. Overall, the effect of Improved Mask R-CNN in plate fault detection is greatly improved.

Figure 14 shows the detection results based on the improved Mask R-CNN model. It can be found that the faults can be identified with high confidence and accurately located in a variety of complex situations such as continuous plate faults, irregular covering, and local busbar heating.

Zoom In Zoom Out Reset image size

However, there are also a small number of false detections, mainly when the background brightness is high. At this time, some background characteristics are similar to the F1, so the non-faulty plate is incorrectly identified as the F1. However, the recognition accuracy of the F2 and the F3 that seriously affect production is high. In an industrial production environment, the priority is to identify the more severe F2 and F3 accurately.