A deep learning-based ensemble method for helmet-wearing detection

Faster RCNN for detecting faces and helmet-wearing.

After images are fed, Faster RCNN firstly extracts image feature maps through a group of basic conv+relu+pooling layer. Next, RPN (Region Proposal Networks) will set a large number of anchors on the scale of the original image, and randomly select 128 positive anchors and 128 negative anchors from all anchors for binary training, and use these anchors and a softmax function to initially extract positive anchors as the candidate area. At this time, the candidate regions are not accurate and require bounding boxes.

For a given image I, we use A to represent the ground-truth anchors. We use A_F and c_F to represent the identified bounding boxes and helmet-wearing confidence scores, respectively, computed by the Faster-RCNN algorithm. If we use F to represent the algorithm, W_F to represent the weight of the network, this approach can be written as follows. (1)${\displaystyle A_F,c_F=F\left(I,W_F\right)}$

If we consider $A_F=F\left(I,W_F\right)\left[0\right]$ and $c_F=F\left(I,W_F\right)\left[1\right]$, we can use (2)${\displaystyle Loss\left(F\left(I,W_F\right)\left[0\right],F\left(I,W_F\right)\left[1\right],A\right)}$ to represent the loss function (Fukushima & Miyake, 1982) when to minimize differences between the detected anchors and the ground-truth.

Thus, when we train this model, the optimization is as follows. (3)${\displaystyle W_F^{\ast }={\text{argmin}}_{{\mathrm{W}}_{\mathrm{F}}}Loss\left(F\left(I,W_F\right)\left[0\right],F\left(I,W_F\right)\left[1\right],A\right)}$

Tiny Face for detecting faces.

The overall idea of Tiny Face is similar to RPN in Faster RCNN, which is a one-stage detection method. The difference is that some scale specific design and multi-scale feature fusion are added, supplemented by image pyramid so that the final detection effect for small faces is better. The training data were changed by three scales, with one-third of the probability respectively and sent to the network for training. Multiple scales could be selected to improve the accuracy rate in the prediction.

For a given image I, we can also use A_T and c_T to represent the identified bounding boxes and confidence scores computed by the Tiny Face algorithm, so if we use T to represent the Tiny Face algorithm and W_T to represent the corresponding weight, we can have (4)${\displaystyle A_T,c_T=T\left(I,W_T\right)}$

However, Tiny Face can only be used to determine whether the detection target contains a human face and cannot directly distinguish whether the target is wearing a helmet. Thus, we propose to use CNN to overcome this disadvantage.

CNN for detecting helmet-wearing.

For anchors determined by Tiny Face, we can use a CNN to detect helmets above the face. We enlarge the face area detected by the Tiny Face and feed it into the CNN model for prediction. The confidence scores indicating whether there is a helmet above the face can be computed by the CNN algorithm (5)${\displaystyle c_C=C\left(A_T,I,W_C\right),}$ where C is a function representing the forward propagation of CNN. Here, C is a composition of two convolution layers and one fully connected layer.

The loss function is again to minimize the difference between detected helmets and the ground-truth (6)${\displaystyle Loss\left(A_T,C\left(A_T,I,W_C\right),A\right).}$

Accuracy of $F\left(I,W_F^{\ast }\right)$.

In this part, we evaluate the accuracy of $T\left(I,W_T^{\ast }\right)$ alone in Algo. 2. Theoretically, the more training steps the model has, the better, but in order to prevent overfitting, we still need to observe the accuracy of the model under different training steps.

In the beginning, we select some images from training dataset to evaluate the model. We trained 5,000 steps and used the model to test the images of the training set, but it was obvious that the effect was not very satisfactory. Because $T\left(I,W_T^{\ast }\right)$ is based on Faster RCNN, which has high accuracy, it is easy to miss the mark of small faces. Therefore, the quality of the model can be preliminarily judged by the number of detected targets, and then we gradually increased the number of training steps.

When the number of training steps reaches 20,000 steps, the number of detected targets in the detection results of 1,000 test set images is basically maintained at about 1,300. As the number of training steps increases, the number of detected targets increases slightly. When the number reaches 60,000 steps, the number of detected targets is 1,523. At this time, precision rate of the model is 87.3%,and the recall rate is 85.9%. When the number of training steps reaches 70,000 steps, the number of detected targets is close to 1,700. At this time, the precision rate of the model is 81.2%, and the recall rate is 86.3%. We find that although the recall rate has a slight increase, but the precision rate is much lower, so we chose the model with 60,000 training steps as the final model. See Table 1 for the accuracy of $F\left(I,W_F^{\ast }\right)$ under different training steps.

Regarding to the scoring threshold, it is 0.5 by default, which means that when the score is lower than 0.5, the result will be discarded. We successively set the threshold to 0.3, 0.4, 0.5, 0.6, 0.7, and tested the validation data to choose the one that works best. Finally, we found that when the threshold is 0.6, the precision rate of the test result is 87.3%, and the recall rate is 85.9%, which is better than other thresholds. After comprehensive consideration, we finally keep 0.6 as the threshold for the ensemble. The ROC curve on the training set is shown in Fig. 7.

When training this model, in order to distinguish whether an individual wearing a helmet, we use two labels: people wearing and without wearing a helmet. It makes the final trained model more accurately distinguish whether the target wears a helmet.

Accuracy of $T\left(I,W_T^{\ast }\right)$.

In this part, we consider the accuracy of $T\left(I,W_T^{\ast }\right)$ alone in Algo. 2. It is basically a trained Tiny Face model. We lowered the scoring threshold of Tiny Face to 0.5, requiring the Tiny Face model to be able to determine the location of the small face, and it does not need it to accurately return the scoring value. The precision rate of face detection was 85.6%, and the recall rate was 69.4%.

Accuracy of $C\left(A_T,I,W_C^{\ast }\right)$.

In this part, we consider the accuracy of $C\left(A_T,I,W_C^{\ast }\right)$ alone in Algo. 2. Function $C\left(A_T,I,W_C^{\ast }\right)$ is basically a CNN model, which requires only one target in an image, so we select over 2,000 images from the training set, cropped the target according to the corresponding anchor labels and get 20,000 images with only one target in each image. We select 18,000 images as training data for CNN, and the other 2,000 images as a validation set to detect the accuracy of CNN, also the cropped images are divided into two sets, people wearing helmets and without wearing helmets. In addition, we rotate some images to get richer training samples.

With cross-validation, we choose to use four pairs of convolution and pooling layers, of which the first layer and the size of the convolution kernel of the second convolution layer are [5,5], and the size of the convolution kernel of the third and fourth convolution layers is [3,3]. The precision rate of the final two-class CNN reached 90.3% when we use it to test the validation set of CNN.

The ROC curve on the training set is shown in Fig. 8.

Accuracy of Ensemble Method $S\left(c_{SF},c_{SC},{\alpha }^{\ast }\right)$.

The ROC curve coverage areas of Faster CNN and A_T, c_C are 0.86and 0.83, respectively. The ensemble method can further improve the accuracy of the final result.

Among the data, c_F and c_C are the results from two base methods, respectively, and the area is the size of the target frame. Obviously, c_F and c_C can be used as the characteristic values of the ensemble method. We test the trained model, and the area under the ROC curve coverage is larger, becoming 0.90. The ROC curve on the training set is shown in Fig. 9.

Obviously, the ROC curve covered by the ensemble method has the largest coverage area, which proves that the ensemble method is effective in our model.

Comparison of different algorithms.

In this part, we demonstrate the effectiveness of our ensemble framework by combining Faster RCNN and Tiny Face+CNN together with ROC curve and PR curve. The ROC and PR curves are calculated from testing results through 5-fold cross-validation as Figs. 10 and 11. From Figs. 10 and 11, we can see that combination with our framework (in green) is better than single algorithms (in black and orange). Our framework can also gain the largest area under the ROC curve (0.83) in Fig. 10, and the largest area under the PR curve (0.93), namely the mAP score. It means our framework works best on average over all the possible threshold choices.

Tables 2 and 3 reveals a similar phenomenon when a reasonable threshold is chosen. It indicates that, with a well-chosen threshold, our framework works better than others in terms of TPR, FPR,FNR, precision, and recall.

Our framework can also be used to integrate other complementary deep learning methods to improve their performance. As an example, we use our framework to combine Mobilenet and TinyFace+CNN, and compare the integrated results with single algorithms. The performance is shown in Figs. 12 and 13. Similar to the previous case, the algorithm performance is generally improved. Our framework also works well when a specific threshold is chosen, as shown in Tables 4 and 5.

Through these experiments, we can find that the integrated framework for two complementary models can improve the performance of single algorithms by increasing the true positive rate, the precision rate, and the recall rate, while reducing the false positive rate and false negative rate.