Face Recognition in an Unconstrained and Real-Time Environment Using Novel BMC-LBPH Methods Incorporates with DJI Vision Sensor

: Face recognition (FR) in an unconstrained environment, such as low light, illumination variations, and bad weather is very challenging and still needs intensive further study. Previously


Introduction
Facial recognition (FR) has drawn more and more attention among researchers and practitioners over the last few decades due to its extensive use in many security fields. Nowadays, FR techniques are extensively used in public places such as airports, shopping malls, and stadiums due to security concerns. For example, FR systems have been used to identify the valid ticket holder in 2008 Olympic games in China [1] and; police identified unknown bodies by facial recognition techniques in California [2]. The existing FR techniques show promising results in favorable conditions (i.e., proper lighting, illumination) using the web camera and small data. However, the performance of FR techniques is not satisfactory in many cases due to the following limitations: large dataset led to 1.
A systematic analysis of LBPH parameter; 2.
Finally, a pilot test for real-time FR in rainy weather using DJI Phantom 4 UAV (DJI Phantom 4 is chosen due to its capability of sensing obstacle and highly capable camera sensor (i.e., effective pixels: 20 M, forward, backward and downward vision system) [11]) based on LBPH and BMC-LBPH FR technique.  The overall datasets used in this study is detailed in Table 1.

Radius and Neighbor of Lbph
Radius and neighbor value are the two most important parameters in the LBPH algorithm. The Radius ranges from 1 to 5, with 1 as the default value [12]. Increasing the radius may help to capture the details on a large scale. On the other hand, the neighbor value should be a positive integer varying from 4 to 24 and has 8 as default value [12]. Ojala et al. (1994) [13] stated that image textures could be described as a 2 p binary, regarded as uniform patterns. More uniform patterns mean that the image contains more texture information. According to Ojala, Pietikäinen, and Harwood (1996) [14], increasing the radius and neighbor size could reduce the number of uniform patterns in the images, which later postulated by another study conducted by Ahonen, Hadid, and Pietikainen (2004) [15]. However, the effect of neighbor value and radius on time complexity regarding face recognition was not justified boldly in previous research. Table 2 presents the percent of uniform patterns in the image with different neighbor values and radius. The neighbor value and the radius increased while the rate of uniform pattern abated. The previous studies did not show how neighborhood and radius affect FR accuracy and computational. Therefore, in this work, the effect of radius and neighborhood in unconstrained weather is statistically measured with computational time and accuracy.

Cell/Grid
Grid/Cell is another parameter of LBPH, in the face recognition method provided by OpenCV. An image contains several cells in horizontal and vertical directions, which can be denoted as grid x in the horizontal direction and grid y in the vertical direction. In Figure 2), the image was divided into multiple grids, horizontally and vertically. Each cell value was used to create the histogram of each region. The cell size was used to divide and normalize the histogram of the image. The histogram determines the binary distribution clustered over the image to produce the output features.  Figure 3 shows a 4-neighborhood LBP operator on a single image. The face region was divided into nine overlapping areas, and a 16-bin histogram was calculated for each area. Then the result was concatenated into a single 144-bin histogram. On the other hand, an LBP u2 8,2 also applied to the whole 19 × 19 size images to produce a 59-bin histogram. In total 203(59 + 144), bin histogram was used as a descriptor of the face region. Note that there was no specific boundary about horizontal and vertical cells on the images. The number of the grid has no specific range, and none of the previous research accounted for these issues. As a result, it was noticed that different studies used unique grid values (i.e., 8 × 8, 18 × 21, and 24 × 24) in their work [16,17] [20] experimented with varying sizes of the window of 3 × 3, 5 × 5, and 7 × 7, and obtained 86.7%, 91.4%, and 94.09% accuracy respectively; however, grid size was disregarded during the study, like it was absent in Reference [19]. Therefore, distinctive cell sizes, including 8 × 8, 16 × 16, and 24 × 24, are tested (inspired by Reference [20]) along with various radius and neighbor value with the focus on LBPH FR techniques in this work.

Mathematical Formulation
Consider an image I(x, y) and let g c = I(x, y) denote the gray level of the pixel(x, y). Let g p denotes the gray value of a sample point of the evenly spaced circular neighborhood of P sampling points and R radius around (x, y): g p = I(x p , y p ) where p = 0,...., p − 1 and Let the local texture of the image I(x, y) characterized by the joint distribution of the gray value of p + 1(p > 0) pixels: Without losing information, it is possible to subtract the central pixel value from the neighborhood: In the next step, the joint distribution is approximated, and the central pixel is assumed to be independent, which allows the factorization of the distribution: Here the first factorization t(g c ) does not contain useful information since the distribution is over I(x, y). Thus, we consider joint distribution differences: t(g 0 − g c , g 1 − g c , ...., g p−1 − g c ). However, collecting texture information from the multidimensional data is still difficult. Ojala, Pietikäinen, and Harwood (1996) [14] proposed a method that uses vector quantization. They used the codebook of 384 codewords to reduce the dimensionality [14]. Vector quantization is also difficult because the codebook must be trained similarly to the other texton-based methods. To get rid of this problem, the only sign of the difference is considered: The generic LBP can be obtained by summing the threshold differences weighted by powers of two. The LBP P,R operator, can be defined as From that equation, the texture can be described as: Figure 4 summarizes the framework to evaluate different LBPH parameters in FR methods. There is no specific experiment that considers the four basic parameters of LBPH (radius, neighbor, grid x, and grid y) in FR methods regarding accuracy and execution time. This paper is the first detailed attempt considering the unconstrained situations. Many of the earlier studies used LBPH parameters between the following range: radius = (1,5), neighbor = (4,24) [12,21]. However, other parameters grid_x and grid_y have no specific parameter range and most of the previous research used the grid value as 8 [12,21]. This study found that increasing the grid value did not increase the accuracy or reduce the execution time, nevertheless, grid value ranges from 8 to 24 was also considered to support the previous disclosure. Hence, experimenting with each of the four parameters of LBPH, will create a overall combination of 400,000 experiment. Thus, as an alternative approach, only following parameters was considered for the whole experiment: radius = (1 to 5), neighbors = (4,8), grid_x, grid_y = (8,8), (16,16), (24,24). Table 3 summarizes the overall result for the maximum and minimum accuracy and computation time in both LU and 5_celebrity databases. In 5_celebrity, the maximum accuracy found for the following parameters: r = 5, n = 8, x = 8, y = 8, and the minimum execution time yielded for the following parameters, r = 3,5; n =4; x = 8; y = 8. In contrast, in the LU database, several parameters yielded the maximum accuracy; and the low computation time eventuated for parameters such as r = 5, n = 4, x = 8, and y = 8.

Effect of LBPH Parameters
From the above experiment, it was found that there are no specific standard parameters with maximum accuracy with low computation in both databases (see Table 3). Moreover, it can be concluded that execution time will not be an issue if a user uses a fast GPU.

Proposed FR Framework
A novel LBPH based method, bilateral median convolution-Local binary pattern histogram (BMC-LBPH), was proposed to improve the FR accuracy in an unconstrained environment. The schematic representation was illustrated in Figure 5. Initially, the image is acquired and converted into a gray-scale scale image, which is time efficient and allows multiple-image manipulation process on the image. Any unnecessary object is removed from the image by selecting the face region and detecting the face using a cascade classifier. Then the simple slice method is used to crop the face for feature extraction. Once the face is cropped, the following filter is used for further image processing-firstly, a bilateral filter is applied to reduce the noise and illumination variation of face image [22][23][24]; secondly, the median blur is used to smoothen the overall image; finally, a convolution filter is used to produce robust feature vectors [25].
The overall experiment was conducted using all the images (100) from 5_celebrity dataset (contains 100 images), and 40 images (20 cloudy and 20 foggy) from LUDB (Other face images captured in rainy and sunny conditions were ignored since those are less challenging, and the existing LBPH and proposed BMC-LBPH showed almost 100% accuracy on those scenarios.) dataset with 5-fold cross-validation; and the performance of proposed BMC-LBPH and existing LBPH were statistically calculated and compared using the following formulas [1]: where, True Positive (t p ) = Face detected and matched with the true labels.
False Positive ( f p ) = Face detected but matched with the false labels. True Negative (t n ) = Not face and not matched with the labels.
False Negative ( f n ) = Face detected but not matched with the labels.  Table 4 presents the performance of existing LBPH and the proposed BMC-LBPH methods in 5_celebrity and LUDB dataset. The proposed methods (BMC-LBPH) outperformed existing LBPH methods in terms of precision and f1 scores in every scenario. However, the recall score for existing methods (LBPH) is slightly better in the LUDB dataset, especially on foggy images.  Figure 6a shows, the proposed BMC-LBPH outperformed existing LBPH FR techniques with higher accuracy on both datasets. However, on a foggy image, existing LBPH's accuracy (96.6%) is slightly higher than the proposed BMC-LBPH (95% accuracy). Similarly, Figure 6b revealed that the computation time of BMC-LBPH is efficient compared to traditional LBPH FR techniques both on 5_celebrity and LUDB.

Real-Time Face Recognition Using UAV
During the literature review, it was noticed that real-time FR systems in bad weather (i.e., foggy, cloudy, rainy) using UAV are either missing or ignored. Therefore, a pilot test was carried out with the initially available drone, DJI Phantom 4. A detailed specification of all necessary parts about the drone can be found in Reference [11]. Drone specifications are the following-DJI Phantom drone with 4 vision sensor; Intelligent Flight Battery-5350 mAh; Remote Controller; Operating Frequency-2.400-2.483 GHz; Gimbal Control range from −900 to +300; Camera-12.4 M pixel. Figure 7 shows a flowchart of the proposed methodology for real-time FR using a DJI Phantom 4 drone in bad weather. Initially, using GPS and DJI GO mobile app, the Phone was connected to the controller ( Figure 8a); with the drone live streaming was induced by the YouTube platform (Figure 8b)-DJI GO app has an option for live streaming using either Youtube/Facebook. If the battery strip on the drone becomes green (Figure 8c), then it is expected that the drone is ready to fly and for the pilot test. Figure 8d, shows recognized frontal face in rainy weather by DJI phantom 4.

Test Analysis
The test was implemented in rainy weather with two subjects. Table 5 presents the overall performance of existing LBPH and proposed BMC-LBPH in real-time FR under rainy weather. The overall precision and f1 score of subjects 1 and 2 for BMC-LBPH are higher than traditional LBPH FR techniques, yet the recall score is 1 for both algorithms. Besides, the accuracy achieved by the proposed BMC-LBPH is higher than that of the existing LBPH FR techniques in real-time for both subjects ( Figure 9). However, in real-time, the computation time for the proposed FR technique is equal to existing LBPH for subject one but slightly higher for subject two.

Discussion
In this work, a schematic examination was performed to evaluate the four LBPH parameters, including radius(r), neighborhood(n), grid x, and grid y. The experiment revealed that default parameters of LBPH (r = 1, n = 8, x = 8, y = 8) performed well both on 5_celebrity and LUDB dataset. Increasing radius and neighbor values resulted in higher calculation time and lowering those values, can not improve FR accuracy. Similarly, changing parameters of grid x and grid y can not improve FR accuracy or computation time at a satisfactory level. Thus, a novel LBPH based FR strategy, Bilateral Median Convolution-Local binary pattern histogram (BMC-LBPH), was proposed and experimented. The proposed method outperformed existing LBPH techniques on both datasets in terms of accuracy. However, in a foggy image, it performed slightly lower than traditional LBPH.
Moreover, a pilot test of a real-time FR in rainy weather was performed to assess the performance of the existing and proposed LBPH methods utilizing a DJI Phantom 4. The proposed LBPH outperformed the current LBPH strategies in real-time FR in unconstrained weather (rainy). The proposed methods increased the accuracy of FR up to 16%. Then again, the execution time for existing and recommended LBPH strategies are practically comparative. However, it is more likely to achieve higher accuracy using other strategies such as using convolutional neural network (CNN), deep CNN, and recurrent neural network (RNN) techniques, which are computationally expensive and require big data. Therefore, it could be inferred that the performance of the proposed BMC-LBPH technique is acceptable in real-time FR in bad weather concerning accuracy and execution time.
However, only two subjects were tested due to the time constraints, and the pilot test was conducted only in rainy weather. More subjects and varying weathers should be considered in the future study.

Conclusions
Real-time face recognition in rainy weather using DJI Phantom 4 drone was successfully implemented with the novel Bilateral Median Convolution-Local binary pattern histogram (BMC-LBPH) methods. The BMC-LBPH methods exceed the existing LBPH algorithm with increased accuracy up to 10% by taking the low computation time. Four LBPH parameters (radius, neighbor, grid x, grid y) were also tested in an unconstrained environment. The results confirmed that the default parameters (radius = 1, neighbors = 8, grid x = 8, grid y = 8) are ideal for accuracy and execution time. The overall experiment was conducted using small datasets (5_celebirty consists of 100 images, and LUDB comprises 40 images); therefore, further improvement of FR's accuracy is feasible using large scale data. Future works include but not limited to: face recognition in different unconstrained situations such as low light, face angle, and motions; testing the proposed BMC-LBPH methods on large scale data incorporates with various weather conditions such as foggy, cloudy, and sunny.