Finger Vein and Inner Knuckle Print Recognition Based on Multilevel Feature Fusion Network

Abstract

Multimodal biometric recognition involves two critical issues: feature representation and multimodal fusion. Traditional feature representation requires complex image preprocessing and different feature-extraction methods for different modalities. Moreover, the multimodal fusion methods used in previous work simply splice the features of different modalities, resulting in an unsatisfactory feature representation. To address these two problems, we propose a Dual-Branch-Net based recognition method with finger vein (FV) and inner knuckle print (IKP). The method combines convolutional neural network (CNN), transfer learning, and triplet loss function to complete feature representation, thereby simplifying and unifying the feature-extraction process of the two modalities. Dual-Branch-Net also achieves deep multilevel fusion of the two modalities’ features. We assess our method on a public FV and IKP homologous multimodal dataset named PolyU-DB. Experimental results show that the proposed method performs best and achieves an equal error rate (EER) of the recognition result of 0.422%.

1. Introduction

Biometric recognition aims to distinguish individuals based on human physiological and behavioral characteristics and is widely applied in Internet-of-Things security [1], financial payment [2], and security systems [3], etc. A biometric system usually includes image acquisition [4], preprocessing [5], feature extraction [6], and matching [7], as shown in Figure 1. Due to the convenience of acquisition, finger-based biometric recognition methods have developed rapidly. With the increasing demand for accurate identity recognition in recent years, multimodal recognition methods have attracted wide attention [8,9,10]. Finger vein (FV) and inner knuckle print (IKP) are two modalities of the finger. FV is the tiny blood vessels inside the finger close to the cortex [11], and IKP is the pattern of the inner surface of the knuckle [12]. We use these two modalities to achieve identity recognition in this study.

There are two main problems to be solved in designing an method based on multimodal biometric recognition: how to extract each modality’s features and how to fuse multiple modalities’ information.

The traditional feature extraction of IKP and FV mainly relies on hand-designed schemes [13,14,15,16,17]. Although traditional feature-extraction methods can achieve good results from specific designs, they are complicated. To solve this problem, we use deep learning models for feature extraction. CNN is suitable for image feature extraction, as the model has a strong generalization ability after pretraining large-scale data. More importantly, the input images of the model do not require complicated preprocessing. We use the Inception-ResNet-v1 [18] model for feature extraction in this study. The experimental results show that the model can extract IKP and FV image features well. In addition, we introduce the triplet loss function [19] as the loss function to optimize the model. The function forces the feature representation distance of different sample images of the same person to be close and that of sample images of different people to be far away. It improves the expression ability of the fusion feature.

Feature-level fusion is the mainstream method for multimodal fusion when using deep neural networks to extract image features [20,21,22,23]. The feature dimension is large, and the operability space of fusion is strong. Some previous works only performed a simple concatenation operation on multimodal features [24,25]. Such fusion method does not consider the features of different stages in the feature-extraction process, which will affect the final recognition performance. Multilevel feature fusion can better represent features. Therefore, we propose to fuse IKP and FV features at different scales and perform multilevel feature fusion. Given the difference between the two modalities, the method takes FV as the main task modality and IKP as the auxiliary task modality. In summary, our contributions are as follows:

• We propose a Dual-Branch-Net network to extract the features of FV and IKP. The features of the FV branch at multiple levels of the network are fused with the corresponding features of the IKP branch, achieving deep fusion between the features of the FV and the IKP.
• We use transfer learning and triplet loss function to optimize the multimodal feature representation and perform identity recognition by measuring the similarity of feature vectors.
• We perform a series of experiments to verify that the recognition performance of our method is superior to both unimodal and other multimodal recognition methods.

2. Related Works

The research related to our work can be mainly divided into two categories: one is feature extraction of biometrics; the other is feature fusion of multiple modalities.

2.1. Feature Extraction

Existing feature-extraction methods are classified into two groups: non-training-based [13,26,27,28] and training-based [22,24,29,30] methods. Most traditional feature-extraction methods do not require training. For example, Kumar and Zhou [13] used the Gabor filter to extract the feature maps of FV images on the preprocessed images and then used the morphological operation to enhance the definition of the feature maps. They also used the local Radon transformation to extract the features of IKP images. Evangelin and Fred [26] used the gray-level co-occurrence matrix (GLCM) to extract the second-order texture features of finger knuckle print (FKP) images. Li et al. [27] proposed a new feature expression method based on local encoding using the generalized symmetric local graph structure (GSLG) to express the position and orientation relationship of domain pixels. Veluchamy and Karlmarx [28] extracted the features of FKP and FV using a repeated line tracking method. Although these traditional feature-extraction methods do not require training of feature extractors, the quality of images affects their extraction methods. Thus, it is necessary to perform complicated preprocessing processes such as region of interest extraction, orientation correction, and image enhancement. In addition, the characteristics of each modality are different, requiring special feature-extraction methods.

Although feature extraction through CNN requires pretraining of the network model, it has become a popular feature-extraction method in biometrics because of its less complex image preprocessing procedure and effective feature extraction. For example, Daas et al. [22] extracted the features of FV and FKP through CNN. Wang et al. [29] first used a five-layer CNN to extract each modal feature and then used principal component analysis (PCA) to standardize the features. Ren et al. [30] proposed a multimodal finger recognition network—namely, FPV-Net—to extract the features of FVs and fingerprints. Kim et al. [24] fine-tuned VGGNet [31] and ResNet [32] and used these pretrained models for the feature extraction of FV and finger shape. Regarding these works, this study utilizes a large-scale pretrained model and triplet loss function to obtain feature representations of multimodal images.

2.2. Multimodal Fusion Method

Multimodal fusion usually includes three levels: pixel level [33], score level [13,20,34,35], and feature level [22,23,30,36]. For example, Kumar and Zhou [13] proposed a score-level fusion function called Holistic Fusion, which considers that the matching score of one modality may be more stable than another. The function results in a similar trend between the total matching score and the matching score of the more stable modality. Noh et al. [20] generated composite images of FV texture and composite images of FV shape to be used as input for the DenseNet [37]. Then, matching was performed using the obtained output matching score. Finally, they performed score fusion under different rules such as weighted product, weighted sum, perceptron, and Bayesian. Alay and Heyam [34] used five convolution layers and two fully connected layers to extract the features of each modality and then inputted them into their softmax classifier to obtain the similarity score. They used two score methods, namely, the arithmetic mean rule and the product rule. Walia et al. [35] achieved optimal score-level fusion by boosting and suppressing concurrent classifiers and resolving conflict among discordant classifiers.

Feature-level fusion can achieve better recognition results than pixel-level and score-level fusion because feature-level fusion preserves more original information [38]. Daas et al. [22] first used the network to extract the features of the FV and FKP images, obtained two 1D feature vectors, and then concatenated the two feature vectors into a full feature vector. Wang et al. [36] proposed a two-channel CNN feature fusion framework. The face image feature and the FV image feature were taken as the input to the self-attention mechanism to obtain their respective attention weights. The vein and face features were combined with their respective attention weights and concatenated to form splicing features. Then the splicing features were convolved to obtain the fused features. Ren et al. [30] used a weighted average to fuse the features of the two modalities through the attention module. They also further fused the shallow feature and deep feature of the network. Wen et al. [23] used a convolutional network to extract the 3D feature tensors of each modality. Next, feature tensors were concatenated in series according to the dimensions of the channels into a total 3D feature tensor, which was then sent to the subsequent network layers for further processing. Compared with these works, we adopt a feature-level fusion strategy fusing features at a deeper level.

3. Proposed Method

This section first introduces the overall framework of fusion recognition based on FV and IKP; then, it introduces the designed Dual-Branch-Net and the loss function used in detail. Finally, it introduces using transfer learning to initialize the network parameters.

3.1. General Framework

Figure 2 shows the framework of the proposed FV and IKP fusion network. The input of the network is a batch of image pairs (image of the FV and its corresponding IKP). After the batch goes through architecture and L2 regularization, the feature representation of N image pairs is obtained. The triplet loss function is adapted in this study to calculate the loss and then optimize the network parameters through backpropagation. Finally, the mapping space corresponding to the network satisfies the following characteristics. In this space, the feature representation distance of different image pairs belonging to the same finger is relatively close. By contrast, the feature representation distance of different image pairs belonging to different fingers is far apart.

We use the feature representation outputted by the network for authentication. Specifically, given image pair A of the subject and image pair B of the target object, if the similarity (expressed by Euclidean distance) between the feature representation of A and that of B is greater than the threshold, the subject and the target object are considered to belong to the same person; otherwise, they belong to different people. In addition, if the similarity between the test subject and multiple target objects is greater than the threshold, the target object category corresponding to the maximum similarity is selected as the final category of the test subject.

3.2. Model

The network architecture in Figure 2 is our proposed fusion model Dual-Branch-Net, and its specific architecture is shown in Figure 3. Dual-Branch-Net uses two branches to extract features from IKP images and FV images, respectively. The network adopts a multilevel deep fusion method to make full use of the two types of modal information. We detail the structure of the branch network and the feature fusion method below.

Dual-Branch-Net uses the Inception-ResNet-v1 [18] network as the base network for the two branches. The Inception-ResNet-v1 network includes three types of basic network modules as follows: (a) Stem—preliminary feature extraction for the original image; (b) Inception-Resnet-A/B/C—for further feature extraction, in which the input features of this module have the same number and size of dimensions as the output features; and (c) Reduction-A/B/C—further feature extraction. With this module, the height and width of the output feature will become smaller than the input feature. Meanwhile, the number of output channels will increase. Considering the recognition performance and computational efficiency, we do not directly use the original Inception-ResNet-v1 network structure but make some adjustments. The details are as follows: the number of Inception-Resnet-A was reduced from 5 to 3, the number of Inception-Resnet-B was reduced from 10 to 6, and the number of Inception-Resnet-C was reduced to 3.

Unlike some works [22,23] mentioned in the introduction, which directly concatenate the features of the two modalities only once, we fuse the features of the two modalities at a deeper level. The network layers at different depths have different receptive fields, and the output features of different network layers represent the features of the original image at different levels. Thus, we perform feature fusion operations at multiple levels of the network. Specifically, we splice the output features of both branch after the Stem modules, the last Inception-Resnet-A module, the last Inception-Resnet-B module, and the last Inception-Resnet-C module, respectively. The spliced features are used as the input of the subsequent network layer of the corresponding branch of the FV image, as shown in Figure 3. We have marked the number of channels, height, and width of the output of each module in the figure. Suppose we use the number of channels (C), height (H), and width (W) to represent the shape of a 3D feature; for two features of shape (C, H, W), we concatenate them into a full feature of shape (C, 2H, W) tensor. Considering that when only single-modality is used for recognition, the recognition effect of the method based on the FV is better than that based on the IKP; so, we fuse the feature of the IKP into the branch corresponding to the FV.

3.3. Loss Function

In the authentication phase, the Euclidean distance between two feature vectors determines whether the test subject matches the target object. Therefore, the selected loss function should make the samples of the same category close in the feature space and the samples of different categories far away in the feature space. The triplet loss function [19] is a function that can make the optimization direction of the network parameters meet this characteristic. Therefore, this paper will introduce the main idea of the triplet loss function.

When using the triplet loss function to optimize the network, some triples need to be constructed for each batch of FV and IKP image pairs (referred to as “samples” hereafter) and corresponding labels from them first. A triplet consists of an anchor sample, a positive sample, and a negative sample. Among them, anchor samples can be selected arbitrarily, samples of the same category as the anchor samples can be regarded as positive samples, and samples of different categories from the anchor samples can be regarded as negative samples. For a batch of image pairs, multiple triples can be constructed.

For each triplet, we first use Dual-Branch-Net to extract features to obtain the corresponding three feature vectors, as shown in Figure 4, and then calculate the loss of the network on the triplet according to formula (1):

$$L(A,P,N)=max(\parallel f\left(A\right)-f\left(P\right){\parallel }^2-\parallel f\left(A\right)-f\left(N\right){\parallel }^2+\alpha ,0)$$

(1)

where A is the anchor sample, P is the positive sample, and N is the negative sample. $f\left(\right)$ represents the Dual-Branch-Net network, and $f\left(x\right)$ represents the feature vector corresponding to the sample x. $\alpha$ is a fixed distance threshold. Only if the distance between the negative sample and the anchor sample and the distance between the positive sample and the anchor sample is not less than $\alpha$ will the loss of the network on the corresponding triplet be 0; otherwise, the loss will be greater than 0. Then, the model will then optimize the network parameters to adjust the feature representation of the sample (the arrow roughly shows the adjustment direction in the Euclidean space in Figure 4).

3.4. Initialize Network Weights

Since the network contains many trainable parameters, the dataset containing FV and IKP images is relatively tiny. If the network parameters are randomly initialized, it is difficult for them to converge to the optimal value. Therefore, we use transfer learning to load the parameter values of the pretrained model. We initialize the Dual-Branch-Net with the weights of the Inception-ResNet-v1 network pretrained on the large-scale dataset CASIA-WebFace [39].

4. Experiments

This section first introduces the experimental environment, dataset, and evaluation metrics. Next, we introduce the content and results of multiple experiments in detail in several subsections: (a) In the comparative experiment in Section 4.2, the proposed method is compared with other multimodal recognition algorithms to verify the effectiveness of the proposed method. (b) In the ablation experiment in Section 4.3, we compare the proposed fusion method with other fusion methods and compare the recognition performance of the multimodal and unimodal methods. Through these experiments, we explore whether the proposed fusion method is appropriate and whether it is necessary to use multimodal information for recognition. (c) In Section 4.4, by comparing different initialization methods of parameters, we explore whether it is necessary to use the parameter values of the pretrained model to initialize the proposed model.

4.1. Experimental Setting

The memory of the experiment environment in this study is 32 GB, the CPU is an Intel Core i9-9940X, the GPU is an Nvidia GeForce RTX 2080 Ti, the programming language is Python, and the deep learning framework is TensorFlow.

During training, network parameters are optimized with stochastic gradient descent (SGD) using an adaptive gradient algorithm (AdaGrad). We set the learning rate to 0.01 and the training number to 500 epochs. We set the batch size to 30 due to GPU memory constraints and the margin value $\alpha$ in the triplet loss value to 0.2.

4.1.1. Dataset

Our experiments use the Hong Kong Polytechnic University public finger image dataset (version 1) [13], referred to as PolyU-DB. Kumar and Zhou [13] divided the collection process of this dataset into two sessions: the data collected in the first stage consist of 6 images of the index finger and middle finger of one hand of 156 people, a total of 1872 images; in the second session, for 105 of the 156 people in the stage, 6 images each of the index and middle fingers of the same hand were collected again, for a total of 1260 images. The final dataset contains 3132 FV and IKP each, belonging to 312 categories. Figure 5 shows some of the images in the PolyU-DB dataset.

During the experiment, we divide 312 categories of the PolyU-DB dataset into training and test sets. Among them, the training set contained 210 categories (2520 FV and IKP image pairs), while the test set contained the remaining 102 categories (612 image pairs), as shown in

Table 1. Dataset division and description.

Dataset	Subject Numbers	Finger Numbers	Image Numbers of each Finger	Total Image Number
Training Set	105	2	12	2520
Test Set	51	2	6	612

. To alleviate the problem that the model may not converge due to the small training set, we adopt image augmentation techniques such as translation clipping and conversion on the training set images. Thus, the data volume of the final training set is five times that of the original training set (12,600 image pairs).

4.1.2. Evaluation Metrics

The experiments use the receiver operating characteristic (ROC) [40] curve and the equal error rate (EER) to evaluate the performance of the fusion recognition system of FV and IKP. We draw the ROC curve according to the value of the false positive rate (FPR) and the value of the true positive rate (TPR) based on different thresholds, which can reflect the balance between the false reject rate (FRR) and the false accept rate (FAR). When the value of FRR is equal to that of FAR in the ROC curve graph, the value of EER is that of FAR. The smaller the value of EER, the fewer instances of matching and recognition errors, and the higher the recognition accuracy.

4.2. Comparative Experiments

To verify the effectiveness of the proposed method, we compare multiple methods, as shown in

Table 2. Results of comparative experiments. Non-training-based represents traditional methods without training, and training-based represents methods based on CNNs. The unimodal methods use only FV, while the multimodal methods use FV and IKP (methods with * use FV and hand shape).

	Non-Training-Based Method	EER/%	Training-Based Method	EER/%
Unimodal	Wide line detector [44]	9.46	SqueezeNet [43]	3.7
	Mean curvature [44]	6.49	Light CNN [43]	10
	Repeated line tracking [44]	4.45	ResNet-50 [43]	5.6
Multimodal	Holistic [13]	2.72	VGGNet-16 * [24]	2.4433
	Nonlinear [13]	2.45	ResNet-50 * [24]	1.0235
			ResNet-101 * [24]	0.7859
			AsymmetricNet  [23]	3.393
			ResNet50-Softmax-Add [22]	1.109
			Dual-Branch-Net (ours)	0.422

. We divide the methods in the table into the following four categories:

(a)

Non-training-based unimodal methods: Including the wide line detector method [15], mean curvature method [16], and repeated line tracking method [17]. These three algorithms were common methods in FV recognition before deep neural networks became the mainstream method for feature extraction.

(b)

Non-training-based multimodal methods: Holistic [13] and nonlinear [13] are two methods using score-level fusion.

(c)

Training-based unimodal methods: Light CNN [41], ResNet-50 [32], and SqueezeNet [42] are all general networks based on CNN. Wimmer et al. [43] used them to realize FV recognition by combining triplet loss.

(d)

Training-based multimodal methods: VGGNet-16 [31], Resnet-50, Resnet-101 [32], ResNet50-Softmax-Add [22], and AsymmetricNet [23]. The former three are also general convolutional networks. Kim et al. [24] combined them with score fusion methods to complete identity recognition. The latter two, like our method, are based on feature-level fusion. We reproduced them for multimodal recognition based on IKP and FV.

We found several phenomena from Table 2. First, the EERs of all multimodal methods are lower than those of the unimodal methods. The main reason behind this phenomenon is that multimodal recognition methods can use more biological information to distinguish individuals from more perspectives. Second, among multimodal recognition methods, most training-based methods can achieve better recognition results than non-training-based methods. This verifies the effectiveness of CNN in this field. Third, our proposed method achieves the lowest EER. Although AsymmetricNet and ResNet50-Softmax-Add both use feature-level fusion methods, their effects are not as good as our proposed Dual-Branch-Net. This is probably because the fusion methods they designed only fuse deep features of multiple modalities. In contrast, Dual-Branch-Net fuses multilevel multimodal features, making the fusion more thorough.

4.3. Ablation Experiments

To verify that the proposed network model can effectively extract the fusion features of FV and IKP, we perform ablation experiments based on different modalities and different fusion methods.

First, to verify the effectiveness of the proposed fusion method, we compare the proposed fusion method with another simple splicing fusion method. Specifically, the features of the two modalities are only fused once at the output end of the last fully connected Inception-Resnet-v1 by concatenation operation. We concatenate the 512-dimensional output features of FV and the 512-dimensional output features of IKP to form 1024-dimensional features. We denote a Dual-Branch-Net with only stage 4 connections as Dual-Branch-Net (only stage 4 fusion). We present the ROC curve of the experiment in Figure 6 and the precise EER value in

Table 3. Results of ablation experiments.

Method	EER/%
Inception-ResNet-v1 (IKP)	2.373
Inception-ResNet-v1 (FV)	2.222
Dual-Branch-Net (only stage 4 fusion)	0.799
Dual-Branch-Net	0.422

. The ROC curve of the Dual-Branch-Net (only stage 4 fusion) method shifts to the lower right compared with that of the Dual-Branch-Net. This indicates that the recognition performance of the simple-fusion method is worse than that of the proposed fusion method. The EER value based on the Dual-Branch-Net (only stage 4 fusion) in Table 3 indicates that the corresponding EER values vary by 0.377%. There is one possible cause for the deterioration of the performance of Dual-Branch-Net (only stage 4 fusion). The feature fusion depth of the multimodal collaborative network output is not enough, as it does not consider the multilevel information fusion of the network. However, Dual-Branch-Net fuses features at different levels. Thus, there is less information loss and more thorough fusion.

Second, we compare our multimodal recognition method with a unimodal recognition method based on IKP or FV to study the necessity of recognition based on multimodal features. In the unimodal experiment, we first use Inception-ResNet-v1 to extract the features of the unimodal image and perform L2 regularization on output features. We then combine the triplet loss function to optimize the network. After that, the trained network model is used to extract the feature of the image pairs. Finally, according to the similarity between the feature vector of the test image and the target image, we can determine whether the experimenter corresponding to the test image is a real matcher or an impostor. From the experimental results in Figure 6 and Table 3, we find that the performance of multimodal recognition of FV and IKP is significantly better than that of the unimodal recognition of FV or IKP. Therefore, the result confirms the feasibility of the proposed multimodal recognition system.

In addition, from the experimental results in Figure 6, the performance of FV recognition is better than that of IKP recognition. Therefore, we use FV as the primary task modality of Dual-Branch-Net, and IKP as the auxiliary task modality.

4.4. Discussion of Pretraining

We conduct experiments with and without pretraining parameters loaded to explore the necessity to initialize the Dual-Branch-Net model. We present the ROC curves of the experiments in Figure 7 and the EER values in

Table 4. Results of experiments with and without pretraining.

Method	EER/%
	Without Pretraining	With Pretraining
Inception-ResNet-v1 (IKP)	6.634	2.373
Inception-ResNet-v1 (FV)	5.764	2.222
Dual-Branch-Net	3.750	0.422

. The methods with the “with pretraining” suffix in Figure 7 represent using the pretrained network weights to initialize the weights. The pretrained network weights are generated by Inception-Resnet training on the large-scale dataset CASIA-WebFace. Meanwhile, the methods with the “without pretraining” suffix randomly initialize the network weights.

Compared with the method of loading pretrained weights, the recognition performance of randomly initializing the network parameters dropped significantly. Numerically, the variation of the EER value is more than 3%. The recognition performance of randomly initializing the network parameters is not even as good as some traditional methods mentioned in Table 2. The main reason for this phenomenon is the large number of model parameters. Specifically, the combined space of model parameter values is large, and the training data used are less, being insufficient to make the model weights converge to the optimal value. Therefore, when using a large-scale network model in the multimodal recognition of FV and IKP, it is necessary to use transfer learning to load the weights of the pretrained model.

5. Conclusions

We proposed a network named Dual-Branch-Net, which can perform identity recognition based on dual modalities of IKP and FV. Unlike previous work that fuses multimodal features using simple splicing, our method combined two features at multiple levels. Moreover, we used both transfer learning and triplet loss to optimize the feature representation of the model. The experimental results on the public dataset PolyU-DB showed that the proposed method can reduce the EER of the recognition result to 0.422%. In the future, we will try to use some model distillation methods to reduce the computational complexity of the proposed method so that low-cost embedded devices can also run the method.