A robust face and ear based multimodal biometric system using sparse representation

Abstract

If fusion rules cannot adapt to the changes of environment and individual users, multimodal systems may perform worse than unimodal systems when one or more modalities encounter data degeneration. This paper develops a robust face and ear based multimodal biometric system using Sparse Representation (SR), which integrates the face and ear at feature level, and can effectively adjust the fusion rule based on reliability difference between the modalities. We first propose a novel index called Sparse Coding Error Ratio (SCER) to measure the reliability difference between face and ear query samples. Then, SCER is utilized to develop an adaptive feature weighting scheme for dynamically reducing the negative effect of the less reliable modality. In multimodal classification phase, SR-based classification techniques are employed, i.e., Sparse Representation based Classification (SRC) and Robust Sparse Coding (RSC). Finally, we derive a category of SR-based multimodal recognition methods, including Multimodal SRC with feature Weighting (MSRCW) and Multimodal RSC with feature Weighting (MRSCW). Experimental results demonstrate that: (a) MSRCW and MRSCW perform significantly better than the unimodal recognition using either face or ear alone, as well as the known multimodal methods; (b) The effectiveness of adaptive feature weighting is verified. MSRCW and MRSCW are very robust to the image degeneration occurring to one of the modalities. Even when face (ear) query sample suffers from 100% random pixel corruption, they can still get the performance close to the ear (face) unimodal recognition; (c) By integrating the advantages of adaptive feature weighting and sparsity-constrained regression, MRSCW seems excellent in tackling the face and ear based multimodal recognition problem.

Introduction

Biometric systems relying on a single trait have to contend with a variety of practical problems like noise, non-university, upper bound on identification accuracy, spoof attacks, etc. [1]. In order to address some of the limitations and improve recognition performance, multiple sources of information are proposed to be combined together to form multimodal biometric systems, which are generally believed to be more reliable and have attracted much attention recently. In the past decade, various multimodal combinations have been reported, including face and fingerprint [2], [3], face and iris [4], fingerprint and iris [3], face and ear [5], [6], [7], [8], [9], [10], etc. Based on the type of information available in a certain module, different levels of fusion can be defined, i.e., sensor level, feature level, and match score, rank, decision levels. Sanderson and Paliwal [11] categorized the fusions performed at the former two levels into pre-classification category, and call the rest as post-classification fusion. Post-classification fusions are fairly popular due to the ease of accessing and processing the match scores, ranks and individual decisions. In contrast, combinations at early stage are relatively difficult because the raw biometric data may contain noisy or redundant data, while features extracted from different biometric traits may be incompatible. Moreover, even the multimodal system using feature level fusion does not work when one or more modalities of query samples are not available [12]. Nevertheless, because of the capability of utilizing more information for classification, pre-classification fusions have drawn more attention in recent years. Especially, feature level fusion can exploit the most discriminative information and eliminate the redundant/adverse information from the raw biometric data, and hence it is expected to provide better performance [4]. In this paper, we will focus on feature level fusion and intend to address some limitations existing in face and ear based multimodal biometric.

Most recently, Sparse Representation based Classification (SRC) techniques have been successfully applied in Face Recognition (FR), and have becoming state-of-the-art techniques in pattern recognition [13], [14], [15], [16], [17]. SRC firstly encodes the query sample with a training sample dictionary and then classifies it to the class which yields the least square coding error. SRC could be seen as a more general model than the previous nearest classifiers, like Nearest Neighbor (NN), Nearest Feature Line (NFL) [18] and Nearest Subspace (NS) [19], [20], [21], and it uses the samples from all classes to collaboratively represent the query sample to overcome the small-sample-size problem in FR [14]. In this paper, we propose to apply SRC techniques, including the original SRC [13] and Robust Sparse Coding model (RSC) [17], to face and ear based multimodal biometric. Two SR-based multimodal methods are developed, namely Multimodal SRC (MSRC) and Multimodal RSC (MRSC). In these methods, appearance-based features of face and ear are separately extracted by using Principal Component Analysis (PCA) [22], and are then directly concatenated in series.

Generally, multimodal biometric systems incorporating more evidences from various modalities can provide better performance than unimodal biometric systems [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [23], [24], [25]. However, if one or more modalities encounter data degeneration, these systems may perform worse than unimodal systems. It mainly results from the fact that most of multimodal systems are based on fixed fusion rules, or their fusion rules cannot effectively adapt to the changes of environment or individual users. In face and ear based multimodal biometric, for achieving good robustness, we propose an adaptive feature weighting scheme based on a novel index called Sparse Coding Error Ratio (SCER). SCER can effectively measure the reliability difference between face and ear query images caused by various factors, such as illumination, pose, expression, corruption, and occlusion. The intuitive motivation is that in many applications face and ear query images are less likely to suffer from degeneration simultaneously, and even if that happens, the degeneration levels are often different. By incorporating the feature weighting scheme into MSRC and MRSC, we derive a second category of SR-based methods, which can dynamically reduce the negative effect of the less reliable modality. They are MSRC with feature Weighting (MSRCW) and MRSC with feature Weighting (MRSCW). Finally, we conduct experiments on two virtual multimodal databases established based on benchmark databases, including the Extended Yale B [26], AR face database [27] and the USTB ear database III [28]. Experimental results demonstrate that, under the conditions without severe degeneration, both categories of SR-based methods not only significantly outperform face and ear unimodal recognition, but also are much better than the multimodal recognition with common classifiers like NN and NFL, including those using serial concatenation and CCA-based feature fusion schemes. Moreover, the SR-based methods with feature weighting, MSRCW and MRSCW, show striking robustness to the image degeneration occurring to one of the modalities. Even when query sample of one modality suffers from 100% random pixel corruption¹, they can still get the performance close to the unimodal recognition with the other modality. It should be noticed that the adaptive feature weighting scheme is not designed to settle the recognition problem when face and ear simultaneously encounter equivalent image degeneration. Therefore, one should not always expect MSRCW or MRSCW to get improvement over MSRC or MRSC, respectively. There is also a result that MRSCW and MRSC perform much better than all the other competing methods in such scenario. This can be attributed to the RSC model, which applies a sparsity-constrained regression process to detect and reduce the effects of corrupted pixels. In our study, by integrating the advantages of adaptive feature weighting and sparsity-constrained regression, MRSCW seems excellent in tackling face and ear based multimodal recognition problem. Overall, our contribution in this paper is threefold, as follows:

• To measure the reliability difference between query samples of the two modalities, a novel index called SCER is introduced, which is critical for adaptive feature fusion;

• An adaptive feature weighting scheme based on SCER is developed for dynamic fusion of face and ear features. The fusion scheme is more flexible, and is proven to be very effective by our extensive experiments;

• SR-based classification techniques are first employed in face and ear based multimodal biometric field. By combining them with adaptive feature weighting, we derive two promising SR-based multimodal methods, namely MSRCW and MRSCW.

The rest of this paper is organized as follows. We outline the related works on face and ear based multimodal biometric in Section 2. Section 3 provides a brief review of SR-based classification techniques, including the original SRC and RSC model. In Section 4, our multimodal biometric system is described in details, including an adaptive feature weighting scheme and two categories of SR-based multimodal recognition methods. Section 5 conducts experiments to evaluate the proposed methods. Finally, concluding remarks are drawn in Section 6.