Statistical non-rigid ICP algorithm and its application to 3D face alignment

Highlights

• A statistical non-rigid ICP method for 3D face alignment is proposed.

• Local fitting in dynamic subdivision framework helps capture subtle facial feature.

• 2D point-driven mesh deformation in pre-processing step helps improve performance.

Abstract

The problem of fitting a 3D facial model to a 3D mesh has received a lot of attention the past 15–20years. The majority of the techniques fit a general model consisting of a simple parameterisable surface or a mean 3D facial shape. The drawback of this approach is that is rather difficult to describe the non-rigid aspect of the face using just a single facial model. One way to capture the 3D facial deformations is by means of a statistical 3D model of the face or its parts. This is particularly evident when we want to capture the deformations of the mouth region. Even though statistical models of face are generally applied for modelling facial intensity, there are few approaches that fit a statistical model of 3D faces. In this paper, in order to capture and describe the non-rigid nature of facial surfaces we build a part-based statistical model of the 3D facial surface and we combine it with non-rigid iterative closest point algorithms. We show that the proposed algorithm largely outperforms state-of-the-art algorithms for 3D face fitting and alignment especially when it comes to the description of the mouth region.

Introduction

Three-dimensional representation of face has always been a valuable source for face recognition and facial behaviour analysis. The huge descriptive power and rich shape variation of 3D human face make it much more informative than the corresponding 2D projection. Due to its complex nature, the structure and topology of the 3D facial scan varies from frame to frame. To obtain a relatively consistent facial representation, it is always preferable to use a 3D deformable facial model. Thus, the building and fitting of the 3D deformable model becomes a key step in retrieving 3D facial information, and not surprisingly, it has received a lot of attention in the past decade [1], [2], [3], [4], [5], [6], [7], [8]. Owing to the recent development on cost-effective depth cameras, such as Kinect 2, Creative Senz3D™ and Intel® RealSense™ Camera (F200), further attention has been drawn on this field of study.

3D face fitting, also referred to as 3D face registration [5], [8], [9] in the literature, aims at aligning two sets of point clouds or meshes, and overlapping the face template as close as possible with the target surface (i.e. facial scan). In the scope of 3D deformable model fitting, the face template refers to the 3D deformable face model. 3D Morphable Model (3DMM) [1] is the most commonly used technique to introduce prior knowledge on 3D human face. 3DMM has been widely adopted in various 3D fitting methods [2], [10], [11], [12], since it can well estimate an unseen 3D facial shape by solving the shape, texture, pose and illumination parameters simultaneously. Fitting of 3D facial models is very important, since they can be used to identify particular facial landmarks or to recognize and define facial deformations that benefits face recognition [3], [13] and facial performance transfer [14]. Furthermore, accurate fitting and modelling of human expressions is able to boost the performance of facial expression recognition[15], [16], [17].

Despite the importance, the majority of existing methodologies use just a simple pre-defined mesh model, parameterised or not, to fit a target mesh [2], [3], [5], [8], [11], [18]. In [2], [11], in order to handle recognition under different pose and lightening conditions, a 3DMM that separates parameters for shape, head pose and illumination is employed. Inspired by this, Amberg et al. [10] developed an expression-invariant 3D face recognition algorithm, in that, they fit an identity/expression separated 3DMM to the facial scan and normalize the resulting face by removing the pose and expression components [10]. Unfortunately, these methodologies may fail to describe properly the complex, non-linear and highly deformable structure of the face. Especially when fitting the data captured by high-resolution face capturing system, such as Di3D (Dimensional Imaging [19]), the results are likely to be over-smooth and thus lose important facial details and micro-expression.

In this paper, we examine the problem of fitting a 3D facial models to high-resolution depth scan. Our key contribution is a new active method for describing and fitting 3D faces, which is achieved by learning a set of local statistical model for facial parts, and combining them with the non-rigid Iterative Closest Point (ICP) algorithm [5]. Besides it, we propose a dynamic local fitting procedure that makes full use of dynamic subdivision framework. To this end, we successfully adopt the proposed active method in the fitting procedure, and show that it manages to accurately model the subtle facial feature. Additionally, we provide a point-driven mesh deformation procedure in the data pre-processing stage that helps to prevent incorrect facial part fitting. It deforms 3D template model under the guidance of the state-of-the-art 2D face alignment algorithm [20].

The remainder of the paper is organized as follows. Section 2 gives a brief introduction to the existing works that are related to this paper, and a short discussion over the merits and demerits of them. Section 3 explains our dynamic subdivision framework as well as the local fitting procedure. Next, in Section 4, we describe the proposed novel Active Non-rigid ICP algorithm. In Section 5, we show the results of our experiments followed by an in-depth discussion. Lastly, we conclude the paper in Section 6.