Phase measurement deflectometry with refraction model and its calibration

The phase measurement deflectometry considering the refraction effect is presented to measure the mirror surface in this paper. In the context of the conventional phase measurement deflectometry, the biplanar structure of the system constructed by spatial multiplexing of a screen or a half mirror with two screens is a compromise of traditional display technology, while they suffer from complex calibration process and low accuracy. To improve the system compactness and efficiency, a novel measurement model consisting of a transparent screen and an ordinary screen is used to determine the incident light. To compensate for the measurement errors caused by transparent screen refraction, the refraction of the transparent screen is characterized by two physical parameters, which can be calibrated thanks to the multi-stereo vision technique. Then, the improved mirror calibration method with the refraction model is proposed to determine the posed relationship of the system. After that, the three-dimensional (3D) information of mirror surface is restored by the radial basis function interpolation with the optimized refraction parameters and posed relationships from the gradient data which is transformed from the normal information. Higher measurement efficiency, higher measurement accuracy and more compactness of the proposed measurement method are verified by the experimental results. © 2018 Optical Society of America under the terms of the OSA Open Access Publishing Agreement

Traditionally, the mirror surface can be reconstructed by the phase measurement deflectometry with two planes according to the following three steps. First, the phase matching points on the screen at two positions are obtained by phase recovery technology [16,17]. Then, the incident light is determined by connecting two phase matching points and the normal information is determined by the incident light and reflected light. Finally, the mirror surface is restored by gradient integration or interpolation method [18][19][20]. In the measurement schemes with linear guide [4,6,12], the incident light is determined by moving the screen along the direction of the linear guide. The mirror surface is reconstructed by the intersection between the reflected light and the incident light. Although the scheme can be used to measure the object with large gradient or discontinuity, the application is restricted because the system calibration is complicated and the error caused by the linear guide is not well considered. Subsequently, the reflection photogrammetry [21] and the phase measurement deflectometry [22] with free screen positions without linear guide are proposed to reconstruct the mirror surface. The free positions of the screen [21,22] for determining the incident light are calibrated by the mirror calibration method [23][24][25][26]. Despite of reduced hardware complexity and easy implementations, the calibration computing complexity of these methods is increased due to the unconstrained movements of the screen. Meanwhile, the biplanar structure constructed by the spatial structure of the screen is proposed. The famous work is the phase measurement deflectometry with two parallel planes, which is proposed by Zhang [8][9][10][11]. Two screens and a beam splitter are used to construct the two parallel planes. The relationship between the height and phase information is established to restore the 3D information of the mirror surface. However, the measurement field of view is limited and the structure of the system is bloated due to the introduction of the beam splitter. In order to improve the measurement accuracy, the two planes are required to be strictly parallel on the hardware structure [10] and the positive parallelism of two planes needs to be improved by the phase correction technique on the software [11].
The biplanar structure of the system constructed by spatial multiplexing [4,21] of a screen or a half mirror [8] is a compromise of conventional display technology in conventional phase measurement deflectometry [10]. Inspired by the naked eye 3D technology [27], multi-layer screens technique which consists of an ordinary screen and a transparent screen is used to construct the biplanar structure. The transparent screen is a new-type display technology developed in recent years. It not only has the display function as an ordinary screen, but also has good light penetrability. This feature of light penetrability can overcome the drawbacks of the ordinary screen and makes it straightforward to construct the biplanar structure in phase measurement deflectometry independent of auxiliary equipments. In conventional phase measurement deflectometry, the deviation of light direction caused by the refraction is not considered, which is a major contributor to the system measurement error [8]. However, the measurement errors caused by the refraction can be effectively inhibited by introducing the refraction model.
A novel phase measurement deflectometry incorporating with the refraction model is proposed to measure mirror surface in this paper. In the aspect of the structure, an ordinary screen and a transparent screen are used to construct the biplanar structure of the system, and their positions do not require special adjustment. The refraction model of the front screen is established to obtain more accurate incident light. Before measuring the mirror surface, the multi-stereo vision technique is utilized to determine the refraction parameters of the front screen. And the improved mirror calibration method considering refraction effects is proposed to determine the posed relationships of the system. Then, the optimized refraction parameters and the posed relationships are used to determine the normal of the mirror surface. The 3D information of the mirror surface is restored by the radial basis function interpolation [18][19][20] from the gradient data which is transformed from the normal information. The proposed method combines the advantages of the scheme with free position of the screen [21,22] and the scheme with two parallel planes [8]. The system measurement accuracy is improved because the light propagation path is accurately determined by introducing the refraction model.
The rest of the paper is arranged as follows: in Sec. 2, the principle of phase measurement deflectometry without refraction model is described. In Sec. 3, the phase measurement deflectometry with refraction model is explained. In Sec. 4, the calibration methods are illustrated. The calibration and the measurement results are shown in Sec. 5. Section 6 shows the conclusions.

Phase measurement deflectometry without refraction model
There are two main schemes for the conventional phase measurement deflectometry without refraction model: 1) the biplanar structure is represented by different positions of a screen [4,21,22]; 2) the biplanar structure consist of two screens and a half mirror [8]. The repeated calibration and time-consuming is the major weakness of scheme 1. The structural defects caused by half mirror and low accuracy due to refraction error are the deficiency of scheme 2. On the contrary, the biplanar structure is constructed by an ordinary screen and a transparent screen in our proposed method. As shown in Fig. 1, the front screen is a transparent screen (the thickness of the front screen is exaggerated) and the back screen is an ordinary screen. When the phase-shifted fringe images are projected by the front screen, the back screen is highlighted as a backlight. When the fringe images are projected by the back screen, the front screen is highlighted as a transparent glass. Obviously, in terms of structural design, the proposed measurement scheme has the following advantages: 1) no repetitive calibration problem; 2) measurement field of view is not limited and the system structure is compact.
The measurement method without refraction model is shown in the Fig. 1, which is described as a cornerstone for our proposed measurement method with refraction model. The coordinate systems of camera, front screen and back screen are defined as  The phase recovery technique [16] is used to obtain the correspondence between the screen position and the camera pixel. b p is defined as The incident light i n intersects with two screens at b p and f p , respectively.
The mirror point o is calculated by the intersection between i n and r n . And the normal is the angle bisector between i n and r n .
i r = − + n n n (3) The normal data n can be transformed to the gradient g . , The radial basis interpolation function [18][19][20] is used to accurately restore the 3D information of mirror surface from gradient data g .

Phase measurement deflectometry with refraction model
Further, in practical applications, according to the law of refraction, the light deflection caused by the front screen is a factor that cannot be ignored. And the accuracy of the incident light can be improved through refraction modeling of the front screen. As shown in Fig. 2, with the refraction model, the imaging process of the pixel b p on the back screen is expressed as follows: first, the light emitted by the pixel b p intersects the back surface of the front screen at the point ' in order to determine the incident light more accurately, it is necessary to introduce the refraction model of the front screen in the phase measurement deflectometry. ' Through the refraction of the front screen, the light intersects the front surface of the front screen is f p along the direction t .
Where, 2 2 cos( ) 1 (cos( ) 1) It can be seen from the above model, the light propagation path can be uniquely determined by two points ( f p and M ) and the refraction parameters ( f d and η ). The correspondence between the two points ( f p and M ) can be determined by the phase recovery technology.

Calibration methods
As explained in the previous content, the refraction parameters of front screen and the posed relationships of the system need to be known in order to calculate the light propagation direction.

Calibration of refraction parameters
The light propagation path with refraction deflection can be determined by refraction parameters ( f d and η ). The multi-stereo vision technique is used to determine the refraction parameters of the front screen. The solution of refraction parameters includes the following four steps.
Step 1: The back screen is highlighted as a backlight. The calibration pattern is projected by the front screen and captured by the camera. The posed relationship between the front screen and the camera is determined by the Perspective-n-Point (PnP) method [28].
Step 2: The front screen is highlighted as a transparent glass. The calibration pattern is projected by the back screen and captured by the camera. The feature points in the calibration pattern are extracted to calculate the refraction parameters.
Step 3: The camera pose is changed multiple times, and then step 1 and 2 are repeated. The two poses of the camera are shown in Fig. 4.
Step 4: The refraction parameters are calculated by the trust-region-reflective algorithm [29,30]. The optimal refraction parameters are corresponding to the minimum cost function [Eq. (9)].

Calibration of posed relationships
The mirror calibration method [23][24][25][26] with a flat mirror is popularly used to determine the posed relationships of the system in conventional phase measurement deflectometry. The calibration of the posed relationship with the mirror calibration method includes three steps [ Fig. 5]: first, the PnP method is used to determine the posed relationship of the virtual image of the screen relative to the camera; second, the relationship between the screen coordinate system and its virtual image coordinate system is established based on the principle of mirror image; finally, the posed relationship of the camera relative to the screen is solved. Obviously, the posed relationship of the front screen relative to the camera can be completely determined by the conventional mirror calibration method. And the mirror calibration method is improved to suit the posed relationship calibration of the back screen relative to the camera. The improved mirror calibration method contains two steps. First, the posed relationship of the back screen relative to the camera is determined by the mirror calibration method without considering refraction of the front screen, which is considered as the initial value for further optimization by Levenberg-Marquardt (L-M) algorithm. Then, the refraction of the front screen is introduced in the cost function [Eq. (10)] of L-M algorithm to calculate the re-projection error. The optimal posed relationship value corresponds to the minimum re-projection error.
where, g is the number of mirror poses, k is the number of camera pixels, ij M is the physical point on the back screen. p is the real pixel coordinate of back screen, and p is the calculated pixel coordinate of the back screen. RT is the posed relationship between the back screen and the camera, and * * R T is the optimal posed relationship value after optimization.

Calibration of refraction parameters and posed relationships
The multi-stereo vision technique is used to determine the refraction parameters of the front screen. Since the camera and screens do not have a common field of view, the auxiliary device is introduced to implement the multi-stereo vision technique, often an auxiliary camera or a flat mirror, as shown in Fig. 4. Depending on the principle of mirror image, the camera and its 'virtual camera' in the mirror are equivalent. Hence, the calibration result of refraction parameters is consistent whether the camera obtains the screen image through the planar mirror or the camera directly obtains the screen image. In order to reduce the introduction of auxiliary equipment, a high precision mirror with flatness 100 nm is utilized to connect the camera with screens and calibrate the refractive parameters, which is also used in the calibration of posed relationships. The flat mirror is placed at 8 positions. It is worth noting that the mirror pose needs to be adjusted so that the camera can observe the same area of the back screen. In each position of the mirror, the fringe images projected by the screens are captured by the camera through the mirror reflection. And then, the absolute phases are restored from these fringe images by the robust Chinese remainder theory (the absolute phases of two mirror pose are shown in Fig. 7). The above steps ensure that the pixel points of two screens and the pixels of camera are mapped together. In each position of the mirror, the posed relationship of the front screen relative to the camera is determined by the PnP method from the correspondence between the pixel positions of the front screen and the pixels of camera. The eight posed relationships are unified to the coordinate system of the front screen. Finally, the refractive parameters of the front screen are calculated by the trust-regionreflective algorithm. The refraction index and the thickness of the front screen are 1.85 and 8.5 mm, respectively. During the calibration of the posed relationships, the flat mirror is placed at 3 different positions [ Fig. 5]. The fringe images projected from two screens are captured by camera successively through mirror reflection. The robust Chinese remainder theory is used to restore the absolute phases [ Fig. 8] from captured fringe images. The posed relationships between the virtual front screen inside the mirror and the camera are determined by the PnP method. And then, the posed relationship of the front screen relative to the camera is obtained by the principle of mirror image directly. At the same time, the normal of mirror and the distance from the camera to mirror at each mirror position are obtained. They are used with the above obtained refraction parameters to determine the posed relationship of the back screen relative to the camera through the L-M algorithm. The optimal posed relationship values correspond to the minimum re-projection error.

Mirror measurement
Optimized calibration parameters are used to measure a flat mirror (Ø76 mm with flatness 100 nm). The robust Chinese remainder theory is used to restore the absolute phase from captured fringe images which are projected from two screens and modulated by the measured mirror.
As shown in Figs. 9(b) and 9(d), the maximum of gradient error is approximately 1.2 × 10 −3 . The radial basis function interpolation is used to restore the 3D mirror surface [ Fig.  10(a)] from the gradient information [Figs. 9(a) and 9(c)]. The reconstruction error of mirror surface is shown in Fig. 10(b). From the reconstruction error, it can be seen that the system measurement accuracy is around 3.2 µm.  If the mirror surface is restored by the radial basis interpolation function without optimized refraction parameters from the gradient data, the reconstruction accuracy of the system is reduced to 8.5µm. A mirror measurement example without refraction parameters is shown in Fig. 11. The same measurement process with the Fig. 10 is applied for measuring a spherical mirror with radius 35.3136 mm. Figure 12(a) describes the reconstructed 3D surface of the spherical mirror. Figure 12(b) shows the reconstruction error, where the maximum deviation is around 2.93µm.
In terms of measurement accuracy, the proposed method outperforms the 0.02-mm measurement error of Xiao [21,31] and 0.023-mm measurement error of Zhang [10]. A transparent screen is used to construct the biplanar structure, which makes the system more compact and more efficient. The measurement accuracy is improved by determining the refraction parameters of the front screen.

Conclusions
The phase measurement deflectometry with refraction model for mirror surface measurement and its calibration methods are presented in this paper. The biplanar structure is constructed by an ordinary screen and a transparent screen in the system. The refraction of the front screen is modeled and calibrated by the multi-stereo vision technique. The light propagation path is accurately determined by the refraction parameters. The improved mirror calibration method with the refraction model is proposed to determine the posed relationship of the system. The mirror surface is reconstructed by the radial basis interpolation function with the optimized refraction parameters and posed relationships from the gradient data which is calculated from the incident light and reflected light. Compared with conventional phase measurement deflectometry, the proposed method has advantages in terms of structure compactness, measurement efficiency and measurement accuracy.