University of Huddersfield Repository 3D shape measurement of discontinuous specular objects based on advanced PMD with bi-telecentric lens

: This paper presents an advanced phase measuring deflectometry (PMD) method based on a novel mathematical model to obtain three dimensional (3D) shape of discontinuous specular object using a bi-telecentric lens. The proposed method uses an LCD screen, a flat beam splitter, a camera with a bi-telecentric lens, and a translating stage. The LCD screen is used to display sinusoidal fringe patterns and can be moved by the stage to two different positions along the normal direction of a reference plane. The camera captures the deformed fringe patterns reflected by the measured specular surface. The splitter realizes the fringe patterns displaying and imaging from the same direction. Using the proposed advanced PMD method, the depth data can be directly calculated from absolute phase, instead of integrating gradient data. In order to calibrate the relative orientation of the LCD screen and the camera, an auxiliary plane mirror is used to reflect the pattern on the LCD screen three times. After the geometric calibration, 3D shape data of the measured specular objects are calculated from the phase differences between the reference plane and the reflected surface. The experimental results show that 3D shape of discontinuous specular object can be effectively and accurately measured from absolute phase data by the proposed advanced PMD method.

3D shape measurement of discontinuous specular objects based on advanced PMD with bi-telecentric lens 1

. Introduction
Specular free-form surfaces are recently widely used in many fields, such as optics, car industry, aerospace, and micro-electro mechanical systems (MEMS). The research of three dimensional (3D) shape measurement for these objects is still in the early stage. The traditional fringe projection techniques [1][2][3][4] can only measure diffused surfaces. In order to measure specular objects by using fringe projection, the surface characteristics need to be changed in advance, for example, coating the surface by powder. This operation will slow the measurement speed and reduce the measurement accuracy.
Phase measuring deflectometry (PMD) [5][6][7][8], which is aimed at measuring specular freeform surfaces, has been widely studied in recent years, because of its advantages of large dynamic range, non-contact operation, full-field measurement, fast acquisition, high precision and automatic data processing. PMD has been applied to measure aspheric mirror [9] and dynamic specular surface [10]. Phase information of the deformed fringe patterns is demodulated to obtain the slope of the measured specular surface and then 3D shape of the measured specular surface can be reconstructed by integrating the gradients. Each point of a reconstructed shape depends on the gradients of surrounding points because of the integration procedure, and the regularity of the surface must be carefully considered [11][12][13]. Because the slope data calculation and the integration procedure are sensitive to the system errors, the obtained 3D shape data are inaccurate. The other drawback is that discontinuous specular objects cannot be measured because of the integration procedure. In order to measure discontinuous specular objects, some researchers have proposed several improved PMD methods. Guo et al [14] presented a measurement method with one camera and one LCD plane that is moved into two or more positions to complete specular measurement by special geometrical structure. However, the accuracy of measurement depends on the precisions of the system parameters and the speed is limited due to the horizontal and vertical fringe patterns projected in the LCD screen. It is difficult to accurately calibrate the measurement system and establish the system structure [15]. Knauer et al [16] proposed a stereo deflectometry to obtain the height and gradient information simultaneously. The system consists of two cameras and one LCD screen. However, the measurement accuracy depends on the process of an iterated algorithm and the precision of the geometric relationship between the cameras and LCD screen. There are two problems in the existing PMD methods. One is a limited depth of field (DOF) of the camera lens because the specular surface under test and the virtual image of the screen have different distances to the camera. The captured fringe pattern image is blurred because of the limited DOF [14,17], so that it is difficult to accurately calibrate the system parameters and measure specular object with a large depth. The other problem is system calibration of PMD. It is an important and basic step to obtain the accurate measurement results, especially to calibrate the relative orientation between the camera and LCD screen [14,[18][19][20]. However, this is a difficult and challenging problem because the camera cannot directly see the LCD screen [21][22][23], as illustrated in Fig. 1. Generally, there are two kinds of geometric calibration method. One is to use a plane mirror with markers on the surface to calibrate the relative orientation between the camera and LCD screen. The known separations between markers determine the position of the mirror [19,20]. Obviously, it is troublesome and inconvenient to accurately determine the position of these markers. The other is to utilize a plane mirror without any markers. The mirror reflects the pattern displayed on the LCD screen several times to determine the relative relationship between the camera and the LCD screen [21][22][23]. All these calibration methods aim at the pinhole camera, without considering the lens distortion. According to the principle of photogrammetry, lens distortion has great effect on high-precision 3D measurement. A bi-telecentric lens has negligible distortion, so that it can be used to improve the measurement system. However, there is not an available method to calibrate the relative orientation between the LCD screen and camera with a bitelecentric imaging lens. . By using a bi-telecentric lens, this paper proposes a new advanced PMD method having unlimited imaging DOF. A novel mathematical model is established for measuring 3D shape of discontinuous specular objects by directly establishing the relationship between absolute phase and depth. The camera captures the deformed fringe patterns reflected by the measured specular surface from the same viewpoint of the displayed LCD screens. An auxiliary plane mirror is used to calibrate the measuring system by reflecting the pattern displayed on the LCD screen three times. Three specular objects have been measured to verify the accuracy and evaluate the performance of the proposed 3D measuring method.
The rest of the paper is organized as follows. In Section 2, the principle of measurement and calibration are demonstrated. Section 3 shows the system calibration and the 3D measurement results of specular objects by using simulation data. In Section 4, the actual experiments on several specular objects are carried out and a quantitative evaluation of the measurement system is given. Finally, Section 5 concludes the paper.

Principle
The proposed measurement method displays sinusoidal fringe patterns onto an LCD screen, which is moved to two different positions along normal direction of a reference plane. A camera captures the deformed fringe patterns reflected by the measured specular surface from the same viewpoint of the screens through a flat beam splitter. A four-step phase shift algorithm and an optimum multiple-fringe numbers selection method [24,25] are used to calculate the wrapped and unwrapped phase data pixel by pixel from the captured sinusoidal fringe patterns. According to the established mathematical model, 3D shape of discontinuous specular surface can be directly reconstructed from the calculated absolute phase map.  Figure 2 shows the measurement principle used to obtain 3D shape of discontinuous specular objects. It consists of an LCD screen, a camera with a bi-telecentric lens, a flat beam splitter, and a reference specular plane. The LCD screen is used as the diffusive structured light source, which can be moved by a translating stage. In order to directly obtain 3D shape of specular objects from the deformed fringe patterns, the sinusoidal fringe patterns need to be displayed on the LCD screen located at two known positions of P1 and P2. The beam splitter is used to change the light path of the camera, so that the optical axis of lens is vertical to the LCD screen. Because of the characteristics of bi-telecentric lens, only the light rays that are approximately parallel to the optical axis of the lens can form the image, as illustrated in Fig.  2. Two light rays are displayed from LCD screens and reflected into the camera by the measured specular object surface and the reference specular plane.

Measurement method
The LCD screen remains parallel to the reference specular plane that is vertical to the optical axis of camera. ∆d is the translating distance of the screen by the stage and d is the distance between the reference specular plane and the LCD screen located at P1. The two incident rays correspond to the same reflection light. The phases of the two incident rays are Because normal vector of the reference plane is parallel to the optical axis of camera, the incident ray coincides with the reflected ray through the reference plane. α is the angle between the other incident ray and the reflected ray through the measured object surface. The period of the displayed fringe pattern is q. Parameter h stands for height of the measured specular surface with respect to the reference plane.
According to the geometric relationship in Fig. 2, the following equations can be deduced.
From Eqs. (1) and (2), height of the measured specular surface is Because d is a constant, it can be moved to the left part. Depth information can be defined as H = d-h, which is the distance between the measured specular surface and the LCD screen located at P 1 . H can be directly calculated from phase data. It clearly shows that depth H can be directly calculated from the captured fringe patterns only if the parameter ∆d and phase information on the reference plane mirror are determined beforehand. Because the optimum three-fringe numbers selection method will be used to independently calculate the absolute phase pixel by pixel, discontinuous specular objects surfaces can be measured by the proposed method. However, when normal of the measured specular point is vertical to the LCD screen, the reference phase equals to the phase of the corresponding point on the measured specular surface. In this case, denominator of Eq. (3) is zero, so that the points whose normal is vertical to the LCD screen on specular objects cannot be measured effectively.

Camera model with bi-telecentric lens
A bi-telecentric lens has the property of purely orthographic projections of scene points and maintains a constant magnification over a specific range of object distances [26][27][28]. In a bitelecentric system, only the light rays that are approximately parallel to the optical axis of the lens pass the aperture stop and form the image. The bi-telecentric lens accurately reproduces dimensional relationships within its telecentric depth, and it is robust to small differences in the distance between the lens and the camera's sensor. The bi-telecentric lens shows a very low distortion and offers a large DOF. A pinhole model of wide angle cameras is not applicable for the bi-telecentric imaging system. The camera model with a bi-telecentric lens is demonstrated in Fig. 3 [29]. The bi-telecentric lens simply performs a magnification in both X and Y directions, while it is not sensitive to the depth in Z direction.
where m is the effective magnification of the bi-telecentric lens; ij r is the element of rotation matrix R and , u v for the telecentric lens from Eq. (4), because the bi-telecentric lens performs parallel projection and there is not a projection center [27,28].

Bi-telecentric imaging model by a plane mirror
Through a plane mirror, the imaging geometric relationship between the camera and mirrored LCD screen is illustrated in Fig. 4.
Markers with known separation in between are generated by software and displayed on the LCD screen. The screen is reflected by the plane mirror placed at three different positions [23] and the virtual image of the markers is captured by the CCD camera. π is defined by its normal vector C j n and its distance j d from C. The relationship between C i p and its reflection C i mj p by mirror j π is known as Householder transformation. However, the bi-telecentric lens is not sensitive to depth in Z direction. Therefore, the third element z t of the translation vector T cannot be calibrated. The real 3D reflection points cannot be estimated. Moreover, the ideal imaging model cannot be used to calibrate posture of the camera with the bi-telecentric lens. In order to address the problem, a new measurement model is described in the following subsection.

Measurement model
where i j t can be represented as Removing i j t from the two equations can obtain ( ) Considering the reference 3D point C i p is unique, its two mirrored 3D points Multiplying ' jj m on the right side of Eq. (10) obtains By applying the orthogonality constraint for the mirrored positions of reference points After removing C i p from Eqs. (6) and (9), a large system of linear equations can be derived Without loss of generality, is used in these equations. 1 r and 2 r are the first and second column vectors of R, that is, The least-squares solution for Z can be computed by * Z A B = ⋅ , where * A is the Moore-Penrose pseudo-inverse of A. The third column vector of rotation matrix 3 r can be computed as follows [30].
Through the above procedure, the rotation matrix R between LCD screen and camera can be calculated. Therefore, the parallel relationship between them can be guaranteed by adjusting the LCD screen according to the calculated rotation matrix R.

Simulation
The simulated experiments were carried out to evaluate the proposed measurement method. To generate the simulated data, the following default values were used. The original distance from the camera to mirrors was 340 mm. The normal vectors ( ) was the angle with respect to each axis, and drawn randomly within the ranges of (−20≤θ x ≤20, −20≤θ y ≤20, −20≤θ z ≤20). The rotation matrix and translation vector from the world coordinate system of LCD screen to the camera coordinate system were set to Using the simulated data, the system was calibrated and then a simulated specular plane and a simulated specular sphere were measured.

Geometric calibration
The normal vectors C j n and rotation matrix R could be estimated by Eqs. (15)- (17). By using the preset value and the estimated results, the errors of R and C j n were quantitatively evaluated. E r is defined as the error by using Riemannian distance [31] between the estimated R and the ground truth value g R ( ) The physical meaning of Riemannian distance reflects appropriately the distance between two poses in a dynamic sense [32]. The Riemannian distance E r defines the minimal absolute value of the angle by which the coordinate system R must be rotated around an arbitrary axis in order to align it with the coordinate system g R [33].
Error of C j n is defined as the estimated angle against the ground truth ,   Figure 6(b) shows the error against different mirror angles. The mirror postures were changed up to ± 20 degrees. When the angle is greater than 15 degrees, the errors of En and Er are not sufficiently obvious. The simulated data were added Gaussian noise with zero-mean and standard deviation σ = 2. These results quantitatively prove that the proposed method is insensitive to a certain noise level and the estimation accuracy increases with increasing the mirror rotation angle. Distance d Δ between the two LCD screens is 30mm and distance d between LCD screen at P1 and the reference specular plane is 100mm. The LCD screen has a resolution of 2048 × 1536 pixels and a pixel size of 15 um × 15 um. The effective magnification of the bitelecentric lens is 0.057. The camera has a pixel size of 3.45 um × 3.45 um. Twelve vertical fringe patterns having the optimum fringe numbers of 64, 63, and 56 were generated with 5% random noise by software, as shown in Fig. 7. A specular plane with size of 100 mm × 100 mm and a specular sphere with radium of 67.434 mm were simulated. 3D shape data of the two simulated specular objects were measured by using the proposed method, as shown in Fig. 8(a) and Fig. 9(a).

Measurement results
To verify the measurement accuracy, the mean squared errors is defined as where c H is the calculated depth value, r H is the preset value, and m is number of the measured points on the specular surface.   Figure 8(b) and 9(b) illustrate absolute error of the reconstructed plane and the reconstructed sphere between the ground truth and the calculated height. Their mean squared errors are 0.011mm and 0.015mm. Figure 9(b) clearly shows that the absolute error is increasing at the region of smooth surface. The main reason is that the measurement method is limited to measure the specular surface that is unparalleled to the LCD screen, as stated in Section 2.1. Only vertical fringe patterns (as shown in Fig. 7) on the LCD screen were used to calculate the absolute phase on the specular surface, so that the phase along the middle column had the same value as that on the reference plane. Therefore, depth data along the middle column was calculated inaccurately. In actual measurement, the problem of inaccurate data can be improved by changing the specular surface with respect to the reference plane, and then the measured data from different positions are merged together. When the reference plane is not parallel to the LCD screen, the angle a between the reference plane and the LCD screen is varied from 0 degrees to 1.5 degrees in increments of 0.5 degrees. Using the simulated measurement system, the relationship between angle a and the mean squared error of the above reconstructed plane is obtained as shown in Fig. 10. Figure 10 shows that with angle a increasing, the mean squared error of the measurement results increases gradually. The main reason for this trend is that with the angle a increasing, the deviations of the measured parameter d and the reference phase increase along x direction, which causing the error of measurement. Therefore, the deviation of angle a should be guaranteed in actual measurements.

Experiments and results
A full-field 3D shape measurement system for specular objects has been setup to test the proposed method. Three separate plat mirrors, a concave mirror and an artificial specular step have been measured to show the feasibility of directly measuring depth information from the calculated absolute phase data.

Hardware system
The developed hardware system consists of a computer, a camera with a bi-telecentric lens, an LCD screen, a flat beam splitter and a translating stage, as illustrated in Fig. 11. The camera is a SVS-Vistek camera with the model number of ceo655CVGE from Germany. It had a CS-mount lens interface, a resolution of 2448 × 2050 pixels, a frame rate of 10 fps at full resolution, and a GigE interface. The bi-telecentric lens has the model number of GCO230105 with magnification 0.057, which has a C-mount interface. Because of CS-mount of the camera, a 5-mm tube is added between the lens and the camera. The LCD screen is model LP097QX2 from LG (Seoul, Korea) and has a resolution of 2048 × 1536 pixels. The linear stage has a position accuracy of 1μm. After the system has been calibrated, parameter d Δ is determined as 50 mm by the stage. Depth information of the reflected object surface could be measured by the developed system.

Process of measurement
The measurement process mainly includes the following four steps, as shown in the following flow chart in Fig. 12.
Step 1: Calibrate the intrinsic parameters of camera with the bi-telecentric lens to decrease the error of system calibration and 3D reconstruction.
Step 2: Establish the measurement system. First, the position of the mirrored object reflected three times by a plane mirror is obtained in the coordination system of camera.
Second, the axis vector ' jj m is calculated by Eq. (14). Third, each normal vector of mirrors located in three positions is estimated by Eq. (15). Fourth, the rotation matrix of the LCD screen is calculated by using Eqs. (16) and (17). The above four steps are repeated until the LCD screen is vertical to the optical axis of camera. Finally, the reference plane mirror is adjusted and the first three steps are repeated until the reference plane mirror is vertical to the optical axis of camera.
Step 3: Obtain reference fringe patterns reflected by the reference plane mirror. In the whole measurement process, this step is performed once to obtain the reference absolute phase which is compared with the deformed absolute phase on the measured specular object.
Step 4: Calculate the absolute phase from the captured deformed fringe patterns reflected by the measured specular object.
Step 5: Calculate the 3D measurement data. Based on Eq. (3), the depth data of the measured object can be calculated using the phase difference and the parameter d Δ that is determined by the translating stage.
The steps 3-5 are detailed in the following chart, as shown in Fig. 13.

Measurement results
After calibrating the geometric parameters, three reflected objects were measured by the developed 3D system. They are three separate plat mirrors, a concave mirror from Micronano Optical Corporation (Beijing, China) which has a radius of 400 mm, and an artificial specular step. Twelve fringe patterns having the optimum fringe numbers of 64, 63, and 56 were generated by software and displayed sequentially on the LCD screens. The reflected fringe patterns by the specular surface are deformed and captured by the camera. Figure 14 shows one of the captured deformed fringe patterns on the specular objects. Because the optimum three-fringe number selection method was used to unwrap the wrapped phase [24], the absolute phase of each pixel is independently determined, as shown in Fig. 15. Figure 16 illustrates the obtained depth information. The results show that the proposed method can directly measure discontinuous specular object surfaces.

Performance analysis
In order to quantitatively evaluate the accuracy of the proposed method, the radius of the measured concave mirror and the distance between neighboring steps of the artificial step were estimated. The radius of the concave mirror was 400.0318 mm calculated from the measured 3D shape data, so that the error was 31.8µm. In order to further verify the accuracy, the error of radius between the actual value and the estimated value was calculated as 38.7 um. The distance between neighboring steps of the artificial step was measured by a coordinate measuring machine (CMM), as the first column shown in Table 1. All the points of one measured step surface were fitted onto a plane to calculate the distance between neighboring steps. The measured distance between neighboring steps was calculated using the average of the distance from all the points obtained on the other step surface to the fitted plane, as the second column shown in Table 1. The absolute error (absolute difference between the average measured distance and the actual distance) and the mean squared error are listed in the third and fourth columns in Table 1. The absolute error in the distance between neighboring steps and the mean squared error were below 25.3 um and 28.5 um, respectively. The experimental results demonstrate that the proposed method can directly reconstruct the 3D shape of specular objects with high precision and reliability.

Conclusion
This paper presents a novel full-field 3D shape measurement method based on a bi-telecentric system for measuring discontinuous specular surfaces by building the direct relationship between absolute phase and depth data. An LCD screen is moved to two different positions by an accurate translating stage during the procedure of measurement. Fringe pattern sets are generated by software and displayed on the LCD screen. The reflected fringe patterns are deformed with respect to the shape and slope of the specular surfaces and captured by a CCD camera. The bi-telecentric lens gives a very low distortion and offers a large DOF. Therefore, the system can measure the specular surface with large depth change. In addition, a new posture calibration method of measurement system has been proposed. An auxiliary plane mirror without markers is introduced and implemented in the calibration procedure. Based on the law of reflection, the relationship between object and the corresponding mirrored object can be directly established. After the parameter of the system has been calibrated, depth information can be directly obtained from the obtained absolute phase. Because depth directly relates to absolute phase without gradient integration, the proposed method can measure discontinuous specular objects. The experimental results show that the system effectively obtains full-field 3D shape information of discontinuous specular objects.
The proposed method has the following advantages. 1). Simple: the system calibration can be completed by using a plane mirror without markers. 2). Accurate and flexible: the measurement system can be calibrated accurately and flexibly due to a very low distortion and a large DOF of the bi-telecentric lens. 3). Discontinuous: discontinuous specular surfaces can be measured because of usage of the optimum three-fringe number selection method and without gradient integration.