Improved geometrical model of fringe projection profilometry

The accuracy performance of fringe projection profilometry (FPP) depends on accurate phase-to-height (PTH) mapping and system calibration. The existing PTH mapping is derived based on the condition that the plane formed by axes of camera and projector is perpendicular to the reference plane, and measurement error occurs when the condition is not met. In this paper, a new geometric model for FPP is presented to lift the condition, resulting in a new PTH mapping relationship. The new model involves seven parameters, and a new system calibration method is proposed to determine their values. Experiments are conducted to verify the performance of the proposed technique, showing a noticeable improvement in the accuracy of 3D shape measurement. Disciplines Engineering | Science and Technology Studies Publication Details Z. Huang, J. Xi, Y. Yu, Q. Guo & L. Song, "Improved geometrical model of fringe projection profilometry," Optics Express, vol. 22, (26) pp. 32220-32232, 2014. This journal article is available at Research Online: http://ro.uow.edu.au/eispapers/3326 Improved geometrical model of fringe projection profilometry Zhengrong Huang, Jiangtao Xi, Yanguang Yu, Qinghua Guo, and Limei Song, School of Electrical Computer and Telecommunication Engineering,University of Wollongong, NSW 2522, Australia Tianjin Key Laboratory of Advanced Technology of Electrical Engineering and Energy, Tianjin Polytechnic University, Tianjin 300087 China * jiangtao@uow.edu.au Abstract: The accuracy performance of fringe projection profilometry (FPP) depends on accurate phase-to-height (PTH) mapping and system calibration. The existing PTH mapping is derived based on the condition that the plane formed by axes of camera and projector is perpendicular to the reference plane, and measurement error occurs when the condition is not met. In this paper, a new geometric model for FPP is presented to lift the condition, resulting in a new PTH mapping relationship. The new model involves seven parameters, and a new system calibration method is proposed to determine their values. Experiments are conducted to verify the performance of the proposed technique, showing a noticeable improvement in the accuracy of 3D shape measurement. 2014 Optical Society of America OCIS codes: (120.2830) Height measurements; (120.4820) Optical systems; (150.1488) Calibration. The accuracy performance of fringe projection profilometry (FPP) depends on accurate phase-to-height (PTH) mapping and system calibration. The existing PTH mapping is derived based on the condition that the plane formed by axes of camera and projector is perpendicular to the reference plane, and measurement error occurs when the condition is not met. In this paper, a new geometric model for FPP is presented to lift the condition, resulting in a new PTH mapping relationship. The new model involves seven parameters, and a new system calibration method is proposed to determine their values. Experiments are conducted to verify the performance of the proposed technique, showing a noticeable improvement in the accuracy of 3D shape measurement. 2014 Optical Society of America OCIS codes: (120.2830) Height measurements; (120.4820) Optical systems; (150.1488) Calibration. References and links 1. M. Takeda and K. Mutoh, “Fourier-transform profilometry for the automatic measurement of 3D object shapes,” Appl. Opt 22, 3397-3982 (1983). 2. X. F. Mao, W. J. Chen and X. Y. Su, “Improved Fourier-transform profilometry,” Appl. Opt 46, 664-668 (2007). 3. Y.S. Xiao, Y. P. Cao and Y. C. Wu, “Improved algorithm for phase-to-height mapping in phase measuring profilometry,” Appl. Opt 51, 1149-1155 (2012). 4. B. Li, N. Karpinsky, S. Zhang, “Novel calibration method for structured-light system with an out-of-focus projector,” Appl. Opt 53, 3415-3426 (2014). 5. A. Asundi and W. Zhou, “Unified calibration technique and it applications in optical triangular profilometry,” Appl. Opt 38(16), 3556-3561 (1999). 6. M. J. Baker, J. T. Xi and J. F. Chicharo “Neural Network digital fringe calibration technique for structured light profilometry,” Appl. Opt 46(8), 1233-1243 (2007). 7. B. M. Chung and Y. C. Park, “Hybrid method for phase-height relationship in 3D shape measurement using fringe pattern projection,” International Journal of Precision Engineering and Manufacturing 15, 407-413 (2014). 8. Q. Hu, P. S. Huang,Q. Fu and F. P. Chiang, “Calibration of a three-dimensional shape measurement system,” Opt. Eng 42(2), 487-493 (2003). 9. H. Du and Z. Y. Wang, “Three-dimensional shape measurement with an arbitrarily arranged fringe projection profilometry system,” Opt. Letters 32, 2438-2440 (2007). 10. E. Zappa and G. Busca, “Fourier-transform profilometry calibration based on an exhaustive geometric model of the system,” Opt. Lasers 47, 754-767 (2009). 11. E. Zappa, G. Busca and P. Sala, “Innovative calibration technique for fringe projection based 3D scanner,” Opt. Lasers 49, 331-340 (2011). 12. L. M. Song, C. M. Chen, Z. Chen, J. T. Xi and Y.G. Yu, “Essential parameter calibration for the 3D scanner with only single camera and projector,” Optoelectronics. Letters 9, 143-147 (2013). 13. Z. Y. Zhang, "A flexible new technique for camera calibration" IEEE Transaction on Pattern Analysis and Machine Intelligence 22(11), 1330-1334 (2000). 14. S. Zhang, “Novel method for structured light system calibration,” Opt. Eng 45(8), 083601-8 (2006).


Introduction
Fringe Projection Profilometry (FPP) has been considered as an enabling technology for noncontact 3-D shape measurement due to such advantages as simple structure and fast measurement [1][2][3][4] .A typical FPP system consists of a digital projector, a camera, a computer and a reference plane.The digital projector generates a group of image patterns of periodic fringes, which are projected respectively onto the reference plane and the object surface to be measured.The camera captures the image patterns reflected from the reference plane and the object surface, the latter of which are deformed version of former.The 3-D information of the object surface shape can be extracted by analyzing the phases of the projected patterns acquired by means of a phase-to-height (PTH) mapping relationship.The PTH mapping is based on the triangulation relationship among the projector, the camera, and the corresponding point on the patterns acquired from the reference plane and the object surface.The effectiveness of the PTH mapping depends on if it matches the structure of FPP system, that is, the positions of the camera, the projector and the reference plane.In early years, the PTH mapping proposed by Takeda, et al. [1] based on a simple model, where the FPP system is assumed to have an ideal structure meeting three conditions, including (1) the optical centers of camera and projector are located at the same distance from the reference plane; (2) the optical axes of the camera and projector are coplanar and the plane is perpendicular to the reference plane; and (3) the optical axis of camera is vertical to reference plane.However, these conditions are not always met in practice, and measurement error will occur if the PTH [1] is employed to recover the 3-D shape.In order to remedy this problem, an improved PTH mapping relationship was proposed by Mao, et al. [2] , where the first condition is removed in that camera and projector can be positioned with different distances from the reference plane.However, the second and third conditions are still required.Recently in 2012, further effort for solving the problem was reported in [3], where a PTH mapping was proposed to remove the first and third condition in that camera and projector are permitted to locate at different distance from the reference plane, and the optical axis of camera is not required to be vertical to the reference plane.However, the second condition remains.To the best of our knowledge, there is not a PTH mapping reported in literature where all these three conditions can be lifted, allowing a flexible positioning of camera and projector.
As the PTH mapping relationship is determined by the system structure, accurate evaluation of the parameters associated with the structure plays an important role, which is carried out by means of system calibration.These parameters include the intrinsic parameters of the camera and the projector, as well as extrinsic parameters associated with the geometrical structure of the FPP.Over the past decades, a number of approaches for system calibration are proposed, e.g., [5-11].Recently in 2013, a method to calibrate five essential parameters associated with the camera and the projector is presented by Song, et al. [12] .However, all the existing work reported in literature was based on the FPP structure meeting the three conditions.To the best of our knowledge, there is not work reported for system calibration with a flexible positioning of the camera and projector, that is, without the restrictions of the three conditions.
In this paper, we firstly propose a model to describe a general structure of FPP, where the three conditions are not required.Such a model involves seven parameters related to the system structure which are all required to be obtained.In order to achieve this, we propose a system calibration method.Before the seven parameters are calibrated, the camera and projector have been calibrated simultaneously.Camera calibration has been widely used for 3-D measuring system.A highly accurate and robust camera calibration method was proposed by Zhang [13] , .Hence this paper adopts Zhang's method [13] .Since the projector cannot take the picture as camera, the camera is used to help it to capture image.The phase-shifting method proposed by Zhang, et al. [14] in 2008 is adopted to transform the points in camera image into projector image.Then Zhang's method [13] is used to calibrate projector by considering it as an inverse camera, since camera and projector share the same optical principle.The innovations of this paper are improving the existing PTH mapping, and presenting the calibration method for our improved PTH mapping.The experiments demonstrate the accuracy and flexibility of 3-D measuring system have been improved based on our method.Such an improved algorithm will be highly suitable to the practical application of 3-D measuring system.

Existing work and problem statement
In this section, we will introduce in the order of the PTH mapping relationship proposed by Takeda, et al. [1] ,an improved PTH mapping by Mao, et al. [2] , the work by Xiao, et al. [3] , and then problem statement.[1]   Figure 1 shows an ideal geometric model proposed in [1].The fringe patterns are projected onto the reference plane by the projector, and then the fringe patterns reflected are captured by the camera.Without loss of generality, we consider a beam of light  

Geometric model proposed in
where 0 f is the frequency of fringe pattern on reference plane.C L is the distance between point O and optical center of camera ; d is the distance between optical centers of camera and projector;    O O is vertical to reference plane.If the three conditions are not satisfied at the same time, the PTH mapping will lead to measuring error.[2]   Figure 2 shows an improved geometric model proposed in [2], which allows the camera to move along z-axis direction, and hence is more flexible than the ideal one in [1].The PTH mapping in [2] is as follows:

Geometric model proposed in
where, in addition to the parameters associated with the model in Eq. ( 1), another three parameters were introduced, including 1 S the distance between the optical centers of camera and projector, 1  the angle between the linear p O F and P C O O , and r the distance between point O and K .2) becomes the same as Eq. ( 1), and hence the PTH mapping in [1] can be considered as a special case of the PTH mapping proposed in [2].In fact, since 1 1 sin S  only represents the difference between the heights of camera and projector relative to reference plane, Fig. 2(a) can be simplified as Fig. 2(b).The geometric model proposed in [2] only removes the first condition that projector and camera must be located at the same height relative to the reference plane.Because the line c O O is vertical to reference plane, the plane including the lines c O O and p O O is vertical to reference plane.Although the geometric model proposed in [2] improves the ideal geometric model, it is still limited by the second and third condition.If these two conditions are not satisfied, the PTH mapping proposed in [2] will still lead to error for measuring object.

Geometric model proposed in [3]
Another improved geometric model in Fig. 3 is proposed in [3], where camera and projector are allowed to be different height relative to reference plane, and optical axis of camera is permitted not to be vertical to reference plane.Hence, compared to the simplified geometric model as Fig. 2 (b), this improved geometric model is more flexible.The PTH mapping in [3]  can be given as: where, in addition to the parameters associated to the model in Eq. ( 2), another one parameter was introduced, that is, 2  the angle between the linear C O O and    , Eq. ( 3) can be the same as Eq. ( 2) indicating that the PTH mapping proposed in [2] is a specific case of the PTH mapping proposed in [3].Although geometric model proposed in [3] improves the geometric model proposed in [2], it requires the lines C O O and p O O are coplanar, and the plane is perpendicular to reference plane.

Problem statement
When the plane formed by the axes of camera and projector is perpendicular to reference plane, the fringe patterns captured by camera are shown as Fig. 4(a).However, it is difficult to satisfy this condition in practice.When this condition is not met, the fringe patterns taken by the camera can be the form in Fig. 4(b), where the fringe patterns are not orthogonal to c uaxis or parallel to c v -axis.If the PTH mapping in [3] is used for such fringe patterns, error will occur for 3-D shape measurement.

Improved phase-to-height (PTH) mapping
Now let us introduce a new PTH mapping which does not require any limitation of these three conditions listed in the section of introduction.As shown in Fig. 5, plane P C O O O is not perpendicular to the reference plane, and hence 0  is not 0 .
From triangles ABD and BCD , AB and BC can be respectively described as tan Substituting Eq. ( 5) into Eq.( 8) yields the following: According to [2], CA and AO can be presented as Substituting Eq. ( 10) into Eq.( 9), we have where, in addition to parameters associated to the model in [3], a new parameter 0  is introduced, which is the angle between the lines C OO and 3 C O O .When 0 0   , Eq. ( 11) is the same as Eq. ( 3).Hence, the geometric model proposed in [3] is a specific case of our geometric model.

System calibration
With the introduction of 0  in the model proposed in Fig. 5, the system must be calibrated in order to determine all the seven parameters associated, including C L , r , 0 f , 0  , 1 S , 1  and 2  .Before the seven parameters are estimated, camera and projector should be calibrated.
Hence, this section introduces camera calibration, projector calibration and calculation of systematic parameters.

Calibration of camera and projector
The camera and projector can be calibrated using the system in Fig. 6 with the aid of calibration plane.In order to describe the mapping relationship among 2-D points on the DMD of projector, 3-D points on calibration board, and 2-D points on the CCD of camera, a number of coordinate system are required, including world coordinate system (WCS), camera coordinate system (CCS), camera image coordinate system (CICS), projector coordinate system (PCS) and projector image coordinate system (PICS).Let   where c A is the matrix containing all the intrinsic parameters of camera, in which  


is the scale factor which denotes the ratio of the physical dimension of an object (in microns) to its size (in pixel).Since camera calibration has been studied extensively with many effective methods developed.In this paper we employ the technique proposed in [13] to calibrate the camera.Fig. 6.Schematic illustration of systematic calibration.
Since the projector can be considered as an inverse camera, the pinhole model can be used to describe the projector.As shown in Fig. 6, the relationship between   where p A is the matrix containing all the intrinsic parameters of projector, in which is the scale factor which denotes the ratio of the physical dimension of an object (in microns) to its size (in pixel).
As the projector cannot take a picture by itself, its calibration must be with the help of camera, and the mapping relationship between the pixels on the DMD and CCD should be determined.
In order to establish the mapping between the pixels on the DMD and CCD, two sets of phase shifted sinusoidal fringe patterns are generated by the projector, one set with vertical patterns and the other with horizontal patterns.Assuming that the resolution of the DMD is p p N M  pixels, and each individual fringe spans V T and H T pixels on the vertical patterns and horizontal patterns, respectively, these patterns can be expressed by the following: Similarly, the captured horizontal fringe patterns are: where


are the phases, which can be retrieved by the following: Note that, due to the arctangent operation in Eq. ( 18), the values of  are wrapped into the range between 0 and 2 .In order to have a unique mapping between the DMD and the CCD pixels, both of them are required to be unwrapped.To this end two sets of the gray code fringe images are projected, one vertical and the other horizontal respectively.These Gray code patterns have the same period as the sinusoidal fringe pattern, but the number of patterns is determined by , where M is the number of fringes in the corresponding sinusoidal pattern.The sinusoidal patterns have 64 fringes, and hence 6 gray code patterns should be used.For a point   In the same way, the six horizontal gray code patterns can be used to unwrap   yielding the following: , where are monotonic with respect to c v and c u , respectively.Compared them with the phases in Eq. ( 14) and Eq. ( 15), a unique point-to-point mapping between the CCD pixels and the DMD pixels can be determined as follows: Hence the mapping relationship between the CCD and DMD can be established.The same procedure will be applied to all the circle of the DMD image, resulting in a set of corresponding points on the board and the DMD, which can be used to calibrate the projector based on method in [13].Note that in order to have enough number of independent equations the calibration board needs to be rotated twice.Although the camera and projector have been calibrated simultaneously, the relationship between systematic parameters and the extrinsic and intrinsic parameters of both camera and projector is not clear.Hence, this relationship must be proposed.The whole derived procedure of this relationship is given as the section of calculation of systematic parameters.

Calculation of systematic parameters
With the camera and projector calibrated, we are now able to determine the seven parameters associated model in Fig. 5, including c L , 1 S , 0 f , 0  , 1  , r and 2  .To achieve this, we set the last position of calibration plane as the position of reference plane.The relationship between coordinates 0  of one point in the CCS and the coordinates   , , wc wc wc x y z of the same point in the WCS can be described as Similarly, the relationship between coordinates   , , where c R is 3 3  rotation matrix of camera, and c T is 3 1  the translation vector of camera.
p R is 3 3  rotation matrix of projector, and p T is 3 1  translation vector of projector.When , the optical centers of the camera and the projector in world coordinate system can be calculated as: Finally, the frequency 0 f of projected fringe patterns on reference plane can be calculated using the method in [12].

Experiments
The experiments are conducted to verify the performance of proposed geometrical model and the calibration approach presented in Section 4. The experimental setup in our lab is shown in Fig. 7, consisting of a computer, a camera, a projector, and calibration board.The resolution of projector is 768 pixels by 1024 pixels, and that of camera is 1024 pixels by 1280 pixels.The calibration board is a black metal plane with 99 engraved circles as shown in Fig. 7. Fig. 7. System calibration equipment in our lab.
The procedure of calibration experiment is as follows.Firstly, a white paper is stuck on the surface of calibration board, and then six vertical patterns and six horizontal patterns are projected onto the covered calibration board.These projected patterns are captured by a camera.Then six vertical gray code patterns and six horizontal gray code patterns are projected onto the covered calibration board, which are then taken by the camera.After these 24 images are projected, we removed the white paper and capture an image of calibration board.Since camera calibration in [13] needs at least three different views of calibration board, the calibration board has been viewed from three different positions, and the gray code phase shifting is used to every view of calibration board.The captured CCD images of calibration board are shown in Fig. 8. Fig. 8 (a).image of calibration board on first position (b).that on second position (c).that on the third position.
When both camera and projector are calibrated, the last position of calibration board is chosen as the reference plane.Then their corresponding extrinsic parameters are used to estimate the seven parameters using the method described in Section 4.2.All the obtained parameters on the new PTH mapping are shown in Table 1.

2.62
Let us now look at the accuracy of the parameters obtained in Table 1.Because their true values are unknown, we employ an indirect method.We measure a cuboid with a flat top surface with its known a priori (i.e., 14.23mm) and hence we can compare the measurement result against the true value.Figure 9 shows the reconstruction results using the ideal geometrical model in [1], the model proposed in [3] and the model in Fig. 5 incorporating the parameter values in Table 1.It is seen that the reconstructed results using the proposed method are much smoother than the models in [1] and [3], and hence the proposed method is the most accurate.Also, the standard deviation of the measurement associated with the proposed method is 0.1238mm (or 0.87%), implying that the parameters obtained in Table 1is also very accurate.In contrast, standard deviation of results in Fig. 9 (a) and Fig. 9 (b) are 0.49 mm, 0.29 mm respectively, which are much higher.Therefore, we can say that a noticeable improvement in the measurement accuracy can be achieved by the proposed model.

Conclusion
In this paper, we proposed a new geometric model for FPP where the plane formed by the axes of camera and projector is not necessarily perpendicular to the reference plane, thus making it much easier for implementing a FPP.Based on the new model, we presented a new PTH mapping relationship in order to improve the measurement accuracy.The new model involves seven parameters, for which we also proposed a new system calibration method to determine the values.Experiments are conducted to verify the performance of the proposed technique, showing a noticeable improvement in the accuracy of 3D shape measurement.

,
between the phase of point D and that of point C ; and   , h x y is the height of object.

Fig. 1 .
Fig. 1.The ideal geometric modal.The Eq. (1) is derived based on three conditions, including (1) the line P C O O is parallel to reference plane; (2) the line p O O and the line C OO are coplanar, and the plane is vertical to
z the corresponding point on the calibration board, and the corresponding point on camera CCD.The relationship between principle point of the camera; c  and c  are the focal length along c u -axis and c v -axis of the CCD; c  is the skewness of c u -axis and c v -axis; c R and c T are the rotation matrix and translation vector containing extrinsic;


the six vertical gray code patterns provide a code by means of the light intensity on the corresponding point, which allows unique determination of the

R 1 S , 1 a
 are the inverse of c R and p R , respectively.From Eq. (24), parameters and r can be expressed as of point O in WCS should be determined.Since point O corresponds to the principle point   .(12),which ignores the lens distortion of camera.Hence, by assuming 0 z  , we have following:

where 1 H
 is the inverse of H .When the coordinates of point O are obtained, parameters c L , 2  and 0  can be expressed as:

Fig. 9 (
Fig. 9(a) Reconstruction based on ideal geometric model (b) Reconstruction based on model in [3] (c) Reconstruction based on proposed model.

Table 1 .
The parameters of our proposed PTH mapping