University of Huddersfield Repository A simple, flexible and automatic 3D calibration method for a phase calculation-based fringe projection imaging system

: An important step of phase calculation-based fringe projection systems is 3D calibration, which builds up the relationship between an absolute phase map and 3D shape data. The existing 3D calibration methods are complicated and hard to implement in practical environments due to the requirement of a precise translating stage or gauge block. This paper presents a 3D calibration method which uses a white plate with discrete markers on the surface. Placing the plate at several random positions can determine the relationship of absolute phase and depth, as well as pixel position and X, Y coordinates. Experimental results and performance evaluations show that the proposed calibration method can easily build up the relationship between absolute phase map and 3D shape data in a simple, flexible and automatic way.


Introduction
Phase calculation-based fringe projection techniques are actively studied in academia and widely applied in industries because of the advantages of non-contact operation, full-field acquisition, high accuracy, fast data processing and low cost [1][2][3].Such techniques project fringe patterns onto the measured object surface.From another viewpoint, the fringe patterns are deformed with respect to the shape of the object surface.By calculating the absolute phase in the deformed fringe pattern, a 3D shape of the object is obtained with high accuracy.Therefore, in order to obtain an accurate 3D shape, an important step is to build up the relationship between the absolute phase map and 3D shape data, known as 3D calibration [4].Existing 3D calibration methods have complicated and hard to implement procedures in practical environments because a precise translating stage or a gauge block is required.Therefore, it is a challenging problem determining how to build up the relationship between phase map and 3D shape data in a simple, flexible and automatic way, especially out of a laboratory environment.
The reported 3D calibration methods of phase calculation-based fringe projection systems can be categorized into model-based [4,5], polynomial [6,7] and least-square [8,9] methods.All the methods need an accurately translated plate or a standard gauge block.The plate needs to be accurately positioned using a high precision translating stage, so the calibration procedure is complicated and mostly implemented on an optical breadboard in the laboratory.When a gauge block is used, the placement of the gauge should satisfy certain conditions and it is difficult to carry out the calibration procedure in the field.Therefore, the existing 3D calibration methods have complicated and hard to implement procedures limited to a laboratory environment.
We previously presented a calibration method for an uneven fringe projection 3D imaging system by using a white plate with discrete markers with known separation on the surface [10].Later, using the same white plate and a checkerboard, a general fringe projection 3D imaging system was accurately calibrated to build up the relationship between absolute phase and depth data [11].Although the white plate and the checkerboard can be randomly placed in the measuring volume, there are still some disadvantages: firstly, all the center positions of marker were determined by manual operation, making this method both labour intensive.Secondly, the relationship between pixel position and transverse x,y coordinates is not implemented, so it is not a real 3D calibration.Thirdly, it needs two calibration plates: a checkerboard and a white plate.Therefore, the existing calibration methods cannot automatically build up the relationship between phase map and 3D shape in an actual application field.This paper will present a simple, flexible and automatic 3D calibration method by using a white plate with discrete hollow ring markers with known separation on the surface.The center location of each marker on the white plate is automatically extracted without any interactive manual operation.The internal parameters of a CCD camera can be calibrated by using the extracted locations of the marker.3D position of the plate is calculated by using the obtained internal parameters of the CCD camera and the known separation between markers.At each plate position, sinusoidal fringe patterns are projected onto the plate surface to obtain phase information.Therefore, absolute phase and depth data of each pixel at different plate positions can be obtained to calculate the coefficients of a polynomial function, which builds up the relationship between absolute phase and depth data.The transverse X and Y coordinates of each pixel position are also calibrated using the internal parameters of the CCD camera and the corresponding depth data.Consequently, the proposed method can easily build up the relationship between phase map and 3D shape data, including not only the absolute phase and depth data, but the pixel positions and transverse coordinates, by using a calibration artifact in a simple, flexible and automatic way.

Principle and method
3D calibration of a phase calculation-based fringe projection imaging system includes two conversions of phase to depth and pixel to coordinates, called depth calibration and transverse calibration, respectively.Depth calibration builds up the relationship between the absolute phase and the depth data, while transverse calibration establishes the relationship between the pixel positions and the X, Y coordinates.A white plate with a scattered surface was designed and manufactured for 3D calibration.There are 9x12 discrete black hollow ring markers on the surface, as illustrated in Fig. 1.The separation of neighboring markers along row and column direction has the same value of 15 mm with an accuracy of 1 µm.Before 3D calibration of the fringe projection system, eight internal parameters of the CCD, including two focal lengths (F u and F v ), two principal point coordinates (P u and P v ), and four image radial and tangential distortion coefficients (K 1 , K 2 , K 3 , and K 4 ), need to be determined by capturing the white plate from several random positions.The eight parameters can be calculated from the extracted location of all the hollow ring markers at different plate positions [12,13].

Marker locations
For each captured image of the white plate the location of all the 9x12 markers is automatically extracted.This is repeated for each position.At first the threshold value of all the markers is automatically selected for one captured image of the white plate without projected fringe pattern.All the pixel coordinates belonging to one marker can be found by binarizing the captured image.An eight-field boundary tracking algorithm extracts the inner and outer edges of one marker to obtain two rings.In order to accurately determine the center of each marker, the ellipse fitting algorithm was used to calculate the center of the two rings [14,15].Figure 2 illustrates the obtained center of each marker by using red dots.The fringe pattern has a variable period when projected onto the reference plane due to the crossed optical axes of the imaging and projection devices.This effect is illustrated in Fig. 3 [11].Therefore, the relationship between absolute phase and depth data is a complicated function of pixel coordinate position of x and y and can be represented by the following equation [4] ( )( )

Depth calibration
where z is the depth relative to M, φ Δ is the difference of the unwrapped absolute phase on the measured object and the plane M, L is the baseline between the CCD camera and the DLP projector, 0 L is the working distance to M, θ is the angle between the optical axis of the projector and the camera, and 0 P is the period of the projected fringe pattern on a virtual plane perpendicular to the projecting axis.It is difficult and complicated to directly calibrate the systematic parameters of L , 0 L , θ , and 0 P .Here, a polynomial-based calibration method will be used to build up the relationship between absolute phase and depth data.
In fact, Eq. ( 1) can be represented by the following polynomial equation  2) on the x and y coordinates, the coefficient set will have different values between pixel positions.Therefore, a Look-Up Table (LUT) needs to be used at each pixel position to build up the relationship between absolute phase and depth data.
Depth calibration means to determine the coefficient set of the polynomial equation which can be realized by placing the white plate at several different positions in the measuring volume.At each position, sinusoidal fringe patterns are projected onto the plate surface to give the absolute phase information of each pixel.The center location [ , ]  u v of each marker in the pixel coordinate system is obtained by the marker location method in subsection 2.1.From the obtained center location of each marker, the coordinates of all the markers in the pixel coordinate system at different positions can be determined.The external parameters R and T of the white plate with regard to the CCD camera are calculated by the following Eq.( 3) where, R is a matrix representing the three rotation angles, [ , , ] x y z T T T T = is a vector representing the three linear translations, [ , , ]   w w w x y z is the coordinate vector of a marker P on the white plate, [ , ] u v is the coordinate vector of P in the pixel coordinate system, A is a matrix of internal parameters of the CCD camera, s is an arbitrary scaling factor and [ ] T denotes the transposition.
The world coordinate of each pixel point on the white plate is obtained by using the external parameters R and T , so that the relative depth of each pixel to a reference plane is obtained.Therefore, the relationship between absolute phase and depth data at each pixel can be built up by the polynomial Eq. ( 2) to give accurate depth calibration.

Transverse calibration
Transverse calibration is to determine the relationship between pixel coordinates and X, Y coordinates.For an actual imaging system, the relationship is nonlinear because of the distortion of optical imaging and projecting lens.Although the MatLab Camera Calibration toolbox can correct for the captured image distortion, there are still some uncorrected distortions from the projecting lens of the DLP projector.Transverse calibration relates to depth information obtained from projected fringe patterns.Therefore, at each pixel position the following two polynomial equations are used to give a high accurate relationship: where 0 0 0 1 1 1 , , , , , a b c a b c are the coefficient set of the system parameters, [ , ] u v is the coordinate vector of a point in the pixel coordinate system, , , r r r x y z are the coordinates of the same point on the white plate in the reference coordinate system.
The same plate in Fig. 1 will be used to transversely calibrate the relationship between pixel positions and X, Y coordinates.With the known separation between neighboring hollow ring markers and the obtained depth, , , r r r x y z of all pixel position in the reference coordinate system can be obtained by the procedure in subsection 2.2.The polynomial coefficients of each pixel can be determined by the obtained , , r r r x y z and pixel coordinates (u, v) of all the points on the white plate.Pixel positions are independent of each other and so two LUTs need to be established to save the coefficients of each pixel to give a highly accurate relationship between pixel positions and X, Y coordinates.

Experimental system
The hardware of the fringe projection imaging system includes a portable DLP video projector, a color 3-CCD camera connected to a personal computer (PC) by Firewire IEEE 1394, as illustrated in Fig. 4. The projector is from BenQ (Model CP270) with one-chip digital micro-mirror device (DMD) and a resolution of up to 1024 × 768 pixels (XGA).The colors red, green, and blue are produced by rapidly spinning a color filter wheel in the projector and synchronously modifying the state of the DMD.The 3-CCD camera from Hitachi (Model HV-F22F) has a resolution of 1360 × 1024 pixels.The camera has a standard zoom lens (Computar) with focal length from 12 to 36 mm and adjustable aperture.The personal computer provides system control.The PC graphics card is setup to drive two monitors, one for the DLP and the other for the control software and viewing the captured data.The system can generate red, green, blue or composite color fringe patterns.In the following experiments, green fringe patterns are used to demonstrate the calibration process.

3D calibration
A white plate with 9x12 discrete black hollow ring markers was designed and manufactured by Ti-Times [16], as illustrated in Fig. 1.The separation of neighboring markers along the row and column direction has the same value of 15mm.Calibrating the camera parameters needs theoretically at least two captured images from different viewpoints.In order to cover the whole measuring volume and to accurately determine the internal parameters of the CCD camera, the plate was positioned at twenty-four random positions and orientations with a large angle between the imaging axis and the plate's normal.At each position, the CCD camera captures one image containing all the 9x12 markers.Using the known separation in the twenty-four captured images, the internal parameters of the CCD camera are obtained by the Camera Calibration Toolbox for Matlab [13].The radial and tangential distortions of the following captured images were corrected by the obtained internal parameters.
For calibrating the fringe projection imaging system, the same plate needs to be randomly placed in the measuring volume.Using more plate positions means better coverage of the measuring volume.This gives a higher accuracy calibration, but will take more time to finish capturing and processing the fringe pattern image data.Therefore, thirty-nine positions were randomly chosen with the plate nearly perpendicular to the imaging axis of the fringe projection imaging system.At each position, twelve sinusoidal fringe patterns with the optimum fringe numbers of 100, 99 and 90 [17] were projected onto the plate surface for absolute phase calculation and one image without projected fringe pattern was captured for In principle, a higher polynomial order can provide more accurate depth data.However, a fifth fitting was found to be sufficient to calibrate the system to the level that could be achieved given the available phase resolution.The coefficient set of the polynomial at one pixel position in Eq. ( 2) is solved by the absolute phase and depth data.The pixel coordinates and the transverse coordinates on the plate from the thirty-nine images were used to determine the coefficient sets of the other two polynomials by using Eq. ( 4).All the obtained coefficient sets are saved into three LUTs for late 3D measurement.In order to evaluate the calibrated fringe projection imaging system, the same plate was placed on an accurate translating stage with resolution of 1μm.The plate was positioned at −18mm, 6mm, −6mm and −18mm with respect to the reference plane M. At each position, the depth data was calculated using the obtained coefficient sets from Eq. ( 2).The profile along one row near the middle is illustrated in Fig. 5 for the four positions.The measured average value, absolute error (absolute difference between the measured average distance and the distance by the stage) and standard deviation value along the middle row for the four positions were listed in the 4th, 7th and 10th column in Table 1, respectively.By utilizing the obtained coefficient sets from Eq. ( 4) and the depth value at each plate position, the transverse X, Y coordinates of all the ring markers on the white plate can be calculated, so the measured distance between neighboring ring markers is obtained.Table 1 lists the average distance between all the markers along the X and Y direction in the 2nd and 3rd column respectively.The absolute error between the measured distance and 15mm is listed in in the 5th and 6th column respectively.The 8th and 9th column show the standard deviation of the measured distance along the X and Y direction.The experimental results on the four plate positions show that the proposed 3D calibration method accurately converts not only absolute phase into depth data but also pixel positions into transverse X and Y coordinates.

Quantitative evaluation
Another evaluation experiment was carried out by measuring a 'step artifact' with a set of known variable geometry steps, as illustrated in Fig. 6(a) with projected green fringe on it.Figures 6(b) and 6(c) are the absolute phase map and measured 3D shape data from the calibrated system respectively.In order to quantitatively evaluate the calibrated system, the distance between the neighboring steps was measured.All the obtained points on one step surface are fitted into a plane.The measured distance between the neighboring steps is the average distance value of all the obtained points on the other step surface to the fitted plane.The actual and measured distance between the neighboring steps, the absolute error (absolute difference between the measured average distance and the actual distance), and the standard deviation are listed in Table 2

Qualitative evaluation
A toy having freeform surface was measured by the calibrated fringe projection imaging system.Twelve sinusoidal fringe patterns having the optimum fringe numbers of 100, 99 and 90 [17] were projected via green channel onto the toy's surface for absolute phase calculation.Figure 7(a) shows one of the captured fringe pattern image.Figure 7(b) illustrates the absolute phase map obtained by using optimum three fringe number selection method.During the process of phase calculation, pixels with a modulation less than 15 grayscales are marked as invalid and these pixels are shown in black.The 3D shape data of the toy were obtained after converting the absolute phase map into depth and transverse data, as shown in Fig. 7(c).

Conclusions
As proposed, a simple, flexible and automatic 3D calibration method was developed for a phase calculation-based fringe projection imaging system.A white plate was used whose surface was marked with discrete hollow ring with a known separation.By placing the plate at several random positions, the internal parameters of the CCD camera can be determined.The plate also can give absolute phase and depth data in the measuring volume to build up their relationship by a polynomial function at each pixel position.At each plate position, the absolute phase of each pixel can be calculated by projecting three fringe pattern sets with the optimum fringe numbers onto the plate surface.From the captured images, the location of all the markers is automatically and accurately extracted without any manual operation, so the relative depth of each pixel to a chosen reference plane can be obtained.Therefore, the coefficient set of the polynomial function are calculated by using the obtained absolute phase and depth data.The pixel positions and X, Y coordinates can be established by the parameters of the CCD camera and the obtained depth data.Four known depths of the plate and a 'step artifact' have been tested and the experimental results show the validity and flexibility of the proposed 3D calibration method.Therefore, the proposed method has the potential for accurate 3D calibration of phase calculation-based fringe projection imaging system with minimal operator skill and in a robust, flexible and automatic manner.
In comparison with the existing 3D calibration method, the proposed one has the following advantages: (1) simplification: because only a white plate is required during calibration, this method can be easily carried out in a shop floor, without limiting to a laboratory environment; (2) flexibility: the white plate can be placed randomly in the measuring field volume, do not need an accurate translating stage; (3) automation: the location of all the markers in the captured images is automatically and accurately extracted without any manual operation; (4) high accuracy and reliability: the white plate with hollow ring markers are produced with micrometer precision, so that the measured data gives high accurate 3D shape.

Fig. 1 .
Fig. 1.A photo image of the manufactured white plate.There are 9x12 black hollow rings on the surface with neighboring separation of 15mm in the horizontal and vertical directions.

Fig. 2 .
Fig. 2. Automatically locating the center of all the markers on a captured image of the white plate represented by red dots.

Fig. 3 .
Fig. 3. Schematic of even fringe projection at the projector and uneven fringe pattern on the reference M. M: Reference plane.

Fig. 4 .
Fig. 4. The hardware setup of the fringe projection imaging system including a DLP projector, a color 3CCD camera and a personal computer.

#
181659 -$15.00USD Received 13 Dec 2012; revised 30 Apr 2013; accepted 1 May 2013; published 10 May 2013 (C) 2013 OSA marker position determination.Fringes on black hollow rings have low modulation and the absolute phase data for them was interpolated from the surrounding pixels.

Fig. 5 .
Fig. 5. Measured depth along middle row direction for four positions of −18mm, −6mm, 6mm and 18mm in (a), (b) (c) and (d).X-axis represents the pixel positions along row direction with a range 1,2,3…, 1024, the vertical axis is the reconstructed depth to the reference surface.

Fig. 6 .
Fig. 6.Illustration of the step artifact and the measured 3D shape data.(a) the step artifact with green projected fringe, (b) absolute phase map, and (c) the measured 3D shape.

Fig. 7 .
Fig. 7.The measured results on a toy having freeform surface.(a) photo of the toy with green projected fringe, (b) absolute phase map, and (c) the measured 3D shape.