Microscopic Three-Dimensional Measurement Based on Telecentric Stereo and Speckle Projection Methods

Abstract

Three-dimensional (3D) measurement of microstructures has become increasingly important, and many microscopic measurement methods have been developed. For the dimension in several millimeters together with the accuracy at sub-pixel or sub-micron level, there is almost no effective measurement method now. Here we present a method combining the microscopic stereo measurement with the digital speckle projection. A microscopy experimental setup mainly composed of two telecentric cameras and an industrial projection module is established and a telecentric binocular stereo reconstruction procedure is carried out. The measurement accuracy has firstly been verified by performing 3D measurements of grid arrays at different locations and cylinder arrays with different height differences. Then two Mitutoyo step masters have been used for further verification. The experimental results show that the proposed method can obtain 3D information of the microstructure with a sub-pixel and even sub-micron measuring accuracy in millimeter scale.

1. Introduction

With the development of micro-electro-mechanical system (MEMS), 3D measurements of microstructures have become more and more important [1,2]. Most microstructures have dimensions ranging from a few microns to several centimeters. Their 3D information, especially the heights, must be measured, and many measurement methods have been developed for this purpose. Among them, non-contact optical methods have been widely used because of their non-destructiveness, flexibility, and high efficiency. Some measurement methods such as digital holography [3], confocal microscopy [4], white-light interferometry [5] and optical fiber probe method [6,7,8], etc., can achieve submicron or even nano-scale measurement accuracy, whereas their measurement ranges are in sub-millimeter, micron, or sub-micron scale.

The microscopic fringe projection method, on the contrary, is studied widely and suitable for measuring microstructures whose dimensions are in the order of millimeters and above with the measurement accuracy from a few to tens of microns [9,10,11]. It can realize well 3D measurement of microstructures with different surfaces such as a gauge block [12], ball grid arrays (BGA) [13,14], coins [15,16], a wafer [17], an earphone diaphragm [18], and a step master [19], and can also perform dynamic measurements [20,21]. Due to a lack of suitable methods for system calibration and removing carrier-phase components from the measurement phases [22], it is difficult for its measurement accuracy to reach two or three microns and below. In the meanwhile, the microscopic stereo measurement method can obtain different measurement ranges and accuracies by changing the microscope objectives used, so that the 3D measurements of microstructures with different sizes and accuracy requirements can be realized [23]. These results depend on the measurement environment (especially the illumination conditions) and objects measured. For microstructures with no apparent features on the surfaces, it is difficult to obtain accurate 3D measurements. In order to improve the measuring robustness, patterns in the form of random speckles can be artificially created on the surfaces of monotonous microstructures [24]. Traditional methods to create patterns on micro-surfaces are mainly by spraying powders [25,26,27] or fluorescent microparticles [28,29], depositing constantan alloy [30], and generating laser speckles [31,32,33]. They are complicated and almost irreversible for the samples, and the microparticles may have a certain influence on 3D measurements. A method of projecting digital speckle patterns by means of a projector can conveniently create random features on the surface without affecting the samples, and has been applied to many macroscopic fields to achieve 3D measurements of simple shapes [34], human face [35], a Venus model [36], human body [37], a shape-complex mask [38], and different types of surfaces [39,40]. Whereas, this speckle projection method is rarely used in the field of microscopy.

Besides, almost all microscopic stereo measurement systems use pinhole microlenses as measuring probes because of the wide ranges in the field of view (FOV) and magnification, and the depths of field (DOF) are very small. The DOF and resolution of these microlenses are mutually constrained. It is insufficient to measure the complete depth of microstructures [13]. The telecentric lenses (including image-side, object-side and bi- telecentric lens), on the contrary, can extend the DOF to millimeters while maintaining high resolution due to the unique affine imaging properties [41,42]. However, telecentric lenses are rarely used in the 3D measurement of microstructures with binocular stereo microscopy except for aligning optical fibers [43].

In view of the above, we present a method combining the microscopic telecentric stereo measurement with the digital random speckle projection to obtain 3D information of microstructures with high accuracy. Two identical monochrome cameras were assembled from bi-telecentric lenses and charge coupled devices (CCDs) and used as image sensors (pixel sizes: 3.45 µm × 3.45 µm) to capture the images of microstructures. The measurement accuracy of the established setup was firstly verified by performing 3D measurements of grid arrays at different locations and cylinder arrays with different height differences. Then two Mitutoyo step masters were employed for further validation. The experimental results proved that the proposed method could obtain 3D information of the microstructures with sub-pixel (cylinder arrays, maximum 1.40 µm, 0.40 pixel size) and even sub-micron (grid arrays and step masters, maximum 0.83 µm) accuracy at least in a measuring range of 3.5 mm × 4.2 mm laterally and 0.6 mm longitudinally.

2. Measurement Principle

2.1. Telecentric Stereo Measurement

The bi-telecentric lens has been studied in detail [41,44]. It is combined with the CCD to form a bi-telecentric camera to perform parallel projection in practice with the model:

$$P_{\infty }=\left[\begin{array}{ccc}m/S_u& 0& u_0\\ 0& m/S_v& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}{r}_{11}& r_{12}& r_{13}& t_x\\ r_{21}& r_{22}& r_{23}& t_y\\ 0& 0& 0& 1\end{array}\right],$$

(1)

where m is the magnification of the bi-telecentric lens (also the intrinsic parameter); (S_u, S_v) are the scale factors in the sensor coordinate directions with units of unit metric length/pixel (generally given by the manufacturers of the sensors); (u₀, v₀) are the coordinates of the image system’s origin in the pixel system, that is, the coordinates of the principal point in pixels (generally taken as the image center); the truncated matrix R′ = [r₁₁, r₁₂, r₁₃; r₂₁, r₂₂, r₂₃] and truncated vector T = [t_x; t_y] are the first two rows of the rotation matrix and translation vector, respectively, and both are the extrinsic parameters. The intrinsic and extrinsic parameters can be calibrated based on the improved telecentric projection model presented in [45]. Then we multiply the intrinsic and extrinsic matrixes in the projection model and get:

$$P_{\infty }=\left[\begin{array}{cccc}{p}_{11}& p_{12}& p_{13}& p_{14}\\ p_{21}& p_{22}& p_{23}& p_{24}\\ 0& 0& 0& 1\end{array}\right],$$

(2)

where p₁₁ = r₁₁ · m/S_u, p₁₂ = r₁₂ · m/S_u, p₁₃ = r₁₃ · m/S_u, p₁₄ = t_x · m/S_u + u₀, p₂₁ = r₂₁ · m/S_v, p₂₂ = r₂₂ · m/S_v, p₂₃ = r₂₃ · m/S_v, p₂₄ = t_y · m/S_v + v₀. These are the projection parameters of one camera which can be calculated directly, and so are the parameters of another camera. The imaging in pixels of a world point P_w(X_w, Y_w, Z_w) in the left telecentric camera (simply called left camera) is considered as p₁(u₁, v₁), and in the right telecentric camera (simply called right camera) is p₂(u₂, v₂). Combining the projection models of the left and right cameras, we can obtain:

$$\left[\begin{array}{c}{u}_1\\ v_1\\ u_2\\ v_2\end{array}\right]=\left[\begin{array}{cccc}{p}_{11}& p_{12}& p_{13}& p_{14}\\ p_{21}& p_{22}& p_{23}& p_{24}\\ p_{31}& p_{32}& p_{33}& p_{34}\\ p_{41}& p_{42}& p_{43}& p_{44}\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right],$$

(3)

where [p₁₁, p₁₂, p₁₃, p₁₄; p₂₁, p₂₂, p₂₃, p₂₄] and [p₃₁, p₃₂, p₃₃, p₃₄; p₄₁, p₄₂, p₄₃, p₄₄] are the projection parameters of the left and right cameras, respectively. p₁(u₁, v₁) and p₂(u₂, v₂) are the matching points in the pixel coordinate systems. Defining this 4 × 4 matrix as Q. Here Q is a full rank matrix, otherwise the world points derived from matching points are not unique, which goes against the basic principle of the binocular stereo vision imaging. Once the matching points are obtained, the coordinates of the world point can be derived from the following formula by performing a reverse operation on Equation (3):

$$\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]=Q^{-1}\left[\begin{array}{c}{u}_1\\ v_1\\ u_2\\ v_2\end{array}\right],$$

(4)

where Q⁻¹ is the inverse matrix of Q.

2.2. Grayscale-Based Global Matching Method

In order to realize the measurement of 3D sizes of microstructures, a grayscale-based global matching method is adopted [46]. A template matching method which minimizes the gray value differences between a kernel of (2w_m +1) × (2w_n +1) pixels in one image (the template) and a displaced copy in another image is used to determine the most possible matching points, where w_m and w_n are the numbers of pixels in row- and column- directions in the pixel coordinate system, respectively. Taking into account the illumination changes between the left and right images, a zero-mean normalized sum of squared difference (ZNSSD) [24] matching criterion is adopted:

$${\varphi }^2={\displaystyle \sum _{x=-w_m}^{w_m}{\displaystyle \sum _{y=-w_n}^{w_n}{(\frac{h_1}{h_2}\cdot I(j_3+x,j_4+y)-I_{rm}\cdot \frac{h_1}{h_2}+I_{lm}-I(j_1+x,j_2+y))}^2}},$$

(5)

where:

$$I_{lm}=\frac{1}{(2w_m+1)\times (2w_n+1)}{\displaystyle \sum _{x=-w_m}^{w_m}{\displaystyle \sum _{y=-w_n}^{w_n}I(j_1+x,j_2+y)}},$$

(6)

$$I_{rm}=\frac{1}{(2w_m+1)\times (2w_n+1)}{\displaystyle \sum _{x=-w_m}^{w_m}{\displaystyle \sum _{y=-w_n}^{w_n}I(j_3+x,j_4+y)}}$$

(7)

$$h_1={\displaystyle \sum _{x=-w_m}^{w_m}{\displaystyle \sum _{y=-w_n}^{w_n}{F}_i}}\cdot G_i,$$

(8)

$$h_2={\displaystyle \sum _{x=-w_m}^{w_m}{\displaystyle \sum _{y=-w_n}^{w_n}{G}_i{}^2}},$$

(9)

$$F_i=I(j_1+x,j_2+y)-I_{lm},$$

(10)

$$G_i=I(j_3+x,j_4+y)-I_{rm}.$$

(11)

Here, C_l(j₁, j₂) and C_r(j₃, j₄) are the centers of the kernel in the left and right images, respectively; x is the increment in row- direction, ranging from −w_m to w_m, and y is the increment in column- direction, ranging from −w_n to w_n. (j₁ + x, j₂ + y) and (j₃ + x, j₄ + y) are the coordinates of the image points P_l (in the kernel in left image) and P_r (in the kernel in right image), respectively, and I(j₁ + x, j₂ + y) and I(j₃ + x, j₄ + y) are the gray values at the points P_l and P_r, respectively. Therefore, I_lm and I_rm are the mean gray values of all pixels in the kernels of left and right images, respectively; F_i and G_i can be considered as the zero-mean gray values at the point P_l and P_r, respectively; h₁ and h₂ are intermediate variables. When φ² takes a minimum value, C_l(j₁, j₂) matches C_r(j₃, j₄). In this way, we can find all the matching points.

There are epipolar line constraints [47] and range constraints in the matching process, so as to avoid a full image search in the reference images and reduce the matching time. When telecentric lenses are used in both the cameras, the epipolar line constraint can be written as [u₂, v₂, 1] · F_A · [u₁, v₁, 1] = 0, where (u₁, v₁) and (u₂, v₂) are respectively the pixel coordinates of the matching points in the left and right images, and F_A is the fundamental matrix of the telecentric binocular stereo measurement system which can describe the intrinsic geometrical properties and can be expressed as [48]:

$$F_A=\left[\begin{array}{ccc}0& 0& a\\ 0& 0& b\\ c& d& 1\end{array}\right].$$

(12)

The range constraint is that a point in the left image matches a point in the right image within a certain area, and the size of this area is suitable for the entire matching process. The four parameters (a, b, c, d) in Equation (12) and the size of the area in the range constraint can be determined by four pairs of non-collinear matching points at least.

2.3. Statistically Random Coding Method

The statistical random coding method, which is easy to implement and miniaturize, has been applied to some commercial products in macro fields successfully, such as iPhone X, VIC series products, Microsoft Kinect V1, Inter RealSense R200, etc. [49]. There are still some fundamental limitations including low spatial resolution, low measurement accuracy, and sensitive to noise, etc. Few people think of applying it for micro measurements. Here we creatively apply it to the telecentric microscopic binocular system so as to realize 3D measurements of microstructures. Firstly, the random speckle coding pattern is generated by a computer; then, this speckle pattern is projected onto the surface of a microstructure by a projector; thirdly, the left and right cameras simultaneously acquire images of the microstructure whose surface is covered with the speckle pattern; fourthly, the matching points are searched by the grayscale-based global matching method described in Section 2.2; finally, the 3D information of the microstructure is calculated by the telecentric stereo measurement method introduced in Section 2.1.

3. Experimental Measurement

An experimental setup combining the microscopic telecentric stereo measurement with the digital random speckle projection, whose measurement range mainly determined by the telecentric cameras was 3.53 mm × 4.22 mm × 2.60 mm, was established as shown in Figure 1a. Two identical bi-telecentric cameras used as image sensors were applied to achieve the binocular stereo measurement with the pixel sizes 3.45 µm × 3.45 µm. An industrial projection module as shown in Figure 1b (PRO4500UV119, Wintech, Beijing, China, with the resolution 912 × 1140, a projection distance of 119 mm, a projection size of 51.6 mm × 32.2 mm, and less than 0.1% projection distortion) was used to project the digital random speckle pattern (Figure 1c) on the surface of the microstructures, and act as an illuminator. Since the projection size was larger, it could completely cover the entire measurement area and the size of the measured projected image section was the same as the measurement range of the telecentric cameras. There was no measurement error between the prospective projected size and actual size. The two bi-telecentric cameras were calibrated based on the improved affine model. More details about other optical components could be found in [45]. The telecentric binocular stereo reconstruction algorithm was performed. A multi-frequency grid distortion target (composed of a grid array) serving as a planar calibration pattern was used for verifying the measurement accuracy of the established setup. This grid array was moved in z-direction controlled by a 3-axis micro-positioning stage, and images at different positions were captured by the two telecentric cameras. 3D measurements of three sets of cylinder arrays were performed for further verification. Two step masters, as shown respectively in Figure 1d (516-499 Ceramic Step Master 300C, Mitutoyo, Kawasaki, Japan, having four designed steps with the nominal values of 20, 50, 100, and 300 µm) and Figure 1e (516-498 Ceramic Step Master 10C, Mitutoyo, Kawasaki, Japan, having four designed steps with the nominal values of 1, 2, 5, and 10 µm), were employed for evaluation and the grayscale-based global matching method was executed. The uncertainty of these nominal steps was 0.20 µm. Limited by the vertical resolution of the established setup, the steps of 1 and 2 µm were not measured.

The relevant parameters of the two bi-telecentric cameras expressed in Equation (1) were calibrated as follows:

$$\left[\begin{array}{c}{u}_1\\ v_1\\ 1\end{array}\right]=\left[\begin{array}{ccc}581.8& 0& 1223.5\\ 0& 581.8& 1023.5\\ 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}-0.0307& 0.9502& -0.3100& -1.2039\\ 0.9993& 0.0358& 0.0110& -1.2872\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right],$$

(13)

$$\left[\begin{array}{c}{u}_2\\ v_2\\ 1\end{array}\right]=\left[\begin{array}{ccc}582.19& 0& 1223.5\\ 0& 582.19& 1023.5\\ 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}-0.0440& 0.9408& 0.3360& -1.0051\\ 0.9990& 0.0427& 0.0114& -1.1256\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right],$$

(14)

and the matrix of Q could be calculated as:

$$Q=\left[\begin{array}{cccc}-17.861& 552.83& -180.36& 523.07\\ 581.39& 20.828& 6.3998& 274.61\\ -25.616& 547.72& 195.62& 638.34\\ 581.61& 24.86& 6.637& 368.19\end{array}\right].$$

(15)

4. Results and Discussion

After matching the corresponding grid points in the left and right images, different positions (where the grid array had been placed) and the spacings between adjacent grid points in x and y directions (both nominal values were 50 µm) could be figured out. Here the size of the measurement area was 2.50 mm × 2.50 mm laterally. Figure 2a shows the surfaces fitted from the reconstructions of the grid arrays at three different locations in an easy-to-view angle, and Figure 2b displays three lines formed by averaging the three surfaces in Figure 2a in the x-direction. In the z-direction, the average measurement results of the positions were 0.02, 499.86 and 599.58 µm, while the corresponding values were 0, 500.0 and 600.0 µm. The absolute errors were 0.02, 0.14 and 0.42 µm, respectively. The measured spacings, absolute errors and standard deviations between the adjacent grid points in x and y directions were also listed in

Table 1. Experimental results on precisely positioned grid arrays.

Position (µm)	Measurements (µm)			Absolute Error (µm)			Standard Deviation (µm)
	x	y	z	x	y	z	x	y	z
0	49.75	50.07	0.02	0.25	0.07	0.02	0.70	0.71	0.70
500.0	49.84	50.08	499.86	0.16	0.08	0.14	0.69	0.73	0.72
600.0	49.92	50.01	599.58	0.08	0.01	0.42	0.72	0.72	0.75

. These errors were mainly affected by the resolution of the hardware devices, calibration error, and imaging noise. Even so, the reconstruction accuracies were still at sub-micron level.

3D reconstructions of the three sets of cylinder arrays were carried out. Figure 3a–c show the images in the left camera, and Figure 3d–f show the corresponding images in the right camera. The three sets of cylinder arrays in red boxes were denoted as C1, C2 and C3, respectively. The designed spacings between adjacent points were correspondingly 90, 150 and 200 µm, and the designed height differences were all 20 µm. Here the sizes of the measured areas were around 1.50 mm × 2.50 mm laterally. After threshold segmentation, edge detection, and feature extraction were performed successively, the corresponding circle center coordinates were extracted. Then, the 3D information of the cylinder arrays C1, C2 and C3 could be calculated and reconstructed, as shown in Figure 3g–i.

The results obtained from the Laser Scanning Confocal Microscope (LSCM, (VK-X200K, KEYENCE, Osaka, Japan), as listed in

Table 2. The reconstruction values compared with the measurement results of LSCM. Here ‘Spacing’ denoted the spacings between the cylinders, ‘Height differences’ denoted the average height differences, and (a), (b) and (c) corresponded to the cylinder arrays C1, C2 and C3.

Items	(a)		(b)		(c)
	Proposed Method	LSCM	Proposed Method	LSCM	Proposed Method	LSCM
Spacing (µm)	90.30	89.60	149.50	150.70	199.00	199.90
Height differences (µm)	20.80	19.60	20.60	19.20	18.50	19.40

, were used for comparison. As could be seen, the maximum absolute errors of the established setup were 1.20 µm (|150.7−149.50|) laterally and 1.40 µm (|19.20−20.60|) longitudinally, proving that these measurement results were still within sub-pixel accuracy and the established setup was effective. The measurement errors were mainly caused by the fact that the circles on the cylinder arrays were not standard and there was a certain error in the extraction of the circle center coordinates.

The measurement results of the two step masters were shown in Figure 4 as depth maps after smoothing and filtering. Figure 4a–f correspond to the 3D point cloud of the step masters with the height differences of 300, 100, 50, 20, 10 and 5 µm, respectively.

The lateral measurement range in this experiment was 3.50 mm × 4.20 mm. The measurement results were given in

Table 3. The average heights measured by the proposed method and Form Talysurf PGI 830 together with the relative errors.

Step	Nominal Value (µm)	Average Height Measured by the Proposed Method (µm)	Average Height Measured by Form Talysurf PGI 830 (µm)	Absolute Deviations (µm)/ Relative Percentage (%)
1	300 ± 0.2	300.16	299.33	0.83/0.28%
2	100 ± 0.2	99.98	99.68	0.30/0.30%
3	50 ± 0.2	50.45	49.94	0.51/1.02%
4	20 ± 0.2	20.13	19.90	0.23/1.16%
5	10 ± 0.2	10.10	10.15	0.05/0.49%
6	5 ± 0.2	5.04	4.92	0.12/2.44%

, and the average heights measured by the Taylor Hobson’s comprehensive measurement system for surface profile (Form Talysurf PGI 830, Taylor Hobson Ltd, Leicester, UK) were also listed for comparison. The absolute deviations and relative percentages of these two methods were calculated. The maximum values of these two indicators were 0.83 µm and 2.44%, respectively, which meant that all measurements were in sub-micron accuracy and the results were reliable. It could be seen from Figure 4 and Table 3 that the established setup combined with the proposed method was able to achieve 3D measurements of micro-steps, and the 3D point cloud data of the microstructures’ surfaces could also be obtained with submicron-level accuracy in the measurement range. The point cloud data showed a slightly tilt (high on the left and low on the right), may ascribing to the underside of the step masters not even or the 3-axis micro-positioning stage slightly tilted after calibration.

We also performed surface analysis of the high- and low-level point cloud data of the step masters whose height differences were 300 and 100 µm. For each level, a plane fitting operation was carried out, and then the fitted plane was subtracted from the measured point cloud to obtain the error distribution. The results along with the corresponding maximum absolute errors, average errors and standard deviations were shown in Figure 5. Here, Figure 5a,b show the error distributions of high- and low-level point cloud data of the 300 µm step master, and Figure 5c,d show those of the 100 µm step master. Although the maximum absolute errors were at most 4.88 µm, the average errors were all in the sub-micron range, and the standard deviations were about 0.1%. This means that the error distribution was relatively even in each level and the proposed method had a sub-micron measurement accuracy in the plane and high measurement reliability. The major sources of these errors could come from the roughness of the surfaces, the random noise of the cameras, and the error transfer during measurement (such as calibration, matching, reconstruction, etc.).

The error caused by the surface roughness can be determined and eliminated by an instrument which is capable of measuring surface roughness more accurately, and other errors can be determined and reduced by measuring a smoother standard surface. Because the original point cloud data were smoothed in the row direction, the error distributions in every level looked the same in the horizontal direction. These results showed that high-precision 3D measurements of microstructures could be obtained in millimeter scale.

5. Conclusions

In this paper, we have proposed a method combining microscopic telecentric stereo measurements with digital speckle projections to achieve 3D measurements of microstructures. An experimental setup containing two identical telecentric cameras was established. An industrial DLP projection module was used to project the digital random speckle patterns and act as an illuminator. The two telecentric cameras used as image sensors were calibrated based on an improved affine model to obtain the intrinsic and extrinsic parameters. A telecentric binocular stereo reconstruction algorithm and a grayscale-based global matching method were then introduced. The measurement accuracy was firstly verified by performing 3D measurements of grid arrays at different locations and cylinder arrays with different height differences. Two Mitutoyo step masters were also used to validate the feasibility and measurement accuracy. The experimental results proved that our setup combined with the proposed method was capable of obtaining 3D information of the microstructure with a sub-pixel (maximum 1.40 µm for cylinder arrays, 0.40 pixel size) and even sub-micron (maximum 0.42 µm for grid arrays and 0.83 µm for step masters) measuring accuracy in millimeter scale.