Applications of Computer Vision in Micro/Nano Observation

Nowadays, micro/nano science and technology has been one of the most attractive research fields. However, real time and accurate observation in micro/nano manipulation is a top important enabling technique. Most recently, with the great development of microscopes and computer vision techniques, real time visualization, including 2D motion measurement and 3D reconstruction, on micro/nano scale is becoming possible.

Volumetric methods usually reconstruct 3D models of external anatomical structures from 2D images. They represent the final volume using a finite set of 3D geometric primitives. Then, from an image sequence acquired around the object to reconstruct, the images are calibrated and the 3D models of the referred object are built using different approaches of volumetric methods. These methods work in the object volumetric space and do not require a matching process between the images used. Thus, typically, the 3D models are built from a sequence of images, acquired using a turntable device and an off-the shelf camera (Teresa et al,in press, 2008). However, in some real applications, we do not need to reconstruct the 3D model of objects, because depth is enough to understand the 3D relationship of scenes. DFS estimates depth from two images of the same scene captured by cameras at different positions and with different postures (Wu, 1999). Because it needs to extract and match feature points in these images, the computational task is so huge. As for DFF, it uses a mapping relation between focus and depth to estimate depth. It obtains a sequence of images with different depth, measures the focus degree using a measurement operator (Bove 1993;Nayar,1992), and attains the desired depth when the measurement value is maximal or minimal. Compared to DFS, DFF is simple in principle, but its estimation accuracy is highly related to the number of images. DFD is first introduced by Pentland in 1987 (Pentland,1987). It has been proved to be an effective depth reconstruction method by using the concept of blurring degree of region images with limit depth of field (Girod & Scherock, 1989;Pentland et al, 1994;Navar et al, 1996). Usually, DFD algorithm captures two images obtained with different camera parameters, measures blurring degree of every point, and estimates depth using the point spread function. During the past years, DFD has become attractive because 1) it requires only two images; 2) it avoids matching and masking problems; 3) it is effective both in the frequency domain and in the spatial domain (Gokstorp,1994;Subbarao & Surya,1994). However, since all above DFD methods need to capture two defocused images with changed camera parameters, they can not be used in applications with high level magnification microscopes, such as micro/nano manipulation, because on these situations, it is destructive to change camera parameters. This is the main reason why DFD has not been used in micro/nano manipulation until now. Here, SSD, LSAD, ZSAD and NCC are all adopted to estimate the motion between two neighbor images in a same image sequence. The theory is shown in Fig. 1. Generally, X(i,j) and Y (i,j) are referred to as the model image and the target image respectively. F(i,j) is the continuous image function, ε x,y represents additive noise, s=(s x , s y ) is the shift between the model image and the target image. , (, ) i j x i j x yy The object functions for SSD, LSAD, ZSAD and NCC are respectively defined as follow, However, the peak position above can only be solved with pixel-level accuracy, and it is not enough in many applications, especially in micro/nano manipulation. In order to attain the sub-pixel resolution, a quadratic curve fitting around the peak s n is usually used to estimate the sub-pixel shift s Δ as follows, where a x , b x , c x , a y , b y , c y are coefficients of the quadratic curves along x and y axis, which are the parameters to be estimated. The shift (,) x y s     is estimated by finding the peak of the following f(x) and f(y).

www.intechopen.com
Applications of Computer Vision in Micro/Nano Observation 531 If three points are used, the estimation results of the shifts can be denoted, However, the precision of the method mentioned above is usually low as it has no additional points to be used. Thus, one simple and effective way to improve the estimation precision is to properly increase the interpolated points and the order of the fitting polynomials.

The searching region
As is known, the searching region is the main factor which influences the computational cost and affects the performance of the BMA. Thus, in this section, with respect to an image sequence, an improved method is proposed to reduce the searching region effectively.
Since our aim here is to estimate the shift in the image sequence, and generally the motion is small between two neighbor images, it is not necessary to calculate R(u, v) with blocks throughout the whole image. Furthermore, the computation procedure not only increases the computational burden but also adds some opportunities of wrong matching when the texture or gray level of the target image is very similar.
Assuming that the largest shift is known, our proposed new BMA can be denoted as following steps, -First, define the initial position. Since the shift between every two images is very small, it is reasonable to take the position of the block in the model image as the initial position in the target image. -Second, define the maximal distance that the block can move in target image as step max_x and step max_y , where step max_x and step max_y can be determined through experience. -Third, move step by step along x and y axis and calculate R(u, v) at each step. During this process, the total distance along x axis should be always equal or lesser than step max_x , the total distance along y axis should be always equal or lesser than step max_y . -Finally, find out the minimal or maximal value of the object function and the corresponding moving steps.
It is clear that using the improved algorithm the computational burden can be greatly reduced because of the reduced searching region. Besides, if the whole image is similar in texture and gray level, or the block is very small, the improved method can also effectively decrease casual matching errors.

The block size
In BMA, the size of a block is another important factor to influence the matching precision and the complexity. If the block is too small, the matching information is limited and it would influence the matching precision greatly. While if the block is too large, the deformation during movement cannot be omitted. Besides, the larger block can result in a large computational burden. Therefore, to research the relationship between the block size and the object function is meaningful.
A standard grid used as the target object is shown in Fig.2, the magnification of the microscope's objective lens is 60  , which means that the underlying image is very smooth. The black squares designate the "blocks". First, we tested the matching error of different object functions on different block sizes. The results are shown in Fig.3, where the vertical axis denotes the estimation errors, with unit of pixel; the horizontal axis denotes the block size, with unit of pixels.
1. The estimation error of NCC is sensitive to the block size and has no obvious rule. Therefore if one wants to estimate the sub-pixel shift between two images by using NCC, the block size should not be the main regulating parameter. 2. The estimation error of LSAD is sensitive to the block size too, the larger the block size is selected, the smaller the estimation errors. Thus, with respect to the LSAD sub-pixel estimation, it is often beneficial to increase the block size. 3. As far as other functions are concerned, including SSD and ZSAD, they have the similar estimation results which can achieve much smaller errors comparatively, and they are not sensitive to block size as NCC and LSAD. 4. As far as the standard grid block, the most appropriate size is 50  50 pixels, which includes an integral corner of a white grid and a little black background. It proves this kind of block both can express the internal difference of the block approximately and can eliminate the coupling between x and y direction.

The sub-pixel fitting precision
Third, it is well known that estimation error of the curve fitting may become smaller when the order of a fitting polynomial increases. In this experiment, the comparison of the subpixel shift estimation error among different fitting polynomials including quadratic curve, cubic curve, quartic curve and spline curve, was conducted. Here, we used the same five fitting points, where the middle point is the peak and the other four points are on its both sides evenly. The main results are shown as Fig.5 to Fig.8, from which the following results can be obtained, No matter what objective functions are used, the estimation error of the quadratic curve is always larger than the original method because of redundancy. While due to high smoothness, the spline curve can achieve an optimal result for all objective functions. 2. For NCC and SSD, the sub-pixel estimation is better enough when the fitting function is the cubic curve. That means, if the higher order fitting function is selected, the improvement in the estimation precision is unclear compared with the introduced computational burden. As far as the LSAD and ZSAD are concerned, the estimation precision improvement of the quartic fitting function is obvious. Thus, it can be concluded that the objective function is the main factor to decide the order of fitting equation. 3. Since the computational burden has been greatly reduced by using the new BMA proposed in section 2.2.1, much higher precision can be achieved by using higher order fitting functions and larger number of fitting points. That can be properly selected based on the preceding conclusions in real applications.
www.intechopen.com Based on the theory and experiment results in 2.2, if we want to attain a matching result with high precision and low computational task using the grid block, the following parameters should be chosen: the reduced searching region, ZSAD object function, 50  50 pixels block size and the quartic sub-pixel fitting function.

The driving characteristic of a piezoelectric actuator
In order to validate the precision of our improved BAM practically, we used the PI nano platform to control the motion of the standard grid and calculated it with our method. Because the platform can output the nano motion, the shift of the grid is known. The KH-7700 microscope was used to capture the images of the grid, and the shift of each step was 50nm. Since the horizontal pixel is 57.47nm, the practical calculation result should be 0.89 pixels. Fig.9 is the result of the integral shift and Fig.10 is the shift of sub-pixel where the vertical axis denotes the motion, with unit of pixel, and the horizontal axis denotes the shift steps; the line with "*" is the true movement and the line with "o" is the calculation movement.
From Fig.9-10, we can see that the precision of our improved sub-pixel motion measurement method is very high, the integral pixel measurement is exactly equal to the true value and the sub-pixel measurement result is close to the true value. Therefore, the method can be used to measure the practical shifts in micro/nano manipulation.
Then, the sub-pixel block matching method of displacement measurement based on computer vision was used to measure the driving characteristic curve of a piezoelectric actuator practically. The result was shown in Fig.11, where the vertical axis denotes the motion, with unit of nm, and the horizontal axis denotes the driving voltage, with unit of V. Fig.11(a) is the driving curve when the voltage increases to 200V and then decreases to 0V smoothly; Fig.11(b) is the driving curve when the voltage increases to 150V and then decreases to 0V; Fig.11(c) is the driving curve when the changing routine of the voltage is 0V-200V-0V,0V-150V-0V,0V-100V-0V,0V-50V-0V; Fig.11(d) is the driving curve when the changing routine of the voltage is 0V-200V-0V,0V-160V-0V,0V-120V-0V,0V-80V-0V,0V-40V-0V.

536
The measurement results of the piezoelectric actuator driving characteristic are consistent with the physics analysis. Furthermore, the proposed method, which is simple in manipulation and credible in measurement results, satisfies the requirement of the micro/nano measurement with high precision.

3D reconstruction
Shape, or depth profile, reconstruction, is based on measurement depth information from 2D images, and now it has been widely used in many fields, such as medicine, robotics, and remote sensing.
All existing DFD algorithms can be divided into two kinds, local DFD algorithm and global DFD algorithm. In local DFD, a window around every pixel point is predefined, and the point's blurring is defined as that of the window (Pentland,1987;Vinay & Subhasis,2007). However, the difficulty of selecting proper size of window is a well known disadvantage of DFD algorithm, because there is a trade-off between having a window that is as large as possible to average out noise, but as small as possible to guarantee that within it (Ens &Lawrence ,1993;Nair & Stewart,1992). As far as global DFD is concerned, its main idea is completely different with local DFD algorithm since it works on the entire image without information of its radiance, or the appearance of the surfaces, and depth. Therefore, it is necessary to construct the depth model and the radiance model simultaneously (Favaro et al 2008(Favaro et al , 2003(Favaro et al 2002. This, however, will bring the problem of huge computation cost. A general method to solve this problem is to simplify the imaging model, for example, assuming the scene contains "sharp edges", that is, there are discontinuities in the scene (Asada et al 1998). Another way is to use a cubic function or structure light to approach the radiance (Nayar et al 1996;Lagnado & Osher,1997). Unfortunately, both local DFD and global DFD are on the basis of attaining two defocused images with different camera parameters which may destroy the camera drastically if the camera's amplification level is high.
In this section a novel DFD method with single fixed optical microscope is proposed to reconstruct the shape of samples on micro/nano scale. In the method, the blurring image model is constructed with the relative blurring and the diffusion equation, and the relation between depth and blurring is discussed from four aspects. The method proposed needs only one microscope with unchanged camera parameters, so the reconstruction process is very simple. The experiments and error analysis results show that it can reconstruct shape on micr/nano scale.

The imaging model for defocus
In the defocus imaging model, a defocused image can be theoretically considered as the summation of some defocused points, and this process can be denoted by the following convolution function normally: uxyt a uxyt a t uxy Exy ,"  "denotes the Laplacian operator, where "  "denotes the gradient operator and "   "is the divergence operator, , T xy xy It is also easy to verify that the variance  is related to the diffusion coefficient a via: Suppose there are two images E 1 (x, y) and E 2 (x, y) for two different focus setting, also, 12    (that is, E 1 (x, y) is more defocused than E 2 (x, y)), then E 2 (x, y) can be written as: where  is a constant between the blurring radius and the blurring degree, b i (i=1,2) is the radius of the burring round : where s denotes depth of the blurring point and D denotes the radius of the lens.

The new shape reconstruction method
Suppose E 1 (x, y), whose depth map is s 1 (x, y), is the defocused image attained before depth variation, and E 2 (x, y) is another defocused image attained after depth variation, in this section, we will propose a new shape from defocus method in which the depth map s 2 (x, y) is attained through a depth change Δs, and the main theory is shown in Fig.12.
In real applications, it is reasonable to discuss the following four cases when the distance between the sample and the microscope is becoming shorter and shorter.
a. s 1 > s 2 >s 0 Fig. 12. The theory of case A In this case, E 1 (x, y) and E 2 (x, y) are on the large side of s 0 , and E 1 (x, y) is more defocused than E 2 (x, y), so it is backward diffusion process from E 1 (x, y) to E 2 (x, y), that is, the diffusion efficient a is negative. The theory is shown as Fig. 13 and the finial depth can be denoted as: As is shown in Fig.14, here, E 1 (x, y) and E 2 (x, y) are on the small side of s 0 , E 1 (x, y) is less defocused than E 2 (x, y), so it is afterward diffusion from E 1 (x, y) to E 2 (x, y), and the diffusion efficient a is positive. The finial depth can be denoted as: This case is a little more complicated than the first two scenarios. E 1 (x, y) is more defocused than E 2 (x, y), but they are not on the same side of s 0 . Suppose s' 2 (x, y) is the symmetrical depth of s 2 (x, y) about s 0 , the process can be transferred from Fig.15(a) to Fig.15(b), and the finial depth can be denoted as: Here, E 1 (x, y) is less defocused than E 2 (x, y), and they are not on the same side of s 0 . Suppose s' 2 (x, y) is the symmetrical depth of s 2 (x, y) about s 0 , the process can be transferred to Fig.16(b), and the finial depth can be denoted as: However, the optimization process above is ill posed (Favaro et al 2008), that is, the minimum may not exist, and even if it exists, it may not be stable with respect to data noise. A common way to regularize the problem is to add a Tikhonov Penalty:   where the additional term imposes a smoothness constraint on the depth map. In practice, we use 0   ,0  k which are all very small, because this term has no practical influence on the cost energy denoted as: Thus the solution process is equal to the following: Eq. (39) is a dynamic optimization which can be solved by the gradient flow, the algorithm can be divided into the following steps (the detailed process can be seen in literature (Favaro et al 2008)): 1. Give camera parameters f, D,  , v,s 0 ;two defocus images E 1 ,E 2 ；a threshold  ；  and optimization step  ; 2. Initialize the depth map with a plan s, to be simple, we can suppose that the initial plane is an equifocal plane; 3. Compute Eq.(28)and attain the relative blurring; 4. Compute Eq.(27) and attain the solution (,, ) uxy t  of diffusion equations ; So if the initial depth is known, maybe it is just a general value, the dynamic depth, as well as the expected shape, can be reconstructed.

Experiment results
In order to validate the new algorithm, we used it to reconstruct the shapes of a nano standard grid which is 500nm high, two AFM cantilevers. We used the microscope of HIROX -7700 shown as Fig.17, and magnify the grid into 7000 times. The rest parameters are as the following: f=0.357mm, s 0 =3.4mm, F-number =2, D= f/2.

Fig. 17. HIROX-7700
In order to investigate the influence of different region size on the algorithm, we tested the grid with three kinds of region size and two kinds of AFM cantilever. As for the grid, through comparing to the true grid, the error maps in each experiment are constructed and the mean square error of the proposed method was calculated to test the precision. When testing the AFM cantilevers, we used PI nano platform to test the reconstruction precision.

Shape reconstruction of the nano grid
Firstly, the experiment using 120×110 pixel grid region was conducted. The results are shown in Fig.18 to Fig.21. Fig.18 is two defocused images in which the left is the image before variation and the right is that after variation; Fig.19 is the constructed 3D shape of the nano grid. In order to investigate the precision of the new algorism, we constructed the error map Φ between the true shape s and the estimated shape s  , and computed the mean square error φ of the whole image. The compute formulas are shown in Eq.(41) and Eq.(42). Fig. 20 is the true shape of the grid and Fig.21 is the error map.
www.intechopen.com Secondly, we tested our algorithm on the grid region of 120×50 pixels, the results are shown as Fig.22-Fig.25. The mean square error is equal to -0.038 and the average error is 10.39 nm.
From the figures, we can see that the error of our construction algorithm is slightly larger at the edge of the image and smaller at other region, this results from the optimization method. However, the average error is only about 2.3% and it can certainly satisfy the demand of micro/nano magnification.

Shape reconstruction of the AFM cantilever
The raise height of the cantilevers was controlled by the Iphysik Instrumente(PI) nano platform. Furthermore, we provided the performance of the algorithm on three kinds of raise height: 500nm, 300nm and 100nm. The theory of the experiment is shown as Fig.26. Firstly, the experiment using the conductive AFM cantilever was conducted. Fig.27 is two defocused images, in which the left is the image before variation and the right is that after variation; Fig. 28(a)- Fig.28(c) are the constructed 3D shapes of the bended cantilever when the PI platform rises 500nm, 300nm and 100nm. From Fig. 28., we can see that when the PI platform rises, the top end of the conductive cantilever bends obviously, and the deflection decreases gradually from the top end to the trailing end until close to a steady value; the bended degree is a monotonic function with the raise height. In order to contrast the bended precision, we choose the section image of them on the same position, and show them in Fig.29. From it, we can see that the deflection height is proportionately increases when the platform rises, and the height difference between the top end and the steady value is exactly equal to the raise height of the PI platform. Secondly, the experiment using the triangle cantilever was conducted. Fig.30 is two defocus images, in which the left is the image before variation and the right is that after variation; Fig.31(a)- Fig.31(c) are the constructed 3D shapes of bended cantilever when different raise height is 500nm, 300nm and 100nm; Fig.31 is the contrast image of these three sections. From them we can get the same conclusions as the last experiment, but the sensitivity of the triangle cantilever is lower because the reconstructed shapes are a little rough.
From these experiments, we can see that, regardless of the cantilever shape, our algorithm all can reconstruct the global bended shape exactly with only two defocused images. The following conclusion can be given: 1. The most obvious bend of the cantilever concentrates on the region near to the tip, it is reasonable because when the PI platform works up, the stress all concentrates on the tip due to our experiment theory. 2. The cantilever's original shape, material and illumination can influence the reconstruction result to some extent. For example, the conductive cantilever is thinner than the triangle cantilever, and the shape reconstruction of it is smoother due to its higher sensitivity; the black edge of the cantilever results in a little error in the result. 3. The raise height is larger, the calculation result is exacter, and the reconstruction image is smoother 4. No matter how much the raise height is, the reconstruction height tends to be steady finally. Furthermore, the height difference between the maximum value and the original value is equal to the raise height of the PI platform. www.intechopen.com

Conclusion
In this chapter, two typical micro vision algorithms were researched to model the process in micro/nano size: sub-pixel 2D motion measurement and DFD 3D reconstruction.
As for 2D motion measurement, this chapter mainly researched the problem of motion measurement based on the sub-pixel estimation for image sequences. Firstly, three important factors, including the searching region, the model size, and the fitting precision of sub-pixel, are analyzed and researched in detail, and the most appropriated parameters are chosen with respect to both the experiment results and the measurement characteristic of micro/nano image sequences. Then, the nano platform with high precision, the microscope with high magnification and the standard grid are used together to validate the measurement precision of this method. Finally, the proposed method is used to measure the driving characteristic curve of a piezoelectric actuator practically. The experimental results of the piezoelectric actuator driving characteristic measurement are consistent with the physics analysis. Also, the proposed method, which is simple in manipulation and credible in measurement results, satisfies the requirement of the micro/nano measurement with high precision.
On the other hand, a global shape reconstruction of the standard nano grid using single optical microscope was researched based on a new DFD method. Our primary contribution is to suppose a new global DFD algorithm. Therefore, it can be used to attain 3D information in one-eye vision, hand-eye system, especially in micro/nano manipulation. The second contribution is proposing a series of experiments to validate the method on micro/nano scale. The results below are significant: the computer vision can be used to reconstruct the global shape of the samples in micro/nano manipulation using defocused images without changing camera parameters.

References
Asada