Subpixel corner point detecting method based on greyscale constraint used for calibrating industrial microscopic systems

: To solve the problems of missed detection, repeated detection and low precision in the process of subpixel corner point detection of black-and-white checkerboard used for calibrating microscopic devices in complexed industrial environment, a new detection method based on greyscale constraint on the four-neighbourhood diagonal was proposed in the paper. By analysing geometric characteristics of the four-neighbourhood region and the grayscale features in diagonal direction, the SINC greyscale distribution was adopted to constrain corner point position, which realised rapid detection of subpixel corner point. Comparing with the existing methods, in the new method proposed the rate of repeated detection and missed detection decreased by about 20% and 2% respectively, which achieved a high rate of detecting accuracy over 99.9%. Meanwhile, the maximum error of corner point detection lowered from ±0.6pix to ±0.3pix, which showed that precision improved by about 50%. Finally, microscopic calibrating experiments used for micro-hole centring system were carried out. The results show centring error was reduced from 10 μm (before calibration) to 3 μm (after calibration), and centring precision was increased by about 75%. It demonstrated that the new method improved the accuracy and precision effectively, which verified its applicable feasibility of microscopic calibration used in industry spots.


Introduction
In complex industrial fields, the process of microscopic calibration is a key factor that affects the precision, accuracy, and stability of running devices. Considering a high precision, rapidity, and simplicity of the process in advanced industrial microscopic calibration, micro black-and-white checkerboard is widely used in the calibration of a plane-based microscopic imaging system [1]. However, to realise high-precision and fast microscopic calibrating, it is very important to extract corner points on its images, because the calibrating precision of the microscopic imaging system is directly affected by the precision and accuracy of corner point detection [2].
The existing corner point detecting methods mainly include line intersection method and greyscale feature method. In line with intersection method, a critical issue is how to extract the edge straight line of the inspected area. For instance, Hu et al. [3] studied edge line extraction by the method of using two successive Radon transform to rectify these grid lines, and solving the algebraic equations to acquire grid corners. Recent researchers like Deng et al. [4] and Antonio et al. [5] have also reported the different methods for edge line extraction by contour modelling. However, their detecting precision is at pixel level, and it is difficult to meet the subpixel precision requirement in the microscopic calibration process.
For obtaining higher detecting precision, domestic and overseas scholars also have studied many corner points extracting method based on greyscale feature, and put forward some effectively improved algorithms. Sirkis [6] points out that the grey-gravity method has the highest detection precision, up to 0.02pixel and much faster. However, due to existing dust pollution and the interference of environmental stray light in practical industrial fields, some suspected grid points occur most often in the microscopic image, which easily lead to improper corner point extraction. Yang et al. [7], Sun [8], Inbarasan [9], and Xing et al. [10] propose a grey-gravity method based on SUSAN operator. In this method, the location of corner points at pixel level is fist solved by adopting SUSAN operator, and then in this location by these processes in turn (template matching, area forming and greygravity solving etc.), the coordinates of corner points at subpixel level are obtained, which has fast calculation and high detecting precision. However, the shape requirement for the circular template is stricter, resulting in less efficient template matching and sensitive to image noises, thus missed detection and repeated detection occurs easily [8]. Bok Y et al. [11] propose an improved cross-template matching approach to attain subpixel detection aimed at the external big fields. It can effectively reduce the influence of the bigger noises on the detecting accuracy and precision in complexed environment, but does not suitable for microscopic calibration in the small field. Besides, the calculation process is complex and low the real time.
In additional, the corner detector introduced by Harris [12] is widely used as a general feature detector. This method analyses the matrix of local derivatives around each point. It has been deeply studied and improved by recent researcher [5,13], for example, Liang et al. [14], Banerjee et al. [15], and Emam et al. [16] present grey-centre method based on Harris operator to implement subpixel corner point detecting. In this method, the location of corner points at pixel level are firstly solved by adopting Harris operator, and then in this location by building the error minimisation function and iterative solving it, the coordinates of subpixel corner points are attained, which has also higher detecting precision. It requires higher image quality, and the image noise can easily lead to repeated detection, which lowers the detecting accuracy.
On the other hand, Guenneau et al. [17], Cui et al. [18], Pang et al [19] put forward quadratic polynomial approximation method based on Harris operator to implement the subpixel corner detection. It firstly employs the quadratic polynomial to approach the response function of the corner point, and then the unknown J. Eng coefficients in the response function are solved by using leastsquare methods. Finally, the extreme point of the response function on the fitting surface is solved, that is the subpixel corner point. The quadratic polynomial approximation method is simple and straightforward, and it is insensitive to image noise, thus the rate of repeated detection is decreased effectively, and has higher detection accuracy. However, the finite polynomial fitting of the response function produces larger residual error in real computation, thus the detection precision is lower than that of greycentre method and the grey-gravity method.
The above analyses show that the grey-gravity method based on SUSAN operator and the grey-centre method based on Harris operator can achieve subpixel-level detection, but missed detection and repeated detection occur easily, thus the rate of detecting accuracy need to be improved. For the quadratic polynomial method, although the accuracy rate is higher than that of the two methods, the precision is lower. In order to be geared to the operating needs, such as simple and rapid on-the-spot detection in complexed industrial environment, aiming at the corner point detection of micro black-and-white checkerboard used for calibrating microscopic devices, the existing corner point detecting methods are difficult to ensure high accuracy rate and stability while satisfying high precision as mentioned in the literature review. Here, a novel corner point detecting method based on the greyscale constraint on four-neighbourhood diagonals is proposed. By analysing of the geometric characteristics of the four neighbourhoods and the maximum greyscale feature in the diagonal direction, the SINC function is adopted to extract the corner position. Then, contrast experiments and error analysis based on different detecting methods are carried out to analyse the accuracy and precision. It can achieve higher accuracy rate and better anti-noise performance while satisfying subpixel precision. Finally, the micro-hole centring applied experiments before and after system calibration are conducted to verify the feasibility of the method in microscopic calibration.

Subpixel corner point detecting method
In the microscopic calibration of the micro-hole centring system, micro black-and-white checkerboard is adopted. In the calibrating process, corner point coordinates in the checkerboard need to be extracted to achieve subpixel level. The steps of this method based on greyscale constraint on the four-neighbourhood diagonal are as follows:

Four-neighbourhood edges fitting
The images of a black-and-white checkerboard are just acquired by the microscopic imaging system, and the images of the checkerboard are obtained after pre-processing with greyscale, binarisation and so on. The black-and-white image of checkerboard is shown below in Fig. 1a. In any four-neighbourhood, D ij , the Canny operator is used for edge extraction. As shown in Fig. 1b, a Cartesian coordinate system xoy is established with the origin at a pixel point o(0,0) in the image. By using the least-square method, four edges of the D ij are obtained. The line equations of four edges in the D ij are as follows: The upper line l i−1 of the neighbour-hooded edge: y = c i−1 x + d i−1 , the lower line l i+1 of the neighbour-hooded edge: y = c i+1 x + d i+1 , the left line m j−1 of the neighbour-hooded edge: y = a j−1 x + b j−1 , the right line m j+1 of the neighbour-hooded edge: y = a j+1 x + b j+1 . Where c i−1 , ci+1 , a j−1 , aj+1 are the slopes and d i−1 , d i+1 , b j−1 , b j+1 are the intercepts, (x, y) is any point in region D ij .

Isometric line constraints in four-neighborhood
As shown in Fig. 1c, e l is the isometric line between the upper line (l i−1 ) and lower line (l i+1 ), and e m is the isometric line between the left line (m j−1 ) and right line (m j+1 ). Then, the intersection of the two lines is regarded as point O ij (x 0ij , y 0ij ). The concrete steps to find the point O ij are as follows: Then at x k , the coordinates of isometric point between the upper line and the lower line are points in in their neighbouring regions, and the isometric line e m is fitted using by the least-square method as shown in (1): where q is the slope and p is the intercept in the fitted isometric line e l . Therefore, the equation of the line e l is shown below: where v is the slope and u is the intercept, the equation of the line e m is shown below: y = vx + u (4) (iii) By solving (2) and (4), the isometric point O ij (x 0ij , y 0ij ) in the four-neighbourhood region D ij , is obtained.

Greyscale constraint in diagonal direction
In the four-neighbourhood D ij , the circular region Z ij as shown in Fig. 1d is acquired with O ij as the centre, with a radius of S (S = 0.5pix). Assume that the greyscale distribution satisfies SINC function in region Z ij , and then the expression of the SINC function is shown in (5).
f (x, y) = sin(x + Δx) x + Δx × sin(y + Δy) y + Δy (5) where Δx, Δy are the offset in the x and y axes. According to line equations of the four-neighbourhood edges in the D ij , two diagonals can be found. As shown in Fig. 1c, the diagonal, l mij with higher average greyscale value is picked, and the slope of l mij is K lmij . Then, as shown in Fig. 1d, construct the line l′ mij with slope K lmij through point O ij (x 0ij , y 0ij ). The equation of the new line l′ mij is shown below: Thus, the maximum point of the SINC function f (x, y) in the field of (x − x 0ij ) 2 + (y − y 0ij ) 2 = S 2 is the subpixel corner point C ij . The objective function is shown below: where ▵ x, ▵ y are the offset in the x and y axes, τ is the Lagrange multiplier, and K lm ij = tan (tan −1 c i − 1 + tan −1 a j − 1 )/2 is the slope of line l mij .
In order to improve the fitting accuracy, it is necessary to sample multiple times for (7), and use the Gauss-Newton or Levenberg-Marquardt [20] method to convert the least square with greyscale constraint on the four-neighbourhood diagonal line to the unconstrained least square. Subsequently, the relevant parameters can be solved and the coordinates of the corner point C ij in subpixel level can be obtained as shown in Fig. 1e.
In the new detecting algorithm, the position and grey features of the detected corner point are limited by the greyscale constraints on the four-neighbourhood diagonal line, which avoids the missed detection and repeated detection caused by the existing dust pollution and the interference of environmental stray light in practical industrial fields. Under the premise of guaranteeing the detecting precision, the detecting accuracy rate of subpixel-level corner point is greatly improved.

Microscopic calibrating process
Establishing the transfer relationship between the pixels coordinates system and world coordinates system of the checkerboard completes the microscopic calibration. Suppose the pixel coordinates of light heart in the image are represented as (u 0 , v 0 ), and the physical size of every pixel in the x and y axes are dx and dy. The relation between the pixel coordinate system and world coordinates system is shown below: [16] u where R stands for rotation matrix and T represents for translation matrix, f is the focal length of the microscope, (X w , Y w , Z w ) presents the corner point coordinates of the checkerboard in the world coordinates, and (u, v) is subpixel corner point coordinates in the pixel coordinates system. Building equations by several points of checkerboard in pixel coordinates system and world coordinates system, the parameter matrixes of the system are solved by leastsquare method, and then the calibrating process is finished.

Corner points detecting experiments
In this experiment, the black-and-white checkerboard is adopted as the calibrating board, its type is 6-23 × 17-0.25-1.6, and the size of each small square in the checkerboard is 0.1 × 0.1 mm. The checkerboard's image is acquired through LabVIEW Ni Vision module, and the original image collected with resolution of 640 × 480 pixels is shown in Fig. 2a. Then, experiments of the corner point detection are conducted by applying the grey-gravity method, grey-centre method, quadratic polynomial approximation method and diagonal greyscale constraint method, respectively, the results of each method are shown in Fig. 2b-e, where the ' + ' stands for the corner position (white point).
The detection results show that there exists repeated detection, missed detection, and error detection to some extent in the greygravity method, the grey-centre method and the quadratic polynomial approximation method. The detection accuracy of above-mentioned the four different methods are compared from the respects of the rates of repeated detection and missed detection and so on, as shown in Fig. 3. Figure 3 show that there exists a total of 144 corners in the original image. In grey-gravity method based on SUSAN operator, 144 corners are detected, no missed detection, but 30 corners are detected repeatedly. In the grey-centre method based on Harris operator, 141 corners are detected, and the corner's number of missed detection and repeated detection are 3 and 26, respectively. In the quadratic polynomial approximation method based on Harris operator, 140 corners are detected, while 4 corner points are missed detection and 11 corners are detected repeatedly. In the diagonal grey constraint method proposed here, 144 corners are correctly detected, without missed detection, repeated detection and error detection.
The analytic results indicate that the rate of repeated detection in the diagonal grey constraint method is decreased by 18.06, 7.64, and 20.83% when comparing with the grey-centre method, the quadratic polynomial approximation method and the grey-gravity method, respectively. The rate of its missed detection in the diagonal grey constraint method is decreased by 2.08, 2.78 and 0%, comparing with above-mentioned other three methods, respectively. Thus, it shows the rate of detecting accuracy of the new method reaches to 100%, which proves that the diagonal grey constraint method effectively improves the accuracy of the corner point detection.

Comparison of detection precision
Error analyses of corner point coordinates are conducted to compare the detection precision in different methods. As shown in Fig. 4 below, the maximum error of corner coordinate is about ±0.6 pix in the grey-centre method, quadratic polynomial approximation method, and grey-gravity method, while the maximum error in the diagonal grey constraint method is about ±0.3 Pix. Compared with the three other methods, the subpixel corners' detecting precision of the method proposed here is increased by about 50%.

Repeatability and anti-interference analysis
To verify its good repeatability and anti-interference of corner point detection in the diagonal greyscale constraint method proposed here, the comparative experiments such as image rotation and brightness changes of the image were conducted. The experimental results of image rotation (5, 30, 55, 85 degrees clockwise) and brightness variation (positive 50,100 nits(cd/cm 2 ), negative 50,100 nits) were shown in Fig. 5a-h, indicated that the image changes of checkerboard did not lead to missed detection, repeated detection, and error detection, and the accuracy still reached to 100%.
The image rotation and brightness variation have not a negative effect on the detecting accuracy of the method. From Figs. 6 and 7, the maximum error of subpixel corner detection was <±0.3 pix after the image changes of checkerboard. Thus, it illustrates the good repeatability and high stability of the diagonal greyscale constraint method, which is more immune to interference from the stray light around the surroundings.

Micro-hole centring experiments
Due to its advantages such as quick, non-contact and so on, the image centring method is applied to the secondary machining micro holes. When applying the method, unavoidable noise and distortion will have significant negative impacts on the centring process. Thus, the microscopic calibration is a necessary step for precision micro-hole centring setup. Acquiring the image of black-  and-white checkerboard by the experimental device [21], the corner-point coordinates of subpixel level are extracted by adopting the diagonal greyscale constraint method. After that, getting the offset of two centres and revising the related parameters to calibrate the system, the experimental results are shown in Fig. 8. If the system does not carry out the calibration process, the centre of the electrode outer circle and the centre of the hole cannot fully coincide, and exists more than 2-pixel error (>10 μm), thus it will influence the secondary processing of the micro-hole, and then damages the micro-hole profile. After calibrating, the centring error of two centres is <0.5 pixel (<3 μm).

Repeatability and anti-interference analysis
The error of corner detection mainly includes algorithm error and microscopic imaging error. For the algorithm error, it mainly comes from least square fitting, the calculation of diagonal slope, and SINC function. When edges of four-neighbourhood are fitted by least squares method, there exists unavoidable error. Applying the mean square error equation and taking 50 samples to get the error of the least square fitting, this is about 0.095 pixel. Calculation of the diagonal slope is based on two least squares fitting, and then the error is about 0.08 pixel according to error transfer function. For SINC function, the maximum error is about 0.13 pixel when the iteration step is 0.01. The theoretical error of system is <0.3 pixel, which is in agreement with the experimental results. For the microscopic imaging error, it is mainly from lens aberration, photosensitive pixel misalignment, perspective error, and the relative position error between workpiece and microscope. Analysing the experimental results, the imaging error is smaller

Conclusion
Considering the complication of environment, high precision of detection, rapidity, and simplicity of the process in advanced industrial microscopic calibration, aiming at the repeated detection and missed detection in the subpixel corner point detection of micro black-and-white checkerboard using microscopic calibration of micro-hole centring system, the paper proposed a novel subpixel corner point detection method based on the greyscale constraint on the four-neighbourhood diagonal. In any four-neighbourhood of the checkerboard, find the diagonal through isometric point. In the diagonal direction, defining SINC greyscale distribution as constraints, the maximum point in the SINC function is the subpixel corner. Then, comparative experiments are conducted in the micro-hole centring setup. The results showed that in the new method the rate of repeated detection and missed detection decreased by about 20 and 2%, respectively, which achieved a detecting accuracy rate of over 99.9%. Meanwhile, the maximum error of corner point detection lowered from ±0.6pix to ±0.3pix, which showed that precision improved by about 50%. Finally, microscopic calibration experiments of micro-hole centring system were carried out. The results show centring error was reduced from 10 μm (before calibration) to 3 μm (after calibration), and centring precision was increased by about 75%. It demonstrated that the new method improved the accuracy and precision effectively, which verified its applicable feasibility of microscopic calibration used in industry spots.