Single image camera calibration in close range photogrammetry for solder joint analysis

: Printed Circuit Boards (PCB) play an important role in the manufacturing of electronic devices. To ensure a correct function of the PCBs a certain amount of solder paste is needed during the placement of components. The aim of the current research is to develop an real-time, closed-loop solution for the analysis of the printing process where solder is printed onto PCBs. Close range photogrammetry allows for determination of the solder volume and a subsequent correction if necessary. Photogrammetry is an image based method for three dimensional reconstruction from two dimensional image data of an object. A precise camera calibration is indispensable for an accurate reconstruction. In our certain application it is not possible to use calibration methods with two dimensional calibration targets. Therefore a special calibration target was developed and manufactured, which allows for single image camera calibration.


INTRODUCTION
Printed Circuit Board (PCB) Jet Printing is a new method in PCB manufacturing.Instead of using stencils, a jet printer is used to deposit up to 500 tiny droplets of solder paste per second through an ejector mechanism onto the board.This technique allows for more flexible layouts and the placement of smaller components.The viscosity of the solder paste can lead to trapped air blisters inside the cartridge.This in turn can lead to deviations in the solder volume or even to missing solder resulting in a cold solder joint and consequently in a faulty function of the board.Automatic optical inspection systems (AOI) can often be found in production lines.In general these systems are only able to detect defects of the board after it was manufactured.This ensures to reject all misprinted boards but does not have the ability to correct wrong or missing solder joints.The aim of the current project is to develop a built-in, closed-loop analysis of the printing process and a correction of missing solder joints or variations of the solder joint volume.This will lead to an increase of the systems efficiency.The image acquisition is done by a gray scale CMOS camera, which is attached to the printing head.The camera moves with the printing head and acquires images of printed areas from various positions.Using a wide-angle lens leads to large overlapping areas in the images, which can be used for photogrammetric reconstructions of the solder joints.This setup reduces the degrees of freedom of the camera to two and is called the "photogrammetric normal case", as shown in figure 1.A precise reconstruction of the data needs an accurate camera model.To be able to pre-and recalibrate the camera within the PCB jet printer only the remaining 2 degrees of freedom can be considered for the calibration.The camera can only be shifted in x-and y-direction.A camera calibration where the camera is shifted and rotated and where a two dimensional calibration pattern is used cannot be performed, because the world points must not lay on a single plane.The current work is dedicated to the modeling of the used camera.Due to the mentioned restrictions a specific calibration target was designed and manufactured.

METHODS
Because the image acquisition will be done during the printing process an appropriate measuring system has to be set up within the printer.In general it would be possible to attach various cameras, but the movement of the printer restricts the possible positions to those with a large working distances.This setup would lead to unfavorable data, because of the restricted resolution of camera systems.The smallest solder joints have a diameter of 300µm and have to be imaged with a short working distance.To acquire areas, where solder paste already has been applied and to solve the distance problem the camera can be attached directly to the printing head.
2.1 Hardware 2.1.1Demonstrator system For the proof of principle and for testing algorithms a laboratory system which can simulate the printers movement and can acquire manufactured PCBs was set up.This system can be seen in figure 2. Two linear slides are used to move the camera to arbitrary positions above the PCB.To have a uniform lighting and a unique appearance of solder joints in every view a bar light is used to illuminate the examination area lateral.This is necessary because a fixed lighting, with respect to the PCB is needed.That means a light attached to the camera, e.g. a ring light, cannot be used, because moving the the camera would also move the light, which result in different illumination angles for each solder joint and thus different appearances of them.This would complicate a feature extraction and matching.Also a fixed illumination system above the PCB is not suitable due to shadows from the camera and printing head.3) is motivated by the restricted degrees of freedom of the camera.Usually a two dimensional calibration pattern (e.g.chess board pattern) is imaged under different camera positions using all six degrees of freedom (rotation and translation).A well known restriction in camera calibration is, that the world points must not lay on a single plane.Since the camera in both the demonstrator system and the production system does have only two degrees of freedom (no rotation and no translation in z-direction), the world points must be distributed in x-, yand z-direction with respect to the world coordinate frame.The developed step block was anodized black.Using a precise laser labeling the circular calibration markers were applied to the target.Besides these markers the target also consists of three special markers (rectangle, cross, triangle) for orientation and origin determination.

Basics
To achieve an exact 3D-reconstruction a precise camera model must be available.Using this model the rays from each two dimensional image point xi xi xi through the camera center C into the world coordinate frame can be defined and intersections of those rays as 3D-positions of solder joint surfaces are calculated.camera models differentiate between interior (intrinsic) and exterior (extrinsic) parameters.While the extrinsics describe the camera location and orientation with respect to the world coordinate frame, the intrinsics determine the cameras imaging properties.In our application a projective camera has to be modeled.This model consists of 6 extrinsic parameters (rotation and translation) and 5 intrinsic parameters.These are the coordinates of the principal point (Hx, Hy), focal length (fx, fy) and the skew s).Starting from the central projection a three dimensional world point Xi Xi Xi is mapped via the focal length to the image point xi xi xi.The projection matrix P that describes that mapping C is the position of the camera center, R the rotation matrix and K the calibration matrix with the interior parameters.Additional parameters have to be taken into account if lenses with distortions are used.Especially the use of wide-angle lenses, like in this application, requires for the correction of radial distortion.The distortion function is dependent from the radius of a pixel to the center of distortion and is defined as: Considering the radial distortion the camera model extends by the two distortion parameters k f 1 and k f 2 .Thus the model consists of linear and nonlinear components.

2D-3D correspondences
The final calibration procedure should work largely automatically.Therefore it is necessary to detect and to classify the calibration markers.For the determination of the origin and the orientation of the calibration target the special markers are detected and analyzed.After an adaptive contrast equalization and the application of an adaptive threshold the filling level of the remaining regions with respect to their bounding box is calculated.The origin is defined as the center of the rectangle which has a filling level close to one.The targets orientation is determined by the centroid of the triangle and its distance to the triangles bounding box.With the known origin and orientation the detected calibration markers can be assigned to their 3D coordinates.Since reflections can influence the centroid position of each marker a better possibility to extract them is the circular hough transform.The used image points are the centers of the detected circles.

Calibration
In general camera calibration determines a 4 × 3 projection matrix P , which is a projective transform from known 3D coordinates Xi Xi Xi to the detected 2D image points xi xi xi.
With the known point correspondences the following system of equations can be set up: Rearranging this system results in (4) Singular value decomposition can be used to solve this system.The derived solution is used as an initial point for a further optimization.Therefore the Levenberg-Marquardt algorithm minimizes the geometric error between measured and calculated image points after equation 3. It can be seen in equation 1 that the projection matrix P can be decomposed into the components of interior and exterior orientation.The calibration matrix K and the rotation matrix R are derived from the RQ-decomposition of the left 3 × 3 of P .

Image correction
The calculated projection matrix P does not take into account how lens distortions, caused by the wide-angle lens, do influence the image coordinates.To correct the image data with respect to the radial distortion the corresponding parameters k f 1 and k f 2 are estimated simultaneously during the Levenberg-Marquardt optimization.Afterwards the image is corrected, where the new image coordinates (xi, ŷi) are calculated as follows: Figure 4: distortion functions Figure 4 shows the distortion function and the application of the function to a regular grid can be seen in figure 5.If the real image is corrected one cannot simply write the pixel values of the original image to the corrected pixel coordinates.This would result in an incomplete image, because more pixels are needed than in the original image.To solve this problem an area mapping from the corrected to the original coordinates has to be performed.This can only be done by the inverse distortion function, which is not simply an inversion of the original distortion function, because it is dependent on the radius of the original image.To achieve the inverse distortion function another optimization is needed to calculate the inverse distortion parameters ki1 and ki2 for the function:

RESULTS
The calculated camera position and rotation with respect to the world coordinate frame can be seen in Figure 6.The camera (red) points perpendicular down to the calibration target (blue calibration markers) and is located above the origin.Which corresponds exactly to the acquisition setup.The calculated distortion functions were used to correct the image data.The resulting effect is shown in figure 7.In the original data of the detected calibration markers are arranged along curves.After the correction   the measurement of solder joint volumes and the feedback to the printing process.After the calibration is implemented successfully within the demonstrator system, corresponding features in overlapping images will be determined.These features, together with the camera model will be used for the three dimensional reconstruction of the applied solder joints.Figure 8 shows an exemplary reconstruction with a freeware tool.The reconstruction is used to evaluate the position and the volume of the solder joints and thus as a feedback for the printer about the printing result.

Figure 1 :
Figure 1: normal case in photogrammetry

Figure 2 :
Figure 2: normal case in photogrammetry

Figure
Figure 3: calibration target

Figure 6 :
Figure 6: camera orientation -visualization of the camera (red) above the 3D coordinates of the calibration markers

Figure 7 :
Figure 7: distortion correction -left: original image, right: correction by lens model

Figure 8 :
Figure 8: exemplary 3D reconstruction of solder paste on a PCB