Photogrammetric measurement of 3D freeform millimetre-sized objects with micro features: an experimental validation of the close-range camera calibration model for narrow angles of view

The measurement of millimetre and micro-scale features is performed by high-cost systems based on technologies with narrow working ranges to accurately control the position of the sensors. Photogrammetry would lower the costs of 3D inspection of micro-features and would be applicable to the inspection of non-removable micro parts of large objects too. Unfortunately, the behaviour of photogrammetry is not known when photogrammetry is applied to micro-features. In this paper, the authors address these issues towards the application of digital close-range photogrammetry (DCRP) to the micro-scale, taking into account that in literature there are research papers stating that an angle of view (AOV) around 10° is the lower limit to the application of the traditional pinhole close-range calibration model (CRCM), which is the basis of DCRP. At first a general calibration procedure is introduced, with the aid of an open-source software library, to calibrate narrow AOV cameras with the CRCM. Subsequently the procedure is validated using a reflex camera with a 60 mm macro lens, equipped with extension tubes (20 and 32 mm) achieving magnification of up to 2 times approximately, to verify literature findings with experimental photogrammetric 3D measurements of millimetre-sized objects with micro-features. The limitation experienced by the laser printing technology, used to produce the bi-dimensional pattern on common paper, has been overcome using an accurate pattern manufactured with a photolithographic process. The results of the experimental activity prove that the CRCM is valid for AOVs down to 3.4° and that DCRP results are comparable with the results of existing and more expensive commercial techniques.


Introduction
The measurement of millimetre-and micro-scale features is performed essentially with optical methods, such as confocal microscopy, interferometry, depth from focus and holographic conoscopy [1][2][3]. Normally these methods result in high-cost technologies and are used on systems that can digitize small features only on small objects due to their narrow working ranges and the need to accurately control the position of the sensors.
The development of digital photography, the continuous evolutions in sensor resolutions and the increasing power of image matching algorithms have also allowed digital closerange photogrammetry (DCRP) to acquire a potential role in millimetre and sub-millimetre-scale measurements. Its use would bring two-fold advantages: (i) portability for the on-field digitization of micro-features as it requires only a camera, preferably on a static support, and (ii) very low costs since consumer cameras and cheap or open-access software are used.
While most research applications of DCRP are related to meso-fabricated and macro-fabricated industrial products according to the classification in [4], a limited number of papers have focused on millimetre-and micro-scale DCRP and these are mainly related to the use of macro-lens DCRP on microscopes or cameras [5][6][7].
Treating micro and millimetric features with a photogrammetric approach implies the use of rather high magnification ratios for photography, resulting in an important narrowing of the depth of field (DOF), defined as the distance between the nearest and farthest objects in a scene that appear sharp in an image, leading to a very difficult calibration [8]. In this context a crucial role is played by the diaphragm aperture of the camera. It is well known that the narrower the aperture, the greater the DOF but the risk of increasing diffraction increases. As a consequence, there is a limit to the magnification, above which photogrammetric algorithms cannot be applied because they require diaphragm apertures which are too narrow. In this condition, we would expect the effect of diffraction to be so great as to prevent an accurate 3D Digitization.
In this context it is evident that a crucial role is played by calibration, which is required to manage in particular both the following issues: the image blur and the accuracy of the pattern.
The achievement of high magnification ratios can imply the use of long-focal-length lenses, which in [9] are believed to need correction on the classical collinearity equation model when the camera angle of view (AOV), defined as the angular extent of a given scene that is imaged by a camera, is smaller than 'around 10°'. In [10] it has been found that calibration accuracy can generally be maintained at the same level if image noise is inversely proportional to focal length. The issue of camera calibration with a narrow AOV is also addressed in [11], where deficiencies are revealed not in the validity of the pinhole camera model but in the algorithms used. In [12] a comprehensive overview of calibration methods is reported, complete with comparative experimental calibration results using commercial software. Additionally many lens distortion models exist with several variations and each distortion model is calibrated by using a different technique [13]. However, in the literature, the standard calibration methods are based on the close-range camera calibration model [14] for lens distortion.
Normally different models are compared considering the well-known parameter called reprojection error (RE) [15], namely the geometric error corresponding to the average image distance, measured in pixels, between a point, projected according to the camera calibration model, and its corresponding measured one. RE is equivalent to the root mean square (RMS) image residuals used in photogrammetric literature.
In the present paper, the authors aim to contribute to the study of the ability of DCRP to correctly describe objects with micro-features involving DCRP/computer vision tools when used to obtain accurate 3D models of small objects. An experimental approach is applied to camera calibration for millimetre-sized objects to study the relation between calibration results, photogrammetric hardware and 3D digitisation results. A general calibration procedure is introduced, based on the open-source software library OpenCV, to calibrate narrow-field cameras with the CRCM. Subsequently the procedure is validated using a reflex camera with a 60 mm macro lens, equipped with extension tubes (20 and 32 mm) and achieving magnification of up to nearly 2 times. The authors have employed particular attention to the accuracy of 3D digitisations performed after the calibrations and to verifying the statements reported in [9] as regards the calibration of narrow AOVs.
The rest of the paper is organized as follows: section 2 describes the theory of the CRCM, how it should be modified according to existing literature and how it is applied in OpenCV and in the present paper. In section 3 the calibration methodology is proposed and its validation with experimental activity is reported for AOVs lower than 10° and magnifications of up to nearly 2 times. In section 3 the results are presented in terms of calibration results (3.1) and their relation with the accuracy of the 3D digitization of two small products with micro features (3.2); in section 4 the results are discussed and section 5 concludes the paper.

Theory
The CRCM distortion model according to the Zhang method described in [16] considers that the mathematical basis of calibration is the well-known extended collinear model, which describes the perspective transformation between the 3D object space and the bi-dimensional image space. Let us consider a generic point P real in the 3D space and its projection onto the sensor plane, through the optical center of the camera (figure 1).
If no lenses are interposed, the projection will be collinear with the optical center of the optical system (namely the inverse pinhole), with P real determining the point P corr. (x , corr y corr ). On the other hand, if a real lens is interposed between the scene and the sensor, the correspondent point on the sensor plane (or image plane) will be a generic point P ( ) x y , that is not collinear.
One of the aims of calibration is to calculate the difference in terms of coordinates Δx and Δy between P and P corr due to distortions, as shown in figure 1, where the principal point ( ) c c , x y is defined as the point on the sensor plane where the optical axis of the lens intersects the sensor plane.
In [17] the Zhang calibration model is implemented considering the terms x and y, defined respectively as: while the coordinates of P corr can be computed as: In [17] the computation of Δx and Δy is divided into the radial distortion and tangential distortion contribution and can be summarized into: where r is the distance between the principal point and the considered point on the image: k k k 1, 2, 3 are the coefficients of radial distortion, and p p 1, 2, are the coefficients of tangential distortion, respectively of the first and second order.
It is important to highlight that in literature it is proposed to treat narrow AOV lenses by modifying the collinear model expressed with equations (5) and (6) in the following way [9]: with Δc defined as a correction value to be computed according to the model presented in [9] and k 1 as the first-order radial distortion coefficient. The authors of that paper state that this modified model is valid for AOVs lower than 10° and the conventional model is not correct. It is noted that, assuming in (8) and (9) Δc as zero, the result is a simplified version of equations (5) and (6), where only the first-order radial distortion affects the calibration and the remaining components are neglected. In fact the linear term Δc reduces the distortion for the exterior points smoothing the distortion curve. For very low levels of radial distortion associated with narrow AOVs, the term is not required, but it is included in the model for completeness.
With the aim of verifying which camera model can be applied when micro features are to be measured with DCRP, the authors of the present paper have studied the calibration of a camera equipped with extension tubes experimentally. Moreover, the traditional CRCM, as implemented in [17] through equations (5) and (6), has been used for studying and validating calibration with DCRP regarding two conditions of narrow AOVs in particular, namely 3.9° and 3.4°. The parameters included in the present study have been: k k k , 1, 2, 3 p p 1, 2, as regards distortion plus principal distance and principal point as constituting the camera matrix [17]; no compensation of sensor plane unflatness has been included into the model since actual electronic manufacturing technologies guarantee the sufficient planarity of the sensor. Consequently accurate photogrammetric measurement results have been found using the traditional CRCM for narrow AOVs.

Experimental activity
The photogrammetric system used in this paper was a Canon 400D camera with a Canon EF-S 60 mm Macro USM lens, sensor size: 22.3 × 14.9 mm, focal length: 60 mm and minimum focus range: 20 cm. The system was preliminarily tested in several versions: lens only and implemented with a set of extension tubes, with the attention focused later on 20 and 32 extension tubes.
The macro lens was chosen to exploit its ability to capture sharp images of small features with small working distances. Long-focal-length cameras exploit the linear increase of magnification as distance decreases, forcing the operator to position the camera at a great distance from the object. Moreover, this configuration involves practical limitations such as the need for large laboratories and difficulties in pointing the object from several points of view. Consequently, the authors have chosen a moderate-focal-length lens, equipped with where z i is the distance between lens and sensor and z 0 the distance from object to lens. In particular, the Canon EF-S 60 is made up of 12 lenses in 8 groups, which are assumed to be considered as a unique lens. Some of the very important advantages of extension tubes are very low costs, flexible and upgradable increase in magnification with virtually any camera lens, no additional glass elements minimizing any potential loss in image quality and exploitation of the central part of the lens, helping to keep distortion at a low level. Their main disadvantages are a narrowing DOF and increase in light intensity required when increasing z i needed.
The calibration software tool chosen by the authors was the calibration module of OpenCV [17], a well-known opensource computer vision library that allows full repeatability of the present work.
The calibration methodology has been designed as follows: assuming to capture photos in landscape mode, and on the basis of a quadrangular shape of the pattern, three main rotation axes are considered: horizontal, diagonal and vertical. In figure 2 a scheme of the scene, one possible pattern and the axes of rotation during calibration are shown. Note that the width and the height of the scene are identified respectively with numbers 3 and 2, since the sensor format of the camera is 3:2. This does not affect the generality of the approach because it can be repeated with all kinds of sensor formats, resizing the format of the pattern adequately.
Like many other reflex camera lenses, the Canon EF-S 60 mm Macro Lens has a settable focusing distance: once it has been set, the DOF depends only on the length of the extension tube. After measuring or computing the DOF for each configuration and considering each rotation axis described above, it is possible to compute the maximum rotation angle for each axis that keeps the edge of the rotated pattern inside the DOF; it is possible to define this angle as the critical angle on that axis. As an example, figure 3 shows the critical angles computed for the Canon 400D with the EF-S 60 Macro Lens with no extension tubes, using the following equation [18] to compute the DOF in millimetres: where C is the diameter of the circle of confusion (CoC) in millimetres, and M is the magnification and N the f-stop, both dimensionless numbers. Like that of DOF, the definition of CoC descends from optics applied to analog photography: CoC can be defined as the largest blur spot that is perceived by the human eye as a point. The definition of CoC is strictly linked to the human eye; consequently the diameter of the circle of confusion depends on visual acuity, viewing conditions and magnification.
In the field of digital photography, a rigorous computation of C is not an easy task and, in equation (11), C has been set at the conventional value of 0.03 mm used for 35 mm films [19] just to provide an example of computation, extended to all the axes, to build the graph shown in figure 3. In this graph the critical angles generated by the horizontal axis are highlighted in red, the critical angle generated by the diagonal axis in green and that generated by the vertical one in purple. In figure 3 the angles are computed considering the camera with no extension tubes, an f/32 aperture and maximum magnification equal to 1 × as declared by the manufacturer, obtained using the minimum focal distance.
These rotation angles should allow the pattern to be kept in the AOV of the camera with no or negligible blurring for the dot recognition software. As regards the extension tubes, the critical angles decrease with the increase of tube length. Unfortunately, the value of the computed DOF is only the first approximation, but it can be used by the authors as a reference for the calibration methodology.
Currently OpenCV supports three types of patterns for calibration, namely (a) Checkerboard, (b) Symmetric circle pattern, and (c) Asymmetric circle pattern. Preliminary tests, partially reported in [20], led to the consideration that circle features are less sensitive to blurring than the corners of a Checkerboard. Among circular dot patterns, the choice of the  authors was to redesign the most recent 'asymmetric' pattern proposed in OpenCV, in order to be fully covered by the sensor, adding two dot columns. The new pattern is compatible with OpenCV routines and its ratio is 1.69, as near as possible to the 3:2 camera format. The recognition of dots is based on the well-known OpenCV blob detection method. Basically it consists in calculating the centroids of the connected components (blobs) from binary images, with subpixel precision. The accuracy of subpixel estimation depends on several factors, such as the blob number of the pixels, the image point spread function, the noise levels and the spatial frequency of the image data. A commonly quoted rule of thumb is 0.1 pixel.
The pattern was scaled to the subsequent sizes 12.2 × 7.5 mm (diagonal d 20 = 14.3 mm) and 11.1 × 6.8 mm (diagonal d 32 = 13 mm), respectively to be used with 20 and 32 mm extension tubes.
In DCRP calibration, the images must include as many perspective views as possible, obtained by images with several, high angles between the principal axis of the camera and the pattern [21], but this is in contrast with a narrow DOF.
The camera was set up with manual focus at the minimum focus distance, namely f 0 = 200 mm without extension tubes: this set-up lets the procedure be repeatable and this configuration has been kept with all the extension tube lengths, setting the minimum focus distance for each extension tube. Focus distances with extension tubes have been measured with digital calipers, which manually measured the distance between the center of the focused pattern and the sensor plane indicator on the camera, resulting in f 20 = 211 mm for the 20 mm tube and f 32 = 220 mm for the 32 mm tube.
For a given rotation angle the maximum AOV is along the diagonal. Then it is possible to compute: where i is equal to the extension tube length, d i is the diagonal of the pattern used for the tube whose length is equal to i and f i is the focus distance as explained above. Applying equation (12), the maximum value of the AOV is equal to 3.9° for the 20 mm tube and 3.4° for the 32 mm tube.
The authors experienced that the OpenCV 2.4.6.0 calibration routines did not manage images with lateral dimensions higher than 1 Mpixel. The original calibration images were resized from 3888 × 2592 to 972 × 648 pixels. The reduction was made with cubic interpolation using open-source GNU Image Manipulation Software and cubic interpolation.
Comparative tests were carried out using the same lens, no tube and infinite focus distance on macro patterns printed on A4 paper sheets. In that case the results between the full size and the reduced size were absolutely the same. Image resolution was divided by four and therefore the precision (in pixels) associated with the 'recognition of circles method' is around 0.4 pixels (considering a subpixel precision of 0.1 for centroid detection).
In figure 4 the experimental set-up is shown, constituted by an illuminated plane, the Canon D400 with extension tubes on a cantilever tripod and the calibration pattern.
For each configuration 16 images were captured following the subsequent rules: excluding the zenithal image, the remaining 15 images were realized, 5 per rotation axis shown in figure 2. The first three images per axis are realized at the angles computed, dividing into three steps the critical angle on that axis, according to equation (11). The remaining two images per axis are realized adding to the critical angle respectively one and two steps more: blurring on external dots did not affect recognition of the dots. The camera was kept in a static position while the pattern was moved. The images captured with each configuration were used to perform OpenCV calibrations.
If any other parameters do not vary, each tube length determines one magnification and one particular size of the calibration pattern that fully covers the AOV. For each tube length the critical angles have been computed and the images captured for calibration.

3D Reconstruction results
Since the focus of this paper is to demonstrate the validity of the camera calibration model for narrow AOVs, the tests were carried out using images of the objects as input of the commercial software Agisoft Photoscan version 0.9.1, changing the calibration intrinsic parameters as resulting from calibration for the smallest aperture f/32, to approximate the pinhole model better since diffraction effects were not evident during calibration.  At the first stage, this software detects points in the source photos which are stable under viewpoint and lighting variations and generates a descriptor for each point based on its local neighborhood. Using a scale-invariant feature transform approach, these descriptors are used to detect correspondences across the photos.
Subsequently camera intrinsic and extrinsic orientation parameters are solved using a greedy algorithm to find approximate camera locations and refine these locations using a bundle-adjustment algorithm. Dense surface reconstruction is performed with algorithms based on pair-wise depth map computation. The main parameters required by Agisoft Photoscan have been set in the following way: accuracy high, image pair preselection generic, maximum 40 000 points per photo, no use of masks, surface type arbitrary, source data dense, medium quality, moderate depth filtering.
The resulting point clouds were all constituted by approximately 500 000 points with a maximum triangle count set to 10 6 points. For close-range non-contact measuring devices, metrological standards have already been developed for their derivation from contact-based techniques, but suffer from checking parameters which are very similar to those attainable from CMMs. As a consequence this field requires more extended work at the international organization level [22]. Thus dimensional accuracy was evaluated by comparing point clouds with commercial computer aided design (CAD) software using the following methodology. As a first case study a calibrating steel sphere with a 10 mm diameter was digitized with the calibrated camera. The scale was obtained by fitting a 10 mm sphere to the point cloud with a CAD procedure, computing the diameter and scaling the point cloud imposing a sphere diameter of 10 mm. Dimensional analysis was carried out by comparing the scaled point cloud to a datum 10 mm sphere with the CAD software. This procedure was repeated with the 20 and 32 mm tubes.
In order to also analyze freeform shapes, as a second case study, a tool insert was digitized. In this case the scale was imposed using an independent measure as the height of the calibrated 4 mm block shown in figure 5. In this second case the 3D point cloud obtained was compared to the point cloud obtained using the 3D scanner MiniConoscan 4000 equipped with a 50 mm HD lens. This scanner is based on conoscopic holography technology with x-y positioning and a 0.015 mm step on the x and y axes. The declared working range in the z axis of such a configuration is however very low: 2 mm. The comparison was carried out after an iterative closest point (ICP) procedure between each photogrammetric point cloud and the conoscopic holography point cloud constituted by 250 000 points approximately.

The 20 mm extension tube
The magnification ratio was experimentally evaluated as the diagonal size of the calibration pattern used divided by the diagonal size of the sensor, and was 1.86 for the 20 mm tube. The first 3D object digitized was the 10 mm sphere, which was textured with a powder spray.
16 thumbnails of the 40 images captured to reconstruct the 3D model are shown in figure 6 and the results of the 3D recostruction is reported in figure 7.
The number of images was set to 40, captured with a ring strategy, ideally dividing the 360 degrees of a complete  rotation into 9-degree angle steps and allowing a high number of frames that cover the whole object without any hidden zones [23]. These steps were manually imposed on the object, keeping the camera fixed since in DCRP it is not necessary to know the position of the camera to solve the measurement problem.
These data have been obtained with the aid of the software Geomagic Studio: after best-fit alignment of the Optimet scan data and Photogrammetric scan data, the former is called Reference and the latter Test. For each point of the Reference the algorithm finds the nearest point of the Test and computes the Euclidean Distance. Each point of the reference point cloud is associated with a distance and the distances are clustered into colored intervals according to the legend on the right side of each figure. The diagram associated with the legend expresses the number of points of the test object corresponding to each computed distance.
The absolute value of the average distance between Test and Reference was equal to 0.001 mm, the absolute maximum distance was equal to 0.001, and the standard deviation was equal to 0.008 mm.
As regards the tool insert, in figure 8, 15 of the 40 images used are shown.
In figure 9 the color map of the distances between photogrammetry and conoscopic holography is shown.
In this case the absolute average distance was smaller than 0.001 mm, the absolute maximum distance was 0.120 mm, and the standard deviation was 0.011 mm.

The 32 mm extension tube
As regards the 32 mm tube, the parameters input for the reconstruction software were the same as those for the 20 mm case. Magnification was experimentally evaluated at 2.05. The results are shown in figure 10, where the 3D reconstruction of the sphere is represented compared to the conoscopic holography point cloud.    The maximum distance and average distance between digitized points and the ideal 10 mm sphere were equal to 0.001 mm, the absolute value of the maximum distance was equal to 0.052 mm and the standard deviation was equal to 0.009 mm.
In figure 11, a 3D comparison between the photogrammetric point cloud and conoscopic holography is shown for the tool insert.
The average distance was smaller than 0.001 mm, but the average of the positive distances was 0.006 mm, the average of the negative distances was −0.006 mm, and the standard deviation was 0.008 mm, showing smaller deviations from conoscopic holography than the 20 mm case.

Discussion
The case studies were designed to evaluate the results of the calibration protocol on practical applications, choosing two different shapes: one sphere of known diameter and one insert for 3D turning machines whose dimensions were not known a priori and digitized with a commercial 3D scanning machine used as the gold standard. As regards the sphere, very similar results were achieved for all the configurations.
It is evident an improvement of accuracy with 32 mm as compared to 20 mm, with average differences from the gold standard which are smaller than 6 μm in absolute value. As a consequence the results are comparable with those of conoscopic holography in terms of accuracy, declared by the manufacturer to be equal to 2.5 μm. While the working range of this technology is 2 mm the photogrammetric technique has a theoretically unlimited working range since it depends on the number and orientation of images. Other very important advantages of DCRP are the low costs required and high transportability.
The very important result of the experimental activity is that it demonstrates how the conventional CRCM is valid for application with a Macro lens down to an AOV equal to 3.4°.

Conclusions
In this paper an experimental study on the validity of the conventional close-range calibration model based on a pinhole when applied to the 3D photogrammetric digitization of small features has been presented.
Calibration and DCRP were applied to magnifications of up to 2.05 and an AOV as narrow as 3.4°. The close-range calibration model proved to be valid down to this condition.
The highest magnification ratio required a photolithographic pattern of the calibration procedure in terms of real dot centres in the pattern, due to inaccuracies in the printed pattern; the pattern was printed on common paper for magnification ratios of up to 1.86. The photogrammetric approach showed an accuracy comparable to that of more expensive micro-scanning systems but with the intrinsic and very powerful advantage of a 3D digitizing system with a practically infinite depth of field. This feature is dependent on the number of images while commercially available systems have limited millimetre depths of view.
Further research must be carried out to focus on finding the limits of the photogrammetric and pinhole camera approach in micro 3D scanning using higher magnification ratios, more accurate manufacturing technologies for the patterns and eccentricity error compensation for calibration dots.