A NOVEL CAMERA CALIBRATION METHOD FOR FISHEYE LENSES USING LINE FEATURES

In this paper, a novel method for the fish-eye lens calibration is presented. The method required only a 2D calibration plane containing straight lines i.e., checker board pattern without a priori knowing the poses of camera with respect to the calibration plane. The image of a line obtained from fish-eye lenses is a conic section. The proposed calibration method uses raw edges, which are pixels of the image line segments, in stead of using curves obtained from fitting conic to image edges. Using raw edges is more flexible and reliable than using conic section because the result from conic fitting can be unstable. The camera model used in this work is radially symmetric model i.e., bivariate non-linear function. However, this approach can use other single view point camera models. The geometric constraint used for calibrating the camera is based on the coincidence between point and line on calibration plane. The performance of the proposed calibration algorithm was assessed using simulated and real data.


INTRODUCTION
Camera calibration is a fundamental procedure in photogrammetry and computer vision for image interpretation and quantitative image analysis.Particularly, the goal of geometric camera calibration is to estimate the geometric characteristic of a camera i.e., the interior orientation of the camera including the coordinates of the principal point and geometric distortion parameters.Knowing the distortion characteristic of a camera, the distortion on the image can be removed, which can make the quantitative image analysis more accurate.
Fisheye lenses have been widely used for measurement applications because of the high information content contained in the images.An advantage of using fisheye lenses is the large Field Of View (FOV) compared to the conventional camera because they are designed to cover the hemisphere field in front of the camera.Unfortunately, their drawback is the severe distortion on the image compared to the conventional cameras.Therefore, the accurate calibration of the camera is highly required if one have to use the fisheye lenses for recover metric information from the environment.
An approach for calibrating fisheye lenes is to model them as the conventional pinhole camera accompanied with the lens distortion model.However, this approach is just an approximation of the fisheye lens model because fishere lenses are designed to obey other projection models (Sturm et al., 2011).
In literature, there exists many works on modelling and calibrating fisheye lenses.Micusík and Padjla (Micusík and Pajdla, 2003) proposed an omnidirectional camera model and calibration algorithm.The projection model was a bivariate non-linear function.By linearizing the projection model, the calibration algorithm was developed based on the epipolar constraint between two views.Estimating the camera parameters was treated as a quadratic eigenvalue problem.An advantage of this work is that only point correspondences between images are required and no calibration structure is needed.
In (Scaramuzza et al., 2006), Scaramuzza et al. proposed a flexible technique for calibrating omnidirectional camera using calibration plane.They replaced the projection model used in (Micusík and Pajdla, 2003) with a polynomial function of the angle between the optical axis and incoming ray.Moreover, the number of images used in the calibration process is not limited to two images.A drawback of this method is the determination of the order of polynomial function which can cause the over-fitting problem.Kannala and Brant (Kannala and Brandt, 2006) also modelled the projection as a polynomial function.An advantage of this work is that radial and tangential distortions are included in the model.In stead of using checker board as the calibration pattern, they utilized the calibration plane with white circles on black background.The centroid of the circles were used as primary data for calibration process.
In (Ying et al., 2006), Ying et al. introduced a novel camera calibration method for fisheye lenses using straight lines.The concept of this work is based on back-projecting the image point into 3D camera coordinate system.The proposed objective function is the orthogonal distance between the back-projected image points and the projection of lines in object space.
Similar to (Ying et al., 2006), Hino et al. (Hino et al., 2009) also used straight lines for calibrating fisheye lenses.The concept of this work is that the images of 3D straight lines on the calibrated image must be straight lines not curves.The calibration process was then treated as a tting problem of the principal component on uncalibrated images.
In this paper, we deal with the fisheye lenses calibration.The primary features used in the calibration process are straight lines.Therefore, the checker board pattern can be used as the calibration structure.The proposed camera calibration method utilizes the model model proposed in (Micusík and Pajdla, 2003) because of minimal number of projection parameters.However, other camera models can be employed in this proposed camera calibration algorithm.The structure of the paper is the following.The camera model and calibration algorithm are described in Sections 2. and 3., respectively.The experimental results is shown in the Section 4. The conclusion of this work is presented in Section 5.

CAMERA MODEL
The concept of the proposed camera calibration method is based on minimizing error in object space in stead of image plane.Therefore, the suitable camera model is the back-projection model.In this work, we use the backward model of omnidirectional camera presented in (Micusík and Pajdla, 2003).This model is utilized because it required only 2 parameters to model the projection of the lenses.
In order to present the model used in the calibration algorithm, lets first discuss about the mapping from the image plane coordinate system to sensor plane coordinate system i.e., on CCD.A point on the image plane is represented by u = (u , v ) and on the sensor plane u = (u , v ).By assuming that the optical axis is perpendicular to the CCD, these two coordinate systems are related by affine transformation: where A ∈ R 2×2 is a 2 by 2 matrix embedding rotation, scale and skew and t ∈ R 2×1 a 2-vector representing the translation.The affine transformation is used to handle the digitization process and small axes misalignments.
Let a scene point X ∈ R 4 in homogeneous coordinate be projected onto the sensor plane at a point u .In (Micusík and Pajdla, 2003), the generic model of an omnidirectional camera is formulated as: where ∼ means equal up to scale and p is the direction vector of the back-projection ray emanating from the viewpoint (origin of the camera coordinate system).Function g : R 2 → R 3 is a non-linear function mapping the point u into the 3D camera coordinate system and P ∈ R 3×4 is a perspective projection matrix: The perspective projection matrix P contains the rotation R ∈ R 3×3 and translation T ∈ R 3×1 between camera and object space coordinate systems, which is, in this case, the coordinate system of the calibration plane.The concept of the generic camera model in ( 2) is illustrated in Figure 1.Particularly, it shows that the back-projection ray obtained from the non-linear function g is coincidence with the scene point X that is transformed into the camera coordinate system via the perspective projection matrix P.
Given the calibration parameter of a camera, the point u on the sensor plane can be back-projected into the 3D camera coordinate system: where f is rotationally symmetric with respect to the sensor axis (optical axis).This model is reasonable when the optical axis is perpendicular to the sensor plane.The choice of the function f in (4) depends on the lens construction (Kumler andBauer, 2000, Bakstein andPajdla, 2002).Let θ be the angle between the projection ray and the optical axis, see Figure 1.The function f is formulated as a function of θ and ρ such that f (u ) = ρ tan(θ) , see Figure 2. In (Micusík and Pajdla, 2003), the relation between θ and ρ was formulated as a function of two parameters: Figure 1: The back-projection of the point u via the non-linear function g is coincident with the scene point X in the camera coordinate system.where ρ = √ u 2 + v 2 .Namely, ρ is the metric distance from the origin of the sensor plane coordinate system to the point u .By substituting (1) and ( 5) into (4), the function g in (4) can be rewritten as: This means that, given pre-calibrated image, an image point can be back-projected into 3D space.The whole process of backprojecting a point u on the image plane is illustrated in Figure 3.

CALIBRATION ALGORITHM
The proposed camera calibration method consists of four steps.
The process begins with the initialization of the interior orientation parameters which are the used for initialize the exterior orientation parameters in the next step.Once both initial parameters are obtained, they will be refined by minimizing the geometric constraint.In this Section, we start with the geometric constraint used in the proposed camera calibration algorithm.The initial-Figure 3: The back-projection of a point on image plane into 3D camera coordinate system.An image point is mapped onto the sensor plane via affine transformation.Given pre-calibrated image, the point on the sensor plane can be back-projected into the 3D camera coordinate system using the non-linear function g.
ization of the interior and exterior orientation parameters are then discussed.The non-linear optimization will be presented at the end of this section.

Geometric constraint
The formulation of the geometric constraint used in this work is based on the preservation of coincidence between point and line.
Particularly, the coincidence on the plane i.e., calibration plane is used.According the coincidence preservation property of the central projection, if the point and line are coincident on the calibration plane, there images are still coincident.Conversely, the back-projection of the point on the image line onto the calibration plane is still coincident with the line on the calibration plane.
To illustrate this concept, let see Figure 4.An image point is back-projected via the backward projection model of the camera.The projection ray with direction p is intersect with the calibration plane at the point H −1 on the calibration plane.If the image point is on the image of the line on the calibration plane, the intersection point must be coincident with the line on the calibration plane.This concept will be used in the initialization of the exterior orientation parameter and the non-linear optimization for estimating both interior and exterior orientation parameters of the camera.
Figure 4: The geometric constraint used in this proposed calibration algorithm.The constraint is based on the coincident relation between the back-projected image point and line in object space i.e., line on calibration plane.

Affine transformation parameters
The initial guess for affine transformation parameters can be obtained using the nominal parameters from camera manufacturers i.e., the size of the CCD grid.The matrix A of the affine transformation (1) can be represented as: The skew parameter is skipped in this affine transformation.The initial estimate of the parameters mx and my can be the size of the CCD grid e.g., 5 µm.The initial solution for the translation part t can be obtained using the center of the image: where (u0, v0) is the center of the image.

Camera distortion parameters
In (Micusík and Pajdla, 2003), the initial guess for the parameter b is 0. Hence, the initial solution for the parameter a can then be obtained using (5).Given b = 0, the equation ( 5) becomes a = θ ρ .The initial solution for both a and b can then be computed: where θm is the maximum view angle which can be computed from the field of view of the lenses and R the radius of the view field circle.The radius R can be obtained by fitting a circle to the boundary of the fisheye image.The radius of the fitted circle R on the image plane is then scaled to be physical quantity i.e., R = mxR .
For lenses with small field of view i.e., conventional pinhole camera, fitting a circle to the boundary of the image can be impossible.To obtain the initial guess for the parameter a, we use the equidistance projection model (Kannala and Brandt, 2006) ρ = fcθ, where fc is the nominal focal length.Therefore, the initial guesses for the parameters a and b of conventional camera are then:

Exterior orientation parameter initialization
Given the initial interior orientation parameters, a point on the image plane can be back-projected into the 3D camera coordinate system.Namely, the image point is transformed to be the back-projection ray by the backward camera model ( 6).Since the camera model is central projection, there exists a planar homography between the back-projected image point and the calibration plane such that: where ũ is the point on the calibration plane, p the back-projection of the corresponding image point and H the homography mapping from the calibration plane to camera coordinate system.
To estimate the homographhy, in this paper, the coincidence between a 2D line on calibration plane and an image point transformed onto the calibration plane via homography is utilized.Let l be the 2D line on the image plane.The coincidence between the line l and the image point that is mapped onto the calibration plane can then be formulated as: where ⊗ is Kronecker product and vec() the vec operator.The non-trivial solution of the above homogeneous system of linear equation can be linearly solved by the Singular Value Decomposition(SVD) technique.Note that the estimated one is the inverse of the homography.
Given the inverse of the homography, initial guest for the exterior orientation parameters can be obtained.Let the homography between the camera coordinate system and calibration plane be: where r i is the i th column of the rotation matrix R. The rotation matrix and translation vector can be estimated: where λ = sign(H 3,3 )/ h 1 , h i is the i th column of the homography H and H 3,3 the element at the third row and third column of the homography.Unfortunately, the obtained rotation matrix is not orthogonal because of estimation error.That is, it does not satisfy the rotation matrix properties which are R = R −1 and det(R) = 1.To fix this problem, SVD of the rotation matrix is used.Let the SVD of the rotation matrix R be R = USV .The condition of rotation matrix can be resolved by modifying the singular values (Gramkow, 2001): (15)

non-linear refinement
Once both interior and exterior parameters are initialized.They are then refined simultaneously via the bundle adjustment process.With the image noise and quantization error, the coincidence cannot be obtained.In the non-liner optimization process, we hypothesize that the orthogonal distance between the line on the calibration plane and the image point that is transformed onto the calibration plane is minimized with optimal interior and exterior orientation parameters.The orthogonal distance is formulated as: where • 2 is the 2-norm of a vector and the selection matrices D1 and D2 are: Note that the objective function or estimation model is the total estimation error.Both interior and exterior orientation parameters can then be obtained by minimizing the objective function in least-square sense.In other words, the sum of squared orthogonal distance between the lines on calibration plane and the image points that are mapped onto the calibration plane is minimized.
The algorithm used for solving the minimization problem is the Levenberg-Marquardt algorithm.The interior orientation parameters are parameterized as scalars and the translations of camera are represented as vectors.For the rotation matrix representation, it is more complicated because there exists various representation in literature.Moreover, some representations require to impose constrains in the estimation model in order to maintain the properties of the rotation matrix.To waive the constraint in the optimization process, the local perturbation model of the rotation matrix is used as mentioned by Triggs et al. (Triggs et al., 2000).

EXPERIMENTS
To test the performance of the proposed camera calibration method, both simulated and real data were used.To evaluate the performance of the proposed camera calibration method, both the reprojection and back-projection errors are reported.

Simulated data
The first experiment for assessing the performance of the proposed calibration algorithm is the experiment with simulated data.
The CCD size of the simulated fisheye lenses was 1 µm.The image resolution [pixel] is 640 × 720.The parameter a and b was setted to be 3.5 and −0.2, respectively.The calibration plane had 14 lines.The camera poses were randomly generated such that the calibration plane was always at the front of camera.Once the lines on the calibration plane were projected onto the images using the predefined parameters, 10 points on each projected line were used for calibration.
To evaluate the performance of the proposed method with respect to noise, Gaussian noise with zero-mean and standard deviation σ is added to to image points.The standard deviation varies from 0.5 to 2 pixels.For each noise level, 200 independent trials were performed and averaged error was computed.
In order to quantitatively test the performance of the proposed methods, the estimated parameters were compared with the ground truth using the averaged re-projection error: where g is the estimated camera parameters i.e., interior orientation parameters and g is the ground truth.The plot of reprojection error as a function of noise level (standard deviation) is shown in Figure 5.It can be observed that the reprojection error increases linearly.
The performance of the proposed camera calibration can also be reported using object space error.There are two types of object space error.The first one is the line based object space error which is the error used as the objective function ( 16).The second one is the point based object space error point based object space error = dEuclidean( u, where u is the point on calibration plane.dEuclidean is the Euclidean distance i.e., the distance between the point u and the non-homogeneous coordinate of the back-projected image point H −1 p.The back-projection errors ( 16) and ( 19) from the experiment are reported in Figures 6 and 7, respectively.It can be observed that both back-projection errors increase linearly as a function of noise level σ.

Real dataset
The proposed camera calibration method was also tested with real dataset.The cameras used in the experiment were Ladybug 3 and Point Grey's Firefly camera equipped with Fujinon YV fisheye lens.For the Ladybug 3, its pixel size is 4.4 µm and focal length is 3.3 mm.The Field Of View (FOV) of the Fujinon UV is about 180 • and the pixel size of the Point Grey's Firefly camera is 6 µm.
The initial solution for the interior orientation parameters were initialized by using information from manufacturer including CCD size and nominal focal length.11 images were taken from Ladybug 3 camera and 28 images from Point Grey's Firefly camera.Points on image line were sampled and used for calibration.The homographies between the images and calibration plane were then initialized using the back-projected points on image lines.
To evaluate the performance of the proposed camera calibration method using real data, the point-based back-projection error in (19) was used.Since equation ( 19) requires the correspondence between the points on the calibration plane and their image points, the corners of squares on the calibration plane were then used for error computation.In Tables 1 and 2, the point-based backprojection errors are reported.The errors reported in those Tables are both the error before and after non-liner optimization.It can be observed that the errors were significantly reduced after nonlinear optimization.2.14 0.25 In order to show the qualitative performance of the proposed camera calibration method, the corner points on the calibration pattern were re-projected onto the image planes.In Figures 8  and 9, the re-projection of the corner points onto Point Grey's Firefly and Ladybug 3 cameras are illustrated.Moreover, the reprojections of the lines on calibration plane onto the image from Ladybug 3 camera is illustrated in Figure 10.

CONCLUSIONS
In this paper, we present a novel model for calibrating fisheye or wide angle lenses.The feature used in this method is straight line.The geometric constraint used in this paper is based on the coincidence between point and line on calibration plane.In stead of using polynomial function for modelling the lens distortion, the proposed calibration method used bivariate function model.Therefore, the proposed calibration model does not have the problem of selecting the suitable order of polynomial function.The performance of the proposed method was evaluated on both synthetic and real data.

Figure 2 :
Figure 2: The diagram showing the relation between the backprojected image point and the function f .

Figure 5 :
Figure 5: The reprojection error versus the random noised added to the image points.The reprojection error was computed using (18).The red shaded area shows the standard deviation of reprojection error.

Figure 6 :
Figure 6: The line-based back-projection error versus the random noised added to the image points.The line-based back-projection error was computed using (16).The shaded area shows the standard deviation of the line-based back-projection error.

Figure 7 :
Figure 7: The point-based back-projection error versus the random noised added to the image points.The point-based backprojection error was computed using (19).The shaded area shows the standard deviation of the point-based back-projection error.

Figure 8 :
Figure 8: The re-projection of the corner points on calibration pattern on Point Grey's Firefly camera equipped with Fujinon YV fisheye lenses.The yellow circles are the detected corner points while the red plus sign the re-projected corner points.

Figure 9 :
Figure 9: The re-projection of the corner points on calibration pattern on Ladybug 3 camera.The yellow circles are the detected corner points while the red plus sign the re-projected corner points.

Figure 10 :
Figure 10: The re-projections of the lines on calibration plane onto the image from Ladybug 3 camera.The re-projected lines are drawn with red color.

Table 1 :
The point-based back-projection error (19) from the experiment with Point Grey's Firefly camera equipped with Fujinon YV fisheye lenses.

Table 2 :
The point-based back-projection error (19) from the experiment with Ladybug 3 camera.