Direct fitting of center collinear circles for equidistant fish-eye camera calibration

Abstract. An equidistant fish-eye camera can be calibrated using a single image of two sets of parallel lines in the scene, which is very convenient for practical applications. The core of the calibration procedure is the fitting of center collinear circles that intersect at two vanishing points. A fast and accurate fitting method that can be used for equidistant fish-eye camera calibration is presented. First, the fitting problem is formulated as a nonlinear least square problem and solved using Levenberg-Marquardt optimization. The objective function is also derived, as well as the Jacobian required by the Levenberg-Marquardt algorithm. It is also demonstrated how to determine the initial estimations. Experimental results on synthetic data have demonstrated the superiority of our method to two existing approaches in terms of speed and accuracy. Results on real data have also demonstrated its effectiveness for fish-eye camera calibration.


Introduction
Because of its large field of view (FOV), the fish-eye camera has been widely used in applications such as robot navigation, visual surveillance and three-dimensional (3-D) reconstruction. 1 However, fish-eye lenses often introduce a large amount of radial distortion, which causes straight lines in the scene mapping to conics in the image.As a result, a calibration procedure is often required for distortion correction. 2 The calibration procedure typically consists of two components, model selection and parameter estimation.In the last decades a number of models have been developed for fish-eye lenses, such as the polynomial model, 3 the division model, 4 the rational function model, 5 and so on.In order to estimate the model parameters, various methods using points, lines and conics have also been proposed in Refs.6 and 7.
The ideal fish-eye lenses are constructed with the aim of complying with the equidistant projection function. 8,9It has been demonstrated that under this model, parallel lines are projected onto circular arcs and all the circles intersect at two vanishing points, thus the centers of the circles are collinear. 10This property is of great importance because it allows fish-eye camera to be calibrated using single image of two sets of parallel lines in the scene, which is very convenient for practical applications.As a result, fitting of center collinear circles lies at the core of fish-eye camera calibration.
Based on the center colinearity property, Geyer et al. 11 proposed a two-step fitting approach.They first fitted the circles separately and then refined the centers of the circles using the colinearity constraint.However, its accuracy is limited due to the fact that the estimated center collinear circles do not necessarily intersect at two points.Recently, Hughes et al. 10 proposed an iterative approach to solve the fitting problem.Instead of fitting the circles separately, they utilized the available data points to fit all the circles simultaneously.On each iteration, they fixed one vanishing point and updated the other one, and enforced all the circles to intersect at the two vanishing points.Obviously this approach is more robust against noise thus can greatly improve the accuracy.
In this paper we propose a direct approach for fitting of center collinear circles.First we build a coordinate system for the circles and give the objective function for nonlinear optimization.Then we use Levenberg-Marquardt (LM) algorithm 12 to solve the optimization problem.The main advantage of our approach is that two vanishing points are updated simultaneously.Experimental results show that the accuracy of our approach is the same as reported earlier, 10 while its speed is much faster.
The remainder of this paper is organized as follows.Section 2 briefly describes the equidistant fish-eye projection.Section 3 presents three fitting approaches, including two existing approaches and the proposed direct approach.Section 4 gives our experimental results on both synthetic and real data, and Sec. 5 offers our conclusion.
2 Equidistant Fish-Eye Projection Ideal fish-eye cameras are manufactured to follow the equidistance mapping function such that the distance between a projected point and the optical center of the image is proportional to the incident angle of the projected ray, scaled only by the equidistance parameter f, as described by the projection equation: 13,14 where r d is the fish-eye radial distance of a projected point from the center, and θ is the incident angle of a ray from the 3-D point being projected to the image plane. 15Figure 1 illustrates the equidistant projection for a simple model of an equidistant camera system.Due to the radial distortion introduced by fish-eye lenses, straight lines in the scene will map to conics in the image.It has been proved that under this model, parallel lines are projected onto circular arcs and all the circles intersect at two vanishing points, 10 as shown in Fig. 2. It has also been demonstrated that if the parameters of two sets of circles are known, all the intrinsic parameters required for fish-eye camera calibration can be determined. 10As a result, estimating the parameters of the circles lies at the core of equidistant fish-eye camera calibration.In the next section, we will present three fitting approaches for parameters estimation, including two existing approaches and the proposed direct approach.
3 Fitting of Center Collinear Circles Fitting of single circle using a set of data points is a wellinvestigated nonlinear least square problem. 16However, little attention has been paid to multiple circles fitting.In this paper, we will focus on the fitting of center collinear circles, i.e., a set of circles whose centers are collinear, as illustrated in Fig. 3.The most important property of center collinear circles is that all of them intersect at two points.Thus fitting of the circles is always coupled with the estimation of the positions of the two points.In this section, we will first briefly describe two existing methods in the literature, and then introduce the proposed fitting approach.

Two-Step Approach
Geyer et al. 11 proposed a two-step approach to estimate the parameters of center collinear circles for catadioptric camera calibration.The approach is briefly described as follows.
1. Separating the data points into several sets of points.
Fitting one circle to each set of points.2. Fitting a line to the centers of all the circles.Updating the centers by projecting them onto this line.
This approach first estimates the parameters of each circle separately, and then enforces the centers of the circles to be collinear.Note that it does not update the radii of the circles, and therefore, it cannot guarantee that all the circles intersect at two points.

Iterative Approach
Recently, Hughes et al. proposed an iterative approach to alternately update one point by fixing the other. 10This approach consists of the following steps.
1. Separating the data points into several sets of points.
Fitting one circle to each set of points.2. Calculating the intersect points of the circles and determining the initial positions of the two points.3. Fixing one point and updating the other using LM optimization.4. Repeating step (3) until convergence is observed or the maximum number of iterations is reached.
It is obvious that when the algorithm terminates, we could obtain the positions of the two points as well as the parameters of the circles.Note that this approach enforces the circles to intersect at two points, thus it is expected to be more accurate than the two-step approach.However, since each time the approach updates only one point, it would take a number of iterations for convergence.

Direct Approach
In this subsection, we present a direct approach for fitting of center collinear circles.We first build a coordinate system to facilitate derivations, and then give the equations of the center collinear circles using the vanishing points constraint.Next we propose to solve the fitting problem using LM Fig. 2 Projection of parallel lines in an equidistant fish-eye camera system.
Fig. 3 The coordinate system for the proposed direct approach.
optimization, and give the objective function as well as its Jacobian.Finally we discuss some issues related to its implementation.Assume the center collinear circles intersect at two vanishing points v 1 and v 2 .After translation T ¼ Tðx; yÞ and rotation R ¼ RðθÞ, we define the midpoint of line segment v 1 v 2 as the origin, and the line determined by v 1 and v 2 as the X-axis, as shown in Fig. 3.In this coordinate system, the center of the circle C i is ð0; b i Þ, and the coordinates of v 1 and v 2 are ð−a; 0Þ and ða; 0Þ, respectively.The equation of circle C i can be expressed as where the radius In order to fit N circles, we should estimate After translation and rotation, a data point p is transformed from ðm; nÞ to ðm 0 ; n 0 Þ, which satisfies If there are N i data points p i;1 ; : : : ; p i;N i belonging to circle C i , and the distance between the k-th transformed point p 0 i;k and its corresponding center is then the fitting error of the points belonging to circle C i is given by and the fitting error of all the data points is Substituting Eqs. ( 4) and ( 6) into Eq.( 7), we get Thus the fitting problem is formulated as a nonlinear least square problem whose objective is finding the minima of Eq. ( 8).LM algorithm 12 is very efficient for the above optimization problem.In order to apply LM algorithm, we should derive the first partial derivative of the objective function, i.e., the Jacobian. 17After some mathematical manipulation, we get the Jacobian Setting the initial estimation properly is very crucial for LM optimization.Since the fitting error increases with the radius of circle, we propose to first fit each circle separately, and then use the intersect points of two smallest circles as the initial estimation of the vanishing points.The initial values used for LM optimization then can be determined by the two vanishing points and the parameters of the circles.After convergence of LM algorithm, we can compute the parameters of the circles.The complete approach is given as follows: 1. Separating the data points into several sets of points.
Fitting one circle to each set of points.2. Finding two smallest circles and calculating the intersect points of them.3. Calculating the initial estimation of the parameters x, y, θ, a, b 1 ; • • • ; b N .4. Updating the parameters using LM optimization until convergence.5. Calculating the parameters of the circles using the optimization result.
Note that our method is also an iterative approach because LM algorithm is used for parameter estimation and LM algorithm often requires a number of iterations.However, in the iterative approach presented in Sec.3.2, the vanishing points are estimated sequentially, while in our approach they are estimated simultaneously.Thus our approach is expected to be faster than the iterative approach.In addition, our approach also enforces the circles to intersect at two points, thus it is expected to be more accurate than the two-step approach.

Experimental Results
We have evaluated our method on both synthetic and real data and have compared it with those proposed. 10,11We use speed and accuracy as performance indicators for comparison.The platform is a PC with Intel CPU i5-2400 3.10 GHz and 8 G RAM.The software environment is Windows 7 Ultimate and Visual Studio 2005.All the testing programs are written in C++ language.

Synthetic Data
In order to stimulate the fish-eye camera calibration, we generate eight center collinear circles (denoted as C1 to C8), and use only arcs of them for parameter estimation, as shown in Fig. 4. The resolution of the synthetic image is 640 × 480, and the parameters of the circles are listed in Table 1.
On each arc we randomly choose 100 points.Gaussian noise with zero-mean and σ standard deviation is added to the points.The noise levels σ are 0, 1, 2, 3, 4, 5, respectively.For each noise level, we perform 100 independent trials, and the mean values and standard deviations of these recovered parameters are computed over each run.The errors compared to the ground truth are defined as where C x ; C y and r are the ground truth, C Ã x;t , C Ã y;t , and r Ã t are the estimated parameters of t-th trail, and T is the number of trails.Although the fitting error increases with the noise level, the comparison results of the three approaches are similar for different noise levels.For the sake of brevity, we only provide the fitting results when σ ¼ 3, as shown in Tables 2-4.The three fitting approaches described in Sec. 3 are indicated by subscripts t, i and d, respectively.
From Tables 2-4, we can see that the fitting results are almost the same for the iterative and the direct approaches, and both of them are much more robust and accurate than the two-step approach.The error and standard derivation of the two-step approach are around 5 to 10 times larger than those of the other two approaches.Thus we can conclude that in terms of fitting accuracy both the iterative and the direct approaches are superior to the two-step approach.
Another observation from Tables 2-4 is that for all the three approaches, the error and standard derivation increase with the radius of the circle, which is consistent with previous studies. 18,19The reason is that for a larger circle, the arc in the image corresponds to a smaller angle.As a result, the accuracy decreases due to a larger amount of occlusion of the circle.On the other hand, because the iterative and the direct approaches use all the data points for parameters estimation, the adverse effect of noise and occlusion can be greatly reduced by the vanishing points constraint.Hence these two approaches are more robust and accurate than the two-step approach.
The time requirements of the three approaches are reported in Table 5.It can be observed that the two-step approach is the fastest one, which only requires less than 16 ms on average when σ ¼ 3. The other two approaches are much slower.The time required by the iterative and the direct approaches are about 700 and 30 times more than that of the two-step approach, respectively.It is reasonable because nonlinear optimization is adopted by the direct approach using all the data points and the center collinear constraint, which costs much more time than just estimating the parameters of each circle separately.The iterative approach often requires several tens of iterations for convergence (as listed in Table 6), and there is also a nonlinear optimization step on each iteration.As a result, the iterative approach is the most time-consuming of all.Our method is more than 20 times faster than the iterative approach, and its average running time is less than half a second, which makes it very promising for practical calibration procedures.

Real Data
In this expirment, we take an image using a fish-eye camera with FOV about 170 deg.A chess board is placed in front of the camera for calibration.The size of each block in the chess board is 15 × 15 mm 2 , and the resolution of the image is  640 × 480, as depicted in Fig. 5. Curve extracting is accomplished by a software package developed by our lab.On both horizontal and vertical directions, only six arcs with the largest lengths are selected for parameters estimation.After the fitting of center collinear circles, we can obtain the positions of horizontal vanishing points v x1 and v x2 and the vertical vanishing points v y1 and v y2 .If the line joins v x1 and v x2 is denoted as l x , and that joins v y1 and v y2 is denoted as l y , then the optical center can be determined as the intersect point of l x and l y , and the focal lengths on horizontal and vertical directions are given by and Then we can use Eq. ( 1) for distortion correction.The details about distortion correction are available. 15ue to lack of the ground truth of parameters of the fisheye camera, we just check the undistorted image to evaluate the calibration results and the proposed approach.The undistorted image is shown in Fig. 6.As can be seen in this figure, straight lines in the scene now map to straight lines in the image.We then extract the 8 × 8 corners of the chess board 20 and compute the reconstruction error.The average difference between the position of each corner and their true position is 1.631 pixel.Because the resolution of each block in the undistorted image is 63.267 × 63.267, the reconstruction error can be computed as 1.631∕63.267× 15 ¼ 0.387 mm.Thus we conclude that our method is effective for fish-eye camera calibration.

Conclusions
In this paper we propose a novel method for fitting of center collinear circles for fish-eye camera calibration.We formulate the fitting problem as a nonlinear least square problem and solve it using LM optimization.Experimental results on synthetic data show that the proposed method is much more accurate than the two-step approach, and it requires much less time than the iterative approach while keeping the fitting Fig. 5 An image captured using a fish-eye camera.performance unspoiled.Results on real data demonstrate its effectiveness for fish-eye camera calibration.Hence we can conclude that the proposed method is very promising for practical applications.

Table 1
Parameters of the synthetic circles.) is the center of the circle.The center of the image (320, 240) is regarded as the origin.r is the radius of the circle.

Table 2
Fitting results of C x when σ ¼ 3.

Table 3
Fitting results of C y when σ ¼ 3.

Table 4
Fitting results of r when σ ¼ 3.

Table 5
The average time (ms) required by the three approaches.Two-step approach 8.27 11.55 13.01 14.35 14.8815.73

Table 6
The average iterations required by the iterative approach for convergence.