Structural-Information-based Robust Corner Point Extraction for Camera Calibration under Lens Distortions and Compression Artifacts

Previous camera calibration methods often use a checkerboard to capture images and estimate the camera parameters from the correspondences between images and the checkerboard. The corner points in the checkerboard images are used as useful features for correspondence matching. Therefore, it is essential to precisely find the corner points in the checkerboard images. In many previous works, the corner points are extracted assuming that the checkerboard images are not distorted by its lens. Instead, image blurring and Gaussian noise on the images are usually considered, but other cases are not dealt with. However, the captured checkerboard images are often corrupted by lens distortions and compression artifacts, which leads to performance degradation of corner point extraction. Moreover, the corner points are extracted individually in the previous methods without considering their geometric relations. To better handle the corner point extraction problem under lens distortions, in our corner point extraction optimization, the distorted locations of the pixels on checkerboard images are corrected with the camera parameters, and the structural constraints for checkerboard image grids are then applied under the line-to-line mapping. Also, to robustly find the blurred edges between corner points due to JPEG compression, an edge surface model is newly proposed that models the transitions with over- and under-shoots around the blurred edges. Extensive experimental results show that our method significantly outperforms the state-of-the-art method with average 88.3% and 54.3% reduction in RMSE for corner point reprojection and camera parameter estimation, respectively under compression and lens distortions for synthetic and real data.


I. INTRODUCTION
Calibration is the task of estimating the projection parameters of a camera. Accurate camera parameter estimation is essential in the fields where 3D information is estimated from images, such as 3D reconstruction, virtual reality, and autonomous driving. For accurate results in the fields, precise camera calibration is required. So, many studies have been made for the camera calibration [1] - [4].
The camera calibration techniques generally relate known target information to the camera parameters to find 2D-to-3D mapping which is a common issue in the fields of computer vision. Two-dimensional boards are conventional tools for the camera calibration. The 2D boards are used not only to calibrate optical cameras but also to calibrate Lidar cameras which obtain depth information [5], [6]. The pattern on the boards usually consists only of black and white colors. Feature points are easily extracted on boundaries of black and white areas from the board images. By matching the feature points on the images to the board in the real world, the camera parameters are estimated [9]. There are various patterns for the camera calibration such as a circular pattern, a checkerboard pattern, a deltille [24] pattern and a timecoded pattern [12]. Among them, the checkerboard pattern is the most widely used. The corner points of the checkerboard VOLUME 4, 2016 images are usually extracted for the feature points. There are various methods of extracting the corner points for the camera calibration [19] - [32].
For an accurate camera calibration result, lens distortion of the camera should be taken into account. The lens distortion is one of the optical aberrations by the camera lens [14], [15]. As a result of this distortion phenomenon, the projection by the camera is not a line-to-line mapping, but a line-to-curve mapping. The complex non-linear mapping does not occur in a distortion-free camera. There are difficulties in producing a completely distortion-free lens. To reduce the lens distortion, a compound lens may be designed or an aspherical lens (whose surface profile is not a portions of a sphere) [16] are required. Placing those lenses inside a camera often requires a large space to increase the volume of the camera. Therefore, the lens distortion is inevitable for small-sized cameras without such compound lenses or aspherical lenses. Also, since the degree of distortion depends on the focal length changes [17], it is very difficult to design a distortionfree lens for a wide range of the changes in focal length. Therefore, most cameras often suffer from the lens distortion.
Despite many efforts, the studies on finding the corner points have mostly been made for images without considering compression distortion. JPEG [11] is widely used for digital cameras, which relies on 8×8 Discrete Cosine Transform (DCT) and quantization on the DCT coefficients. The quantization causes distortion which leads to over-and under-shoot (called Gibbs phenomenon) along the edges of the checkerboard image. This makes it difficult to find the corner points of the checkerboard image precisely, thus leading to unreliable camera calibration.
In this paper, we propose a novel and elaborate corner point extraction method that incorporates the structural information under both lens distortion and compression artifacts, which leads to outperforming the previous corner point extraction methods with significant margins.

II. RELATED WORK
Harris corner detector method [19] is the most widely known method where the gradient values of the pixels are used to determine whether the pixels belong to edges, corners, or flat areas. However, it did not provide subpixel-level coordinates of the corner points. So, some methods extended the Harris corner detector method to estimate the subpixel-level corner points [20], [21]. Harris detector finds the corner points integer-pixel levels where the patches with the corner points tend to have large eigenvalues of Hessian matrices of their pixels. Han et al. [21] extended Harris detector to find subpixel level corner points by fitting Gaussian functions centered on the estimated integer-level corner points. Sánchez et al. [20] used a secondary polynomial function instead of Gaussian functions to find sub-pixel level corner points.
Geiger's method [22] [23] has been widely adopted in corner point extraction tools such as those in OpenCV T M [7] and MATLAB T M [8]. By using the fact that the difference vectors of two pixels on the edges are perpendicular to the gradients of the edges, the corner points by Geiger's method are determined as the points that minimize the square sum of the inner products of the difference vectors between their neighbor pixels and corner point candidates and the gradients at the neighbor pixels. Although Geiger's method is simple and fast for corner point extraction, the gradient values are sensitive to noise.
Saddle points in checkerboard images can be extracted as the corner points [24], [31], [32]. For the saddle points, the pixels near the corner points are fitted to a polynomial function. The polynomial surface fitting is suitable only for very small areas near the corner points, not in the region where the brightness values are steady. That is, the polynomial fitting error becomes dramatically larger as the size of the fitting surface area increases. However, with smaller fitting surface areas, the surface fitting becomes sensitive to the noise of the pixels in the areas.
Zhao et al. [26] utilized the property that the distributions of brightness values around the corner points are symmetrical. In Zhao's method, the weights of pixels are set based on how much symmetrical the distribution of their neighboring pixels is. The weighted averages of the pixels are the corner points in Zhao's method. The method works well when the distributions of the weights are of very sharp shapes centered at the corner points, so the weights of pixels near the corner points often have larger values, which makes it vulnerable to blurring near the corner points.
Duda et al. [28] utilized the property of lines through the corner points. In the method, the line segments of an equal length centered at a pixel location in a checkerboard image are set, and the maximum and the minimum among the sums of pixel values along the line segments are found. The difference between the two extreme values is maximized when the pixel passed by the line segments is a corner point because the line segment through the only bright areas and the line segment through the only dark areas meet at the corner point. A difference map of the two extreme values at neighboring pixel locations is near the corner point. The local maximum of the map is calculated as a corner point in subpixel-level via Gaussian peak fitting. When generating the difference map near the corner point, the pixels in the flat area away from the corner point are mostly used, rather than those near the corner point. So, the accuracy of the result may be degraded by using the pixels away from the corner point. It can also be a disadvantage that the map is composed of the maximum difference values of two extreme values which are sensitive to disturbances in pixel values.
Yang et al. [25] fitted the edges of the checkerboard images to a surface model and extracted the intersections of the edge surfaces as the corner points. The edge fitting was performed under the assumption that the edges were straight. The performance of the method might be degraded when the edges are curved by lens distortions.
If the corner points are obtained using structural information from the edges, they can be more robustly estimated. There are some methods to utilize the edges for the camera calibration. Liu et al. [30] proposed a corner point extraction algorithm that utilized the curvature of curved edges by extending Geiger's method. In Chen's method [27], the curved edges in checkerboard images distorted by the lens are extracted, and the resulting corner points are determined as the intersection points of the extracted curved edges. Zhou et al. [29] proposed a method that extracted feature points on the edges, rather than the corner points, for camera calibration. In Zhou's method, the feature points on the edges are extracted from local regions near the edges. However, the methods [30], [27], [29] handle the edges independently and do not consider the structural information of the grid formed by the edges, especially for corner point extraction.
The previous methods have extracted the corner points on the checkerboard images individually or have utilized the relations between the corner points and the edges partially. However, for better camera calibration, the corner points in the checkerboard images should be extracted more accurately. So, the structural property of the corner points should be considered. Unlike the previous methods, we consider the structural information such as the grid structure formed by the edges to robustly estimate corner points even under the lens distortions as well as compress distortions.

III. SUMMARY
In this paper, our contributions for corner point extraction are summarized as follows: • We firstly propose a corner point extraction method that utilizes the structural information with the geometric property of the edges formed by the structures of the projected grids in the checkerboard images. The corner points on the checkerboard images should be intersection points of the edges, and the edges should meet at vanishing points. The cross-ratios between each checkerboard image and the checkerboard should be preserved after camera projection [10]. When extracting the corner points and the edges in the checkerboard images, these conditions must be satisfied. So, all the corner points and edges in a single checkerboard image should be expressed by parameters of the camera projection, rather than being treated independently.
• For corner point extraction, the lens distortions are taken into account in conjunction with the structural constraints. Camera projection under the lens distortions results in a line-to-curve mapping, which causes the corner point extraction to be more complicated. With estimated parameters for the lens distortions, we correct the distorted checkerboard images and apply the structural constraints under the line-to-line mapping. We estimate the camera projection parameters which determine the edges and the corner points, and the distortion parameters to make the line-to-line mapping work well.
• We propose a novel edge surface model to robustly find the corner points of the edges corrupted by JPEG compression. Since, by the JPEG compression, high-frequency components of the checkerboard images are discarded due to quantization of their DCT coefficients, the corner points and the edges of the checkerboard images are damaged due to the blurriness (transition) with over/under-shoot along the edges, called Gibbs phenomenon. Our edge surface model takes into account Gibbs phenomenon in precisely finding the edges, thus leading to accurate estimation of corner points.

IV. CAMERA MODEL
We used a camera model with the lens distortion. The pin hole model [10] is described as follows: K is an internal camera calibration matrix with a focal length f , the numbers of pixels per unit distance in image coordinate (s x , s y ), a principal point (p x , p y ), and a skew term α. R and T are external parameters which are a rotation matrix and a translation vector, respectively. (X, Y, Z) is the location of a 3D point, and (x, y) is the location of a 2D point projected from the 3D point to an image plane. Lens distortion is a kind of optical aberration, and a straight object is bent in an image due to the distortion. There are two main distortions: radial and tangential distortions [14], [15]. The radial distortion is caused by the refraction of the lens. The tangential distortion occurs when the image plane and the lens are not in parallel. A camera model with the lens distortions [33] is given as: where r 2 = x 2 + y 2 and ( x, y) is a coordinate on a 2D plane in the 3D real world, which is projected from a 3D point via x y 1 is a coordinate on a distorted 2D plane in the 3D real world by the lens distortion with a set of distortion coefficients D = {k 1 , k 2 , t 1 , t 2 } where k 1 , k 2 indicate the coefficients of radial distortion, and t 1 , t 2 are the coefficients of tangential distortion. Finally the coordinate (x d , y d ) in a distorted image is defined by transforming ( x d , y d ) as: We also consider a distortion model for fish-eye lens cameras which are designed to create ultra-wide-angle views [18]. The fish-eye lens distortion model [7] is described as: where r 2 =x 2 +ŷ 2 and (x,ŷ) is a coordinate on a 2D plane in the 3D real world, which is projected from a 3D point via coordinate on a distorted 2D plane in the 3D real world by the fish-eye lens distortion with a set of distortion coefficients Fig. 1 shows the block diagram of our corner point extraction-based camera calibration. For each checkerboard image, its corner points are extracted via the corner point extraction block that utilizes the structural information of the checkerboard grid under the lens and compression distortions. Based on the estimated corner points of all the checkerboard images, the camera calibration is performed via the Camera Parameter Estimation block using Zhang's method [9]. In Zhang's method, the corner points on a checkerboard are projected to the checkerboard images by the estimated camera parameters. The camera parameters are found in the sense that a cost function is minimized. The cost function is defined as the square sum of the distances between the estimated corner points and the projected points. Next, we explain our proposed edge surface model under the JPEG compression distortion, the structural information, and the lens distortion for precise corner point extraction.

A. AN EDGE SURFACE MODEL
The corner points are the most basic features of checkerboard images for camera calibration. Many studies have performed camera calibration based on corner points [9], [22], [24], [30], [31], [32]. However, image blur makes it difficult to find corner points accurately. To remedy this, Yang et al. [25] used an error function model [13] to handle the Gaussian blurred surfaces along the edges of the checkerboard images. Chen et al. [27] applied a morphological filter and extracted subpixel-level coordinates of edges for polynomial fitting, thus finding the intersections of the edges as corner points. However, the checkerboard images are corrupted from  checkerboard image. Such pixel variations around the edge in Fig. 2 make it difficult to find the edge and its corner points accurately. In this paper, we first consider the JPEG compression distortion as well as lens distortions for edges in the corner point extraction problem. In order to robustly find the corner points under JPEG distortion, we design a novel edge surface model to be applied for the vicinity zone (Z v ) of each edge as shown in Fig. 2. We first perform normalization of the pixel values in Z v to estimate the parameter values of our edge surface model. The normalization procedure is as follows: (i) As shown in Fig.  3, the pixels with small fluctuations of the values in each bright area (a blue-dotted rounded box), which are located in a distance from the edges, are modeled as a surface by a second-order polynomial, as denoted by F W (x, y). The distance is defined to get pixels on flat areas whose values are little damaged by the compression. Since the edge widths in our checkerboard images are often smaller than 5 pixels, we set the distance value to 5. Similarly, the pixels with small fluctuations in a dark area (a red-dotted rounded box) are modeled also as a surface using another second-order polynomial, as denoted by F B (x, y); (ii) Once we get the two surface models (F W (x, y), F B (x, y)), we normalize the pixel values of the pixels in Z v by expanding F W (x, y) and F B (x, y) into Z v where (x, y) ∈ Z v and applying them as follows: where p = [x, y] T is a pixel location in Z v , and V (p) is the pixel value at p. It can be noted in Eq.
. Then, the normalized over-and under-shoot pixel values are larger than 1 and smaller than -1, respectively, which are the red dots in an 1-D profile of the normalized pixels as shown in Fig. 2c. Our edge surface model under the JPEG compression distortion is defined in Z v as: where S is an amplitude of over/under-shoot, a is a steepness factor of the edge surface on the edge surface model, d is a distance in pixel unit between a coordinate of pixel p = [x, y] T and line γ = [γ x , γ y , γ c ] T where γ x x + γ y y + γ c = 0 and γ 2 x + γ 2 y = 1. As shown in Fig. 2, the over-and under-shoot values are located at d = π/a and −π/a, respectively, as in (6) and edge surface model φ is defined between d = −π/a and +π/a. The pixels (outside Z v ) from line γ in distances of |d| > π/a are discarded for the parameter estimation of edge surface model. The estimation of edge surface model parameters (a, γ) can be found within our corner point extraction optimization according to (9) in the following subsection. The values of S are the mean values of absolute over-/under-shoot values. Note that the gradient values of the edge surface model near d = 0 are almost constant along the edge. As the |d| increases, the magnitudes of the gradient values of the edge surface model decrease so as to fit the model to the over/under-shoot. In our experiments, we also compare the edge surface model in Eq. (6) to a simple version with a plane which is given by where Z v is defined as |d| < π/a. For an N X ×N Y checkerboard grid, we have total N esm (= N X (N Y −1)+N Y (N X −1)) edge surface models since there is one edge surface model for the edge between two corner points. For a vertical line of the checkerboard, there are N v (= N Y − 1) edge surface models on N v edges corresponding to the vertical line. If there is no lens distortion, the edge surface models corresponding to a vertical line share a single γ because the edges are generated by projections of the single vertical line in checkerboard. The edge surface models with the other vertical lines are the same. This constraint is also applied equally to the edge surface models corresponding to the horizontal lines of the checkerboard.

B. STRUCTURAL INFORMATION FOR EDGE SURFACE MODEL PARAMETER ESTIMATION
In our method, corner points are found not independently but jointly by utilizing structural information of the checkerboard grid. The independent findings of corner points are vulnerable to local damages due to compression distortions, which causes degradation of camera calibration performance. Instead, the structural information can be effectively used to robustly find the corner points where the entire corner points and edges are jointly found over a whole checkerboard image, which can overcome the local distortions.
The structural information we are dealing with is originated from the grid of the checkerboard. Actually, with VOLUME 4, 2016 the lens distortion, we have to deal with complex nonlinear relations for the structural constraints: (i) The corner points on the checkerboard images should be the intersection points of the edges; (ii) the edges should meet at vanishing points; and (iii) The cross-ratios between each checkerboard image and the checkerboard should be preserved after camera projection. We first consider a distortion-free camera to simplify the corner point extraction problem and expand it for the camera with the lens distortions in the next subsection.
The corner points in the checkerboard images should be intersection points of the γ's in (6). So, rather than finding the corner points directly from the checkerboard images, the γ's are obtained from the checkerboard images, and the intersections of the γ's are set as the corner points. The structural constraints of the γ's include that the γ's meet at vanishing points, and the cross-ratios between each checkerboard image and the checkerboard are preserved. These constraints on γ's are from the fact that the γ's on each checkerboard image are projected from the grid lines of the checkerboard by a single homography. So, to satisfy the constraints, all the γ's in each checkerboard image are expressed as where l n is the n-th line in the checkerboard, n = 1, ..., N X +N Y , and H ∈ R 3×3 is a homography [10] with 8 parameters. From the edge surface model in (6) and the γ's in (8), the corner points can be found via the following optimization of the corner point extraction as where n is an index of the lines on the checkerboard including vertical and horizontal lines, m is an index of the edges along the n-th line where an edge is defined as a line segment between two corner points on a same line, N L (=N X +N Y ) is the total number of lines on a checkerboard, and M n (=N X -1 or N Y -1) is the number of the edge surface models along the n-th line. φ i n,m (·) and a i n,m indicate the m-th edge surface model and its steepness factor, respectively, along the n-th line in the i-th checkerboard image. J i n,m is a set of pixels in Z v (See Fig. 3) and G i (p) is the normalized pixel value by Eq. (5) at pixel location p in the i-th checkerboard image. It should be noted that the S value of each edge surface model is computed after the pixels within a predefined distance (±1.5 pixels) around an edge are assigned to J i n,m before solving the corner point extraction optimization problem in (9). The S value for each edge surface model is determined as the mean value of absolute over-and under-shoot pixel values for J i n,m (See Fig. 2c). In conclusion, from (9), the corner point extraction optimization forces all the γ's and the corner points to be precisely estimated by obeying the structural constraints.

C. CONSIDERATION OF LENS DISTORTIONS
Without any lens distortion, the line-to-line projection can be done by a homography in (8) at camera projection. However, under lens distortion, (8) is no longer valid. So, the distorted locations of the pixels by the lens distortions need to be corrected before applying the homography in (8). When estimating H and A = {a n,m }, we have to find the camera parameters K and the distortion coefficients D, both of which are related to the lens distortions [33]. After the distorted pixel locations are corrected, the curved edges become straight lines, and the straight lines are projected from the checkerboard grid by H. The desired parameters for corner points can be estimated via the optimization based on Levenberg-Marquardt (LM) method [34], [35] as where p c is a corrected location of p by K i and D i for the ith checkerboard image. G i c (p c ) is a normalized pixel value at p c of the corrected i-th checkerboard image. The others are the same as those in (9). Through the optimization in (10), K i , D i , H i and A i are estimated. The cost function of our method converges well in the optimization. So the damping factor used in the LM method is empirically set to be small, λ = 10 −20 , with its associated constant, ν = 20.
For the optimization, we need to estimate the internal parameter K, a set of distortion coefficients D, a homography H, and a set of steepness factors A. The initial values of the homography are estimated by matching the edge pixels on the checkerboard images with the known lines of the checkerboard. Please note that the distortion is ignored in this case. The estimated homography is not accurate, but good enough for the initial guess. The principle point of K is set to be the center of images. The skew term α in (1) is set to zero. Then the remaining parameters in (1) are f s x and f s y which are the products of the focal lengths and the number per unit distance. Also, we assume s x = s y = s. In order to find f ·s, we use the following relation among the internal/external parameters and the homography as: For the equation above, there are 8 unknowns (a scale factor η, f · s in K, θ x , θ y , θ z in R = [r 1 r 2 r 3 ], and t x , t y , t z in T ), but we have 9 equations. Note that r 1 is a 3×1 vector whose elements are functions of θ x and θ y , and r 2 is a 3×1 vector whose elements are functions of θ x , θ y , and θ z [36]. So, we can determine the internal parameter f · s which is the only needed variable to compute the initial values. The initial distortion coefficients for D = {k 1 , k 2 , t 1 , t 2 }. in (2) and D f e = {ρ 1 , ρ 2 , ρ 3 , ρ 4 } in (4) are all set to zeros. Each of the steepness factors A = {a i n,m } in (10) initially tries with 0.5 and -0.5 for which the one that results in a lower cost value is used as its initial value.
Based on these estimated camera parameters from (10), we can find the corner points on the i-th checkerboard image by projecting the corner points on the checkerboard (See Fig. 4). With the corner points of all checkerboard images, we finally estimate the camera internal parameters K, the lens distortion coefficients D, and the external parameters, {R i }, {T i } using Zhang's method [9] as shown in Fig. 1.

VI. EXPERIMENTAL RESULTS
Experiments are conducted with synthetic and real data. The eight previous methods are compared with ours, which include Geiger's [22] [28], and Zhou's [29] methods. In Zhou's [29] method, the points along the edges are extracted for camera calibration. We used a built-in function in MATLAB T M for Geiger's method, and an open source code of Ha's method. The codes for the other methods are implemented by ourselves.

A. EXPERIMENTS WITH SYNTHETIC DATA
We conducted two experiments with the synthetic data. One is an ablation test to see the effectiveness of each contribution in our method for corner point extraction performance and the camera calibration. The other one is the comparison with the previous methods. For the comparison test, cameras with and without a fish-eye lens are utilized.
The dimension of a square of the checkerboard is 40mm×40mm, and the number of the corner points is 56 (= 7×8). The checkerboard is located approximately 800mm away from the camera. The ground truth parameters of the non fish-eye lens camera are f s x = 1,280, f s y = 1,260, α = 1, p x = 510, p y = 355, k 1 = −0.15, k 2 = −0.01, t 1 = −0.015, and t 2 = 0.01. For the fish-eye lens camera, ρ 1 = −2.4, ρ 2 = −0.18, ρ 3 = −0.12, and ρ 4 = −0.036. The internal parameters of the fish-eye lens camera are the same as the internal parameters of the non fish-eye lens camera. The image size is 1,000×700. A total of 100 checkerboard images was used where 20 checkerboard images were randomly selected for the cameras calibration by repeating the camera calibration 40 times.
To evaluate the accuracy of the estimated corner points, the root mean square errors (RMSE) are measured in pixel units as where N C is the total number of the corner points of all checkerboard images, c i and c gt i are the i-th estimated and ground-truth corner points, respectively.
By the camera calibration, we estimate K, D, {R i } and {T i }. To evaluate the accuracy of the parameters, the error of the reprojected corner points are measured in pixel unit as where P j is the j-th corner points on the checkerboard in the real world, P gt i,j is ground truth coordinate of the j-th corner points on the i-th checkerboard image, N I is the number of images for the camera calibration, N P is the number of corner points on the checkerboard, and f P returns coordinate of projections of P j by K, D, R i , and T i according to Eq. (2).
The K, and D are the results of the camera calibration which are the camera's unique characteristic values. So, to see the performance of each camera calibration method, these parameters should be evaluated. The errors of the estimated K's, and D's are measured in pixel unit as where f K (X, K, D) returns 2D coordinates of a projection of 3D points, {X}, in a region of interest (ROI) by K, and D according to Eq. (2) with R = I and T = 0. When we evaluate Zhou's method with the others, only these evaluation method is utilized because the corner points are not extracted in Zhou's method.

1) Ablation Study on Proposed Method
Our contributions include 1) the edge surface model to handle the JPEG compression distortion for corner point extraction and 2) the utilization of structural information to deal with the lens distortion. To see the effectiveness of the two contributions, we inspect the four variants (A, B, C and D) of our methods as shown in Table 1. The two variants (A and C) use the plane model in Eq. (7) instead of the edge surface model in Eq. (6). The other two variants (A and B) do not rely on the structural information. They assume that the edges are VOLUME 4, 2016 straight, and each edge surface is individually estimated from the pixels around each edge. So, the intersection lines of the surfaces are searched and the intersections of the lines are extracted as the corner points. Note that the variant D is our full model with the two contributions that are optimized in Eq. (10). The quality factor (Q) of JPEG compression is set to 40 for experiments. The camera for the experiments is the non fish-eye lens camera.  Table 2 show the performances of the four variants. As shown in Table 2, the variants C and D show significantly lower E C , E P , and E K values compared to the other two variants A and B that do not incorporate the structural information for corner point extraction. This implies that our proposed usage of structural information plays a significant role in precise corner point extraction. In comparison between two variants A (C) and B (D), the variant B (D) has lower E C , E P , and E K values than the variant A (C). From this, our edge surface model works positively to further improve the corner point extraction performance. Among those variants, the variant D is our final proposed method (Proposed-ESM). ESM stands for the edge surface model. We will compare it with the previous methods in the next subsection. The variant C (Proposed-Plane) is also used for the comparison test. The model parameters of 'Proposed-ESM' and 'Proposed-Plane' are found according to our optimization in Eq. (10).

2) Performance Comparison with Previous Methods
In comparison with the previous methods, to see the robustness of our method to the compression distortion, we used various Q values of JPEG compression and various lens distortion coefficients. Note that the lower Q values result in more compression distortions with smaller visual quality. The camera for the experiments is the non fish-eye lens camera. Table 3 compares the performances of corner point extraction methods for various Q values. In Table 3, our methods, 'Proposed-Plane' and 'Proposed-ESM' indicate a simple model with the plane in Eq. (7) and the edge surface model in Eq. (6), respectively. As shown in Table 3, our proposed methods yield significantly lower RMSE values for all Q values with at least one order in magnitude, compared to the other methods. Table 4 shows errors of reprojection of checkerboard corner points. Since the reprojected corner points of all the methods satisfy the structural information, less errors tend to occur than those in Table 3. The proposed methods also appear less than the other methods. Fig. 5 shows the scatter plots of reprojection errors for the estimated corner points by the eight corner point extraction methods for Q = 40. As shown, our proposed method with edge surface model clearly outperforms all the other methods with the smallest error. Also, a simplified version of our method, which uses a plane model instead of edge surface model, also significantly outperforms all the previous methods. The dispersions of the reprojected corner points in Fig. 5 are measured as the standard deviation values in Table 4. Table  5 shows the performance of camera calibration for various methods. As shown, both of our methods, 'Proposed-plane' and 'Proposed-ESM' outperform all the previous methods with large margins for all Q values. The superiority of our 'Proposed-ESM' method comes from the utilization of structural information for corner point extraction and the edge surface model to cope with the compression distortion. For smaller Q values, the 'Proposed-ESM' more significantly outperforms the 'Proposed-Plane' because the 'Proposed-Plane' suffers from over-and under-shoot problems of pixels in Z v due to more severe JPEG distortions. Fig. 6 shows the accuracy maps of camera calibration based on the estimated K's and D's and Table 6 shows the numerical RMSE errors of the estimated camera parameters for Q = 40. As shown, our methods clearly outperform all the previous methods for camera calibration with help of more precise corner point extraction performance. Tables 7, 8 and 9 show the performances of the corner point estimation methods for Q = 40 under various lens distortions with k 1 = −0.15n, k 2 = −0.01n, t 1 = −0.015n and t 2 = 0.01n for n = 1, 2, 3 and 4. As shown, our methods show the superiority in corner point extraction, thus leading to more accurate camera calibration performances compared to the other methods in all cases. It is worthwhile to note that Yang's method exhibited the highest RMSE although it incorporates an error fuction surface model into the corner point extraction problem. This is because it does not take into the lens distortions for the resulting curved edges. Table 10 shows the performances of the corner point estimation for the synthetic images obtained for the fish-eye lens camera. The Q value is set to 40 for the test. The fish-eye lens often causes to generate large lens distortions with wide angle views. Even for this situation, it can be seen in Table 10 that our method outperforms all the other methods in RMSE. As shown, Chen's method shows the worst performance because their method is based on polynomial curve fitting to the edges of checkerboard images and is not sufficient to model the curved edges caused by the fish-eye lens.

B. EXPERIMENTS WITH REAL DATA
Experiments with real data are often difficult to evaluate estimated parameters because the true parameter values are not available. To overcome this difficulty, we evaluate the camera parameters by reconstructing a cuboid with four checkerboards on its sides (Fig. 7b). The number of corner points in the checkerboard for calibration (Fig. 7a) is 110 (= 10×11), and the dimension of a checkerboard square is 38mm×38mm. From a single cuboid image with two checkerboards (Fig. 7b), we can reconstruct two facets of the cuboid image (Fig. 7c) based on the estimated camera parameters. For the cuboid images, we have taken four images for the cuboid in approximately 1.2m distance away from the camera. The angles and rotations of the cuboid are somewhat restricted to have similar areas for the two faces of the cuboid, as shown in Fig. 7b. So, angles and rotations were determined such that each of the four cuboid images includes two facets of the cuboid and the boundary between the two facets, as shown in Fig. 7b. Evaluation can be performed in terms of distances in RMSE between corner points after reconstruction. A Camel T M 's WC100 camera is used. The size of the image is 1,920×1,080. A total of 50 checkerboard images was used where 20 checkerboard images were randomly selected for each camera calibration by repeating the camera calibrations 40 times.  Table 11 shows the reconstruction performances in terms of RMSE in millimeter units. Our methods outperform all the other methods by yielding more accurate camera parameter estimations that lead to more accurate cuboid reconstruction.

VII. CONCLUSION
We first proposed a novel corner point estimation for accurate camera calibration by jointly considering both compression and lens distortions. We incorporate an elaborate edge surface model to handle the over-and under-shoots of pixel values along the edges of checkerboard images due to compression distortion. Furthermore, the structural information of the corner points and the edges from the checkerboard grid is utilized to precisely find the corner points. Extensive experimental results have verified that our method outperformed all previous methods with significantly large margins, especially for corner point estimation on synthetic data. This has led to more accurate camera calibration.
As future work, it might be worthwhile to explore our edge surface model for different image data such as multispectral color images having blurred corners from compression.