Optimization of the stereo system calibration parameters for photogrammetric measurements in the conditions of field experiments

Optical methods for deformation diagnostics and surface shape measurement are often used in scientific research and industry. Most of these methods are based on the triangulation of a set of two-dimensional points from different images corresponding to the three-dimensional points of an object in space. Triangulation is based on the stereo system calibration parameters, which are determined before the experiment. Measurements during conditions with increased vibration loads can lead to a change in the relative position of the cameras of the stereo system (decalibration). This leads to a change in the actual calibration parameters and an increase in the measurement error. This work aims to solve the problem of increasing the measurement accuracy of the photogrammetric method in the case of high vibration loads. For this, it is proposed to use an optimization algorithm for calibration parameters to minimize the reprojection error of three-dimensional points calculated using triangulation. The paper presents the results of a computer simulation of decalibration of a video camera stereo system, an algorithm for optimizing the external parameters of a stereo system, and an assessment of its performance.


Introduction
The task of surface deformation diagnostics often occurs in the automotive industry, building, and aviation. Such diagnostic is usually performed using close-range photogrammetry, a measurement technique that is used to determine the geometry, shift, and deformation of construction using images. Measurement of the 3D shape of a surface is usually done by two video cameras (stereo system), with which a pair of images are captured. On this pair, two-dimensional coordinates of the same points of the measured object are determined. Next, a pair of points with 2D coordinates is triangulated into one point with 3D coordinates. For this, information about the relative position of the cameras in the stereo system is used. Calibration usually is carried out before measurements. Its result is a set of external and internal parameters of the stereo system.
Photogrammetric measurements can often be carried out during high vibration loads. Vibration causes a change in the position of the cameras and the direction of their line of sight. Thus, the calibration parameters of a stereo system are changing during measurements (decalibration), and its error is increasing. When designing an experimental setup, all possible measures are taken to minimize the influence of decalibration on the measurement results, but they cannot be completely excluded. That is especially true for measurements when the base of the stereo system reaches more than one meter. Displacement of the cameras relative to each other can be corrected by carrying out the recalibration procedure, which consists of optimizing the parameters of the calibration matrix to minimize the triangulation error. That approach was already used in other works. In [1] recalibration for a system consisting of several cameras is described. External parameters are recalculated for a camera changing its position using information from cameras that remained motionless. In [2], according to a similar principle for the PIV stereo system, the position of the laser knife is corrected, but not the relative position of the cameras. In [3] for the PIV tomographic system, the mutual arrangement of the cameras is corrected, but a camera model is based on the polynomial function. In [4] the scheme of flight measurements for wing deformations based on the DIC method is described. It is based on two groups of coupled cameras located in the vertical tail of the aircraft. This approach provides the measurement of deformations relative to a stationary aircraft frame of reference attached to the fuselage. In [5], the results of flight measurements of VUT100 Cobra aircraft wing deformations are presented. The presence of the stereo system decalibration effect during a flight increases the measurement error. Based on the experimental images, a recalibration procedure was carried out. It makes it possible to significantly reduce the triangulation error. The main disadvantage of this approach is the lack of verification of measurement results.

Nelder-Mead algorithm
It is proposed to use multidimensional optimization methods to determine the effective values of the calibration parameters of the stereo system during the experiment to solve the problem of decalibration. Methods of the direct search for the minimum of an objective function (or zero-order methods) use the information only about the values of this function. Computational experiments and comparative analysis of methods based on the results of such experiments are common ways to estimate the effectiveness of direct search methods. However, it should be borne in mind that this analysis may not in all cases lead to unambiguous conclusions about the advantages of one method over another, since they can behave differently at different stages of the minimization process.
Zero-order methods include methods that do not use derivatives to select the direction of descent: Gauss method, method of rotating directions (Rosenbrock); deformable polyhedron method (simplex search); Hook Jeeves method, Powell method. In this work, the Nelder-Mead algorithm was chosen for the recalibration process. This is the most well-known method among the non-directional strategy methods. The method is based on the fact that the experimental sample containing the smallest number of points is a simplex.
A regular simplex in an N-dimensional space is a polyhedron formed by N + 1 equidistant pointsthe vertices of the simplex. An important property of the simplex is that a new simplex can be constructed on any face of the original one by reflecting a vertex relative to the center of gravity of all other vertices of the simplex.
Calculating the values of the objective function at the simplex vertices, information about the nature of the change in this function can be obtained. The search for the minimum point of the objective function using correct simplexes is performed as follows: • at each iteration, the objective function is calculated at all points of the simplex, and their ordering is performed in ascending order of values; • then a sequential attempt is made to construct new simplices with the best values of the objective function, by reflecting the points with the worst values; • if a sequential attempt to reflect the two worst vertices fails, then the simplex is compressed to the point with the smallest value and a new iteration is performed; • the search is stopped when the difference between the function values at the simplex points becomes sufficiently small. Choosing the optimal parameters for each specific objective function can be done after an additional study with the help of a computational experiment.
Advantages of the Nelder-Mead algorithm are simplicity, a small number of preset parameters, a simple search strategy, calculating only function values, a small amount of memory required. Disadvantages of the method: the method works efficiently for N ≤ 6, the algorithm is based on cyclic motion along with coordinates. This can lead to the degeneration of the algorithm into an infinite sequence of exploratory searches without pattern matching.

Simulation of a stereo system decalibration
The measurement error introduced by decalibration can be evaluated to estimate the possibility of optimizing the calibration parameters of a stereo system. For this, computer modeling was carried out. The decalibration simulation process is as follows: 1. The camera parameters are set, both internal (focal lengths, pixel size, and coordinates of the principal point) and external (vector of displacement and rotation of cameras relative to each other). All these parameters are determined in the experiment, at the stage of camera calibration.
2. The measurement object is formed as a set of points in 3-dimensional space. 3. Without displacement (decalibration), by projecting 3D points onto the image plane of the cameras, 2D points are calculated, which act as actually received images from the cameras. 4. A known offset (decalibration) is set as the vector of displacement and/or rotation of the cameras relative to each other.
5. Taking into account the changed external parameters of the cameras and the 2D points obtained in item 3, the 3D points of the object are calculated by triangulation.
6. Then the obtained 3D points are again projected onto the image plane of the cameras, as a result of which we get reprojected 2D.
7. Further, comparing the original 2D points and the reprojected ones, the magnitude of the reprojection error is estimated.
If the offset value is zero, then the reprojection error will be close to zero. The non-zero value is due only to machine error.  Figure 1 shows the result of decalibration modeling as the dependence of the 3D points reprojection error on the change in the external parameters of the stereo system. The following values were used in the simulation: the ratio of focus to pixel size is equal to 7500, the center point of the camera [1250, 980], displacement of one camera relative to the other T x = -315 mm, T y = 115 mm, T z = 40 mm, rotation of one camera relative to the another one around the x-axis is 3.5°, the y-axis is 12°, the z-axis is -1°. Parameters of the measured object: distance to the object 1.5 m, object size 30×20 cm, the number of points is 10×10, the angle of inclination of the object is 45°. The object is located opposite the left camera, which is taken as the origin of the coordinate reference system. These parameters were taken as an example of the actual calibration parameters of the stereo system and the measured object used in laboratory experiments.

Evaluation of recalibration results
By presetting the initial offset, for example, the rotation around the y-axis by 1.5 °, the minimum of the objective function will be moved on some dependencies. That means that simple optimization by displacement or rotation by one axis will not lead to a global minimum of it, i.e. no correct or close to correct solution will be received.
Therefore, in the optimization process, the objective function of the reprojection error depending on six parameters is minimized. The parameters are: T x , T y , T z are the elements of the displacement vector, and α, β, γ are the angles of rotation of one camera relative to the other. In step 4, the offsets are performed by the Nelder-Mead method, and then steps 5, 6, 7, etc. are repeated until we reach the maximum iteration value or the reprojection error value is less than the specified one.
An example of the dependence of reprojection error on displacement along the x-axis and rotation around the x-axis (with and without optimization) is shown in figure 2. The vertical axis (mean square deviation (RMS) in pixels) is displayed in logarithmic scale, in red the result without optimization, and in green the result with optimization. Optimization was performed by the Nelder-Mead method in conjunction with the penalty function method. The number of attempts for each of the displacements is equal to 12, the maximum number of iterations in each attempt is equal to 500, the steps for calculating displacements are 0.1 mm and 0.02°. It can be seen from the figure above that the reprojection error as a result of optimization has significantly decreased, by about two orders of magnitude. It is not worth counting on the same error reduction in physical modeling, since the error of reprojection without displacements in real cases is much higher.

Conclusions
The paper shows one of the approaches to optimizing the reprojection error function to find the actual parameters of the stereo system calibration. Optimization is carried out using the Nelder-Mead algorithm for a function of six variables: displacement amplitudes and angles of rotation along three axes. It is impossible to determine the accuracy characteristic of the algorithm directly since it depends on many factors, but it is possible to estimate the achieved values of the objective function as a result of optimization. For example, at the calibration stage, the average reprojection error is 0.75 pixels, which means that the average deviation of the original 2D coordinates from those reprojected along the two axes x and y is 0.75 pixels. As a result of optimization, it is likely that the values of the reprojection error are the same as at the calibration stage and even smaller values.
In the work, a computer simulation of the decalibration process was carried out with the subsequent application of the recalibration procedure. In the course of the latter, the reprojection error has decreased significantly. However, it cannot be said unequivocally that recalibration allows one to determine the ongoing changes in the position of the cameras. For this, it is necessary to carry out additional computer and physical modeling.