A review on egomotion by means of differential epipolar geometry applied to the movement of a mobile robot

Abstract

The estimation of camera egomotion is an old problem in computer vision. Since the 1980s, many approaches based on both the discrete and the differential epipolar constraint have been proposed. The discrete case is used mainly in self-calibrated stereoscopic systems, whereas the differential case deals with a single moving camera. This article surveys several methods for 3D motion estimation unifying the mathematics convention which are then adapted to the common case of a mobile robot moving on a plane. Experimental results are given on synthetic data covering more than 0.5 million estimations. These surveyed algorithms have been programmed and are available on the Internet.

Introduction

Camera calibration is the first step toward computational computer vision. The use of precisely calibrated cameras makes the measurement of distances in a metric world from their projections on the image plane possible. The camera model is a mathematical description of the geometric relationship between the 3D geometric entities and their 2D projections on the image plane consisting of a set of internal camera parameters which describes the internal geometry and optics of the camera, and a set of extrinsic parameters which describes the position and orientation of the camera in the scene. Perspective cameras can be represented by several models depending on the desired level of accuracy.

Given a 3D point $\text{p}$ in metric coordinates with respect to the world coordinate system {W}, its projection $\text{m}$ in pixels with respect to the image coordinate system {I} can be computed through some linear (sometimes non-linear) equations.² This set of equations encapsulates several transformations which are broken down into four steps (see Fig. 1).

• First the coordinates of point $\text{p}$ in the world coordinate system are transformed into the camera coordinate system by using an Euclidean transformation.

• Next, it is necessary to carry out the projection of point $\text{p}$ onto the image plane by using a projective transformation, obtaining point $\text{q}$.

• The third step models lens distortion causing a disparity of the real projection on the image plane. Then, point $\text{q}$ is transformed into the real projection $\text{m}$.

• Finally, the last step consists of transforming the $\text{m}$ point from the metric coordinate system of the camera into the image coordinate system of the computer in pixels.

Small variations in the definition of the geometric transformations used imply the use of different camera models, resulting in different calibration techniques. For instance, the technique proposed by Hall [1] in 1982 is based on an implicit linear camera calibration by computing the 3×4 transformation matrix which relates 3D object points with their 2D image projections. The latter work of Faugeras [2], proposed in 1986, was based on extracting the physical parameters of the camera from this sort of transformation technique. Some years later, work proposed by Faugeras was adaptated to include radial lens distortion [3]. Two other interesting contributions are the widely used method proposed by Tsai [4], is based on a two-step technique modelling only radial lens distortion, and the complete model of Weng [5], proposed in 1992, which includes three different types of lens distortion. Research efforts are still being carried out on obtaining new camera models to improve both accuracy in computing the optical ray and in extracting the camera parameters which best model reality. For additional details concerning camera calibration methods, please check the recent calibration survey [6].

When we get into the binocular case (that is two views from a stereoscopic system or two different views from a single moving camera) another interesting relationship is defined in the so-called epipolar geometry. This information is contained in the fundamental matrix which includes the intrinsic parameters of both cameras and the position and orientation of one camera with respect to the other. The fundamental matrix can be used to simplify the matching process between the viewpoints and to get the camera parameters in active systems where optical and geometrical characteristics might change dynamically depending on the imaging scene. In this case, the camera parameters can be extracted by using Kruppa equations [7]. Moreover, the epipolar geometry can be considered from both a continuous and a discrete point of view.

Probably the most well-known viewpoint is the discrete epipolar constraint formulated by Longuet–Higgins [8], Huang [9] and Faugeras [10]. In this case the relative 3D displacement between both views is recovered by the epipolar constraint from a set of correspondences in both image planes. Then, given an object point $\text{p}$ with respect to one of the two camera coordinate system and its 2D projections $\text{q}$ and $\text{q}\text{′}$ on both image planes (in metric coordinates), the 3 points define a plane Π which intersects both image planes at the epipolar lines $\text{l}{}_{\mathrm{<mtext>q</mtext><mtext>\prime </mtext>}}$ and $\text{l}{}_{\mathrm{<mtext>q</mtext>}}\text{′}$, respectively, as shown in Fig. 2. Note that the same plane Π can be computed using both focal points $\text{c}{}_{\mathrm{<mtext>o</mtext>}}$ and $\text{c}{}_{\mathrm{<mtext>o</mtext>}}\text{′}$ and a single 2D projection, which is the principle used to reduce the correspondence problem to a single search along the epipolar line. Moreover, the intersection of all the epipolar lines defines an epipole on both image planes, which can also be extracted by intersecting the line defined by both focal points $\text{c}{}_{\mathrm{<mtext>o</mtext>}}$ and $\text{c}{}_{\mathrm{<mtext>o</mtext>}}\text{′}$ on both image planes. All the epipolar geometry is contained in the so-called Fundamental matrix [8] as shown in Eq. (1)

$$\text{m}{}^{\mathrm{<mtext>T</mtext>}}\text{Fm}\text{′=0,}$$

where the fundamental matrix depends on the intrinsic parameters of both cameras and the rigid transformation between them

$$\text{F}\text{=}\text{A}{}^{\mathrm{<mtext>-</mtext><mtext>T</mtext>}}\text{R}{}^{\mathrm{<mtext>T</mtext>}}\text{t}\text{̂}\text{A}\text{′}{}^{\mathrm{-1}}\text{.}$$

When the intrinsic camera parameters are known, it is possible to simplify , , obtaining,

$$\text{q}{}^{\mathrm{<mtext>T</mtext>}}\text{Eq}\text{′=0,}$$

where $\text{q}\text{=}\text{A}{}^{\mathrm{-1}}\text{m}\text{,}\text{E}\text{=}\text{R}{}^{\mathrm{<mtext>T</mtext>}}\text{t}\text{̂}\text{,}\text{q}\text{′=}\text{A}\text{′}{}^{\mathrm{-1}}\text{m}\text{′}$. The matrix $\text{E}$ is called essential [9].

Many papers describe different methods to estimate the fundamental matrix [11], [12], [13], [14].

The differential case is the infinitesimal version of the discrete case, in which both views are always given from a single moving camera. If the velocity of the camera is low enough and the frame rate is very high, the relative displacement between two consecutive images becomes very small. The 2D displacement of image points can then be obtained from an image sequence using the optical flow. In this case, the 3D camera motion is described by a rigid motion using a rotation matrix and a translation vector, as in (Fig. 3)

$$\text{p}\text{(t)=}\text{R}\text{(t)}\text{p}\text{(0)+}\text{t}\text{(t),}$$

where differentiating

$$\text{p}\text{̇}\text{(t)=}\text{R}\text{̇}\text{(t)}\text{p}\text{(0)+}\text{t}\text{̇}\text{(t).}$$

Then, replacing the parameter $\text{p}\text{(0)}$ to $\text{R}{}^{\mathrm{-1}}\text{(t)[}\text{p}\text{(t)−}\text{t}\text{(t)]}$ in Eq. (5), the following equation is obtained:

$$\text{p}\text{̇}\text{(t)=}\text{R}\text{̇}\text{(t)}\text{R}{}^{\mathrm{-1}}\text{(t)}\text{p}\text{(t)+}\text{t}\text{̇}\text{(t)−}\text{R}\text{̇}\text{(t)}\text{R}{}^{\mathrm{-1}}\text{(t)}\text{t}\text{(t),}$$

which leads to the following differential epipolar constraint:

$$\text{q}{}^{\mathrm{<mtext>T</mtext>}}\text{υ}\text{̂}\text{q}\text{̇}\text{+}\text{q}{}^{\mathrm{<mtext>T</mtext>}}\text{ω}\text{̂}\text{υ}\text{̂}\text{q}\text{=0.}$$

$\text{ω}\text{=(ω}{}_1\text{,ω}{}_2\text{,ω}{}_3\text{)}{}^{\mathrm{<mtext>T</mtext>}}$ is the angular velocity of the camera and $\text{υ}\text{=(υ}{}_1\text{,υ}{}_2\text{,υ}{}_3\text{)}{}^{\mathrm{<mtext>T</mtext>}}$ is the linear velocity of the camera. By projecting $\text{p}$ and $\text{p}\text{̇}$ in the image plane, $\text{q}$ in camera coordinates and its corresponding optical flow $\text{q}\text{̇}$ are obtained.

For a complete demonstration, the reader is directed to Haralick's book, Chapter 15 [15], where the movement of a rigid body related to a camera is explained. In our case, the demonstration is used to describe the movement of a camera related to a static object, in which only the sign of the obtained velocities differs from the previous one. Nevertheless, Eq. (7) can also be demonstrated in different ways as explained in Viéville [16] and Brooks [17]. Also, another equivalent form of Eq. (7) is shown in Eq. (8). In this case, since matrix $\text{s}$ is symmetric, the number of unknowns is reduced to six

$$\text{q}{}^{\mathrm{<mtext>T</mtext>}}\text{υ}\text{̂}\text{q}\text{̇}\text{+}\text{q}{}^{\mathrm{<mtext>T</mtext>}}\text{S}\text{q}\text{=0,}$$

where

$$\text{S}\text{=}\text{1}\text{2}\text{(}\text{ω}\text{̂}\text{υ}\text{̂}\text{+}\text{υ}\text{̂}\text{ω}\text{̂}\text{).}$$

The existence of two forms indicates that a redundancy exists in Eq. (7) (for a demonstration see Viéville [16], Brooks [17] and Ma [18]). Several books describe the optical flow such as Trucco and Verri [19], and the article published by Barron et al. [20] gives a state-of-the-art in optical flow estimation.

When comparing the discrete and differential methods, the discrete epipolar equation incorporates a single matrix, whereas the differential epipolar equation incorporates two matrices. These matrices encode information about the linear and angular velocities of the camera [15].

Approaches to motion estimation can be classified into discrete and differential methods depending on whether they use a set of point correspondences or optical flow. Another possible classification takes into account the estimation techniques used for motion recovery (linear or non-linear techniques). In Table 1, the algorithms are summarized and classified in terms of their nature (discrete and differential case), and estimation method (linear and non-linear technique).

This article analyzes several different algorithms for camera motion estimation based on differential image motion. The surveyed methods have been compared with them and experimental results are given. Moreover, this article analyzes the adaptation of general methods used in free 3D movement to planar motion which corresponds to the common case of a robot moving on a plane with the aim of studying how much accuracy improves by constraining the camera movement. Hence, this article focuses on linear techniques, as the motion has to be recovered in real-time.

This article is structured as follows. Section 2 describes up to 12 algorithms for 3D motion estimation based on optical flow. Section 3 focuses on the estimation of planar motion by constraining the free movement explained in the previous section. Then, Section 4 deals with the experimental results obtained. The article ends with conclusions.