Matching techniques to compute image motion

doi:10.1016/S0262-8856(99)00018-9

Image and Vision Computing

Volume 18, Issue 3, February 2000, Pages 247-260

https://doi.org/10.1016/S0262-8856(99)00018-9 Get rights and content

Abstract

This paper describes a thorough analysis of the pattern matching techniques used to compute image motion from a sequence of two or more images. Several correlation/distance measures are tested, and problems in displacement estimation are investigated. As a byproduct of this analysis, several novel techniques are presented which improve the accuracy of flow vector estimation and reduce the computational cost by using filters, multi-scale approach and mask sub-sampling. Further, new algorithms to obtain a sub-pixel accuracy of the flow are proposed. A large amount of experimental tests have been performed to compare all the techniques proposed, in order to understand which are the most useful for practical applications, and the results obtained are very accurate, showing that correlation-based flow computation is suitable for practical and real-time applications.

Introduction

Window-matching or correlation-based techniques are the most intuitive and perhaps also the most widely applied techniques to compute the optical flow from an image sequence, i.e. to estimate the 2D motion projected on the image plane by the objects moving in the 3D scene [1], [2], [3], [4], [5], [6]. Optical flow estimation has many practical and industrial applications, i.e. for object tracking, assisted driving or surveillance systems, obstacle detection, image stabilisation or video compression [7], [8], [9], [10], [11]. In spite of this fact, few works analysing the performances and the possible enhancements of these algorithms have been presented [1], [2], [6] so that a more detailed analysis of this simple and widely used optical flow technique seemed to us necessary. The aim of this paper is to give a clear overview of window matching algorithms and to suggest new solutions to improve on their shortcomings (such as the computational cost, the pixel precision, and so on).

The paper is organised as follows: Section 2 gives an overview of correlation-based techniques discussing advantages and drawbacks, Section 3 introduces the distance or similarity measures we applied to our algorithms. Section 4 discusses the matching error due to high frequencies and search space quantisation. Section 5 introduces several techniques in order to have the best results in matching, reducing the complexity and increasing the accuracy obtaining also a sub-pixel motion estimation. Section 6 presents the experimental results, with comparisons of algorithms on well-known test image sequences.

Section snippets

Overview: advantages and drawbacks

Correlation-based methods are based on the analysis of the gray level pattern around the point of interest and on the search for the most similar pattern in the successive image. In a few words, having defined a window $W(x →)$ around the point $x →,$ we consider similar windows W′(x+i,y+j) shifted by the possible integer values in pixels in a search space S composed by the i, j such as −Δ<i<Δ and −Δ<j<Δ. The optical flow, i.e. the estimated image displacement is taken as the shift corresponding to

Distance–similarity measures

Many ways of measuring difference or similarity between the gray-level pattern can be used. In our work we compare squared windows of N×N and compute motion between a window centred in (x,y) in the image I₁ and a window shifted by (i,j) in the image I₂. The most used distance measures are reported in Table 1. The widely used sum of absolute differences (SAD) and SSD can be modified to consider the effect of global gray-level variations, setting the average gray level difference equal to 0

High frequencies and quantisation

Even if correlation-based techniques are not affected by the aliasing problem as differential ones, signal quantisation introduces error in flow computation due to high frequencies. If the frequency of the signal has the same order of magnitude as the sampling frequency and the displacements to be computed are not exactly integer (i.e. a multiple of the sampling step), correlation may lead to completely wrong results as well. Let us show it with a simple example: we consider a 1D sinusoidal

Improving the method

The previous sections have shown several features of correlation algorithms also showing some problems of the method; in this section we will introduce several algorithms to solve these problems and to improve the algorithm performances.

Experimental results

To test the algorithms, we computed optical flows on synthetic or calibrated image sequences with the true displacements known at every pixel location. We measured the average differences between the computed flow $v →$ and the true motion $v → ′$ using the angular distance introduced by Barron et al. [2]: $dist (v →, v → ′)= arccos uu′+vv′+1 (| v → |^{2} +1)(| v → ′|^{2} +1)$

Discussion

The use of techniques based on pattern matching performed on image regions at different times is common in practical applications, but a few algorithms of this kind are considered in the literature reviews [2]. In this paper several variations of this kind of algorithms and some tricks to reduce the major drawbacks of the method (i.e. the computational complexity and the integer values of the estimates) are presented. All the solutions proposed have been tested on the test images used by the

Acknowledgements

Special thanks to Pietro Parodi for reading the manuscript and useful hints and to Marco Cappello and Marco Campani for technial help. Thanks also to Vincent Torre, Alessandro Verri for useful discussions.

References (19)

B.K.P. Horn et al.
Determining optical flow
Artificial Intelligence
(1981)
H. Nagel
Displacement vectors derived from 2nd order intensity variations in image sequences
Comput. Vision Graph. Image Process
(1983)
W. Enkelmann
Obstacle detection by evaluation of optical flow fields from image sequences
IVC
(1991)
A. Giachetti
On line analysis of echocardiographic images sequences
Med. Image Anal.
(1998)
P. Anadan
computational framework and an algorithm for the measurement of visual motion
Int. J. Comput. Vision
(1989)
J.L. Barron et al.
Performance of optical flow techniques Int
Int. J. Comput. Vision
(1994)
B. Lucas et al.
An interative image registration technique with an application to stereo vision
Proc. DARPA Image Und. Workshop
(1981)
R. Singh, An estimation-theoric framework for image flow computation, Proc. Third ICCV, Osaka, 1990, pp....
N. Ancona et al.
Optical Flow from 1D Correlation: application to a simple time-to-crash detector
Int. J. Comp. Vision
(1995)

There are more references available in the full text version of this article.

Cited by (194)

Early-age shrinkage assessment of cementitious materials: A critical review
2024, Cement and Concrete Composites
With the advent of formwork-free construction technologies like 3D concrete printing, evaluating early-age shrinkage in cementitious materials has become more critical. This paper reviews different test methods for assessing early-age shrinkage. These methods are classified as contact-based, which includes measurement systems that must remain in physical contact with the test specimen or embedded sensors placed inside the fresh concrete, and non-contact-based, which includes optical-based approaches either using white light (digital image correlation) or laser beams. The working principles, advantages, and challenges of the techniques, particularly while handling fresh concrete are discussed. The different mechanisms contributing to early-age shrinkage are reviewed and the common shrinkage mitigation strategies are also pointed out.
Measuring landslide-driven ground displacements with high-resolution surface models and optical flow
2023, Computers and Geosciences
This study proposes and evaluates the performance of two image matching algorithms applied to hillshades derived from consecutive high-resolution digital surface models (DSMs) to measure surface displacements on landslides. The method is applied on Te Horo, a slow-moving landslide in the Dart Valley of New Zealand’s Southern Alps/Kā Tiritiri o te Moana using pairs of hillshades derived from Airborne and Satellite Photogrammetric Mapping (APM/SPM) of imagery captured in 2018 and 2020. This novel approach uses the consistency of displacement predictions generated from multiple hillshade pairs to gauge match quality and mask unreliable displacement predictions. The study compares a widely used normalised cross-correlation algorithm (NCC) alongside an optical flow approach for image matching. The performances of both algorithms are assessed against manually derived displacements of prominent surface features. We demonstrate the effectiveness of the masking approach as well as the good performance of the optical flow algorithm in delivering dense, accurate displacement measurements efficiently, particularly when high resolution DSMs are available. The results show that the main translational body of the landslide was displaced at rates averaging 25–55 mm day⁻¹ over the 2018–2020 period.
Interactive Structural Topology Optimization with Subjective Scoring and Drawing Systems
2023, CAD Computer Aided Design
Topology optimization techniques can create efficient and innovative structural designs by redistributing underutilized materials to the most-needed locations. These techniques are typically performed based purely on structural performance without considering factors like aesthetics and other design requirements. Hence, the obtained structural designs may not be suitable for specific practical applications. This study presents a new topology optimization method, SP-BESO, by considering the subjective preferences (SP) of the designer. Here, subjective scoring and drawing systems are introduced into the popular bi-directional evolutionary structural optimization (BESO) technique. The proposed SP-BESO method allows users to iteratively and interactively create topologically different and structurally efficient solutions by explicitly scoring and drawing their subjective preferences. Hence, users do not need to passively accept the optimization results. A user-friendly digital design tool, iBESO, is developed, which contains four optimizers to simultaneously perform the proposed SP-BESO method to assist in the design exploration task. A variety of 2D examples are tested using the iBESO software to demonstrate the effectiveness of the proposed SP-BESO method. It is found that the combination of parameters used in the scoring and drawing systems controls the formation of final structural topologies toward performance-driven or preference-driven designs. The utilization of the proposed SP-BESO method in potential practical applications is also demonstrated.
Multi-path decoder U-Net: A weakly trained real-time segmentation network for object detection and localization in ultrasound scans
2023, Computerized Medical Imaging and Graphics
Detecting and localizing an anatomical structure of interest within the field of view of an ultrasound scan is an essential step in many diagnostic and therapeutic procedures. However, ultrasound scans suffer from high levels of variabilities across sonographers and patients, making it challenging for sonographers to accurately identify and locate these structures without extensive experience. Segmentation-based convolutional neural networks (CNNs) have been proposed as a solution to assist sonographers in this task. Despite their accuracy, these networks require pixel-wise annotations for training; an expensive and labor-intensive operation that requires the expertise of an experienced practitioner to identify the precise outline of the structures of interest. This complicates, delays, and increases the cost of network training and deployment. To address this problem, we propose a multi-path decoder U-Net architecture that is trained on bounding box segmentation maps; not requiring pixel-wise annotations. We show that the network can be trained on small training sets, which is the case in medical imaging datasets; reducing the cost and time needed for deployment and use in clinical settings. The multi-path decoder design allows for better training of deeper layers and earlier attention to the target anatomical structures of interest. This architecture offers up to a 7% relative improvement compared to the U-Net architecture in localization and detection performance, with an increase of only 0.75% in the number of parameters. Its performance is on par with, or slightly better than, the more computationally expensive U-Net++, which has 20% more parameters; making the proposed architecture a more computationally efficient alternative for real-time object detection and localization in ultrasound scans.
DIC-Net: Upgrade the performance of traditional DIC with Hermite dataset and convolution neural network
2023, Optics and Lasers in Engineering
Digital image correlation (DIC) is a non-contact optical method that tracks the speckle pattern on specimen surface to calculate the displacement and strain by image correlation algorithm. Although the traditional DIC method can conveniently measure surface deformation, it still has many limitations: (1) the accuracy of displacement and strain calculation needs to be improved in the case of high deformation gradient; (2) under match or over-match can hardly be avoided when the filters are used to reconstruct smooth displacement or strain field, and (3) boundary effect remains unresolved in computing the deformation near the boundary of region of interest or the discontinuous area (e.g. area near crack tip or crack face). Recently, the deep learning based DIC (Deep-DIC) has revealed its attractive ability in handling above issues in traditional DIC, and impressive results have been achieved. The mean value of the absolute error (MAE) on the test set has been optimized to 0.0361 pixels using existing Deep-DIC approaches, which are accompanied by a real-time measurement speed. The network structure and training dataset are two key factors for the deep learning method. However, the current working networks have been modified from other image tasks and cannot fully adapt to the demands of the DIC tasks, and the dataset they generated still has evident flaws, limiting the method's accuracy and generalization performance which is utilized to assess performance on samples outside the training set. In this paper, we firstly proposed a new Hermite dataset that is created by using the high-order Hermite element to take account more complex deformation, then a new network architecture designed for the DIC task has been developed to extract richer deformation features. A test set of 3216 examples containing six different modes of displacement is used to compare the performance of our network with others. The proposed DIC-Net-d achieves the lowest MAE in the test set. Meanwhile, in the Star5 image sets from DIC-Challenge, the proposed DIC-Net-d achieves a spatial resolution of 17.25 pixels and a noise level of 0.0136 which outperforms existing traditional and non-traditional methods. Finally, the strain network trained by our Hermite dataset is also successful in predicting the strain field of Star6 in the DIC challenge. The experiment results show the superiority of the proposed Hermite dataset and new network with respect to other Deep-DIC methods.
A state-of-the-art review of mechanical characteristics and cracking processes of pre-cracked rocks under quasi-static compression
2022, Journal of Rock Mechanics and Geotechnical Engineering
Citation Excerpt :
These subsets contain many pixels which can be used for distinguishing themselves from others (Pan et al., 2009). To determine the correlation degree of subsets between reference and deformed images, the correlation criteria were developed (Grün, 1985; Giachetti, 2000; Pan et al., 2010). When the position with the maximum correlation coefficient was determined, the relationship between the reference and deformed images was identified, and the full-field displacement and strain could be subsequently calculated (Blaber et al., 2015).
The mechanical characteristics and failure behavior of rocks containing flaws or discontinuities have received wide attention in the field of rock mechanics. When external loads are applied to rock materials, stress-induced cracks would initiate and propagate from the flaws, ultimately leading to the irreversible failure of rocks. To investigate the cracking behavior and the effect of flaw geometries on the mechanical properties of rock materials, a series of samples containing one, two and multiple flaws have been widely investigated in the laboratory. In this paper, the experimental results for pre-cracked rocks under quasi-static compression were systematically reviewed. The progressive failure process of intact rocks is briefly described to reveal the background for experiments on samples with flaws. Then, the nondestructive measurement techniques utilized in experiments, such as acoustic emission (AE), X-ray computed tomography (CT), and digital image correlation (DIC), are summarized. The mechanical characteristics of rocks with different flaw geometries and under different loading conditions, including the geometry of pre-existing flaws, flaw filling condition and confining pressure, are discussed. Furthermore, the cracking process is evaluated from the perspective of crack initiation, coalescence, and failure patterns.

View all citing articles on Scopus

View full text

Matching techniques to compute image motion

Abstract

Introduction

Section snippets

Overview: advantages and drawbacks

Distance–similarity measures

High frequencies and quantisation

Improving the method

Experimental results

Discussion

Acknowledgements

Artificial Intelligence

Comput. Vision Graph. Image Process

IVC

Med. Image Anal.

computational framework and an algorithm for the measurement of visual motion

Int. J. Comput. Vision

Performance of optical flow techniques Int

Int. J. Comput. Vision

An interative image registration technique with an application to stereo vision

Proc. DARPA Image Und. Workshop

Optical Flow from 1D Correlation: application to a simple time-to-crash detector

Int. J. Comp. Vision