Review articleSoft computing strategy for stereo matching of multi spectral urban very high resolution IKONOS images
Graphical abstract
Highlight
► Fuzzy thresholding proposed method realizes at the same time both segmentation and thresholding. ► It allows good results of buildings extraction step without requiring a high solution cost or other technological resources. ► Hopfield neural stereo matching method is based on new constraints, it is initialized by a classical matching technic. ► Hopfield neural stereo matching proposed method improves matching rate and decrease ambiguities. ► All proposed strategy exploits soft computing proprieties to achieve simplicity, good results and low solution cost.
Introduction
In the last two decades, the stereo matching issue has attracted a lot of researches, and several approaches have been proposed [1], [2], [3], [4], [5], [6].
These approaches can be broadly classified into two categories: the area-based matching techniques and the feature-based matching techniques. In the first category, the matching process is applied directly to the intensity profiles of the two images, it allows an overall dense restitution but it manages with difficulties the discontinuities and homogeneous zones, the absence of semantic information harms a good management of coherence and scene structure [7]. In the second category, features are first extracted from the images and the matching process is applied to the features like edge elements [8], [9], the line segments [10], [11], [12] and the homogeneous regions [13], [14], [15], [16].
In this paper, we are interested by “buildings” which constitute a basic part of urban landscapes. They are keys elements of urban morphological analysis, so, buildings stereo matching supposes that the second category of approaches is the most appropriate in our application [17]. We choose “region” as a primitive because many of the shortcomings inherent in approaches based on points or lines can be overcome by taking more developed entities [18], [19], [20], [21], [22], [23]. The higher dimensionality of regions makes them richer of target object's geometric properties such as shape and size, and photometric properties such as colour. The higher dimensional character of regions also makes their matching more stable to small illumination and viewpoint changes across given images [15].
However, “a building” is so variable and it is not the only object in our pairs of high resolution multi spectral Ikonos images which include cars, roads, vegetation, etc. Also, shapes and colours of buildings can be close or different in the same image, from right to left image or from one pair of images to another which complicate more and more their extraction. Buildings extraction must be reliable, especially if we have to do other treatment after it, like stereo matching which depends on the quality of extraction results.
There are a lot of building extraction techniques applied to aerial or satellite images like [24] whom used laser remote sensing data to develop a method based on the standard deviation to distinguish between trees and buildings using the height variation at the periphery of the objects present in the data. Sohn and Dowman extracted buildings tracks automatically from a combination of the Ikonos imagery with pan-sharpened multi spectral bands and lidar data [25]. Lafarge and al presented an automatic buildings extraction method that involved digital elevation models based on an object approach. Using this method, a rough approximation of all relevant building footprints was first calculated from marked point processes. The resulting rectangular footprints were then normalized by improving the connections between neighboring rectangles and detecting any roof height discontinuities [26].
All these techniques cited above require a high computational effort or need other technological resources like digital elevation model, Lidar or Laser data, etc. To overcome these difficulties and in order to realize both buildings extraction and stereo matching, we propose in this paper a soft computing strategy able to exploit the given tolerance of imprecision, partial truth, and to achieve tractability, robustness and low solution cost.
Concerning buildings extraction step, we have to detect these last as only interesting regions in order to match them, so this process can be viewed as “color image thresholding problem”, for that, we propose an algorithm based on fuzzy logic (fuzzy clustering method) having the particularity to realize automatically at the same time both segmentation and thresholding. Compared to other tresholding techniques like: global thresholding (otsu method) and K-means thresholding, a proposed soft computing technique gives best results.
For stereo matching step, we choose a feature-based matching approach. It exists many optimisation techniques which allows to find homologous couples using soft computing, we mention for example relaxation method used by Brockers [27] and by Sidib [28], genetic algorithm used by Goulermas and Liatsis [29] and Hopfield neural network used by Jan Jae Lee to put in correspondence points [30], by Nasrabadi to put in correspondence characteristics points [31], by Nichari who uses as primitive the edge points [32] and by pajares and al which identified edge segments as features [33].
Generally, all these works mentioned above require some constraints to guide stereo matching process such as: the similarity, the continuity, the order and the epipolar constraint, etc. The implementation of these constraints is not always very easy, in particular the epipolar one which requires the knowledge of the intrinsic and extrinsic parameters of the acquisition system [34]. As we do not have these parameters and in order to overcome these restrictions, we are inspired by similarity constraint to propose in this work Hopfiel neural stereo matching technique using new constraints including geometric and photometric regions properties: surface, elongation, perimeter, colour and gravity center coordinates. This network will be initialized by simple method which we called classical matching technique.
Section snippets
Principle
In our stereo matching application, it is necessary to extract buildings before putting them in correspondence, for that, we apply thresholding process, this one can be seen as the simplest form of segmentation or more general as a two class clustering procedure. Because the importance of this process, scientific community has proposed a lot of methods and technics of image thresholding [35], however, there is no single method that can be considered “good” for all images [36], nor are all
Selection of possible candidate for stereo matching
After buildings extraction step applied to right and left images, we carry out stereo matching step in order to find homologous regions, however, it is a difficult search procedure, so, to reduce false matches, some matching constraints must be imposed. In the present work, we consider the new constraints that include geometric and photometric regions properties such as surface, elongation, perimeter, average of colour and gravity center position criterion. Soft computing technique is used for
Stereo multi spectral IKONOS images
The pair of stereo sample images is generated by IKONS 2 satellite, we obtain them from Space imaging company via internet, there are 1 meter multispectral images composed by three bands RGB (reed, green and blue), coded by 8 bits per pixel per band, size of each image is 2001 × 2001 pixels (Fig. 2). They have a ratio base/height about 0.53.
We have only one pair of stereo images which contains various real world landscapes (urban, suburban, rural, etc.) and we are interested by buildings stereo
Conclusion
In this paper, we present a fast and effective soft computing strategy for stereo matching of multi spectral urban very high resolution Ikonos pairs of images. We are interested by buildings, for that, we apply at the first step a fuzzy thresholding algorithm for automatically extracting buildings from pairs of images. Based on fuzzy clustering method, we proposed an unsupervised iterative algorithm which needs only a knowledge of class number, it has the particularity to realize both
References (41)
- et al.
Robust approach for disparity estimation in stereo vision
Image and Vision Computing
(2004) - et al.
Incremental reconstruction of 3D scenes from multiple, complex images
Artificial intelligence
(1986) - et al.
A new region matching for color stereo images
Pattern Recognition Letters
(2007) - et al.
A new region matching method for stereoscopic images
Pattern Recognition Letters
(2000) - et al.
Automatic building extraction from laser scanning data: an input tool for disaster management
Advances in Space Research
(2004) - et al.
Data fusion of high-resolution satellite imagery and LiDAR data for automatic building extraction
ISPRS Journal of Photogrammetry & Remote Sensing
(2007) - et al.
Automatic building extraction from DEMs using an object approach and application to the 3D-city modeling
ISPRS Journal of Photogrammetry & Remote Sensing
(2008) - et al.
Stereo correspondence using the Hopfield neural network of a new energy function
Pattern Recognition
(1994) - et al.
Relaxation by Hopfield network in stereo image matching
Pattern Recognition
(1998) - et al.
A review on image segmentation techniques
Pattern Recognition
(1993)
Epipolar plane image analysis: an approach to determining structure from motion
International Journal of Computer Vision
Computing visual correspondence with occlusions using graph cuts
Stereo by intra and inter-scanline search using dynamic programming
IEEE Transactions on Pattern Analysis and Machine Intelligence
Stereo matching with nonlinear diffusion
International Journal of Computer Vision
A taxonomy and evaluation of dense two-frame stereo correspondence algorithms
International Journal of Computer Vision
Matching and retrieval of distorted and occluded shapes using dynamic programming
IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI
Reconnaissance d’objets volumiques par mise en correspondance d’indices visuels
Traitement du signal TS
A global matching framework for stereo computation
Cited by (6)
New pseudo-CT generation approach from magnetic resonance imaging using a local texture descriptor
2018, Journal of Biomedical Physics and EngineeringNew neural buildings stereo matching method applied to very high resolution ikonos images
2018, Computer Vision: Concepts, Methodologies, Tools, and ApplicationsRemoving Shadows Using RGB Color Space in Pairs of Optical Satellite Images
2017, Journal of the Indian Society of Remote SensingNew shadow detection and removal approach to improve neural stereo correspondence of dense urban VHR remote sensing images
2016, European Journal of Remote SensingStereo Matching Based on Immune Neural Network in Abdomen Reconstruction
2015, Mathematical Problems in EngineeringNew neural buildings stereo matching method applied to very high resolution Ikonos images
2014, Handbook of Research on Artificial Intelligence Techniques and Algorithms