Journal of Visual Communication and Image Representation
Image fusion based on visual salient features and the cross-contrast☆
Introduction
Image fusion is an active research area in optical signal processing, the objective of image fusion is to combine useful information from several images of the same picture or scene [1]. Therefore, multiple different images of one same scene may be acquired by different image sensors under different optic conditions or at different times to integrate different data so as to obtain more information [2]. Because the fused image contains the main features of several images which captured by different sensors, the target object in the same scene can be observed and distinguished more clearly, more comprehensively, more reliably. Now, as an important image analysis and computer vision technology, image fusion has widely applied to target recognition, computer vision, remote sensing, robot, medical image processing, military application, etc. Meanwhile, image fusion can provide more effective information for further computer image processing, such as high efficiency video processing, image classification, image segmentation, object recognition and detection [3], [4], [5], [6], [7], [8], [9].
In recent years, many effective image fusion methods have been proposed, such as the method based on Multi-scale transform (MST) [10], the method based on ICA or PCA [11], the method based on neural networks [12], the method based on SIFT [13] and the method based on morphological component [14]. Multi-scale transform (MST)-based fusion methods are the most popular and important tools in image processing, which are also effectively used for image fusion. There are many classical MST-based fusion methods such as pyramid-based ones, wavelet-based ones and multi-scale geometric analysis (MGA)-based ones. Pyramid-based ones include Laplacian pyramid (LP) [15], [16], ratio of low-pass pyramid (RP) [17] and gradient pyramid (GP) [18], [19]. The wavelet-based ones include discrete wavelet transform (DWT) [10], [20], stationary wavelet transform (SWT) [21], [22], [23], [24] and dual-tree complex wavelet transform(DTCWT) [25]. The multi-scale geometric analysis (MGA)-based ones include curvelet transform (CVT) [26], [27], ridgelet transform [28], nonsubsampled contourlet transform (NSCT) [29], [30], [31] and nonsubsampled shearlet transformation(NSST) [32], [33], [34]. In general, the MST-based fusion methods consist of the following three steps [35], [36]. First, the original images are decomposed into a multi-scale transform domain. Secondly, the transformed coefficients are merged with a given fusion rule. Finally, the fused image is reconstructed by performing the corresponding inverse transform over the merged coefficients. Therefore, it’s obvious that the fusion rule of high-pass and low-pass subband image plays a crucial role for the result of image fusion. Moreover, transform domain also has a great impact on the fused results.
Human is primarily dependent on visual sense to obtain information from the outside world. The studies of human visual system (Human Visual system, HVS) have shown that during observating and understanding a image, HVS is usually more concerned about salient features of the image [37], [38], [39]. Some analysis methods based on visual saliency also have been proposed to quickly detect salient area or targets in an image [40], [41], [42], [43]. In this paper, three feature maps will be constructed based on visual saliency which are the local energy, the contrast and the gradient respectively, and low frequency subband coefficients are fused utilizing these visual feature salient maps. Then, a cross-contrast fusion method is used to get bandpass directional subband coefficients, and the cross-contrast represents the ratio between the local gray mean of the bandpass directional subband coefficients and the local gray mean of the fused low frequency subband coefficients. A comparative study of different MST-based methods is reported in [44], where Li et al. found that the NSCT-based method can generally achieve the best results. Therefore, in this paper, NSCT has been selected as MST-based fusion method. This paper is organized as follows. The following section briefly explains the principle of NSCT, and the Section 3 introduces image fusion based on nonsubsampled contourlet transform. the Section 4 introduces image fusion algorithm based on visual salient features and the cross-contrast. In Section 5, the results and analysis of experiments are presented. Finally, our conclusions are given in Section 6. Further, for brevity, in the subsequent part of this paper we use the abbreviation LFS and BDS, and define low frequency subband coefficients as LFS coefficients and bandpass directional subband coefficients as BDS coefficients
Section snippets
Non-subsampled contourlet transform
The tools of multiscale geometric analysis have been broadly used in image fusion. Nowadays, wavelet transform is an efficient tool to express the one-dimensional (1-D) piecewise smooth signals, but in the case of two-dimensional (2-D) signals, it cannot efficiently preserve edges of a nature image. In addition, separable wavelets are deficient in capturing only limited directional information and feature of multi-dimensional signals.
To overcome the drawbacks of wavelet in dealing with higher
The image fusion based on nonsubsampled contourlet transform
Based on the above theory, NSCT can effectively be applied to image fusion. The image fusion based on nonsubsampled contourlet transform is usually done by the following steps [48].
Fusion algorithm based on visual salient features and the cross-contrast
In this section, the proposed algorithm in this paper will be discussed in detail. A fused image f is assumed to be generated from a pair of original images f1 and f2 that have already been registered perfectly. In image fusion based on NSCT, rules of fusion play a decisive role for quality of the fused image. In this paper, LFS coefficients are selected based on visual salient features, and for the selection of BDS coefficients, a cross-contrast method is used. The schematic diagram of the
Experimental results and analysis
To verify the effective performance of the proposed method, multifocus images and Visible-infrared images from different applications are used in this paper. For comparison purposes, some other fusion methods is also selected to perform fusion such as the DWT-based method, NMF-based method and the NSCT-based method, in all of which lowpass subband coefficients and bandpass subband coefficients are merged by the averaging scheme and the absolute maximum choosing scheme respectively. For
Conclusion
For MST-based image fusion method, the fusion rule of high-pass and low-pass subband coefficients plays a crucial role for the result of image fusion. Moreover, transform domain also has a great impact on the fused results. In this paper, to extract and combine the features of the original images, a novel algorithm based on visual salient features and the cross-contrast is proposed for multi-scale transform. Decomposition and reconstruction of the multiscale image and the fusion rule are the
Acknowledgments
We sincerely thank the reviewers and editors for their carefully checking our manuscript and providing constructive suggestions. Project (KYTZ201322) Supported by the Scientific Research Foundation of CUIT. We have benefited from the images supplied by TNO Human Factors Research Institute in the Netherlands.
References (49)
- et al.
Multi-focus image fusion using PCNN
Pattern Recogn.
(2010) - et al.
Multi-focus image fusion with dense SIFT
Information Fusion
(2015) - et al.
Image fusion with morphological component analysis
Information Fusion
(2014) Image fusion by a ratio of low-pass pyramid
Pattern Recogn. Lett.
(1989)- et al.
Multi-focus image fusion using a bilateral gradient-based sharpness criterion
Opt. Commun.
(2011) - et al.
Multifocus image fusion scheme using focused region detection and multiresolution
Opt. Commun.
(2011) - et al.
Remote sensing image fusion using the curvelet transform
Information Fusion
(2007) - et al.
Multifocus image fusion using the nonsubsampled contourlet transform
Signal Processing
(2009) - et al.
A novel algorithm of image fusion using shearlets
Opt. Commun.
(2011) A general framework for multiresolution image fusion: from pixels to regions
Information Fusion
(2003)
A general framework for image fusion based on multi-scale transform and sparse representation
Information Fusion
Key frame extraction based on visual attention model
J. Vis. Commun. Image Represent.
Axiomatic approach to computational attention
Pattern Recogn.
Performance comparison of different multi-resolution transforms for image fusion
Information Fusion
Image fusion through feature extraction by using sequentially combined toggle and top-hat based contrast operator
Appl. Opt.
Merging thermal and visual images by a contrast pyramid
Opt. Eng.
Multisource classification of remotely sensed data: fusion of Landsat TM and SAR images
IEEE Trans. Geosci. Remote Sens.
A novel image fusion framework for night-vision navigation and surveillance
SIViP
Unsupervised image segmentation and image fusion for multi-beam/multi-aspect sidescan sonar images
Oceans
Feature-based image fusion scheme for satellite recognition
Information Fusion
A highly parallel framework for HEVC coding unit partitioning tree decision on many-core processors
IEEE Signal Process. Lett.
Efficient parallel framework for HEVC motion estimation on many-core processors
IEEE Trans. Circuits Syst. Video Technol.
Cited by (18)
An improved approach for medical image fusion using sparse representation and Siamese convolutional neural network
2022, Biomedical Signal Processing and ControlCitation Excerpt :The multiscale transform (MST) fusion technique emerged recently and is employed for computer vision and image processing areas. Moreover, there are various transforms based fusion images, for instance a discrete wavelet (DWT)-based [7], complex wavelet transform (CWT)-based [8], dual-tree complex wavelet transform (DTCWT)-based [9], the non-subsampled contourlet transform (NSCT)-based method [10], curvelet transform (CVT)-based [11], contourlet transform based [12], dense SIFT-based [13], shear-let transform-based [14], singular value decomposition (SVD) [15], and sparse representation based techniques [16]. However, the shortcomings of these methods are always observed in the design of the transform basis, namely activity level measurement and fusion rule.
On the use of joint sparse representation for image fusion quality evaluation and analysis
2019, Journal of Visual Communication and Image RepresentationCitation Excerpt :Instead, transform domain methods fuse source images in various transform decomposition domains, this can avoid the spatial discontinuities resulting by spatial domain methods effectively [3,4]. A variety of Multi-Scale Transform (MST) tools, including various pyramid decompositions and smooth filters [5], various discrete wavelet transform (DWT) [6] and stationary wavelet transform[7], Dense SIFT [8], shearlet transform[9]and non-subsampled contourlet transform (NSCT) [10,11], singular value decomposition (SVD) [12], etc., have been employed. But selecting an optimal transform basis for different fusion scenes and applications is still a demanding task because each transform has its own limitations corresponding to context of input images [1,13].
Infrared and visible image fusion methods and applications: A survey
2019, Information FusionCitation Excerpt :The feature-level image fusion methods usually extract the features of source images at first, and then obtain the fused image based on the extracted features for some specific purposes, which have the advantages of designing more intelligent semantic fusion rules based on the extracted features compared with pixel-level fusion strategies [210]. Therefore, feature-level fusion methods have also been adopted for infrared and visible image fusion [72,191,207,209,211,212]. For example, Li et al. proposed a fusion scheme based on nonsubsampled contourlet transform and low-level visual features [72].
An optimized non-subsampled shearlet transform-based image fusion using Hessian features and unsharp masking
2018, Journal of Visual Communication and Image RepresentationA unified image fusion framework with flexible bilevel paradigm integration
2023, Visual ComputerRemote sensing imaging analysis and ubiquitous cloud-based mobile edge computing based intelligent forecast of forest tourism demand
2023, Distributed and Parallel Databases
- ☆
This paper has been recommended for acceptance by M.T. Sun.