Image fusion based on visual salient features and the cross-contrast

Highlights

• Low frequency subband coefficients are selected based on visual salient features.

• Bandpass directional subband coefficients are selected by the cross-contrast.

• Three maps of visual salient features are constructed based on visual saliency.

Abstract

To extract and combine the features of the original images, a novel algorithm based on visual salient features and the cross-contrast is proposed in this paper. Original images were decomposed into low frequency subband coefficients and bandpass direction subband coefficients by using the nonsubsampled contourlet transform. Three maps of visual salient features are constructed based on visual salient features the local energy, the contrast and the gradient respectively, and low-frequency subband coefficients are got by utilizing these visual saliency maps. The cross-contrast is obtained by computing the ratio between the local gray mean of bandpass direction subband coefficients and the local gray mean of fused low-frequency subband coefficients. Bandpass direction subband coefficients is goted by the cross-contrast. Comparison experiments have been performed on different image sets, and experimental results demonstrate that the proposed method performs better in both subjective and objective qualities.

Introduction

Image fusion is an active research area in optical signal processing, the objective of image fusion is to combine useful information from several images of the same picture or scene [1]. Therefore, multiple different images of one same scene may be acquired by different image sensors under different optic conditions or at different times to integrate different data so as to obtain more information [2]. Because the fused image contains the main features of several images which captured by different sensors, the target object in the same scene can be observed and distinguished more clearly, more comprehensively, more reliably. Now, as an important image analysis and computer vision technology, image fusion has widely applied to target recognition, computer vision, remote sensing, robot, medical image processing, military application, etc. Meanwhile, image fusion can provide more effective information for further computer image processing, such as high efficiency video processing, image classification, image segmentation, object recognition and detection [3], [4], [5], [6], [7], [8], [9].

In recent years, many effective image fusion methods have been proposed, such as the method based on Multi-scale transform (MST) [10], the method based on ICA or PCA [11], the method based on neural networks [12], the method based on SIFT [13] and the method based on morphological component [14]. Multi-scale transform (MST)-based fusion methods are the most popular and important tools in image processing, which are also effectively used for image fusion. There are many classical MST-based fusion methods such as pyramid-based ones, wavelet-based ones and multi-scale geometric analysis (MGA)-based ones. Pyramid-based ones include Laplacian pyramid (LP) [15], [16], ratio of low-pass pyramid (RP) [17] and gradient pyramid (GP) [18], [19]. The wavelet-based ones include discrete wavelet transform (DWT) [10], [20], stationary wavelet transform (SWT) [21], [22], [23], [24] and dual-tree complex wavelet transform(DTCWT) [25]. The multi-scale geometric analysis (MGA)-based ones include curvelet transform (CVT) [26], [27], ridgelet transform [28], nonsubsampled contourlet transform (NSCT) [29], [30], [31] and nonsubsampled shearlet transformation(NSST) [32], [33], [34]. In general, the MST-based fusion methods consist of the following three steps [35], [36]. First, the original images are decomposed into a multi-scale transform domain. Secondly, the transformed coefficients are merged with a given fusion rule. Finally, the fused image is reconstructed by performing the corresponding inverse transform over the merged coefficients. Therefore, it’s obvious that the fusion rule of high-pass and low-pass subband image plays a crucial role for the result of image fusion. Moreover, transform domain also has a great impact on the fused results.

Human is primarily dependent on visual sense to obtain information from the outside world. The studies of human visual system (Human Visual system, HVS) have shown that during observating and understanding a image, HVS is usually more concerned about salient features of the image [37], [38], [39]. Some analysis methods based on visual saliency also have been proposed to quickly detect salient area or targets in an image [40], [41], [42], [43]. In this paper, three feature maps will be constructed based on visual saliency which are the local energy, the contrast and the gradient respectively, and low frequency subband coefficients are fused utilizing these visual feature salient maps. Then, a cross-contrast fusion method is used to get bandpass directional subband coefficients, and the cross-contrast represents the ratio between the local gray mean of the bandpass directional subband coefficients and the local gray mean of the fused low frequency subband coefficients. A comparative study of different MST-based methods is reported in [44], where Li et al. found that the NSCT-based method can generally achieve the best results. Therefore, in this paper, NSCT has been selected as MST-based fusion method. This paper is organized as follows. The following section briefly explains the principle of NSCT, and the Section 3 introduces image fusion based on nonsubsampled contourlet transform. the Section 4 introduces image fusion algorithm based on visual salient features and the cross-contrast. In Section 5, the results and analysis of experiments are presented. Finally, our conclusions are given in Section 6. Further, for brevity, in the subsequent part of this paper we use the abbreviation LFS and BDS, and define low frequency subband coefficients as LFS coefficients and bandpass directional subband coefficients as BDS coefficients