A new metric based on extended spatial frequency and its application to DWT based fusion algorithms
Introduction
Image fusion is a tool that serves to combine multiple-source imagery by using advanced image processing techniques. Specifically, it aims for the integration of disparate and complementary data in order to enhance the information apparent in the respective source images, as well as to increase the reliability of interpretation. This leads to more accurate data [1] and increased utility [2], [3]. In addition, it has been stated that fused data provides for more robust aspects of operational performance such as increased confidence, reduced ambiguity, improved reliability and improved classification [2], [4].
This paper focuses on the so-called “pixel-level” fusion process, where a composite image has to be built of several (typically two) input images. A general framework of image fusion can be found in [5]. In pixel-level image fusion, some general requirements [6] are imposed on the fusion result: (1) the fusion process should preserve all relevant information of the input imagery in the composite image (pattern conservation); and (2) the fusion scheme should not introduce any artifacts or inconsistencies which would distract the human observer or subsequent processing stages. Consequently, quantitative evaluation of the quality of fused imagery is considered very important for an objective comparison of the performance of the respective fusion algorithms. In addition, a quantitative metric may potentially be used as feedback to the fusion algorithm to further improve the fused image quality.
Through the practical applications of image fusion in medical imaging, remote sensing, nighttime operations and multi-spectral imaging, many fusion algorithms have been developed. Two common fusion methods are the discrete wavelet transform (DWT) [7], [8], [9], [10] and various pyramids (such as Laplacian, contrast, gradient and morphological pyramids) [11], [12]. However, only a few metrics are available for quantitative evaluation of the quality of fused imagery. For example, the root mean square error (RMSE) is the natural measure of image quality if there is a “ground truth” image available; however, for realistic image fusion problems there are no ground truths. Beauchemin et al. [13] presented a method using local variance for image fusion assessment that still requires a comparison with the measurement of ground truth. Leung et al. [14] proposed the image noise index (INI), based on entropy calculation, to measure fused image quality; this method requires the exact reverse process of an image fusion procedure, which is impractical for most fusion processes such as DWT or pyramid methods. Piella and Heijmans [15] recently presented a new metric for image fusion—the image quality index (IQI), which measures how similar the fused image is to both input images. The values of IQI are within [0, 1] and have an ideal value of 1 (if two input images are identical). The IQI metric has been shown to be consistent with other methods for evaluating fused image quality [16]. However, the IQI does not give the error direction, which would indicate whether the fused image is under- or over-fused.
An image measure termed “spatial frequency” (SF) [20] can indicate how much information is contained in an image and thus may be used as a fusion rule (determining which input should be selected in the fused image) [17], but it cannot directly be used to measure the fused image quality. From the definition of SF (given in Section 2.3), we know that the SF metric is sensitive to gray level changes, although it cannot distinguish useful information from noise or artifacts. Thus, to use the spatial frequency value as a fusion metric, a standard or reference value computed from the input images must be constructed for the purpose of comparison. The new image quality metric presented in this paper, termed as “the ratio of SF error (rSFe)”, is a relative measurement regardless of the type of image to be analyzed. The accuracy of the rSFe metric can be verified with the currently used measurements of RMSE and IQI. In addition, the rSFe value can be further back propagated to the fusion algorithm (BP fusion) such that the fusion parameter adjustment is directed to perform the next loop of fusion, a process that is repeated until an optimized fusion result is achieved.
An advanced wavelet transform (aDWT) method that incorporates principal component analysis (PCA) and morphological processing into a regular DWT fusion algorithm was recently presented [16]. Furthermore, in [16], experimental results showed an important relationship between the fused image quality and the wavelet properties, that is, a higher level of DWT decomposition (with smaller image resolution at higher scale) or a lower order of wavelets (with shorter length) usually resulted in a more sharpened fused image. This means that we can use the level of DWT decomposition and the length of a wavelet as control parameters of an iterative DWT-based fusion algorithm. Together with the rSFe metric, an iterative BP-aDWT can be realized. In the current experiments, a regular DWT and also a Laplacian pyramid (which has been shown to be better in fusing images when compared to other pyramids [16]) were also implemented with the BP-fusion procedure and both methods were compared to the results of BP-aDWT. To further verify the evaluation results of quantitative metrics, a qualitative experiment carried out with human observers was designed and analyzed.
The subsequent sections of this paper are organized as follows. The currently used metrics—RMSE, IQI and SF are described in Section 2, where the new metric, rSFe, is also introduced. Next is a full description of Laplacian pyramid method, the aDWT, and the iterative BP-aDWT directed by the rSFe metric. Lastly, the experimental results and discussion of both quantitative and qualitative analyses are presented, followed by conclusions.
Section snippets
Image quality metrics
As mentioned in the introduction, the general requirements of an image fusion process are that it should preserve all valid and useful pattern information from the source images, while at the same time it should not introduce artifacts that could interfere with subsequent analyses [18]. However, it is nearly impossible to combine images without introducing some form of distortion. In the current body of literature, image fusion results tend to be evaluated either subjectively (with human
Laplacian pyramid
Image pyramids have been described for multi-resolution image analysis and have been proposed as a model for binocular fusion in human vision [18], [21]. An image pyramid can be described as collection of low- or band-pass copies of an original image in which both the band-limit and sample density are reduced in regular steps. The basic strategy of image fusion based on pyramids is to use a feature selection rule to construct a fused pyramid representation from the pyramid representations of
Experimental results and discussion
The following experiments were performed to investigate: (1) whether the metric of rSFe is consistent with other metrics such as RMSE and IQI; (2) whether there is an optimal value of rSFe among the results of varying two parameters (Ld and Lw); (3) whether the optimal value of rSFe can be easily achieved by separately varying the two parameters; (4) when judged with the metric IQI, how the iterative BP-aDWT performs compared to a regular DWT algorithm or Laplacian pyramid; and (5) whether the
Conclusions
We presented a new metric for image fusion based on an extended definition of spatial frequency, which is the ratio of spatial frequency error (rSFe). Experiments showed that evaluation with rSFe is very consistent with the RMSE (root mean square error) and IQI (image quality index) metrics, but the rSFe metric is sensitive to small changes in image quality and can also provide more information of the fusion process—under-fused (rSFe < 0) or over-fused (rSFe > 0)—that makes it useful for iterative
Acknowledgements
This work was supported by grant #N00014-03-1-0224 from the Office of Naval Research. We wish to thank Lex Toet and the TNO Human Factors Research Institute who provided the night-vision imagery. Finally, thanks are given to the anonymous reviewers for many helpful and constructive suggestions.
References (38)
- et al.
Multisensor image fusion using the wavelet transform
Graph. Models Image Process.
(1995) - et al.
Combination of images with diverse focuses using the spatial frequency
Infusionstherapie
(2001) Image fusion by a ratio of low pass pyramid
Pattern Recogn. Lett.
(1989)A morphological pyramid image decomposition
Pattern Recogn. Lett.
(1989)- et al.
Application of morphological pyramids: fusion of MR and CT phantoms
J. Visual Commun. Image Represent.
(1995) - et al.
Image fusion based on median filters and SOFM neural networks: a three-step scheme
Signal Process.
(2001) - et al.
Sensor noise effects on signal-level image fusion performance
Inform. Fusion
(2003) - L.D. Keys, N.J. Schmidt, B.E. Phillips, A prototype example of sensor fusion used for a siting analysis, in: Technical...
- R.H. Rogers, L. Wood, The history and status of merging multiple sensor data: an overview, in: Technical Papers 1990,...
- et al.
Perceptual ability with real-world nighttime scenes: imageintensified, infrared, and fused-color imagery
Hum. Factors
(1999)
Human perception of sensor-fused imagery
Review article: multisensor image fusion in remote sensing: concepts, methods and applications
Int. J. Remote Sens.
Pixel level fusion of image sequences using wavelet frames
Contrast-based image fusion using the discrete wavelet transform
Opt. Eng.
Image merging and data fusion by means of the discrete two-dimensional wavelet transform
J. Opt. Soc. Am. A
Image fusion with additive multiresolution wavelet decomposition: applications to spot1 landsat images
J. Opt. Soc. Am. A
Assessment of pyramid-based multisensor image data fusion
Proc. SPIE
Cited by (251)
Medical image fusion method based on saliency measurement improvement and local structure similarity correction
2024, Biomedical Signal Processing and ControlMulti-image transformer for multi-focus image fusion
2023, Signal Processing: Image CommunicationCDRNet: Cascaded dense residual network for grayscale and pseudocolor medical image fusion
2023, Computer Methods and Programs in BiomedicineRDCa-Net: Residual dense channel attention symmetric network for infrared and visible image fusion
2023, Infrared Physics and TechnologyInfrared-visible image fusion method based on sparse and prior joint saliency detection and LatLRR-FPDE
2023, Digital Signal Processing: A Review JournalIntegrated MPCAM: Multi-PSF learning for large depth-of-field computational imaging
2023, Information FusionCitation Excerpt :Considering that most of these methods are run on two images, and for the sake of fairness, the following comparison experiments are all conducted with two input images (raw image 1 and 4). Apart from subjective evaluation, SAM (spectral angle mapping) [56], rSFe (ratio of spatial frequency error) [57], MG (mean gradient), PSNR and SSIM metrics are adopted for objective evaluation, as listed in Table 3. Moreover, the time costs of different methods are provided for efficiency analysis, as listed in Table 3.