Elsevier

Information Fusion

Volume 8, Issue 2, April 2007, Pages 177-192
Information Fusion

A new metric based on extended spatial frequency and its application to DWT based fusion algorithms

https://doi.org/10.1016/j.inffus.2005.04.003Get rights and content

Abstract

A new quantitative metric is proposed to objectively evaluate the quality of fused imagery. The measured value of the proposed metric is used as feedback to a fusion algorithm such that the image quality of the fused image can potentially be improved. This new metric, called the ratio of spatial frequency error (rSFe), is derived from the definition of a previous measure termed “spatial frequency” (SF) that reflects local intensity variation. In this work, (1) the concept of SF is first extended by adding two diagonal SFs, then, (2) a reference SF (SFR) is computed from the input images, and finally, (3) the error SF (SFE) (subtracting the fusion SF from the reference SF), or the ratio of SF error (rSFe = SFE/SFR), is used as a fusion quality metric. The rSFe (which can be positive or negative) indicates the direction of fusion error—over-fused (if rSFe > 0) or under-fused (if rSFe < 0). Thus, the rSFe value can be back propagated to the fusion algorithm (BP fusion), thereby directing further parameter adjustments in order to achieve a better-fused image. The accuracy of the rSFe is verified with other quantitative measurements such as the root mean square error (RMSE) and the image quality index (IQI), as well as with a qualitative perceptual evaluation based on a standard psychophysical paradigm. An advanced wavelet transform (aDWT) method that incorporates principal component analysis (PCA) and morphological processing into a regular DWT fusion algorithm is implemented with two adjustable parameters—the number of levels of DWT decompositions and the length of the selected wavelet. Results with aDWT were compared to those with a regular DWT and with a Laplacian pyramid. After analyzing several inhomogeneous image groups, experimental results showed that the proposed metric, rSFe, is consistent with RMSE and IQI, and is especially powerful and efficient for realizing the iterative BP fusion in order to achieve a better image quality. Human perceptual assessment was measured and found to strongly support the assertion that the aDWT offers a significant improvement over the DWT and pyramid methods.

Introduction

Image fusion is a tool that serves to combine multiple-source imagery by using advanced image processing techniques. Specifically, it aims for the integration of disparate and complementary data in order to enhance the information apparent in the respective source images, as well as to increase the reliability of interpretation. This leads to more accurate data [1] and increased utility [2], [3]. In addition, it has been stated that fused data provides for more robust aspects of operational performance such as increased confidence, reduced ambiguity, improved reliability and improved classification [2], [4].

This paper focuses on the so-called “pixel-level” fusion process, where a composite image has to be built of several (typically two) input images. A general framework of image fusion can be found in [5]. In pixel-level image fusion, some general requirements [6] are imposed on the fusion result: (1) the fusion process should preserve all relevant information of the input imagery in the composite image (pattern conservation); and (2) the fusion scheme should not introduce any artifacts or inconsistencies which would distract the human observer or subsequent processing stages. Consequently, quantitative evaluation of the quality of fused imagery is considered very important for an objective comparison of the performance of the respective fusion algorithms. In addition, a quantitative metric may potentially be used as feedback to the fusion algorithm to further improve the fused image quality.

Through the practical applications of image fusion in medical imaging, remote sensing, nighttime operations and multi-spectral imaging, many fusion algorithms have been developed. Two common fusion methods are the discrete wavelet transform (DWT) [7], [8], [9], [10] and various pyramids (such as Laplacian, contrast, gradient and morphological pyramids) [11], [12]. However, only a few metrics are available for quantitative evaluation of the quality of fused imagery. For example, the root mean square error (RMSE) is the natural measure of image quality if there is a “ground truth” image available; however, for realistic image fusion problems there are no ground truths. Beauchemin et al. [13] presented a method using local variance for image fusion assessment that still requires a comparison with the measurement of ground truth. Leung et al. [14] proposed the image noise index (INI), based on entropy calculation, to measure fused image quality; this method requires the exact reverse process of an image fusion procedure, which is impractical for most fusion processes such as DWT or pyramid methods. Piella and Heijmans [15] recently presented a new metric for image fusion—the image quality index (IQI), which measures how similar the fused image is to both input images. The values of IQI are within [0, 1] and have an ideal value of 1 (if two input images are identical). The IQI metric has been shown to be consistent with other methods for evaluating fused image quality [16]. However, the IQI does not give the error direction, which would indicate whether the fused image is under- or over-fused.

An image measure termed “spatial frequency” (SF) [20] can indicate how much information is contained in an image and thus may be used as a fusion rule (determining which input should be selected in the fused image) [17], but it cannot directly be used to measure the fused image quality. From the definition of SF (given in Section 2.3), we know that the SF metric is sensitive to gray level changes, although it cannot distinguish useful information from noise or artifacts. Thus, to use the spatial frequency value as a fusion metric, a standard or reference value computed from the input images must be constructed for the purpose of comparison. The new image quality metric presented in this paper, termed as “the ratio of SF error (rSFe)”, is a relative measurement regardless of the type of image to be analyzed. The accuracy of the rSFe metric can be verified with the currently used measurements of RMSE and IQI. In addition, the rSFe value can be further back propagated to the fusion algorithm (BP fusion) such that the fusion parameter adjustment is directed to perform the next loop of fusion, a process that is repeated until an optimized fusion result is achieved.

An advanced wavelet transform (aDWT) method that incorporates principal component analysis (PCA) and morphological processing into a regular DWT fusion algorithm was recently presented [16]. Furthermore, in [16], experimental results showed an important relationship between the fused image quality and the wavelet properties, that is, a higher level of DWT decomposition (with smaller image resolution at higher scale) or a lower order of wavelets (with shorter length) usually resulted in a more sharpened fused image. This means that we can use the level of DWT decomposition and the length of a wavelet as control parameters of an iterative DWT-based fusion algorithm. Together with the rSFe metric, an iterative BP-aDWT can be realized. In the current experiments, a regular DWT and also a Laplacian pyramid (which has been shown to be better in fusing images when compared to other pyramids [16]) were also implemented with the BP-fusion procedure and both methods were compared to the results of BP-aDWT. To further verify the evaluation results of quantitative metrics, a qualitative experiment carried out with human observers was designed and analyzed.

The subsequent sections of this paper are organized as follows. The currently used metrics—RMSE, IQI and SF are described in Section 2, where the new metric, rSFe, is also introduced. Next is a full description of Laplacian pyramid method, the aDWT, and the iterative BP-aDWT directed by the rSFe metric. Lastly, the experimental results and discussion of both quantitative and qualitative analyses are presented, followed by conclusions.

Section snippets

Image quality metrics

As mentioned in the introduction, the general requirements of an image fusion process are that it should preserve all valid and useful pattern information from the source images, while at the same time it should not introduce artifacts that could interfere with subsequent analyses [18]. However, it is nearly impossible to combine images without introducing some form of distortion. In the current body of literature, image fusion results tend to be evaluated either subjectively (with human

Laplacian pyramid

Image pyramids have been described for multi-resolution image analysis and have been proposed as a model for binocular fusion in human vision [18], [21]. An image pyramid can be described as collection of low- or band-pass copies of an original image in which both the band-limit and sample density are reduced in regular steps. The basic strategy of image fusion based on pyramids is to use a feature selection rule to construct a fused pyramid representation from the pyramid representations of

Experimental results and discussion

The following experiments were performed to investigate: (1) whether the metric of rSFe is consistent with other metrics such as RMSE and IQI; (2) whether there is an optimal value of rSFe among the results of varying two parameters (Ld and Lw); (3) whether the optimal value of rSFe can be easily achieved by separately varying the two parameters; (4) when judged with the metric IQI, how the iterative BP-aDWT performs compared to a regular DWT algorithm or Laplacian pyramid; and (5) whether the

Conclusions

We presented a new metric for image fusion based on an extended definition of spatial frequency, which is the ratio of spatial frequency error (rSFe). Experiments showed that evaluation with rSFe is very consistent with the RMSE (root mean square error) and IQI (image quality index) metrics, but the rSFe metric is sensitive to small changes in image quality and can also provide more information of the fusion process—under-fused (rSFe < 0) or over-fused (rSFe > 0)—that makes it useful for iterative

Acknowledgements

This work was supported by grant #N00014-03-1-0224 from the Office of Naval Research. We wish to thank Lex Toet and the TNO Human Factors Research Institute who provided the night-vision imagery. Finally, thanks are given to the anonymous reviewers for many helpful and constructive suggestions.

References (38)

  • E.A. Essock et al.

    Human perception of sensor-fused imagery

  • C. Pohl et al.

    Review article: multisensor image fusion in remote sensing: concepts, methods and applications

    Int. J. Remote Sens.

    (1998)
  • O. Rockinger

    Pixel level fusion of image sequences using wavelet frames

  • T. Pu et al.

    Contrast-based image fusion using the discrete wavelet transform

    Opt. Eng.

    (2000)
  • D.A. Yocky

    Image merging and data fusion by means of the discrete two-dimensional wavelet transform

    J. Opt. Soc. Am. A

    (1995)
  • J. Nunez et al.

    Image fusion with additive multiresolution wavelet decomposition: applications to spot1 landsat images

    J. Opt. Soc. Am. A

    (1999)
  • F. Jahard, D.A. Fish, A.A. Rio, C.P. Thompson, Far/near infrared adapted pyramid-based fusion for automotive night...
  • B. Ajazzi et al.

    Assessment of pyramid-based multisensor image data fusion

    Proc. SPIE

    (1998)
  • M. Beauchemin, K.B. Fung, X. Geng, A method based on local variance for quality assessment of multiresolution image...
  • Cited by (251)

    • Multi-image transformer for multi-focus image fusion

      2023, Signal Processing: Image Communication
    • Integrated MPCAM: Multi-PSF learning for large depth-of-field computational imaging

      2023, Information Fusion
      Citation Excerpt :

      Considering that most of these methods are run on two images, and for the sake of fairness, the following comparison experiments are all conducted with two input images (raw image 1 and 4). Apart from subjective evaluation, SAM (spectral angle mapping) [56], rSFe (ratio of spatial frequency error) [57], MG (mean gradient), PSNR and SSIM metrics are adopted for objective evaluation, as listed in Table 3. Moreover, the time costs of different methods are provided for efficiency analysis, as listed in Table 3.

    View all citing articles on Scopus
    View full text