An image fusion method based on the Hermite transform applied to nuclear medicine and x-ray tomographic modalities

Image fusion is an interesting processing task that has reached great significance for medical image analysis. In general, the combination of medical images coming from different modalities is a common practice that significantly helps in the process of diagnosis and detection of several diseases. In this work, we present a novel method for image fusion based on the Hermite transform which consists of a powerful tool that projects an input image into the space defined by the Hermite polynomials. The proposed approach is performed in three main stages. 1) The HT is applied to the input images, 2) The resulting coefficients are fused using the maximum and average intensity rules, and 3) The inverse HT is performed to obtain the final fused image. The method is applied and evaluated using several single photon emission computed tomography and computed tomography studies taken for bone structures. Typical metrics were used to assess the proposed framework. We demonstrate that this methodology is able to efficiently fuse images coming from different modalities, particularly, nuclear medicine and x-ray tomographic techniques. With the Hermite transform, image features are successfully extracted which becomes fundamental in the process of image fusing.


Introduction
Image fusion tasks have gained much relevance for medical image processing applications since specialists normally use information gathered from different modalities with the aim of performing better analysis and diagnoses. The main objective of image fusion techniques is to combine, in the same space, images coming from different modalities [1]. The resulting fused image must contain the most significance information obtained from the input data. The specific application of single photon emission computed tomography (SPECT) and computed tomography (CT) images comes from the fact that metabolic and anatomical data can be used together to improve and help in the process of illness diagnosis which is naturally performed by the physicians. While SPECT provides information of the metabolic behavior of the human body, CT images are fundamental for evaluating structural and anatomical information [2]. Both are frequently combined and used together to evaluate some diseases such as thyroid problems, bone affections and cancer metastases [3].
On the other hand, the fusion task has been mainly focused on using image decomposition methods to perform the corresponding combination [1]. Among the most used decomposition-based methods, we can mention those based on the wavelet, contourlet and sheartled transforms [4]. Most of the multimodal fusion methods applied to medical images reported in the literature have been designed to evaluate mostly brain structures [5]. There exist many other methods used for fusing multimodal information which do not use image decomposition [6]. Although these methods demonstrated being very efficient in some cases, we believe that the fusion process should be performed depending on the local texture information. In this sense, transform-based methods are very suitable since they provide image features with a great diversity of details.
In this work, we propose a fusion scheme based on the Hermite transform (HT) which consists of powerful tool that has been applied to many tasks in the field of medical imaging [9][10][11]. Image features like edges and zero crossing, as well noise removing can be competently performed by using this transformation. The image fusion is carried out in three main process: the HT is applied to both input images, the resulting coefficients are subsequently fused, and the inverse transform is then computed. Our method is assessed by using several SPECT and CT image examples. Some metrics were also used for performance quantification.

Methodology
The proposed method includes two main stages: the HT and the fusion rules used for the combination of the coefficients. Both methods are described as follows.

The Hermite transform
Let f(x, y) a two-dimensional function representing an input image. The HT, introduced in [8], can be computed using Equation (1).
where [8]. Coefficients are computed for n = 0,1,2, … , N and m = 0,1, … , n where n is the transform order. Figure 1(a) illustrates an example of the HT obtained for a CT input image. This image shows the bone structure at the hip level. The HT has been calculated until order n = 2. It can be seen that some image features have been extracted from the original input. In the same way, Figure 1(b) illustrates the HT coefficients obtained for a SPECT image example. In both cases, coefficients show some texture features of the input images.  Considering the distribution illustrated in Figure 1, the dashed red line highlights the zero-order coefficient. The dashed blue line indicates the first-order coefficients and the dashed yellow line shows the second-order coefficients. It can be also mentioned that coefficients of order n = 1 provide edge information of the input images while zero crossing details can be obtained from coefficients of order n = 2. The first order coefficient is simply a smoothed version of the input image, which means that low frequency information is coded.
The calculation of HT transform can be extended to higher orders to provide more features of the input data. In general, the complete set of coefficients obtained composes the texture pattern used for further analysis. The inverse HT can be computed through Equation (2).

Coefficients fusion rule
Once the HT is applied to both input images, the obtained coefficients are then combined using the fusion rule described in Equation (3).
where L !"#,# :;<=> and L !"#,# => correspond to coefficients of n order for the input SPECT and CT images, respectively. As expressed in Equation (3), for the low frequency coefficients we used the average rule for the fusion process. The goal is to preserve the information coming from both input data. In SPECT images, the most relevant information related to the patient's disease is coded as intensity levels in a specific region. For details coefficients, n ≥ 1, we use the rule of the maximum value. The objective is to highlight the boundary of the structures of interest. The resulting combined coefficients, L !"#,# 9 , are posteriorly used to obtain the inverse HT which provides the final fused image.

Results
SPECT and CT images were used for evaluation. Both images are considered to be mutually registered before performing the image fusion. Images were acquired from five different patients. Quantitative analysis was also performed. We used several metrics for this purpose: entropy (E), mutual information and the edge preserving index (QAB/F). We refer the reader to [5,12] for more details about the definition of these metrics. Table 1 illustrates the obtained results using the mentioned measures. In general, quantitative results show that the proposed method is competitive since reached values are typical for fusion methods with good performance.

Discussion
The proposed method was evaluated using three typical metrics of the state of art. The entropy is a statistical metric that can be used to measure the image information content. The higher the entropy is, the more information can be found. It can be seen, from Table 1, that entropy is relatively high for all evaluated cases, considering a measure which is ranging from 0 to 1, being 1 the highest possible entropy. On the other hand, the mutual information metric measures how much content from one image can be found in other [11]. Results in Table 1 shows that all examples present similar values of mutual information and it demonstrates an acceptable performance obtained with the proposed approach. The edge preserving index is other metric commonly used for evaluating fusion methods. It tries to assess the success of the edge transferring from the input images to the fused one. Results reported in Table 1 also show the good performance of the proposed method, obviously considering the typical values that indicate acceptable results when using this metric. I refer the reader to [11] to investigate more about interpretation of the used metrics.
Qualitative assessment is also important for performance evaluation. It is known that fusion-based methods are normally applied to tasks whose final objective includes visual inspection of the fused image. The application presented in this paper is one these tasks since the final image can be used by the physicians in the diagnosis process. As mentioned, the final fused image must contain detailed information of the bone structure which is naturally provided by the CT studies, and metabolic information provided by the SPECT images.
It can be observed, from images presented in Figure 2, Figure 3 and Figure 4, that the most relevant information from both studies is transferred to the fused images. The edge features representing the bone structures and the bright spots which indicate the metabolic behavior of the bone tissues are clearly presented in the final fused image. Visual results are presented for the three most common views in which the spot coming from the SPECT images are successfully combined with the high frequency details provided by the CT images. The correct combination mainly depends on the image features used for fusion. In this sense, coefficients of the HT provide rich and separated information about different image features and texture. The average rule for the low frequency components and the maximum value criterium are very suitable for fusing the Hermite coefficients of both input images.