Medical Image Fusion Based on Low-Level Features

Medical image fusion is an important technique to address the limited depth of the optical lens for a completely informative focused image. It can well improve the accuracy of diagnosis and assessment of medical problems. However, the difficulty of many traditional fusion methods in preserving all the significant features of the source images compromises the clinical accuracy of medical problems. Thus, we propose a novel medical image fusion method with a low-level feature to deal with the problem. We decompose the source images into base layers and detail layers with local binary pattern operators for obtaining low-level features. The low-level features of the base and detail layers are applied to construct weight maps by using saliency detection. The weight map optimized by fast guided filtering guides the fusion of base and detail layers to maintain the spatial consistency between the source images and their corresponding layers. The recombination of the fused base and detail layers constructs the final fused image. The experimental results demonstrated that the proposed method achieved a state-of-the-art performance for multifocus images.


Introduction
The depth-of-field limitations may potentially limit the complete and accurate understanding of the medical problem of the human body, organs, and cells and even the performance of medical diagnostics and analysis [1][2][3]. Medical image fusion is an essential technique for combining multiple images with complementary information to provide more comprehensive descriptions of the medical problems [4]. To date, medical image fusion has become a relevant research field due to its efficiency and wide applications in medical analysis. The growing appeal of high-performance medical diagnostic devices prompts the development of low-cost computing and imaging techniques. There are many medical image fusion methods proposed to address the problems mentioned above. These methods include two categories: spatial domain methods and transform domain methods [5].
The spatial domain methods deal with pixels or regions in spatial domains directly based on the pixel intensities [6]. The fundamental problem of spatial domain methods is the selection of the clearest image pixels or regions from the source images in order to construct the fused image. The regions with greater energy or larger changes of pixels are considered to be in focus during the fusion process. The spatial domain methods mainly include pixel-based methods [6][7][8] and region-based methods [9][10][11]. These methods are simple and fast. However, the pixel-based methods are sensitive to noise, which may potentially lead to the incorrect choice of pixels. Due to difficulties in the selections of block sizes or segmentation algorithms [12,13], the region-based methods suffer from blocking artifacts. Also, the visual quality of the final fused image could be compromised when blur and sharp pixels are segmented to the focused regions [5].
The transform domain methods deal with the coefficients of the transformed source images in the transform domain. The transform domain methods that approximate and detail coefficients at different scales integrate these coefficients to new multiscale representation by employing various fusion rules. And finally, an inverse transformation is performed on these coefficients to reconstruct a fused image [4]. Based on the hypothesis that the greater the frequency content in the transform domain, the higher the corresponding contrasts in the spatial domain, the regions with greater coefficients of high frequency are accepted as being in focus during the fusion process. These methods mainly include the following: the Laplacian pyramid (LAP) [14], discrete wavelet transform (DWT) [15], curvelet transform (CVT) [16], contourlet transform (CT) [17], and non-subsampled contourlet (NSCT) [18]. These methods can achieve better signal-to-noise ratios [19]. However, they suffer from time and space consumption, loss of contrast, selection of the decomposition level, and decomposition type. The simple fusion rules of these methods cannot always successfully identify the detailed and structured information from coefficients, and may cause the degradation of image quality [20]. Due to spatial inconsistencies, these methods are unable to effectively preserve the edge and texture information, which may lead to halo artifacts near the edges, as well as spurious data and distortions.
In recent years, deep learning as a novel technique is applied to medical image fusion for better fusion performance. Liu et al. [21] fused the medical image by integrating MCA and convolutional sparse representation (CSR) into a unified optimization framework. It can outperform some benchmarking and state-of-the-art SR-based fusion methods. Xia et al. [22] proposed a novel fusion scheme for multimodal medical images which utilizes both the features of the multiscale transformation and deep convolutional neural network. Hou et al. [23] designs a novel fusion scheme for CT and MRI medical images based on convolutional neural networks and a dual-channel spiking cortical model. Ding et al. [24] fused medical images by combining convolutional neural networks and non-subsampled shear-let transform to simultaneously cover the advantages of them both for medical image fusion. Wang et al. [25] proposed a medical image algorithm based on the Siamese convolutional network and contrast pyramid. These algorithms can effectively preserve the detailed structure information of source images and achieve good human visual effects. These methods have achieved better fusion performance. However, the need for tuning millions of parameters during the training stage, as well as the difficulty of exploration, seriously affected the fusion quality.
Recently, edge-preserving filter-based fusion methods are introduced to solve the problems mentioned above and they simplify the representation of source images while retaining the robust edges [26]. They include the guided filtering method [27], L 0 gradient minimization method [28], cross-bilateral filter (BF) method [29], weighted least square filter method [30], and rolling guidance method [31]. The guided filtering method retained the spatial consistency of the base and detail layers by using a weighted average technique. The L 0 gradient minimization method preserved and enhanced human visual system interests by using visual weight maps. The crossbilateral filter method fused the source images by using the detailed information of the source images. The rolling guidance method prevented noise and image distortion through the spiking cortical model, while the iterative guided filtering method suppressed the noise by using a guided filter in an iterative mode. All these methods achieved excellent performance for medical image fusion. This paper mainly focuses on the improvement of the conventional image fusion method.
As we know, low-level features such as color, texture, edge shape, and structure are the significant features for image representation. These features are the key to image saliency detection and image understanding. It is significant for medical analysis and decisions according to medical images. The low-level feature is the salient feature for focused region detection. It can improve the accuracy of the focused region detection by using low-level features. Then, more structured and detailed information can be transferred into the fused image from the source images. Thus, high-quality fusion result can be obtained. To further improve the medical image fusion quality by using a low-level feature, we propose a novel medical image fusion scheme with a low-level feature. In the beginning, the proposed method decomposes the source images into base and detail layers by using local binary patterns operators for the low-level feature. Then, weight maps of the base and detail layers are constructed based on the low-level feature. Thirdly, the base and detail layers are fused according to the optimized weight map with a fast guided filter. Finally, the fused base and detail layers are combined to produce the final fused image. The fast guided filter (FGF) is one of several popular filters for edge-preserving smoothing, which is independent of the filter size. Due to its flexibility and speed of computation, it is feasible to use in different real-time applications. The objective of this paper is to investigate its potential applications in medical image fusion.
This paper's main contributions fell into the following three points: (1) We propose a novel multifocus image fusion method with a low-level feature (2) We propose a novel weight construction method according to low-level feature saliency and spatial context (3) We extract the low-level feature by using local binary patterns operators The rest of this paper is organized as follows: Section 2 explains the basic concept of FGF which discusses its feasibility and advantages for medical image fusion, whereas Section 3 defines the fusion method with FGF. Section 4 discusses experimental results followed by conclusions and future work in Section 5.

Related Works
2.1. Fast Guided Filter. FGF is one of several popular techniques for edge-aware image filtering, whose computing time is independent of the filter size [26]. In this study, the FGF is applied to medical image fusion. In theory, FGF is driven by a local linear model and the relation between the guidance image I and filter output image O in a local square window ω k centered at the pixel k is defined as follows: where i denotes a pixel and a k and b k are liner coefficients in 2 Computational and Mathematical Methods in Medicine ω k . The size of ω k is ð2r + 1Þð2r + 1Þ. The linear coefficients a k and b k are used to minimize the squared difference between the filter input image P and the filter output image O. The linear coefficients a k and b k can be obtained by linear regression as follows: where u k and σ k are the mean and variance of I in the window ω k , respectively, jωj denotes the pixel number in k, and P denotes the mean value of P in ω k . ε is a regularization parameter, which controls the degree of smoothness on P.
The filtering output image O is defined as follows: where a i and b i are two smoothed maps and denote the mean value of a and b, respectively, in ω i . The main computation of FGF is for the smoothed maps of a i and b i . In order to improve the computational efficiency, all the box filters are performed on the low-resolution maps. Moreover, a i and b i are bilinearly upsampled to the original size. Figure 1 lists the comparison of BF and FGF for image filtering. The enlarged region of the filtering results for the source image demonstrates that FGF can well inhabit gradient-reversal artifacts and preserve the lowlevel features. It is known that a suitable fusion method should transfer most of the useful low-level feature, such as edges, texture, and structure information, from the source images to the fused image. FGF is a novel edge-preserving smoothing technique, which can effectively remove noise, weak edges, and small details while preserving the overall low-level features of an image. Activity level measurements and focus regions or pixel extractions are two essential problems affecting the fusion quality. As mentioned above, the transform domainbased fusion methods use predesigned bases to represent the source images. The spatial domain-based fusion methods use activity-level measurements based on high-pass spatial filtering. FGF is able to well preserve the low-level feature such as colors, boundary, structures, edges, and textures corresponding to those of the source images. The low-level feature of base and detail layers can be used to perform more effective activity-level measurements and accurately discriminate the focused regions from the defocused regions. The fusion of the source images can be transformed into the subfusion of base and detail layers. FGF can well extract the lowlevel feature from the source images. Therefore, it was determined that it is feasible to apply FGF to medical image fusion. The advantages of the FGF-based fusion method over other existing methods are fourfold: (1) FGF can effectively suppress gradient-reversal artifacts [15] and produce visually pleasing edge profiles, (2) FGF is independent of the filter size and well suited for real applications with highly computed efficiency, (3) FGF is flexible and easy to implement, and (4) the FGF-based fusion method can be adapted to different field applications.

Feasible and Superiority.
Considering the advantages of FGF mentioned above, we propose a novel FGF-based fusion method to extract the lowlevel features for saliency map construction and guide the medical image fusion, as detailed in Section 3.

Proposed Approach
3.1. Fusion Algorithm. We assumed that there are two registered source images I 1 and I 2 , which are preregistered. As shown in Figure 2, the proposed fusion algorithm consists of four main steps: Step 1. (two-scale image decomposition).
The local binary pattern (LBP) operators [32] are applied to decompose the source images I 1 ,I 2 into base layers B 1 ,B 2 and detail layers D 1 ,D 2 respectively. It can be defined as follows: where I is the source image, B is the base layer of I, and D is the detail layer of I.
Laplacian filtering (LF) and Gaussian low-pass filtering (GLF) are performed on the source images I 1 , I 2 to obtain the saliency maps S 1 and S 2 , respectively, which are compared to construct weight maps P 1 and P 2 , respectively. Weight maps P 1 and P 2 are optimized for W B 1 , W B 2 and W D 1 , W D 2 , respectively, with FGF guided by source images I 1 and I 2 , respectively.
Following the fusion rules and optimized weight maps, B 1 and B 2 are integrated to obtain F B , which denotes the fused base layerD 1 . Moreover, D 2 is integrated to obtain the fused detail layer F D .
3.2. Fusion Rule. As mentioned above, the low-level features of the source images such as colors, boundary, structures, edges, and textures can be used to perform more effective activity-level measurements and accurately discriminate the focused regions from the defocused regions. The relationship between the source images "skull" and their corresponding layers is shown in Figure 3. It can be seen that the low-level features of the source images are corresponding to the lowlevel feature of different layers, such as the structure, the textures, and the edges of the tissue.

Computational and Mathematical Methods in Medicine
In order to measure the activity level, we apply Laplacian filtering and Gaussian low-pass filtering to detect the salient feature and construct the saliency maps of the source images. The Laplacian filter highlights the change in light intensity surrounding a pixel. It can extract the outline of the target and generalize the details. Laplacian filtering is used for obtaining the high-pass component of the source image. Gaussian low-pass filter is a linear smoothing filter, which is suitable for eliminating Gaussian noise and is widely used in the denoising process of image processing. Gaussian filtering is a process of weighted averaging of the entire image. The value of each pixel is obtained by weighted averaging of itself and other pixel values in the neighborhood. In this study, Gaussian low-pass filtering is used for computing the local average of the absolute value of the obtained high-pass component, which constructs the saliency maps S. The saliency map is defined as follows: where S is the saliency map. L is the Laplacian filtering. G is Gaussian low-pass filtering. The obtained saliency maps represent the saliency level of detail information. The pixel maximum of the saliency maps is computed to construct the corresponding weight maps P, which are defined as follows: where "1" in weight maps indicated that the pixel location ð i, jÞ in source images was in focus. However, the weight maps are not well consistent with the object boundaries in the source images. It may produce artifacts in the fused image. As shown in Figure 3, the base layers are spatially smooth and the detail layers have large moment of detailed information. Spatial consistency demands that those pixels with similar color or brightness tend to have similar weights. Thus, the weights for base layers should be spatially smooth and the weights for detail layers should be sharp. FGF is applied to optimize the binary maps with the source image I serving as a guidance image. A large filter size and large blur degree are used for the fusion of base layers. A small filter size and small blur degree are used for the fusion of detail layers. Optimized weight maps W are defined as follows: Then, the base and detail layers obtained by using LBP are fused with optimized weighted maps W. The multilayers of the source images are fused as follows: where F B and F D represent the fused base layer and the fused detail layer, respectively.

Experimental Results
In this section, some commonly used testing image sets are used to assess the performance of the proposed method. To be more objective in the performance assessment, the proposed method is compared with some of the existing fusion algorithms in terms of visual quality and quantitative evaluation.  [34], multiscale image decomposition-(MSID-) based fusion algorithm [35], cross-bilateral filtering-(CBF-) based fusion algorithm [24], and guided filtering fusion-(GFF-) based algorithm [22].

Parameter Setting.
The decomposition level of the DWT was 4. Also, the pyramid filter "9-7" and orientation filter "7-9" with {4, 4, 3} levels of decomposition were set for the NSCT. The parameters of the recently proposed fusion methods, such as CTD, MSID, CBF, and GFF, are found to be the same with the corresponding papers. The local window radius and the regularization parameters of the FGF are set as r 1 = 45, r 2 = 7:5, ε 1 = 0:25, and ε 2 = 10 -6 .

Evaluation Metrics.
Four commonly used evaluation metrics, i.e., mean square error (MSE), mutual information (MI) [40], Q AB/F [41], and Q Y [42], are used to evaluate the effectiveness of the proposed method. These metrics measure the information preservation ability of the fusion method.   CTD is an improved technique for the problems of L1regularized optimization. It can well extract the low-level features such as structural patterns and latent detail information of source images with cartoon-texture decomposition. The source image can be split into cartoon components and texture components with a split Bregman algorithm. The salient low-level features are compared to construct the decision map for the fusion of cartoon and texture components. However, the construction of the decision map for cartoon and texture components is affected by the imprecise morphological operation, such as erosion and dilation. Thus, the lowlevel feature of the source image cannot be well transferred to the fused image. The corresponding artifacts of CTD can be seen in the edge regions of the enlarged detail regions (Figures 6(a)-6(d)).
MSID is based on saliency detection and multiscale image decomposition. MSID is efficient to emphasize visual saliency by extracting low-level features. It can improve the capacity of focused region detection. Weight maps of this algorithm are capable of detecting and identifying focused and defocused regions of the source images. However, the    The cross-bilateral filter considers both gray-level similarities and geometric closeness of the neighboring pixels without smoothing edges, but it uses one image for finding the kernel and the other to the filter and vice versa. CBF applies joint bilateral filtering to extract low-level features for suppressing the gradient reversal artifacts of the bilateral filter. It fuses source images by weighted average using the weights computed from the detail layers. However, the fusion of different layers based on weight maps may be affected by the combination of the pixels with different intensities. The low contrast and visible blurs can be seen in the fused images, such as the incompleteness of the soft tissue of the brain ( Figure 5 GFF is based on a two-scale decomposition of an image into a base layer containing large-scale variations in intensity and a detail layer capturing small-scale details. GFF is an efficient fusion method. It combines pixel saliency and the spatial context for medical image fusion. It improves the fusion performance by combining the different layers with optimized saliency maps constructed from the low-level features. However, the weight map operations used in the fusion 9 Computational and Mathematical Methods in Medicine process can result in the loss of some low-level features of the source images due to inaccurate weight values. It can be seen in the fused images, such as the blurring artifacts, e.g., the corresponding enlarged detail regions (Figures 8(g) and 12(g)).
The proposed method decomposes the source images into base layers and detail layers by using local binary pattern operators. The saliency low-level features such as texture, edge, and structure are extracted by Laplacian filtering and Gaussian low-pass filtering. These features are compared to construct the saliency maps, which optimized to weight maps for the fusion of different layers by using fast guided filtering. Fast guided filtering is adopted as a local filtering for optimization of the weight maps. Different filter sizes and blur degrees control the pixel saliency and spatial consistency by adjusting the value of the parameters in the fast guided filter. Different layers are fused with corresponding weight maps based on the guidance of the source images. The pixel saliency and spatial consistency can be well improved, which ensures the better fusion performance of the proposed method. The visual quality assessment of the fusion results obtained by different methods has demonstrated that the   Table 1 are shown in Figure 13. The average running time of the compared algorithms is listed in Table 2.
It is easy to see that the MI and Q AB/F values of CTD and the proposed are higher than those of other fusion methods.  Table 1 and the change of the trend chart (MI, Q Y , and Q AB/F ) demonstrate the proposed method has achieved better performance. Table 2 demonstrates that the NSCT required the longest computational times, followed by CBF and CTD. The reason is that the processing of coefficients consumes most of the fusion time. The statistic model of CBF consumes most of the fusion time. The sliding window of CTD consumes the running time. As previously mentioned, this study demonstrates that the proposed method costs little time for the medical images.

Conclusion
This study presented a novel medical image fusion method based on the low-level feature. The fast guided filtering is applied to the saliency map of the source images to construct the weight map of the multilayers of the source images. The base and detail layers obtained by local binary pattern operators are fused according to their corresponding weight maps, and the fused base and detail layers are combined to produce the final fused image. This study's experiments are performed on nine pairs of medical images. The experimental results demonstrate that the proposed method obtains a state-of-the-art performance in both qualitative and quantitative evaluations. The optimization in parameter adaptability is exciting and worthwhile for further investigations.

Data Availability
The medical image data used to support the findings of this study have been deposited in references [20,30].

Conflicts of Interest
The authors declare that they have no conflicts of interest.