A Comprehensive Review of Multimodal Medical Image Fusion Techniques

- The multimodal image fusion is the process of combining relevant information from multiple imaging modalities. A fused image which contains recovering description than the one provided by any image fusion techniques are most widely used for real-world applications like agriculture, robotics and informatics, aeronautical, military, medical, pedestrian detection, etc. We try to give an outline of multimodal medical image fusion methods, developed during the period of time. The fusion of medical images in many combinations assists in utilizing it for medical diagnostics and examination. There is an incredible progress within the fields of deep learning, AI and bio-inspired optimization techniques. Effective utilization of these techniques is often used to further improve the effectiveness of image fusion algorithms.


Introduction
Image fusion is a process of mixing appropriate information from more images to reinforce the knowledge content and improves the performance of visual perception systems. Image fusion is an enrichment technique that aims to mix images obtained by diverse sorts of sensors to get a strong or useful image which will facilitate succeeding process or help in deciding.
The fusion methods are effective image extraction and suitable fusion principles, that extract useful information from source pictures and integrated within the fused image without presenting any artifact within the sensor outputs, processing and explosion of data are being developed rapidly.
The aspects of complex applications require complete information, certain scenarios for enhanced understanding of varied conditions. The sensors of an equivalent type obtain information from just one aspect and unable to supply all essential information.
Hence fusion techniques play significant role in recent applications and computer visualization. The imaging techniques with different medical image modalities such as Single-Photon Emission Computed Tomography (SPECT), Positron Emission Tomography (PET), Computed Tomography (CT) and Magnetic Resonance Imaging (MRI) are used for image fusion process. It provides clinical information of the human body's organizational features, soft tissues and so on.
The image fusion technology is budding, the process involved in image fusion methods are disintegration of images and reconstructing it, image fusion guidelines and image quality assessments. The input images are initially decayed into a sequence of sub-images such as high and low frequency components by using image decomposition algorithm. Next, image fusion rules are used to fuse all the features extracted through the high and low frequency components into sub-images at diverse resolutions.
Then, image reconstruction algorithms are used to regenerate fused images. Finally, evaluated the quality of merged images. The resultant fused images should (a) retain complete information of source images; (b) not produce unreal data and (c) not have any bad circumstances, such as mis-recording and noise.

Image Fusion Types
The medical image fusion schemes are generally based on the sensors used to capture an image, namely single sensor, multi-sensor, multi-view fusion and multi-focus fusion.

Single Sensor
It captures the important world as a series of images and therefore the a group of images are merged together to get a replacement image with finest information. For instance, in an illumination variant and noise full atmosphere, a clinical operator like detector operator might not be ready to detect objects of his interest which might be highlighted within the resultant amalgamated image.
The disadvantage of this kind of systems lie behind the restrictions of the imaging sensing element that are getting used in alternative sensing space, beneath the conditions throughout that the system will operate, its dynamic vary, resolution, etc and all restricted by the competence of the sensing element [1].
For example, a visible-band sensing element just like camera is appropriate for a bright lighted setting like Advances in Computing, Communication, Automation and Biomedical Technology daylight scenes however is not suitable for poorly lighted things found throughout getting dark or below not sensible conditions like in fog or rain.

Multi Sensor
A multi detector image fusion overcomes the restrictions of 1 detector image fusion by merging pictures from many sensors to create a composite image, which associate in Nursing infrared camera is attendant the camera and their individual images area unit incorporated to urge a amalgamate image and this approach overcome these issues.
The camera is appropriate for daylight scenes. The infrared camera is suitable in poorly light environments and is used in military space, computer vision like in object detection, robotics and medical imaging [2].

Multi View
Multi view fusion pictures, including images from various models such as panchromatic, multispectral, visible, infrared, remote sensing, provide several or separate views at the same time.

Multi Focus
With its focal length, the multi focus image fusion scheme has 3D views. The original image can be divided into regions, such that at least one image channel focuses on each region.

Image Fusion Classification
The techniques used for medical image fusion are often categorized into three, based on feature extraction, level of decision and pixel intensity.

Feature Extraction
The feature-based image fusion algorithms typically combine multiple input multimodalities with the region of interest. Extracts from the images such as edges, corners, texture parameters, lines, etc from input modalities are different features here and mix them into one function image for further processing [3].

Decision Level
The fusion techniques based on selection rely primarily on the training data set and fusion efficiency. Typically, the dictionary learning fusion uses the methodology of judgment level. The results obtained from various algorithms are denoted instead of choices as confidences, then it is called soft fusion, and otherwise it is called hard fusion

Pixel Intensity
Medical image fusion techniques based on pixels are split into sub-stages such as pre-processing, decomposition, fusion rules and evaluation of fusion efficiency. Thanks to the main advantages over feature-based and decisionbased techniques, pixel level image fusion is preferably used in medical image fusion. Visual clarity in pixel intensity-based image fusion is highly informative [4].

Medical Imaging Modalities
Computed tomography, magnetic resonance imaging, positron emission tomography and single-photon emission computed tomography are the most common medical imaging modalities used for the fusion process.

Computerized Tomography
In earlier days, Computerized Tomography (CT) was recognized as a CAT scan or computerized axial tomography scan. CT is a technique for gaining and remaking the image of a thin cross-segment on the premise of estimations of X-ray weakening. CT is used as a modern medicine representing one of the main noninvasive diagnostic techniques.
CT is a medical imaging technique that provides a very large amount of information about tissue density and anatomical features that also allow tumour detection. The characteristics of CT images are: it implants with less distortion and provides details regarding dense structures, such as bones and demonstrates superior ability to slight differences in tissue contrast. CT is one of the primary modalities included for fusion [5].

Magnetic Resonance Imaging
Magnetic resonance imaging (MRI) is a medical imaging process that uses signs of a magnetic field and radio frequency (RF) to image anatomical structures, the proximity of disease, and various organic capabilities within the human body. In the diagnosis of brain tumours, MRI also plays an important role and is one of the most effective and commonly used imaging modes in medical studies.
The main characteristics of MRI are: it produces "slices" that represent the human body by applying magnetic signals and provides information about pathological soft tissues. The problem of radiation is not influenced by the MRI, which is its main advantage. MRI offers information and information on the brain, spinal cord, strokes and other disorders in the body's organs and soft tissues.
The limitation of multi-modal imaging is to overcome by using image fusion, thus, allowing recreation and forecast of the missing data from MRI. The MRI alongside the use of other modalities, have seemed to enhance the imaging exactness and pragmatic clinical relevance.

Positron Emission Tomography
Positron Emission Tomography (PET) is a dynamic piece of atomic drug imaging, non-intrusive imaging framework offering the portrayal and assessment of prechosen tracer digestion. Functional nuclear-based imaging methods in PET medicine are used to observe metabolism processes occurring in the human body that help diagnose disease.
The fundamental qualities of PET images are: it gives useful data about the human mind and allows recording the adjustments in the solid cerebrum movement and gives indications of different illnesses. PET has become one of the widely used clinical tools for imaging of any part of the body. PET is also helpful for diagnosis and care related to the brain.

Single Photon Emission Computed Tomography
Single Photon Emission Computed Tomography (SPECT) is a type of atomic pharmaceutical tomographic imaging that uses gamma rays to study the flow of blood to tissues and organs. SPECT scan analyses the function of the internal organs. It gives genuine 3D data and is commonly exhibited as cross-sectional slices through the patient. SPECT scans are one of the mostly used scans for tissues outside the brain, where the location of tissues is significantly variable.

Medical Image Fusion Algorithms
Medical image fusion algorithms are classified based on morphological operators, human value system operators, sub-band decomposition, artificial neural networks, fuzzy logic, neuro-fuzzy and factorized thresholding.

Morphological Operators
The morphology operators are utilized by the restorative imaging group to spot spatially significant data from the medical images. After recording the knowledge source images, a pyramidal information structure is produced using each image by consecutively smoothing and inspecting each image.
The smoothing operation is completed by a morphological (opening) function, which preserves image structure and has a plus of computational productivity. A morphological difference pyramid is then built for every of the above pyramids, which contain just the features existing between each successive scale.
These features from various modalities are utilized as a neighborhood of image fusion. Medical images are inclined to noise and detecting blunders when the features are identified erroneously. K-L transforms, averaging, morphology towers and morphology pyramids are used by the operators to achieve the info fusion as an example. These methodologies are unusually sensitive to the variability of the inter-image due to irregularities, commotion, scale, and condition of the features [6].

Human Value System Operators
Human value System (HVS) operator models permit to backtrack the new emblematic image portrayal during a considerably characterized manner implying with a selected end goal to make elective translations. This model permits information concerning the fusion of image, geometric and spatial properties that interestingly characterize anatomical structures.
There is an increase in applications where domaindependent learning is suitable for image fusion, as an example, segmentation, small scale calcification finding, tissue characterization, cerebrum analysis, classifier fusion, carcinoma tumor discovery, and depiction and acknowledgment of anatomical mind question [7].
The benefit of HVS is that it is capable of contrasting the images with the established concepts of human vision, while the downside is that the impediment imposed by human choice in images that are vulnerable to large pixel power changeability.

Sub-band Decomposition
The basic thought used by the wavelet-based fusion of images is to remove and implant the detailed data from one image into another. The accurate image data is typically within the high recurrence, and the frequencies can be chosen by wavelets in both space and time. As far as the features of the images are concerned, the resulting fused image has the colossal characteristics that increase the image standard.
There are a few of models for fusion, the only being the substitution of pixels from one image to a different. There exists a few of numerical models for fusion, as an example, the essential addition operation and aggregator capacities to more eccentric logical models.
The signal is decomposed into scaled and moved versions of the selected mother wavelet or frequency domain in wavelet theory and wavelet will provide good resolution in both time and frequency domains.
The DWT has numerous noteworthy qualities, for instance, localization, compact representation, and direction that make the DWT-based fusion schemes generally better than the pyramid-based methodologies [8].
Super-resolution, medical image pseudo coloring, feature level image fusion, diagnosis, lifting scheme, 3D conformal radiotherapy treatment preparation, segmentation and color simulation are the uses of wavelets in image fusion.

Artificial Neural Networks
Artificial Neural Networks (ANN) are encouraged by the likelihood of natural neural systems which will gain from contributions for preparing features and for deciding on worldwide choices. Although ANN offers simplification as far as having the ability to use the thought of coaching, the heartiness of ANN techniques is restricted by the character of the training information and therefore the exactness of conjunction of the training procedure.
To reinforce the features character, the strength of the ANN, hybrids of neural systems and consecutive handling; other fusion methods are utilized. Some of the illustrations is that the wavelet-neural system, neural fuzzy and fuzzy genetic neural system-rough set.

Fuzzy Logic
In increasing the standard of the frame, medical image enhancement is extremely significant. For enhancing images, for example, histogram equalization, grey-level transformation, there are several crisp algorithms. However since medical images involve several uncertainties, they do not improve the image quality properly.
To eradicate the anomaly present in images, fuzzy sets play a major role in image processing. In the form of membership function within the interval (0, 1), Fuzzy sets take uncertainty where zero speaks to no membership and one speaks to complete membership [9].

Neuro-Fuzzy
A Neuro-fuzzy method for categorizing and deblocking anomalies from Brain fMRI. Study of the function deblocking critical data process that leads to categorization. Texture and Wavelet functions are used to diagnose the image class as discriminating elements.
Using the feed forward Back propagation neural network, the categorization stage discriminates against conventional and pathological fMRI slices. For feature extraction, the classified abnormal images are added and are compared with ground truth data [10].

Factorized Thresholding
The pixel intensity was assisted by segmentation to emphasize the diagnosis of the tumour and stroke affected region within the given MRI brain image. The segmented results of both original and fused images are then assessed to predict the specific tumour and stroke area [11].

Conclusion
A brief research on the subject of multimodal medical image fusion and the basic principle used in it was discussed in this review paper. First, based on capturing an image, the paper gives a definition of different types of fusion. Then offer an idea behind the technology classification used in the fusion of medical images. Provide a description for various modalities of medical imaging. The paper covers the various algorithms used for medical image fusion in depth as well. The advantages and disadvantages of each of the strategies were also included. The paper also reveals that neurofuzzy and factorized threshold medical image fusion is the better form of fusion.