The Effect of Basic Fusion Techniques in Deep Ensemble Learning-Based Models For COVID-19 Diagnosis

The coronavirus (COVID-19), declared a global epidemic (pandemic), is a new viral respiratory disease. The disease is transmitted from person to person through droplets or contact. It is very important to detect the disease early with rapid diagnosis rates to prevent the spread of the disease. However, long-term pathological laboratory tests and low diagnosis rates in test results led researchers to apply techniques. Radiological imaging has begun to be used to monitor COVID-19 disease as well as being useful in detecting various lung diseases. The application of deep learning techniques together with radiological imaging has a very important place in the correct detection of this disease. This study investigated the effect of basic fusion functions on classification performance on ensemble learning algorithms using the COVID-19 X-ray

The coronavirus , declared a global epidemic (pandemic), is a new viral respiratory disease.The disease is transmitted from person to person through droplets or contact.It is very important to detect the disease early with rapid diagnosis rates to prevent the spread of the disease.However, long-term pathological laboratory tests and low diagnosis rates in test results led researchers to apply techniques.Radiological imaging has begun to be used to monitor COVID-19 disease as well as being useful in detecting various lung diseases.The application of deep learning techniques together with radiological imaging has a very important place in the correct detection of this disease.This study investigated the effect of basic fusion functions on classification performance on ensemble learning algorithms using the COVID-19 X-ray

Introduction
The World Health Organization (WHO) has named the deadly virus originating in China "COVID-19".The institution used the following abbreviations while giving this name; "co" means corona, "vi" means a virus, and "d" comes from the initials of the English word "disease".The number 19 in COVID-19 indicates December 31, the date the disease was first identified.In retrospect, coronaviruses have caused 3 epidemics in the last 20 years: MERS, SARS, and COVID-19 (Li et al., 2020).
The WHO used the term COVID-19 to describe the disease caused by the virus, making the virus the official name SARS-COV-2 (Severe Acute Respiratory Syndrome-Coronavirus-2). On January 31, 2020, COVID-19 was declared a global health emergency by the WHO.Due to the virus, a pandemic, that is, a global epidemic, was declared on March 11, 2020.Clinical symptoms of the disease include shortness of breath, cough, fatigue, fever, sore throat-head-muscle pain (Singhal, 2020).
The most common testing technique used for diagnosing COVID-19 is RT-PCR, a real-time reverse testing technique.However, RT-PCR infrastructure is insufficient in most regions with the COVID-19 outbreak.RT-PCR testing is considered time-consuming and tedious with a complex manual procedure.The standard screening tools for early detection and diagnosis of thoracic and lung diseases, including COVID-19, are radiological imaging techniques such as chest x-ray (CXR) and computed tomography (CT) (Ozturk et al., 2020).The advantages of radiology techniques are that they can be detected in the early stages of the disease even if the scan is negative in some cases, and the disadvantages are that X-ray analysis requires a radiologist and manual reading is time-consuming.Therefore, it is necessary to develop an automatic analysis system that will detect abnormalities in scans to save time for healthcare professionals (Ouchicha et al., 2020).Thus, on March 16, 2020, America's white house announced that it is encouraging experts and researchers to use artificial intelligence (AI) techniques to combat the new COVID-19 pandemic (Alimadadi et al., 2020).
Two important scientific communities have emerged to combat COVID-19.The first is a collection of Artificial Intelligence (AI) in the form of automatic COVID-19 detection from a dataset of computed tomography (CT) scans and X-ray images.The second are mathematicians and epidemiologists who develop complex patterns of virus spread and transmission (Shuja et al., 2020).
This study uses commonly used deep learning approaches, fusion techniques, and X-Ray images to create an effective model.Among these pre-trained deep learning approaches, ResNet-50, ResNet-152, VGG-19, and DenseNet201 methods were selected (Narin et al., 2021).Three different scenarios were applied with these methods.First, patients with COVID-19 were classified among three different conditions (Normal,and Pneumonia).Secondly, cases with COVID-19 were classified among 2 different conditions Normal).As the third, cases with COVID-19 were classified among 2 different conditions Pneumonia).Two new models were created using these models.Our first model ENSEMBLE-1 consists of ResNet152, VGG-19, DenseNet201 models, and our second model ENSEMBLE-2 consists of ResNet50, VGG-19, DenseNet201 models.The results of these models are combined with the fusion methods at the decision level, and the effects of these methods on the classification performance are examined.
The remainder of this paper is organized as follows: Section 2 describes related work on COVID-19 and CNN models.Section 3 introduces the details of the methodology, and the results are discussed in Section 4. Finally, the paper ends with a general conclusion.

Related Work
When chest X-ray and CT studies in the literature are examined, it is seen that handmade and recently CNN-based methods have been used.Convolutional neural network (CNN) has shown promising results in the diagnosis and classification of various diseases.However, CNN needs a huge amount of labeled datasets for initial training.When it comes to medical images, it is difficult to obtain a large number of labeled images.In such cases, pre-trained CNNs can use large numbers of images (such as ImageNet).Pre-trained CNN has been successful in predicting COVID-19.Scientists were motivated by the need for rapid interpretation of X-ray images.Therefore, they proposed deep learning models (especially convolutional neural networks) to detect COVID-19-infected cases from chest X-ray imaging.
Autoregressive Integrated Moving Average (ARIMA), Nonlinear Autoregression Neural Network (NARNN), and Long Short-Term Memory (LSTM) approaches have been used to predict COVID-19 (Kırbas et al., 2020).Studies also present a new model called DarkCovidNet, which is based on the DarkNet model using X-Ray images (Ozturk et al., 2020).DarkCovidNet has been tested on binary classification (COVID-19andNormal) and multi-class classification  for COVID-19 detection.DarkCovidNet achieved an accuracy of 98.08% for binary classes and 87.02% for multi-class situations.Abraham et al. proposed a method for high-speed recognition of COVID-19 from chest X-ray images, using features from multiple CNNs.The method used a feature selection approach based on correlation and the BayesNet classifier (Abraham and Nair, 2020).Ouchicha et al. used a neural network-focused model that uses local and global features of chest X-ray images using two parallel layers with various kernel sizes.With this model, called CVDNet, they achieved an average accuracy of 97.20% (Ouchicha et al., 2020).
In addition, manual techniques were used together with CNN models.Detection and diagnosis were performed using MLP on fractal features and CNN on CXR.After the images were first converted to 1-dimensional vectors, the covariance matrix was calculated and fractal features were extracted with the help of eigenvalues and eigenvectors.The CNN architecture achieved a higher accuracy of 93.2% (Hassantabar et al., 2020).Studies have used pre-trained models  for feature extraction from chest X-ray images.Using a Support Vector Machine (SVM) classifier implemented with different kernel parameters, they achieved 94.7% accuracy on ResNet and SVM with linear kernel (Ismael and Şengür, 2021).
It is challenging to find labeled images of new diseases such as COVID-19, as training CNN requires large amounts of data.Transfer learning methods were also used to solve this problem (Xu et al., 2020).Using X-Ray images, they compared the performance of GoogleNet, ResNet101, Xception, and MobileNetv2 with the transfer learning method to solve the medical image imbalance problem.
Tuncer et al. tried a COVID-19 classification method using a residual sample local binary model called ResExLBP.However, a limited number of positive and negative labeled data were used in the study (Tuncer et al., 2020).Nilanjan et al. developed a new method to detect Pneumonia by calculating features from chest X-ray using a modified VGGNet19 (Visual Geometry Group Network) (Dey et al., 2021).This method achieved 97.94% classification accuracy.Ardakani et al. applied the deep learning technique to manage COVID-19 in routine clinical practice using CT images.They used 10-CNNs to distinguish COVID-19 disease from healthy individuals and presented the characteristics of these networks (Ardakani et al., 2020).
In the study where the performances of 5 different machine learning and deep learning algorithms were compared (Özbay and Altunbey Özbay, 2020), 98.1% accuracy was obtained with the Convolutional Neural Network algorithm.In another experimental study (Bozkurt, 2021), 98.17% success was achieved with the DenseNet121 model.In another study (Korkmaz and Atila B, 2018) that detects lung infection caused by Covid-19 using deep learning techniques, three different classes were used, and the classification success was found to be 97%.
The fusion method has been applied before and has been found to work well in many areas.The main contribution of this study is testing this method on current data such as Covid-19.The basic fusion functions have achieved superior classification performance, especially with the max and product functions.

X-ray Image DataSet
The data set was created by combining two data sets (Ozturk et al., 2020).The COVID-19 X-ray image database using images from various open-access sources was used in the study (Cohen et al., 2020).The authors compiled radiology images of COVID-19 cases from various sources (Radiology Society (RSNA), Radiopaedia, etc.) for research purposes, and most of the studies on COVID-19 use images from this source.This dataset is constantly updated with data shared by researchers from different regions.In addition, the ChestX-ray8 dataset provided by (X.Wang et al., 2017) was used for images of Pneumonia and a normal chest.A total of 1125 pictures were collected from these two sources.The dataset comprises 500 normal, 500 pneumonia cases, and 125 COVID-19 cases.The summary of the dataset used in the study is given in Table 1.Some chest X-ray image samples from the prepared data set are given in Figure 1.The architecture of the VGG (Visual Geometry Group) network (VGG19) includes 19 trainable weighted layers (Simonyan and Zisserman, 2014).The block structure of the VGG network consists of a combination of convolution and maximum pooling operations.VGG blocks start with two convolutional layers with 64 and 128 filters, respectively.Then, the third layer contains 256 filters (Figure 3

Ensemble Learning and Fusion Rules
Ensemble Learning is a machine learning prototype created by weak learners coming together to solve a similar problem.This learning method enables to create a model with more than one learner instead of training the model with a single learner.The output of each individual or single learner is taken as 'votes'.The final output (decision) is based on the majority response (Figure 4).Ensemble methods attempt to generate a set of hypotheses and combine them for use.Ensemble-based classification systems require a scoring mechanism to combine the prediction results of the elements of the ensemble and produce the final output.For this purpose, different fusion approaches such as max, majority vote, sum, mean, and product are used.wherein, multi-dimensional vector (y i1 ,y i2 ,…y iT ) represents the predicted output of y i on the training data sample and y i,j represents the output of y i on class c j .T is the number of learners with a value of 4.
Given the input feature vector , the .output of the classifier is shown as Eq. 2 for the three classes.
y(X) = Fusion(y 1 (X), y 2 (X), y 3 (X), y 4 (X)) The output of all basic learners can be represented as a decision profile in the form of a 4x3 matrix (Eq.4).

DP (𝑋
Additionally, the fusion result y() is a 3-dimensional vector represented by the measure layer form (d represents the decision for each class) shown in Eq. 5.

𝑦(𝑋
The fusion rules given in Table 3 are applied to each column of DP () and give the fusion output as a result of the operation.The ensemble scoring by fusion rules is given by Eq.1.4. Second, the models are trained to classify two classes (COVID-19 and Normal), as seen in Table 5.Third, the models are trained to classify two classes (COVID-19 and Pneumonia), as seen in Table 6.A dataset of 1125 images (125 COVID-19, 500 Pneumonia, and 500 Normal) was used to develop the model.The dataset was separated by 5-fold cross-validation with 80% training and 20% validation (Figure 5).Both the binary and multiclassification performance of the models were evaluated for each fold, and finally, the classification performances of the basic fusion methods were compared.The biggest strength of our ensemble deep learning system is that two selected models make their predictions according to fusion methods.Our model also reduces the misclassification error by predicting the new sample with multiple ensemble models.
Among the fusion methods, the Max function achieved the highest classification performance in three classification scenarios for both Ens-1 and Ens-2 (Table 4).Figure 6 shows the accuracy rates of the fusion functions for each fold for Ens-1 (left) and Ens-2 (right).While all fusion methods continued close to the same mean in Ens-1, the Max function showed the highest performance.Similarly, while the max function showed the highest performance in Ens-2, the lowest performance was obtained with Mode and Product.Binary classification performances are higher for each fusion function than for 3-classification (Table 5).The classification performance for COVID-19 vs Normal is 99% for Ens-1 and Ens-2.Among the fusion methods, the Sum function obtained the lowest value, while the other functions achieved the level of 99% (Table 5).

Figure 1 .
Figure 1.Chest X-ray images samples from the prepared dataset (Top: Normal, Medium: COVID-19, Bottom: Pneumonia) 3.2.Deep Learning Methods Deep learning, a sub-branch of Machine Learning, is a study area that covers artificial neural networks and similar machine learning algorithms that contain one or more hidden layers.Convolutional neural networks (CNN), on the other hand, are a sub-branch of deep learning.CNN is generally used to analyze visual information.A CNN has three basic types of layers: Convolutional Layer, Pooling Layer, and Fully-connected layer.Multiple convolution+pooling can be done in succession.Then there are several fully connected layers.Feature extraction occurs in both convolutional and pooling

Figure 2 .
Figure 2. ResNet-50 block diagram Deep convolutional networks use a deeper number of layers to improve classification and recognition accuracy, solving complex tasks for image classification.In neural networks, however, this causes the accuracy to saturate in the deep layers and then degrade.Residual training deals with this problem.
-a).VGG19 has been initialized with ImageNet weights and is preferred as a transfer learning model.Densely Connected Convolutional Networks (DenseNet) is a deep CNN proposed by Gao Huang et al(Huang et al., 2017).In DenseNet, each layer is fed-forward to all other layers, so DenseNet has n(n+1)/2 connections in total (Figure3-b).The feature maps of the previous layers are used as input to the active layer, and the feature map of the active layer is used as the input to the next layer.

Figure 4 .
Figure 4.The flow diagram of the study.The proposed framework is presented in Figure 4. Here, 4 CNN-based classifiers are seen.These classifiers are considered base learners.Each learner y i will predict a result y ensemble from the class set {c 1 :COVID-19,c 2 :Pneumonia,c 3 :Normal}.The ensemble strategy is formulated as follows. () =  argmax  ∑ =1 techniques to classify the COVID-19 disease has become quite common.Especially in binary classification, classification studies between healthy and COVID-19 are common.It is also very important to distinguish other types of lung disease from COVID-19 positive.Therefore, binary-classification (COVID-19 vs Normal, COVID-19 vs Pneumonia) and multi-classification (Normal, COVID-19, Pneumonia) were preferred in this study.An ensemble usually consists of basic learners.In our approach, these key learners were chosen as ResNet50, ResNet152, VGG-19, and DenseNet201.In this study, a bagging approach is considered to combine different deep learning models with basic fusion methods to increase classification accuracy.Two ensemble models were created to combine different deep learning models; Ensemble-1 (ENS-1) ve Ensemble-2 (ENS-2).Ensemble-1 model consists of ResNet152, VGG-19, DenseNet201 models, and Ensemble-2 model consists of ResNet50, VGG-19, DenseNet201 models.Three different scenarios were applied to these models.First, the models were trained for three classes (COVID-19, Pneumonia, and Normal) to classify X-Ray images into three categories, as seen in Table

Figure 5 .
Figure 5. 5-fold cross-validation used to split the dataset into training and validation

Figure 6 .
Figure 6.Multi-classification results of basic fusion functions

Table 6 .
ENSEMBLE Table 7 reaches 85% for Max, and 84% for Mode, Sum, Mean, and Product with basic fusion functions.These values reached 86% with the Max function applied in Ens-2.As can be seen from Table7, the model with the lowest 3classification performance was DenseNet-201 with 82%.

Table 8 .
Performance parameters of 4 different models on each fold for binary classification (COVID-19 vs Normal) Binary classification performances of the models remained stable at 98% and 99% levels.The ResNet-152 and DenseNet-201 models showed the lowest performance in both cases (COVID-19 and Normal and COVID-19 and Pneumonia).ResNet-50 and VGG-19 showed the highest performance in both cases (Table8, Table 9).

Table 9 .
All performance parameters of 4 different models on each fold for binary classification

Table 10 .
Comparison of accuracy rates of COVID-19 classification methods using chest X-ray images