Predicting DoS and DDoS attacks in network security scenarios using a hybrid deep learning model

3.1 CICIDS-001 dataset

The CICIDS-001 dataset is a recent and realistic dataset that contains network traffic data for IDSs in a cloud environment. The dataset was created by Ring et al. in 2018 using the OpenStack platform to simulate a network with several clients and servers, such as email servers and web servers [24]. The primary objective of using this dataset was to assess the model’s accuracy and its ability to handle diverse attack scenarios. The dataset captures both normal and malicious traffic, with a variety of attacks injected into the network by using predefined profiles or custom scripts. The dataset is divided into four parts, each corresponding to one week of data collection. The dataset is in a flow-based format, using the unidirectional NetFlow standard 2 to record the statistics of each network flow. The dataset contains 14 attributes, namely, 10 default NetFlow attributes and 4 additional attributes that provide more information about the flow direction, status, label, and sublabel. The dataset contains approximately 2.8 million records, with 79 types of attacks classified into four categories: port scan, ping scan, denial of service (DoS), and brute force (Table 1). It is publicly available online and can be applied for testing IDSs and evaluating it, especially for anomaly-based detection techniques [25].

3.2 CIC-IDS2017 dataset

This dataset was generated by the Canadian Institute for Cybersecurity (CIC) in 2017 to provide a realistic and up-to-date benchmark for NIDS. It consists of two parts: PCAP files and CSV files. The PCAP files contain the raw network packets captured from a simulated network environment, while the CSV files contain the extracted features and labels for each network flow. The dataset covers both normal and malicious traffic, with a total of eight attack scenarios: brute force, FTP, heartbleed, DoS, brute force SSH, web attack, infiltration, botnet, and DDoS (Table 2). It is generated by using a user profiling system called the B-Profile, which can mimic the abstract behavior of human interactions based on different protocols, such as HTTP, FTP and email. The dataset contains 16,545,685 records with 80 features for each record. The features include much information, such as timestamps, protocols, source, destination IP addresses, and derived information, such as packet length, duration, flags, payload size, and entropy. It was publicly available on CIC website and Kaggle. Our focus with this dataset was to address class imbalance and enhance precision. It can be used for various purposes, such as testing and evaluating NIDS techniques, conducting feature analysis and selection, applying ML algorithms, and performing data mining and visualization [26].

3.3 CIC-IDS2018 dataset

This dataset was initialized by the Canadian Institute for Cybersecurity (CIC) in 2017 to provide a realistic and up-to-date benchmark for networks (NIDS). The dataset consists of two parts: PCAP files and CSV files. The PCAP files contain the raw network packets captured from a simulated network environment, while the CSV files contain the extracted features and labels for each network flow. The dataset covers both normal and malicious traffic, with a total of eight attack scenarios: heartbleed, (DDoS), brute force FTP, web attack, infiltration, brute force SSH, DoS, and botnet (Table 3). The dataset was generated by using a user profiling system called B-Profile, which can mimic the abstract behavior of human interactions based on different protocols [13]. The dataset contains 16,545,685 records with 80 features for each record. Itb includes basic information such as source and addresses, destination IP, protocols, ports, and timestamps, as well as derived information such as packet length, duration, flags, payload size, and entropy. The dataset is publicly available on Kaggle. It can be used for various purposes, such as testing and evaluating NIDS techniques, conducting feature analysis and selection, applying ML algorithms, and performing data mining and visualization [27].

3.4 Comparison of the CICIDS-2018 and CICIDS-2017 datasets

The CICIDS-2018 dataset is a current and reliable dataset that contains various types of network intrusions and cyberattacks. It is an improved version of the CICIDS-2017 dataset, which was also produced by the Amazon Web Services platform. The CICIDS-2018 dataset has more data sources, more attack scenarios, and more extracted features than the CICIDS-2017 dataset. It also has a larger size and a higher diversity of network traffic. The CICIDS-2018 dataset consists of 80 statistical features that are computed in forward and backward directions, such as flow, packet length, and duration. The dataset has about five million records and is available online in PCAP and CSV formats. In this study, we used the CSV format [10]. The CICIDS-2018 dataset includes two patterns and seven attack techniques, such as brute force, heartbleed, DDoS attacks, DoS attacks, infiltration, botnet, brute force SSH, and web attacks [15]. The dataset architecture includes 50 systems, while the attacking entities include 31 servers. Table 4 shows the comparison between the two datasets. However, the number of web attacks in CICIDS-2018 is very low [28].

3.5 Preprocessing

Figure 1 shows multiple preprocessing steps, such as:

1. Drop null data: a process of removing the rows or columns that have missing values and reducing the noise and uncertainty in the data. Missing values can reduce the quality and accuracy of the model, as they can introduce bias, errors, or inconsistencies. By dropping null values, the data become more complete, consistent, maintain a balanced representation of classes, and reliable. However, dropping null values also has some drawbacks, such as losing information, reducing the sample size, or changing the distribution of the data [29].

2. Drop duplicate data: duplicated data are data that have the same values or attributes in more than one row or record in a dataset. Duplicate instances can skew the class distribution, leading to imbalance and reducing the quality and accuracy of the data analysis and modeling, increasing the storage space and processing time of the data, and introducing errors or inconsistencies in the data. Therefore, it is important to identify and remove duplicated data from a dataset before performing any data processing or analysis tasks.

3. Data type conversion: a process of changing the data type of the columns from int 64 and float 64 to int 32 and float 32, respectively. This can help reduce the memory usage and improve the performance of the data processing and analysis [30,31].

4. Handle imbalanced data: random undersampling is a technique that can be used to deal with imbalanced data. It can decrease the computational and memory requirement of the models, as it reduces the size of the dataset. It prevents the models from being biased toward the majority class, as it makes the classes more balanced. In addition, the performance of the models is improved, as it reduces the risk of overfitting on the majority class.

   (1) Imbalance ratio = maxi { Ci } mini { Ci } .

   The Ci parameter in the equation specifies the data size in the dataset. It is the ratio of the maximum number of samples in the dataset to the minimum number of samples.

5. Label encoding: this preprocessing is to convert the data categorical labels into numerical values, which can help simplify and standardize the data representation and avoid errors or inconsistencies when using different ML algorithms.

6. Missing value imputation: the purpose of this preprocessing is to fill in the missing values (NaN) in the input features with the mean value of each column and to normalize the feature values by removing the mean and reducing to unit variance [32,33].

7. Feature scaling: a technique that transforms the values of the features into a common range or scale, such as 0 to 1 or −1 to 1. This can help to enhance the performance and efficiency of ML algorithms.

8. One-hot encoding: this is the process of modifying the class labels in the target data into binary vectors that have a 1 in the position corresponding to the label and 0 s elsewhere. It can help simplify and standardize the data representation and avoid errors or inconsistencies when using different ML algorithms.

9. Correlation-based FS: choose a group of features that have less correlation with one another and strong correlation with the target variable to maintain or increase predictive power. It can help avoid multicollinearity issues, such as high variance, instability, or bias. The results of attack remains after FS are shown in Figure 2 [34].

Figure 1

Preprocessing data.

Figure 2

Attacks in each dataset after selecting the best ones: (a) CICIDS-001 attacks, (b) CIC-IDS2017 attacks, and (c) CIC-IDS2018 attacks.

4.1 CNN architecture

It is a type of multilayer perceptron. It basically consists of two main layers in the form of supervised learning. It operates in the form of convolution and subsampling in feature extraction. It proceeds like multilayer perceptrons in the classification algorithm [38].

CNN can be used for predicting cyberattacks by learning the patterns and features of network traffic data that indicate normal or malicious behavior. It can perform feature extraction and classification on network traffic data, which can be represented as one-dimensional sequences of numbers or symbols. It can apply convolutional filters and pooling layers to extract the most relevant and discriminative features from the data, such as packet length, duration, flags, payload size, and entropy, as in Figure 3. Then, fully connected layers and output layers are used to classify the network traffic data into different categories, such as normal, DoS, port scan, and brute force. SoftMax or sigmoid functions can also be used to output the probability of each category for each network flow [39]. The CNN can be trained and tested on labeled network traffic datasets, such as CICIDS-001 1 or CIC-IDS2017, which contain both normal and malicious traffic with different types of attacks. It can achieve high accuracy and robustness in predicting cyberattacks by using DL techniques, such as dropout, batch normalization, or residual connections [40,41].

Figure 3

Convolution neural network (CNN).

4.2 Long short-time memory (LSTM)

LSTM, a RNN, was proposed by Schmidhuber and Hochreiter in 1997. In the model, the data to be remembered are transferred to the next stage with the help of gates and cell state or forgotten by labeling it as unimportant. The data from the previous layer pass through the sigmoid function and are processed according to being in the range of 0 to 1, as in Figure 4. The LSTM algorithm was chosen because of its success on datasets with long-term dependency relationships [42]:

where i _t refers to the training data, W is the weight value, σ is the activation function, O _t is the output gate, B is the bias, f _t is the forget gate, C _t is the cellular cell, x _t is the input information, and h _t is the output information [43].

Figure 4

Structure of LSTM model.

4.3 XGBoost

XGBoost is a popular ML technique that can be used for network intrusion detection or other domains that involve complex and high-dimensional data. Network intrusion detection is a challenging task that requires accurate and efficient classification of network traffic into normal or malicious categories. XGBoost can achieve this by using a combination of tree-based learners that can capture the nonlinear relationships and interactions in the data. XGBoost also uses an additive training strategy that can iteratively update the model by adding new trees that minimize the loss function. XGBoost also uses regularization terms that can control the complexity and diversity of the model and prevent overfitting or underfitting [44,45]:

XGBoost has several parameters and hyperparameters that need to be tuned to optimize the performance and efficiency of the model. These parameters include the learning rate, which controls the step size of each update; the maximum depth, which control the tree’s depth; the tree’s number, which defines the iterations number; and the subsampling ratio, which controls the fraction of data used for each tree. These parameters are useful for handling the trade-off between bias and variance of the model and should be chosen carefully based on the data characteristics and objectives [46], as shown in Algorithm 1.

XGBoost has some advantages over other ML techniques or methods that are commonly used for network intrusion detection or other domains. For example, compared to Random Forest, which is another ensemble method based on trees, XGBoost can improve the accuracy and efficiency by using gradient boosting instead of bagging. Compared to SVMs, which are linear models that use kernels to map data into higher dimensions, XGBoost can handle nonlinear data more effectively and efficiently by using trees instead of kernels. Compared to DNNs, which are complex models that use multiple layers of neurons to learn features and patterns from data, XGBoost can achieve similar or better performance with less computational cost and complexity by using trees instead of neurons [47].

5.1 CNN architecture

A neural network (CNN) is a type of multilayer perceptron. It basically consists of two main layers in the form of supervised learning. These layers extract spatial features from raw network traffic data. They identify patterns related to network behavior, such as packet sequences and spatial correlations. It operates in the form of convolution and subsampling in feature extraction. It proceeds like multilayer perceptrons in the classification algorithm. The components of the CNN model are as follows:

• Two convolutional layers are added to the model with 32 filters and an activation function rectified linear unit (ReLU). These layers perform feature extraction from the input data by convolving the kernels with the data. The task of the CNN layers is to capture spatial patterns and local features in the data, which has the shape (number of samples, number of features, 1), and applies convolution operations to extract features from the data.

• A max 4 pooling layer uses the maximum value in each size 4 window to minimize the dimensionality of the convolutional layer’s output.

• A flatten layer reshapes the output from the max pooling layer into a one-dimensional vector that can be fed into a classifier.

5.2 Long short-time memory (LSTM)

LSTM, a RNN, was proposed by Schmidhuber and Hochreiter in 1997. In the model, the data to be remembered are transferred to the next stage with the help of gates and cell state or forgotten by labeling it as unimportant. LSTM processes sequential data, capturing temporal dependencies. It excels at recognizing long-term patterns, essential for detecting anomalies and attacks. The data from the previous layer pass through the sigmoid function and are processed according to being in the range of 0 to 1. The LSTM algorithm was chosen because of its success on datasets with long-term dependency relationships.

• A sequential model is a linear stack of layers that can be easily built using the sequential class from Keras model.

• The LSTM layers are stacked in sequence, with 64 units in each layer. These layers capture temporal dependencies in sequential data. The task of the LSTM layers is to learn long-term patterns and context from the input, with a shape (number of samples, number of features, 1), and apply recurrent operations to learn temporal dependencies from the data.

• Another LSTM layer with 64 units, the input to it, is the output from the previous LSTM layer and applies recurrent operations to learn higher-level temporal dependencies from the data. This layer returns only the last output for each input.

• A dense layer with five units and a softmax activation function is the output layer for multiclass classification. Linear transformation is applied to the output taken from the last LSTM layer, followed by a softmax function to make a probability distribution for the five classes.

• A compile method determines the optimizer, loss function, and metrics to be used when configuring the model for training. The optimizer in this instance is the adaptive learning rate optimization algorithm adam, which is appropriate for DL models. Categorical_crossentropy, a popular option for multiclass classification issues, serves as the loss function.

• A fit method uses feature vectors and the one-hot encoded labels as parameter to train the model on the training data by passing the as arguments. The method also specifies the number of epochs, batch size, and shuffle option to use during training.

5.3 XGBoost

This algorithm is based on the gradient boosting framework. It is used for various types of tasks such as regression, classification, and ranking tasks. It combines the predictions from several weak learners (trees) to create an effective ensemble model. It handles large-scale and imbalanced data efficiently. In the context of detecting anomalies and other attacks in network based on traffic data:

1. Model training: XGBoost can be trained on a dataset containing network traffic data, where features might include various types of information extracted from the traffic such as packet size and protocol type.

2. Labeling: the training dataset should be labeled with normal behavior and different types of known attacks or anomalies.

3. Feature relevance: rating that XGBoost generates can be used to determine which features are most predictive, which can help in understanding which features are most indicative of an attack or anomaly.

4. Anomaly detection: once trained, the model can predict whether new/unseen network traffic data is normal or an anomaly/attack.

Based on learned patterns found from the training set of data, the extracted features from CNN and LSTM are fed into XGBoost for final classification. XGBoost’s speed and accuracy contribute greatly to the model’s overall performance. The elements of the XGBoost model are as follows:

• The XGBClassifier is a class provided by the XGBoost library. It serves as an interface to the XGBoost algorithm, which is a powerful and efficient tree-based ensemble learning method.

• A parameter objective='multi:softmax' specifies the learning task and the corresponding loss function. In this case, the task is multiclass classification, and the loss function is softmax. It calculates the variation between the actual one-hot encoded labels and the expected probability distribution.

• Parameter num_class=5 specifies the number of classes in the classification problem. This parameter is needed for multiclass classification tasks.

• A fit method trains the model on the training data by passing the feature vectors from the LSTM model and the class labels as arguments. The method also converts the one-hot encoded labels into integer labels by using the np.argmax function from the np module, which returns the index of the maximum value along a given axis (Table 5).

Algorithm 2: Pseudo code for hybrid prediction algorithm

Input: Online Datasets (CICIDS-001, CIC-IDS2017, CIC-IDS2018)

Output: Prediction Model

1: Dataset Preprocessing

Drop Null data

Drop Duplicate data

Data type Conversion

Handle Imbalanced data

Imbalance ration = maxi { Ci } mini { Ci }

Label Encoding

Missing value imputation

2: Feature Selection

Feature Scaling

One-hot encoding

Correlation-based feature selection

3: Build CNN model (Filters: 32, AF: ReLU, Kernel size: 3, pool size: 4)

  cnn_model= Sequential(conv(filters=32, activation='relu', kernel_size=3, In put=(X_train), MaxPooling(pool_size=4))

 cnn_model.predict(X_train))

 cnn_model.predict(X_test)

4: Convert labels to one-hot encoded format

5: Build LSTM model (Units: 64, Dense layer:5 units, activation: SoftMax)

         lstm_model=Sequential (LSTM (64, input =(X_train_cnn), return_sequences=True),

     LSTM(64),

     Dense(5, softmax) )

 lstm_model. compile (categorical_crossentropy, adam, accuracy)

  lstm_model.fit(X_train_cnn, Y_train_one_hot, epochs=10, batch_size=32,shuffle=False)

 lstm_model.predict(X_train_cnn)

 lstm_model.predict(X_test_cnn)

6: Build XGBoost model ( Multi-class: SoftMax , Number of classes: 5)

  xgb_model = XGBClassifier(objective='multi:softmax', num_class=5)

  xgb_model.fit(X_train_lstm, np.argmax(Y_train_one_hot)

7: Final Prediction

            xgb_model.predict(X_test_lstm)

            8: Output: Evaluate the Model

              F1-score(y_test, y_pred)

              confusion_matrix(y_test, y_pred)

              precision(y_test, y_pred)

              recall(y_test, y_pred)

9: Test Model with new instance

6.1 Environment of the proposed system

The environment is a combination of hardware and software components that enable the execution of a specific task or application [48,49]. Here is a possible description of the environment:

1. Hardware:

   The hardware components consist of the mac OS 11.2, with CPU (134 GB), RAM (16 GB), and (T4 GPU) runtime environment, which provides additional services and libraries for running a specific type of application, such as a graphical user interface or a ML model.

2. Software:

   The software components consist of various libraries and frameworks that provide functions and tools for developing and running the application. The Python libraries and frameworks used in the proposed system are shown in Table 6.

6.2 Proposed model performance

In this part, we delve deeper into the results obtained from our proposed hybrid model, which seamlessly integrates CNN, LSTM, and XGBoost algorithms for network intrusion detection. Our evaluation focuses on three widely recognized datasets: CICIDS-001, CICIDS2017, and CIC-ID2018. The proposed system achieves high results with accuracy, F1-score, precision, recall, and it is used for predicting and classifying network attacks in the CICIDS-001, CICIDS-2017, and CICIDS-2018 datasets. The system performs better than the current approaches that employ only CNN or LSTM models. The results can be analyzed and compared as follows:

For the CICIDS-001 dataset, the primary purpose of this experiment was to assess the model’s accuracy and its ability to handle diverse attack scenarios. The system achieves an accuracy of 0.98, a F1-score of 0.98, a precision of 0.98, a recall of 0.98, and a weighted average of all classes. The system also achieves high scores for each individual class, such as 0.99 for DDoS, 0.99 for DoS GoldenEye, and 0.99 for PortScan (Table 7). These results indicate that the system can effectively detect and classify different types of attacks in the dataset, such as DDoS, DoS Hulk, DoS GoldenEye, and PortScan. Figures 6 and 7 show the the training and loss functions and confusion matrix for this dataset.

Figure 6

Training and loss for CIC-IDS2017.

Figure 7

CICIDS-001 confusion matrix.

For the CICIDS-2017 dataset, the model aimed to address class imbalance and enhance precision. Its achieves an accuracy of 0.99, a precision of 0.99, a recall of 0.99, and a F1-score of 0.99 for the weighted average of all classes. The system also achieves high scores for each individual class, such as 0.99 for DDoS, 1.00 for DoS GoldenEye, and 1.00 for PortScan (Table 8). These results obtained that the system can effectively detect and classify multiple types of attacks in the dataset, such as DDoS, DoS Hulk, DoS GoldenEye, and PortScan. Figures 8 and 9 show the training, loss function, and confusion matrix when applying the model on CIC-IDS2017.

Figure 8

Training and loss for CIC-IDS2017.

Figure 9

CIC-IDS2017 confusion matrix.

For the CICIDS-2018 dataset, the system achieves a precision of 0.99, an accuracy of 0.99, a F1-score of 0.99, and recall of 0.99 for the weighted average of all classes. The system also achieves high scores for each individual class, such as 1.00 for DoS attacks-GoldenEye, 1.00 for DoS attacks-Slowloris, and 0.99 for Benign (Table 9). These results obtained that the system can effectively detect and classify different types of attacks in the dataset. Figures 10 and 11 show the training and loss function and confusion matrix when applying the model on CIC-IDS2018.

Figure 10

Training and loss for CIC-IDS2018.

Figure 11

CIC-IDS2018 Confusion matrix.

6.3 Evaluation metrics

This metrics are used to compare the system’s output with the expected or true output and to quantify the errors or discrepancies [50]. The evaluation metrics used in the proposed system are as follows:

1. Accuracy: the percent of the number of correct predictions made by the accuracy rate classifier to the overall amount of data in the dataset. The accuracy rate measures how often the classifier makes an accurate prediction. The accuracy rate has a value between 0 and 1.0 represents the worst rate, and 1 represents the best rate. The accuracy rate is calculated as shown in the following equation:

   (11) Accuracy = True Positive + True Negative True Positive + True Negative + False Positive + False Negative .

2. Precision: the percent of successfully detected attacks to all attacks that the system was able to identify. It indicates how accurate the system is in detecting botnet attacks and avoiding false positives. The equation for precision is as follows:

   (12) Precision = True Positive True Positive + False Positive .

3. Recall: the percent of successfully detected attacks to the all attacks in the data. It indicates how sensitive the system is in detecting attacks and avoiding false negatives. The equation for recall is as follows:

   (13) Recall = True Positive True Positive + False Negative .

4. F1 score: the mean of harmonic of precision and recall. It indicates how balanced the system is in detecting attacks and avoiding both false positives and false negatives. The equation for the F1 score is as follows:

where true positive represents the total amount of attacks that the system correctly detects, true negative denotes the total amount of normal activities that that the system correctly detects, false positive indicated the number of normal activities that the system wrongly detects, and false negative represents the number of attacks that are wrongly identified as normal activities by the system [51].

6.4 Model applications

The hybrid CNNLSTM–XGBoost model holds enormous promise for network security across diverse applications. It excels in early intrusion detection, swiftly identifying abnormal traffic patterns and minimizing damage. When extended to internet of things networks, it safeguards connected devices, ensuring data integrity. Furthermore, the model’s ability to identify malware detection prevents propagation and data exfiltration. By analyzing email headers and URLs, it successfully detects phishing attempts by looking at email headers and URLs, preserving users from social engineering attacks. Additionally, its role in insider threat detection mitigates internal risks and secures sensitive information. Detecting zero-day vulnerabilities and integrating with SIEM systems enhances real-time threat response. Customization for specific industries ensures targeted threat insights, empowering proactive defense strategies and optimizing resource allocation [52].