HyproBert: A Fake News Detection Model Based on Deep Hypercontext

Abstract

News media agencies are known to publish misinformation, disinformation, and propaganda for the sake of money, higher news propagation, political influence, or other unfair reasons. The exponential increase in the use of social media has also contributed to the frequent spread of fake news. This study extends the concept of symmetry into deep learning approaches for advanced natural language processing, thereby improving the identification of fake news and propaganda. A hybrid HyproBert model for automatic fake news detection is proposed in this paper. To begin, the proposed HyproBert model uses DistilBERT for tokenization and word embeddings. The embeddings are provided as input to the convolution layer to highlight and extract the spatial features. Subsequently, the output is provided to BiGRU to extract the contextual features. The CapsNet, along with the self-attention layer, proceeds to the output of BiGRU to model the hierarchy relationship among the spatial features. Finally, a dense layer is implemented to combine all the features for classification. The proposed HyproBert model is evaluated using two fake news datasets (ISOT and FA-KES). As a result, HyproBert achieved a higher performance compared to other baseline and state-of-the-art models.

1. Introduction

Over the years, the increase in the usage of social media platforms has contributed to facilitating the phenomenon of spreading false information. The spread of fake news can be linked to achieving several goals, which can be targeted toward specific individuals, groups, politics, regions, religions, ethnicity, culture, race, etc. It can also be connected with illegal and fraudulent activities [1,2]. The generation and propagation of fake news has a huge negative impact on the economy, peace, and health all over the world. It is considered a major threat to journalism. It was reported in the year 2013 that the American stock market had lost over USD 130 billion over fake news of its president being injured in an explosion. In 2016, the US elections were packed with fake news glorifying and defaming both presidential candidates. One such fake news claim proclaimed “Pope Francis endorsed President Trump”. This resulted in huge turmoil among the rivals and started riots among the supporters. In recent years, fake news related to the spread and effects of COVID has disturbed the harmony in society, created social instability, and resulted in the economic collapse of several governments.

To overcome the adverse effects of fake news, its propagation needs to be stopped. There is a lot of research in place to moderate social media content with the help of human content moderators. Machine Learning (ML) algorithms, along with previously moderated data, are effective for fake news detection [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23]. However, machine learning models do not provide a significant level of efficacy and improvement over time for various datasets. Keeping in view the success of Deep Learning (DL) in various domains [24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41], this research highlights the significance of deep learning techniques for Natural Language Processing (NLP) tasks. The proposed model is a hybrid of deep models, adept placements, and hyperparameter tuning for an effective solution for fake news detection. It utilizes the most valuable expertise of deep learning models in a structured manner to provide an automated, efficient, and effective classification solution for identification and detection of fake news. The main contributions of this research in the field of fake news detection include:

i.

The introduction of a novel deep neural network model, $HyproBert$, for fake news detection;

ii.

Extraction and evaluation of content attributes at multiple orientations based on deep hypercontext;

iii.

Analysis of the developments in hyperparameter optimization on $HyproBert$.

Section 2 describes the literature review of fake news detection along with the limitations in this domain. Section 3 includes the methodology of the proposed HyproBert model. Section 4 includes the experiments and results. Section 5 discusses the evaluation results and comparative analysis. The research along with a description of future work is concluded in Section 6.

2. Literature Review

In this section, a review of various existing machine learning and deep learning techniques being used for detection of fake news is presented. This section also summarizes the status and limitations of fake news detection in the news media.

The research on fake news detection is divided into two types: supervised learning and unsupervised learning. Ozbay and Alatas [3] suggested a two-step method to identify false news using text information; the first stage entails preprocessing, and the second stage involves applying the textual feature vector to 23 clever supervised classifiers for further experimental assessment. By applying multiple word embedding approaches across five datasets in three languages sourced from different social media sites, Faustini Covoes [4] created a model for detecting false news that is independent of language and platform. To this end, Guo et al. [5] thoroughly reviewed the advancements in the field of bogus information detection, detailing the many problems, unique methodologies, and specifics of available datasets.

For the first time in the domain of fake news detection, Ozbay and Alatas [6] employed two metaheuristic algorithms for the identification of fake news. Grey Wolf Optimization (GWO) resulted in higher performance for social media data. Kumar et al. [9] used five different classifiers and thirteen different attributes to categorize tweets; they also used Particle Swarm Optimization (PSO) to extract an ideal feature set from the tweets’ text and improve classifier performance.

Perez-Rosas et al. [7] created two new datasets for automatic identification of false news, each including seven different types of news related content. They laid out an extensive framework and patterns for distinguishing between authentic and fake news. In order to identify fake news, Ahmed et al. [8] developed a novel dataset called ISOT that was compiled from actual news stories around the globe. Linear Support Vector Machine (LSVM) classifier along with Term Frequency–Inverse Document Frequency (TF–IDF) evaluation for feature vector representation were utilized for the classification of fake news. However, these methods require bespoke features, which adds to their production and time requirements. Akyol et al. [10] determined applicability of Random Forest (RF), Gradient Boost Tree (GBT), and Multilayer Perceptron (MLP) for the concept of fake news detection.

With the rise of deep learning theory, an increasing number of researchers employ it to detect fake news. Recent research has been progressively more focused on unsupervised and semi-supervised detection methods. The capacity of deep learning models to automatically extract high-level properties from news articles is a major draw for academics; this makes these methods particularly useful for diagnosing fake news [11,12,13,14,15,16,17,18,19,20]. Goldani et al. [19] recommend the use of embedding models and Convolutional Neural Networks (CNN) to detect bogus news. During model training, static and dynamic word embeddings were compared and gradually updated. Their approach was evaluated using two public datasets. ISOT datasets improved accuracy by 7.9% and 2.1%, respectively.

Ma et al. [14] developed a Recurrent Neural Network (RNN) to represent the textual data sequence for rumor detection. Using text data, Asghar et al. [11] created a deep learning model that recognizes rumors by combining CNN with Bidirectional Long Short-Term Memory (Bi-LSTM). In order to effectively manage the textual contents in a bi-directional method, Kaliyar et al. [13] developed a hybrid of the CNN and BERT (Bidirectional Encoder Representations from Transformers) model, named FakeBERT.

Yu et al. [18] emphasized the shortcomings of RNN-based algorithms for the early identification of disinformation. They suggested a CNN model for extracting significant features and spotting propaganda. To identify and categorize bogus news, Shu et al. [15] took a pragmatic approach by establishing a co-attention method to find the top 1000 most important lines in the material and combine them with the top 1000 most significant user reactions. Autoencoder-based Unsupervised Fake News Detection (UFNDA) was proposed by Li et al. [20]. Their methodology combines the context and content data from news to produce a feature vector that will improve the identification of false news. For early rumor detection, Chen et al. [12] developed an attention-based RNN to accumulate unique language properties over time. Wang [16] offered a new dataset named LIAR. He suggested a deep learning model that combines CNN and Bi-LSTM to extract textual features and meta-data features, respectively. Yin et al. [17] employed Support Vector Machine (SVM) classifiers to determine correlation among data. CNN and Principal Component Analysis (PCA) are utilized as the feature extraction tools. In another study, CNN, Long Short-Term Memory, and Convolutional Long Short-Term Memory (C-LSTM) are all used in the ensemble learning model for fake news detection presented by Toumi and Bouramoul [42]. The hybrid technique of CNN and RNN was proposed by Nasir et al. [43]. At first, CNN was employed for feature extraction close to the point of interest, whereas RNN was used for discovering a correlation. The difficulty of optimizing RNN is exacerbated by the vanishing and exploding gradient problem [44].

Current Status and Limitations

The current literature analysis concludes with the reality of false news and its uncontrolled spread. The extraction of deep semantic and contextual information is crucial for the identification and classification of propaganda. Extracting context from an instance’s several orientations is a key function of the Capsule Network (CapsNet) [45]. CapsNets are also utilized for NLP in various studies, including text-based categorization [46], misleading headlines recognition [47], and opinion categorization [48]. It has also been utilized specifically for fake news detection [49,50], but these studies lack the effective exploration of CapsNet. While DL models have seen extensive use in analyzing news articles, they have yet to work out the following: retaining long-term word dependencies, using a parallelization technique in training, accepting bidirectional input sentences, and maintaining an attention mechanism. The proposed HyproBert model integrates the convolution layer, BiGRU, and CapsNet with self-attention to extract the spatial features along with contextual information and hidden representation from multiple orientations. In the current work, DistilBERT, a lighter and more efficient version of the BERT model has been implemented to extract context-based high-level characteristics from news text. A BERT-based model outperforms alternatives in terms of training loss decay time [13]. Adopting bidirectional pretrained word embeddings accelerates model training and improves classification performance [21]. Additionally, the proposed HyproBert model integrates the convolution layer, Bidirectional- Gated Recurrent Unit (BiGRU), and CapsNet with self-attention to extract the spatial features along with contextual information and hidden representation from multiple orientations. The BiGRU model is superior in its ability to capture the semantic combination of text and to store extended context data [50]. A CapsNet equipped with a self-attention mechanism is employed to simplify the routing of CapsNet at each layer [49]. The goal of the proposed HyproBert model is to improve the performance of classification by offering an interesting, effective, and efficient model to detect fake news.

3. Methodology

In this section, a detailed description of the proposed model HyproBert for fake news detection in news articles is presented.

The proposed HyproBert model is presented in Figure 1. The title of the news article along with the complete text is utilized for the tokenization process. Afterward, embedding vectors are generated to represent features in the solution space for further classification. Initially, CNN is applied along with operations to extract the spatial features from the embeddings. BiGRU is applied sequentially to the output of CNN to gather contextual information. Capsule networks are used along with the attention layer to enhance the spatial features [46]. Although CNN has extracted the spatial features from the contents, the capsule network identifies the spatial relations between the extracted features to model the hierarchical relationships within the data [45]. The proposed HyproBert model is a combination of essential, efficient, and effective DL models. The sequential processing of input extracts, enhances, and correlates highly valuable features semantically and contextually [51]. The introduction of the attention layer between the BiGRU and CapsNet enlightens context awareness [46] and helps CapsNet to identify semantic representation and hidden contextual information [49]. The details of the steps involved, from preprocessing data to the final output of the proposed HyproBert model, are provided in the following section.

3.1. Data Preprocessing

In order to utilize a dataset, it must be cleansed according to the requirements of the model. It will significantly help the model to understand the data in a better manner and increase its performance. Generally, it includes steps, such as stop word removal, tokenization, removal of special characters and spaces, numbers to words, sentence segmentation, etc. The datasets under consideration are real-world news datasets, and they contain several URLs. In this research, we examine text semantics for deep textual context. As a basic preprocessing step, we removed URLs, stop words, and other noise from the given data before feeding it into the proposed model.

3.2. The Proposed HyproBert Model

The components of the proposed HyproBert model along with the function and significance of each component is described as follows.

3.2.1. Input Layer

After preprocessing, the data are sent into the HyproBert input layer. The input layer is responsible for separating the title and text of a news article into smaller tokens. Furthermore, each token is indexed with a unique number using a dictionary. Additionally, padding is used to keep the length of the input text constant. Finally, numerical vectors are created from all the text from the news article N, converted into tokens t, and indexed with a dictionary D, such that $N\in D^{1\times t}$. $DistilBertTokenizer$ is used for end-to-end tokenization.

3.2.2. Embedding Layer

The embedding layer is responsible for learning the word representation based on training over immense real-world data. It also finds that words that are connected conceptually have comparable vector representations. In this research, we use DistilBERT embedding, which is an approximate version of the BERT. It performs around 97% similar to the BERT model and uses the approximation of the posterior distribution generated by the BERT model [21]. It has over 110 million parameters and is still much smaller than BERT. The number of transformer layers is also reduced to six layers. Furthermore, the training and prediction times are significantly smaller. This makes DistilBERT the ultimate choice for the proposed model.

3.2.3. Convolutional Layer

Convolutional layers are the fundamental constituents of convolutional neural networks [18]. The output of neurons is controlled by the convolutional layer. By figuring out the scalar product of their weights and the density region of the input, the neurons are linked to parts of the input that are close to them [19].

The convolution layer is utilized in HyproBert to extract the spatial features from the embedding layer. In experimentation, a one-dimensional convolution layer along with 128 filters of 3 sizes is employed. The local spatial features are combined in a pooling operation to generate high-order features. $MaxPooling$ function is utilized for the pooling operation. $ReLU$ is used as an activation function to improve the performance. The feature sequence can be represented as $S_i=S(\left(W_n.x_n\right)+b)$), where i is the index of the feature sequence, $W_n$ is the filter weight, $x_n$ is the window size, and b is the bias weight.

3.2.4. Bigru Layer

The gated recurrent unit (GRU) is the fundamental building block of the Gated Recurrent Neural Network (GRNN). At each iteration, GRNN takes a textual vector as input and uses the previous iteration’s output vector as a weight in a weighted sum that is used to update the nodes in its hidden layer. The present context is built using a bias vector to selectively retain or discard related information. The operation of a hidden layer is managed by the following equations:

$$\begin{array}{c}{I}_t=\sigma \left(W_I·\left[y_{t-1},x_t\right]\right)\\ K_t=\sigma \left(W_k·\left[y_{t-1},x_t\right]\right)\\ {\tilde{y}}_t=tanh\left(W·\left[r_t\ast y_{t-1},x_t\right]\right)\\ y_t=\left(1-I_t\right)\ast y_{t-1}+I_t\ast {\tilde{y}}_t\end{array}$$

(1)

where $I_t$ is the update gate, $K_t$ is the reset gate, $\sigma$ is a sigmoid function, $tanh$ is a hyperbolic tangent function, and $W_I$, $W_k$, and $r_t$ are training parameters. A $BiGRU$ network is capable of understanding and learning relationships between current, past, and future data, which is effective for extracting deep features from the input sequence [50]. The structure of $BiGRU$ is presented in Figure 2.

The proposed model HyproBert utilizes BiGRU for its specialties in processing context and extracting sentence representations with forwarding and backward direction propagation [50]. The BiGRU layer is applied directly to the output of the convolution layer. A forward GRU processes the input sequence of ${(S}_1,S_2,S_3,\dots ,S_{128})$, and the backward GRU processes the input sequence of ${(S}_{128},S_{127},S_{126},\dots ,S_1)$. Later, both GRUs are integrated to combine the collected context information. As a result, the output of the BiGRU layer has an improved feature representation of propaganda incorporated in the input news text.

3.2.5. Attention Layer

The contextual information and representations from BiGRU are stored in a fixed-size vector. In the case of large input sequences, the related information cannot be fully stored in the vector, which hinders the understanding of the model, hence decreasing the overall efficiency. The attention mechanisms are known to focus on the most important parts of input while generating dynamic adaptive weights [22,23]. Furthermore, self-attention (SA) includes the interaction of inputs all together to generate an output for a single input, as shown in Figure 3. SA is a powerful tool for identifying self-awareness and dependencies amongst input sequences [46]. SA considers the entire context of vector representations and substitutes the words on the principle of location and associated weights.

In the proposed model, HyproBert, SA is used to highlight the context awareness and relevant features to detect the hidden relations in news data. This will result in a better understanding of the data and increase classification accuracy.

$$Attention(Q,K,V)=Softmax\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)V$$

(2)

where Q is query, k is key, v is the value, and $d_k$ is the linear dimension of k.The HyproBert model uses the self-attention process; therefore, it sets Q = K = V. This offers the benefit that the present position’s information and the information of all other places may be computed to capture the interdependence throughout the full sequence. If the input is a sentence, for instance, each word must be attention-computed alongside all other words in the phrase.

3.2.6. Capsule Network Layer

Generally, CNN is known to mislay important information during the classification process. A higher performance of CNN requires hyperparameter tuning, which is a cumbersome and manual task. Hinton et al. [52] presented the CapsNet, a novel neural network architecture. It collects the syntactically enhanced characteristics from the input data while considering the positions and order of other words in a sentence. It has the ability to distinguish between full and partial relations in textual data. It outperforms a CNN in terms of recognizing a representation of the data and underlying relevant information in the input text [46,47,48]. Over the years, it has shown outstanding results for text categorization and information retrieval.

CapsNet is a combination of several capsules connected as neurons to detect semantic and syntactic information from input data. The capsules are presented as vectors of classification probability, and the direction of the capsules represents the position of text in the input. Additionally, the weight of hidden features is adjusted by the dynamic routing algorithm [53]. This improves and controls the limits on attention and connections between capsules to optimize the capsule network.

The proposed $HyproBert$ model employs the capsule network to develop and focus the semantic and syntactic awareness of the news text input [49]. The output from the attention layer is provided as input for CapsNet. Initially, a nonlinear function is used to convert the input to a feature capsule $F_i$, which is then used to produce a prediction vector $P_{j\parallel i}$ along with a weight matrix $W_{ij}\forall P_{j\parallel i}={W_{ij}F}_i$ to forecast the relationship between the layers of the input and the outcome. Furthermore, the coupling coefficients $C_{ij}$ are calculated using the dynamic routing algorithm, represented in Equation (5). The vector $T_{ij}$ is updated throughout the number of iterations $\exists T_{ij}\parallel T_{ij}+R_j^{M}f(F_i,{\theta }_j)$. This process provides attention to the high weights of related propaganda words and overlooks the irrelevant, less effective words, as shown in Figure 4. As a result, the output of the capsule has a higher-level contextual representation considering various orientations. The output of the capsule can be represented as Equation (3), where $B_{ij}$ and $B_{if}$ represent the standard exponential function for the input vector and the output vector, respectively. The final output of capsule $R_j$ contains a local ordering of words considering various orientations; it is represented in Equation (5).

$$C_{ij}=\frac{exp\left({\mathit{B}}_{ij}\right)}{{\sum }_{f}exp\left({\mathit{B}}_{if}\right)}$$

(3)

$$O_j=\sum _{i=1}^n{C}_{ij}{P}_{j\mid i}$$

(4)

$$R_j=\frac{{∥O_j∥}^2}{1+{∥O_j∥}^2}\frac{O_j}{∥O_j∥}$$

(5)

3.2.7. Dense and Output Layer

The proposed HyproBert model uses the output of the capsule network layer as input to the fully connected layer. This layer classifies the input by applying the sigmoid activation function. The collection of higher probability scores determines the news article to be fake news and vice versa.

The orderly processing of the proposed HyproBert model is presented in Algorithm 1. Firstly, the processing of HyproBert is set to sequential. The news article’s title and the text body are separated for tokenization. The tokens are then used for word embeddings. DistilBERT is used for both tokenization and embedding [21]. A single layer of one-dimensional CNN executes the word embedding vectors to extract the spatial features from the data. An optimized dropout of 0.4 is allotted to the model. Later, two layers of BiGRU are deployed. The self-attention layer, along with the capsule layer with optimized hyperparameters, is deployed. A dense layer with sigmoid activation is deployed to integrate the features required for the classification process. Finally, HyproBert is compiled using the Adam optimizer.

Algorithm 1$HyproBert$

Require:  News articles Ensure:  Fake news  1: Model ← Sequential ()  2: Model ← model.add(DistilBertTokenizer())  3: Model ← model.add(Distillbert embeddings())  4: Model ← model.add(Conv1D(128, 3, activation = ReLU))  5: Model ← model.add(DropOut(0.4))  6: Model ← model.add(BiGRU(128))  7: Model ← model.add(BiGRU(128))  8: Model ← model.add(Self-attention)  9: Model ← model.add(Capsule(3, 5, 4))  10: Model ← model.add(Dense(1, activation = Sigmoid))  11: Model ← Model.compile(binary_crossentropy, Adam,)

4. Experiments and Results

The experimental details are presented in this section. It includes datasets, experiments, hyperparameter settings, evaluation metrics, comparative analysis, and results.

4.1. Dataset

In this research, we validated the proposed HyproBert model using two datasets i.e., FA-KES (https://zenodo.org/record/2607278 (accessed on 9 May 2022)) and ISOT (https://www.uvic.ca/engineering/ece/isot/datasets/fake-news/index.php (accessed on 9 May 2022)). The specifications of the employed datasets are provided in the

Table 1. Multi-label text news dataset specifications.

Dataset	Total	Topic	# Of Articles	Class
FA-KES	804	miscellaneous	378	Fake
		miscellaneous	426	True
ISOT	44898	government news	1570	Fake
		Middle East	778	Fake
		US news	783	Fake
		left news	4459	Fake
		politics	6841	Fake
		news	9050	Fake
		world news	10,145	True
		politics news	11,272	True

.

The FA-KES [54] is a fake news dataset. It contains 804 news articles. These news articles are related to the Syrian war and contain title, date, location, and news text information. All the articles are marked, i.e., fake news is marked withzero, and real news is marked as one. It is considered a well-balanced dataset as 47% of news articles are fake news and the rest (53%) are real news.

The ISOT [55] is also a fake news detection dataset collected by the ISOT lab at the University of Victoria. It consists of 45,000 news articles. The collected news is related to politics and world news between 2016 and 2017, containing the title, date, topic, and news body text. The source for true news is the Reuters website, and fake news is collected from various sources, including PolitiFact and Wikipedia. This is also considered a well-balanced dataset, with almost an equal distribution of fake news and true news.

4.2. Experimental Settings

The proposed HyproBert model is developed and evaluated on $Jupyter$ (https://jupyter.org/ (accessed on 26 May 2022)) notebook based on $GoogleColab$ (https://colab.research.google.com/ (accessed on 26 May 2022)) using $Python$ (https://www.python.org/ (accessed on 26 May 2022)) programming language. $Colab$ is a cloud-based tool that provides the environment to execute code along with the required GPUs and TPU for high performance. The reason for choosing Python as a preferred programming language is its fast execution and vast support libraries for building machine learning models.

We applied $pandas$ (https://pandas.pydata.org/ (accessed on 30 May 2022)) and $NumPy$ (https://numpy.org/ (accessed on 30 May 2022)), which are open-source libraries that provide data structures and data analysis tools for data manipulation and analysis. We applied $NLTK$ (https://www.nltk.org/ (accessed on 30 May 2022)), a natural language toolkit in the extraction and characterization phase. $TensorFlow$ (https://www.tensorflow.org/ (accessed on 30 May 2022)) is an open-source library package developed by Google to help researchers to build and test deep learning models. We have used the sequential model of the $Keras$ (https://keras.io/ (accessed on 30 May 2022)) TensorFlow deep learning Python library to build and train the proposed HyproBert model. Additionally, $Matplotlib$ (https://matplotlib.org/ (accessed on 30 May 2022)) a package of python is used for plotting graphs and charts.

4.3. Hyperparameter Settings

The datasets under consideration (FA-KES and ISOT) are divided into training and testing subsets, with 80% and 20%, respectively. The optimized hyperparameters for the convolution layer are set as the number of filters to be 128 and a pool size of 02. Furthermore, the number of neurons employed by BiGRU is 128. An attention layer is implemented to extract the hierarchical context representation using the weights of the keywords [25]. Finally, the capsule network with three routing iterations and five capsules with four dimensions is utilized. Additionally, the proposed HyproBert model uses a dropout of 0:4, a batch size of 128, an epoch of 100, and sigmoid as an activation function and Adam as an optimizer.

4.4. Comparison Models

The performance of the proposed HyproBert model is compared with baseline machine learning and state-of-the-art deep learning models including Logistic Regression (LR), Random Forest (RF), Multinomial Naive Bayes (MNB), K-Nearest Neighbors (KNN’s), CNN, RNN, LSTM, BiLSTM, and BiGRU. Baseline models were tested with their default parameters, while the state-of-the-art models are evaluated using the settings provided by the related authors. The efficacy of the proposed HyproBert model is evaluated using various a hyperparameter setting to optimize the performance for the fake news detection. The proposed HyproBert model is evaluated using standard evaluation matrices, i.e., $Accuracy$, $Precision$, $Recall$, and $F_{Measure}$ [56].

4.5. Results

The result of the proposed HyproBert model on FA-KES and ISOT datasets along with the comparison is shown in

Table 2. Results of all models on the FA-KES dataset.

Dataset	Accuracy	Precision	Recall	F_Measure
LR	49.82	50.46	49.65	49.27
RF	53.92	56.39	53.42	54.14
MNB	38.88	39.40	38.74	32.86
KNNs	57.62	58.53	57.17	57.30
AB	47.57	49.12	47.27	47.12
DT(Elhadad et al. [57])	58.63	63.35	58.21	50.85
CNN	50.68	55.74	50.35	48.83
RNN	50.75	51.27	50.53	50.43
LSTM	59.66	58.24	59.17	58.92
BiLSTM	59.75	53.42	59.45	58.92
BiGRU	60.85	57.97	60.28	60.44
BiLSTM-CNN+task-specific word embedding [11]	60.9	61.48	58.69	60.19
CALLATRUMORS [12]	60.45	61.52	59.84	60.12
FakeBERT [13]	60.95	61.87	59.89	60.43
Ensemble model [42]	54.74	55.25	54.18	54.27
Hybrid CNN-RNN [43]	60	59	60	59
HyproBert	61.15	60.40	61.07	61.09

and

Table 3. Results of all models on the ISOT dataset.

Dataset	Accuracy	Precision	Recall	F_Measure
LR	52.38	50.15	52.34	42.92
RF	92.42	92.25	92.04	92.18
MNB	60.25	60.34	60.35	60.47
KNNs	60.14	67.35	61.57	56.39
AB	92.29	91.49	91.34	91.25
DT(Elhadad et al. [57])	100	–	–	–
CNN	99.24	98.45	98.72	98.44
RNN	98.74	98.21	98.42	98.19
LSTM	98.33	98.44	97.25	97.64
BiLSTM	98.43	98.61	97.11	98.75
BiGRU	98.68	98.17	96.73	98.21
BiLSTM-CNN+task-specific word embedding [11]	98.82	98.26	97.28	98.24
CALLATRUMORS [12]	97.47	96.59	97.17	97.28
FakeBERT [13]	99.12	98.8 99	98.93
Ensemble model [42]	87.17	89.47	86.50	86.53
Hybrid CNN-RNN [43]	99	99	99	99
HyproBert	99.30	99.12	99.14	99.20

below. The results reveal that the HyproBert model surpassed the performance benchmarks for the FA-KES and ISOT dataset in terms of $Accuracy$, $Precision$, $Recall$, and $F_{Measure}$.

Although, Elhadad et al. [57] reported 100% accuracy with their Decision Tree classifier on the ISOT dataset, the accuracy of their approaches varied from 85% to 100%. Only 25,200 articles from the ISOT dataset were used; therefore it appears they only used a portion of it. The proposed HyproBert method performs better than the Elhadad et al. [57] state-of-the-art method. When it came to accuracy, Random Forests performed significantly better than other baseline approaches on both datasets. For FA-KES, KNNs performed best, while Decision Trees performed better for ISOT among baseline models. Hybrid CNN-RNN [43] used a combination of convolutional and recurrent neural networks which resulted in competitive performance among the state-of-the-art models. The ensemble model [42] presented a decay in performance for both datasets.

The proposed HyproBert model outperformed the baseline models and state-of-the-art models. Various factors contributed to achieving the high performance of HyproBert, including the selection of the hyperparameter values, which played a vital role in higher accuracy [6]. The choice of one-dimensional CNN executed the extraction of spatial features efficiently and effectively [23]. The BiGRU was deployed in a multi-layer manner, which captures and integrates different textual features simultaneously [50]. By including input gates, output gates, and forget gates, it was able to circumvent the issue of long-term reliance. The multi-layer BiGRU provided significantly better feature extraction. However, increasing the number of BiGRU layers will increase the processing cost exponentially. Therefore, the optimal number of layers is obtained from experimentation. To remember the background details that BiGRU has supplied, the attention mechanism computes an adaptive weight that changes constantly [23]. It strengthens the generalization ability and solves the overfitting problem [46]. A self-attention layer along with CapsNet is also added to the ensemble to dynamically highlight the syntactically explicit features considering various orientations [49].

The classification performance of balanced datasets is often superior to that of imbalanced datasets [58]. Conventional classifiers may not be adequate for imbalanced classifications because most classifiers assume the classified dataset is balanced to achieve maximum accuracy. However, due to a lack of data, minority-class samples have low prediction accuracy. Classifiers have a strong tendency to favor samples from the majority class and ignore those from the minority class [59,60]. In this context, the usage of a deep neural network appears promising, as it can comprehend the essential distribution of the dataset and therefore avoid overfitting.

5. Discussions

As seen in the previous section, the proposed HyproBert model outperformed all the baseline and state-of-the-art models for fake news detection. The study of baseline models provided the intuition for the development of HyproBert. The best performing baseline models and advanced models are integrated and tuned in a logical manner to solve a fake news detection problem effectively, efficiently, and correctly. There are various reasons that contribute to the high performance of HyproBert. Various arrangements and hyperparameters were involved to obtain the optimal outcomes from HyproBert. These optimal values were gathered using extensive experimentation for each parameter involved. The details and effects of these parameters are described in the section below.

5.1. Convolutional Layer Optimization

Hyperparameters play a vital role in the performance of deep neural networks. An optimal set of hyperparameters produces a highly optimized output. The filter size is an important parameter in detecting the spatial features from the input. Various configurations of filter size are experimented with to achieve the optimal filter size. The optimal size of filters is determined to be three for both datasets under consideration. The effects of the filter size on the performance of the proposed HyproBert model are presented in Figure 5.

Another important parameter to enhance the performance of the proposed HyproBert model is the activation function. The activation function is responsible for the activation of neurons in a deep neural model. Two activation functions, SoftMax and sigmoid, were considered for experimentation. The results presented in Figure 6 have revealed that the sigmoid activation function enhances the performance of the proposed HyproBert model for both datasets due to the binary nature of the classification solution under consideration.

The number of epochs is also considered an important factor in improving the performance of the proposed HyproBert model. Increasing the number of epochs does not always result in more elevated performance. Therefore, we need to detect the optimal number of epochs to satisfy the training requirements of HyproBert. The optimal number of epochs was determined as 100 and 200 for the FA-KES and ISOT datasets, respectively. The effects of the number of epochs on the proposed HyproBert model are presented in the Figure 7.

5.2. Bigru Optimization

The number of neurons in a BiGRU highly affected the classification performance of the proposed HyproBert model. The experimentation included the number of neuron values from 32, 64, 128, and 256. The optimal number of neurons was determined to be 128 for both datasets. The impact of the number of neurons of BiGRU on HyproBert is represented in Figure 8.

A single BiGRU is very effective in the classification process. The performance of BiGRU can be enhanced by deploying it in a multi-layer manner. Increasing the number of layers of BiGRU will increase resource utilization as well [50]. Therefore, to a certain extent, it is considered a trade-off between available resources and performance. The optimal number of BiGRU layers was determined to be two for both datasets. The results of the increasing number of layers on the $Accuracy$, $Precision$, $Recall$, and $F_{Measure}$ are presented in

Table 4. BiGRU layers with HyproBert on the FA-KES dataset.

# of Layers	Accuracy	Precision	Recall	F_Measure
1	59.97	59.98	59.62	59.74
2	61.10	60.89	60.60	59.97
3	61.14	59.86	59.63	59.74
4	61.15	60.40	61.07	61.09

and

Table 5. BiGRU layers with HyproBert on the ISOT dataset.

# of Layers	Accuracy	Precision	Recall	F_Measure
1	99.07	98.63	97.62	98.74
2	99.10	98.93	97.60	98.96
3	99.11	98.61	97.63	98.99
4	99.30	99.12	99.14	99.20

for the FA-KES and ISOT datasets, respectively.

The experimentation also included testing various multi-layers of BIGRU to find the optimal number of layers of iterations and training time for the proposed HyproBert model. The effect of increasing Bi-GRU layers and the number of iterations is represented in Figure 9.

5.3. Capsnet Optimization

Just like with conventional neural networks, the performance of CapsNet heavily depends on its hyperparameters. Generally, the determination of hyperparameters requires experimentation because there is no specific optimal value for each hyperparameter. The optimal values vary for different data and solution types. One such hyperparameter is the number of capsules. The optimal number of capsules was determined to be five for both datasets. The capsule is the set of neurons that highlight a specific property. The effects of the number of capsules for the proposed HyproBert model are presented in Figure 10.

The next important consideration is the dimensions of the capsule. The lower dimension of the capsule will contain fewer features, and it will lose most of the information. A higher-dimensional capsule will undoubtedly contain more important features, but the computation complexities will increase. Therefore, an optimal number of dimensions is needed for optimal output. The dimension of a capsule indicates the length of the output vector. The dimension values, including two, four, and eight were tested for both datasets under consideration. After considering various options during the experimentation, the optimal dimension of the capsule was determined to be four for both datasets. The effects of the dimensions of a capsule on the proposed HyproBert model are presented in Figure 11.

The number of routing iterations is also a hyperparameter of CapsNet. It is responsible for connecting the capsules of the consecutive layers. The experimentation process for evaluating the optimal value of routing iteration was performed on both datasets under consideration. The testing range of values included 2, 3, 4, and 5. The optimal value was determined to be three iterations for both datasets. The effects of the number of routing iterations on the proposed HyproBert model are presented in Figure 12.

5.4. Imbalanced Data Study

The discrete imbalanced data problem affects many real-world applications and hinders the effectiveness of machine learning models when identifying samples from the marginalized group. There have been several investigations on this issue, with a concentration on imbalanced statistical datasets. However, imbalanced textual datasets have received relatively little attention [60]. The ISOT dataset is well-balanced, featuring both fake and real news articles [55]. The focus of this part of study is on skewed data, so we merged the ISOT dataset with 10,000 real news form the RCV1-v2 [61] dataset and named the resulting collection ISOT-R10.

Ranking metrics, threshold metrics, and probability metrics are the three types of performance evaluation measures [62]. In classification evaluation, basic threshold metrics are more frequently used. As the outcomes of evaluation metrics are crucial for making decisions, picking the right metrics is of utmost importance. There is evidence that relying just on accuracy as a criterion for evaluating classification in unbalanced datasets might lead to erroneous conclusions [63,64]. In a comparison of the F-measure and accuracy measures, [65] demonstrated that both can produce overly optimistic findings in the binary unbalanced data classification problem, which can lead to poor conclusions. According to [66], when datasets are not evenly split, the minority class experiences a higher rate of classification errors. The ROC-AUC [67] can be used to demonstrate the proposed hyproBert model’s ability to distinguish between the two categories, as the model performs a binary classification. When using unbalanced datasets, the area under the receiver operating characteristic ROC-AUC, is a good metric for gauging the efficacy of a model. Furthermore, metrics such as accuracy, precision, sensitivity, specificity, and AUC were considered for model evaluation in light of the imbalanced dataset evaluations [67].

As with our prior ISOT evaluations, we classified news samples in the ISOT-R10 dataset using the same baseline and state-of-the-art models.

Table 6. Results of all models on the ISOT-R10 dataset.

Dataset	Accuracy	Precision	Sensitivity	Specificity	AUC
LR	44.72	48.98	44.37	41.62	42.38
RF	82.64	82.91	82.14	77.52	80.91
MNB	48.92	50.41	49.34	46.28	45.22
KNNs	49.25	47.85	49.92	47.56	47.02
AB	72.96	76.61	73.88	70.36	70.07
DT(Elhadad et al. [57])	84.39	85.41	85.21	81.01	82.78
CNN	69.89	68.36	71.58	67.29	67.36
RNN	77	78	77	70	73.5
LSTM	88.34	91.23	90.51	84.52	86.37
BiLSTM	89.23	90.74	89.75	87.37	87.64
BiGRU	93.91	95.35	95.04	93.14	91.98
BiLSTM-CNN+task-specific word embedding [11]	93.83	95.82	95.11	93.27	92.13
CALLATRUMORS [12]	92.26	95.93	93.21	88.26	91.59
FakeBERT [13]	93.28	94.81	94.21	90.36	90.86
Ensemble model [42]	69.37	74.34	72.19	67.59	68.44
Hybrid CNN-RNN [43]	92.96	94.78	92.17	88.40	89.69
HyproBert	95.67	97.12	95.88	92.94	94.14

demonstrates that accuracy was impacted by the oversampling of real news in the ISOT-R10 dataset, but the proposed HyproBert model is still significantly superior to other models in terms of performance measures.

Furthermore, to monitor the effects of imbalanced data on the performance of HyproBert, we added 10,000 more real news in each step twice to the ISOT-R10 dataset and named the updated dataset as ISOT-R20 and ISOT-R30, respectively. The comparison of the performance on imbalance data is provided in

Table 7. Comparison of results of imbalanced data on HyproBert.

Dataset	Accuracy	Precision	Sensitivity	Specificity	AUC
ISOT	99.30	99.12	99.07	99.03	99.20
ISOT-R10	95.67	97.13	95.88	92.94	94.14
ISOT-R20	94.73	95.98	93.91	91.48	93.58
ISOT-R30	92.18	93.46	90.78	90.12	92.04

.

The findings of the unbalanced studies imply that the imbalanced data offers an additional challenge, as the majority of classifiers will demonstrate bias toward the class that comprises the majority of the data. A high-class imbalance is inherently prevalent in many applications that are used in the real world [68]. Implementing feature extraction and oversampling algorithms to address the challenge of binary unbalanced data classification will further improve the classification using imbalanced datasets [60].

5.5. Ablation Study

An ablation study was performed to evaluate the effectiveness of the designed architecture of the proposed model HyproBert. The architecture of the HyproBert was altered with different hybrid models in three combinations to measure the best possible outcome. The ablation architectures were evaluated with the FA-KES and ISOT datasets and the results are presented in the

Table 8. Ablation study architecture and results.

Datasets	BiGRU + CapsNet	BiLSTM + CapsNet	Conv + BiLSTM	Conv + BiGRU
		+Conv	+Attention + CapsNet	+Attention + CapsNet
FA-KES	52.67	51.83	54.03	61.15
ISOT	72.10	69.35	83.89	99.30

.

The first ablation architecture is BiGRU + CapsNet; BiGRU collects the contextual information, and CapsNet perceives the hidden relations amongst the identified features. The second ablation architecture is BiLSTM + CapsNet + Conv. BiLSTM processes data in the cell state as well as hidden states, which can be costly. The third ablation architecture is Conv + BiLSTM + Attention + CapsNet; BiGRU is replaced by the BiLSTM. The fourth is the hybrid of the proposed HyproBert model. The three identified models for the said study along with the proposed model and resultant F1 score are presented in the table. The results suggest that the proposed architecture of HyproBert is effective and performs better than its counter-ablation study architectures. The receiver operating characteristics ROC curve determines the usefulness of the binary classifier system, presented in Figure 13. The area under curve AUC determines the performance of the model, i.e., greater area means greater performance.

Throughout this body of work, we have investigated the efficacy of a variety of machine learning models as well as deep learning methods. Following our analysis, we concluded that the accuracy of detecting fake news for the FA-KES and ISOT datasets is not up to par. Consequently, a pretrained model for word embeddings is utilized in this research. DistilBERT is based on bidirectional transformer encoders. The BERT-based encoder is a robust feature extractor that yields good results in NLP tasks. The training loss decays faster with the BERT-based model than with the other word embedding model [21]. It is concluded that adopting bidirectional pretrained word embedding results in faster model training, reduced cross-entropy loss, and improved classification task performance.

Additionally, the BiGRU model can more effectively capture the semantic combination of text and store longer context information [50]. In addition, we stacked BiGRU units layer by layer to create a system with multiple layers. The output of the first layer is used without alteration as the input of the second layer. Therefore, the multi-layer BiGRU feature improvement contributed to the model’s improved classification. Furthermore, the layer-by-layer complexity of CapsNet’s routing is reduced by employing CapsNet with a self-attention mechanism in the proposed hyprobert model [49]. It increases the proportion of computations that could be performed simultaneously while reducing the number of required sequential processes [53]. Self-attention routing is able to capture the capsule’s global information, which results in fewer misclassification errors. A detailed analysis of the proposed hyprobert model has resulted in improved classification performance.

This research is limited to content-based features, which may not be sufficient to improve the performance of shorter news, such as that seen on social media. Future studies should also examine a comprehensive examination of news origins and context. Classification using imbalanced datasets can be enhanced through the use of feature extraction and oversampling algorithms, both of which aim to rectify the issue of binary unbalanced data classification [68]. In addition, we are working to incorporate meta data and supporting information, such as related images and social context, into our future publications.

6. Conclusions

A hybrid model based on deep hypercontext for fake news detection, HyproBert, is presented in this paper. HyproBert uses the distill-Bert for embeddings, the convolution neural network for spatial data extraction, the BiGRU for contextual data extraction, and the CapsNet with self-attention for the hierarchical understanding of full and partial relations among data. The CNN, BiGRU, and CapsNet are equipped with optimized hyperparameters. The proposed HyproBert model is evaluated using FA-KES and ISOT datasets. HyproBert results in high performance; it surpassed the performance of the baseline models. The results indicate that the HyproBert has an accuracy of 61.1% and 99.3% for the FA-KES and ISOT datasets, respectively. In the future, a multimodal approach using text and images simultaneously for better classification and fake news detection can be explored.