COVID-19 Vaccine Hesitancy: A Global Public Health and Risk Modelling Framework Using an Environmental Deep Neural Network, Sentiment Classification with Text Mining and Emotional Reactions from COVID-19 Vaccination Tweets

Popular social media platforms, such as Twitter, have become an excellent source of information with their swift information dissemination. Individuals with different backgrounds convey their opinions through social media platforms. Consequently, these platforms have become a profound instrument for collecting enormous datasets. We believe that compiling, organizing, exploring, and analyzing data from social media platforms, such as Twitter, can offer various perspectives to public health organizations and decision makers in identifying factors that contribute to vaccine hesitancy. In this study, public tweets were downloaded daily from Tweeter using the Tweeter API. Before performing computation, the tweets were preprocessed and labeled. Vocabulary normalization was based on stemming and lemmatization. The NRCLexicon technique was deployed to convert the tweets into ten classes: positive sentiment, negative sentiment, and eight basic emotions (joy, trust, fear, surprise, anticipation, anger, disgust, and sadness). t-test was used to check the statistical significance of the relationships among the basic emotions. Our analysis shows that the p-values of joy–sadness, trust–disgust, fear–anger, surprise–anticipation, and negative–positive relations are close to zero. Finally, neural network architectures, including 1DCNN, LSTM, Multiple-Layer Perceptron, and BERT, were trained and tested in a COVID-19 multi-classification of sentiments and emotions (positive, negative, joy, sadness, trust, disgust, fear, anger, surprise, and anticipation). Our experiment attained an accuracy of 88.6% for 1DCNN at 1744 s, 89.93% accuracy for LSTM at 27,597 s, while MLP achieved an accuracy of 84.78% at 203 s. The study results show that the BERT model performed the best, with an accuracy of 96.71% at 8429 s.


Background
In 2019, in Wuhan, China, coronavirus disease 2019 (COVID- 19) was reported for the first time [1,2], and it later spread rapidly to 72 countries [3] then started to deteriorate children's learning process [4]. Before the Chinese government quarantined Wuhan (on 23 January 2020), it was estimated that COVID-19 was spread to 369 cities in China [5]. The COVID-19 pandemic which started at the end of 2019 remained unbated in 2021. Coronavirus disease 2019 (COVID- 19), which started at the end of 2019 and had spread effects [25]. Consequently, many researchers and public health experts have been continuously gathering dataset to have a better understanding on vaccine hesitancy [26].
Nowadays, the Internet has been expanding its use to provide information related to health [27]. Many researchers gather rich and free information on COVID-19 vaccination in societies from the Internet [28]. Social media has been an astonishing source of information, where societies continuously express their beliefs through social networks [28,29]. Furthermore, the application of text mining to collect datasets from social media has been a meaningful tool [30], which makes it possible that text mining can be an alternative method to analyze public opinions on COVID-19 vaccine hesitancy [31]. In addition, social media have also played a major role in educating anti-vaccine groups, and traditional channels, such as national television and national and local newspapers, continuously promote vaccine information to their audiences [32]. In this study, extracted emotions from COVID-19 datasets were gathered from social media for sentiment analysis.
This paper is organized as follows: Section 2 reviews current literature utilizing deep learning to understand COVID-19 hesitancy, Section 3 explains the methodology of the research study, and Section 4 describes the experimental results. The conclusion is presented in Section 5. Future work is presented in Section 6, and acknowledgments are presented in Section 7.

Previous Studies
There have been various studies on COVID-19 vaccine hesitancy utilizing text-based sentiment analysis since the outbreak of the pandemic in 2019. Many previous studies used machine learning techniques to analyze vaccine hesitancy based on Twitter datasets [33][34][35][36][37]. Some researchers applied a deep learning approach to conduct text-based sentiment analysis on social media platforms [38][39][40], while other researchers used multiple techniques combining machine and deep learning approaches to analyze public opinions on COVID-19 based on social media datasets, such as from Twitter, to achieve better model performance [41][42][43][44][45].
Assessing the use of supervised learning in sentiment analysis [41] expounds on the need for identifying sentiments as a guide to better decision making due to the presence of subjective opinions and biases in reporting on COVID-19. This work tries to answer questions on how well supervised learning can perform in sentiment analysis through feature engineering. The authors of [41], using five classifiers, demonstrated how machine learning can be used for methodical investigation of Twitter sentiments expressed by social media users by using Radom Forest, XGBoost, and Support Vector classifiers. In their study, a comparative approach of deep learning is suggested to evaluate the performance of COVID-19 Twitter sentiment analysis, especially in addressing the complexity associated with the decidability of supervised learning.
In related literature, the impact of divergence support on COVID-19 vaccination is also assessed. Sentiment analysis of online platforms has emerged as one of the best indicators in getting an idea of what's on people's minds and, subsequently, what may be trending due to public opinions on an issue that may be contested. The authors of [38] used deep learning methods to analyze COVID-19 vaccination responses and demonstrated the use of Natural Language Processing tools for the predictive performance of RNN, LSTM, and Bi-LSTM in understanding public opinions. The author showed the importance of sentiment analysis in measuring a critical parameter (vaccination), which by far is known to be most impactful in the fight against COVID-19. The aim of this study was to understand the dominant feelings before and after vaccination to create modified approach. This study, among other comparative studies, demonstrates the leverage of sentiment analysis, especially in the application of decision science.
Sentiment analysis on COVID-19 based on geographical location and setting has also been studied. Liu's examination [46] used geolocation to compare online COVID-19 Twitter sentiments between urban and rural residents. In the cluster analysis which used trained word2vec models, urban and rural sentiments based on hashtags and opinion objectives were also investigated in the study. The inference of the study, which deployed geospatial and timing information, found a significant statistical contrast in sentiments for urban and rural Twitter users. The study results demonstrated the leverage of geographically targeted measures in epidemic prevention.
The use of deep learning for online sentiment analysis is also expounded by Hernandez [47] in a performance comparison of multilingual BERT pre-trained models. This study on the topic of COVID-19 in Mexico used Spanish-language parameters to compare pretrained Spanish and multilingual BERT models against other machine learning algorithms, such as Naïve Bayes, SV, Logistic Regression, and Decision Trees. In the paper, the authors presented the sentiments of the Mexican population on COVID-19 using lexicalbased data labeling and a semi-supervised technique. Their work demonstrates the use of bi-directional transformers and how efficiently the method can classify tweets with respect both positive and negative sentiments. However, this study also acknowledges the need for models that can recognize semantics such as irony and sarcasm, and words that have different meanings, since this would be a challenge while classifying contrary opinions.
Two popular sentiment methods to compute text-based sentiment analysis in these works are TextBlob and VADER. Some research studies used TextBlob sentiment computation [41,48] to investigate COVID-19 vaccine hesitancy using Twitter datasets. Other research studies [49,50] applied VADER to examine vaccine hesitancy. Table 1 represents a comparative study of text-based sentiment analysis to understand COVID-19 vaccine hesitancy based on Twitter datasets and utilizing deep learning approach. Some of the projects used deep learning techniques only, while others combined machines and deep learning methods. Comparing prior studies enhances our research in analyzing various areas that have not been considered in these prior studies. Table 1 is a representation of various studies with different deep learning approaches using public tweets to understand COVID-19 vaccine hesitancy.

Existing Gap in Literature
Considering previous research on COVID-19 vaccine hesitancy using a deep learning approach, as shown in Table 1, there seems to be a paucity in the literature that needs to be addressed: Some previous research projects were limited to only sentiment analysis. We believe that extracting emotions, in addition to its sentiments, will enhance the study. II. Some previous works were based on datasets downloaded from other institutions.
We believe there is a need for a shift in the dataset used to analyze the study with updated records. Consequently, addressing updated COVID-19 vaccination hesitancy tweets is another crucial matter to be considered. III. Building complex emotions could enhance the discussions on COVID-19 vaccine hesitancy. Most previous studies were limited in scope on classifications. We believe that integrating the topic with another field, such as psychology, could enhance the research study.
From the gaps in the previous studies on COVID-19 vaccine hesitancy, as represented in Table 1, this study utilized text-based sentiment analysis and deep learning techniques to investigate COVID-19 public tweets. Our goal is to enhance our understanding of public opinions on COVID-19 vaccine hesitancy. Therefore, we designed, developed, and evaluated various models to ascertain the best deep learning model for the COVID-19 public tweets dataset.

Overview
The experimental design of our project is shown in Figure 1. The first step, as shown in the figure, was text preprocessing. In this process, the dataset is cleaned by removing the URLs, punctuations, and retweets. In the preprocessing stage, the emojis in the dataset were converted to words. We also performed tokenization and removed stop words before performing stemming and lemmatization. Removing collection words was performed as well. After the text preprocessing, we classified the tweets using NRCLexicon into two sentiments and eight emotions. We computed word frequency for data exploration. Finally, the last step was to build neural networks using 1DCNN, LSTM, Multiple-Layer Perceptron (MLP), and BERT.

Data Collection
The dataset was collected from Twitter using API search words #vaccine or #COVID-19. We collected English public tweets globally. One of the limitations is that API search can only be retrieved in about seven to nine days, so we could only collect public tweets for up to nine days. We collected the dataset daily from 26 September 2021 to 27 March 2022. Using Twitter API search during this time frame, we collected one hundred fifty-six thousand five hundred thirty-nine public tweets and saved them as CSV (Comma-Separated Values). Figure 2 represents the number of tweets collected on a daily basis. We notice that at the beginning of February 2022, the number of tweets decreases tremendously. We collected an average of one thousand tweets every day prior February 2022. Figure 2 shows that there is a decline in the number of tweets related to COVID-19 from February 2022 onward. There is a possibility that people started discussing other topics on Twitter rather than COVID-19. In the U.S., for example, people could be discussing tax return until mid-April. In addition, the end of February 2022 was the beginning of the Ukraine war. There is a possibility that people around the globe shifted the discussion from COVID-19 to topics related to the Ukraine War topic. Furthermore, it had been two years that we were dealing with COVID-19. The is a likelihood that people started feeling happy that COVID-19 cases had been decreasing, so the COVID-19 topic was no longer a critical issue.

Data Collection
The dataset was collected from Twitter using API search words #vaccine or #COVID-19. We collected English public tweets globally. One of the limitations is that API search can only be retrieved in about seven to nine days, so we could only collect public tweets for up to nine days. We collected the dataset daily from 26 September 2021 to 27 March 2022. Using Twitter API search during this time frame, we collected one hundred fifty-six thousand five hundred thirty-nine public tweets and saved them as CSV (Comma-Separated Values). Figure 2 represents the number of tweets collected on a daily basis. We notice that at the beginning of February 2022, the number of tweets decreases tremendously. We collected an average of one thousand tweets every day prior February 2022. Figure 2 shows that there is a decline in the number of tweets related to COVID-19 from February 2022 onward. There is a possibility that people started discussing other topics on Twitter rather than COVID-19. In the U.S., for example, people could be discussing tax return until mid-April. In addition, the end of February 2022 was the beginning of the Ukraine war. There is a possibility that people around the globe shifted the discussion from COVID-19 to topics related to the Ukraine War topic. Furthermore, it had been two years that we were dealing with COVID-19. The is a likelihood that people started feeling happy that COVID-19 cases had been decreasing, so the COVID-19 topic was no longer a critical issue.

Text Preprocessing
Text preprocessing is essential in Natural Language Processing (NLP) before performing text sentiment classification. In this step, texts are prepared to be trainable for machine learning algorithms.
The first step is removing retweets. A retweet is a tweet shared from one user to another. This will result in tweet duplications. Removing retweets will normalize the dataset. A dataset with normal distribution will improve the fitness of the model used. Another benefit of removing retweets is to minimize the space required to run the experiment. Consequently, we removed retweets to improve the classification models.
The second step is to remove URLs and punctuation. It is important to clean the URLs since each tweet has a link. Keeping links in the dataset will skew word frequency. Furthermore, removing non-useful signs and punctuation will eliminate unnecessary items in the dataset. Because URLs and punctuations have no meaning, removing unnecessary items may improve model efficacy.
The next step is converting emojis into words. We used the library emoji.demojize() in sklearn to convert emojis into words. This is necessary because some people tend to express their feelings using emojis. Transforming emojis into words might improve tweets' sentiment analysis.
Next, we performed tokenization and normalization. Tokenization is a process of splitting words in each tweet, so it will be helpful to perform text-based sentiment analysis. Normalization is a process to normalize the collection of words by lowering the case as normalized words. Each word from the tweets is stored in a collection of words before computing its polarity.
After normalizing phrases, we removed stop words from the collection of words. Removing English stop words, such as 'am', 'is', 'aren't', 'hasn't', etc. will be helpful in computing text-based sentiments since these stop words are not valuable in this analysis. In addition, having stop words in the word collection will skew the frequency of the words. Consequently, removing them might improve text-based sentiment analysis.
The next step is stemming and lemmatization. Stemming is a process of transforming words into their base or root form. Words such as learning, learns, or learned are transformed to learn. Reducing the form of words to their base forms will homogenize the structure of the collection of words. We used PorterStemmer in this experiment. Unlike stemming, lemmatization is a process of reducing the form of words into the root of their dictionary forms. The purpose of performing lemmatization is to remove inflectional endings and return them to their dictionary base forms. WordNetLemmatizer was used for the experiment. Studies have shown that this process could enhance text-based sentiment analysis [6].
The final step of text preprocessing is to remove words used to search in the tweet collection. Keeping a collection of words, such as COVID-19, vaccine, and corona, will increase the skewness of word frequency. Consequently, removing these words will help normalize the distribution of the dataset.

NRCLexicon
The NRC Emotion Lexicon is a directory of English phrases and their relations with eight basic emotions and two sentiments (positive and negative) [56]. A lexicon is an emotional analysis that contains a set of words that will classify texts according to Plutchik's Wheel of Emotion [57]. These eight basic emotions are joy, trust, fear, surprise, anticipation, anger, disgust, and sadness, which are popularly called Plutchik's Wheel of Emotions [58]. The NRCLexicon technique classifies tweets associated with the NRC Emotion Lexicon directory into ten classes: positive sentiment, negative sentiment, and eight basic emotions (joy, trust, fear, surprise, anticipation, anger, disgust, and sadness) [59]. The 8 basic emotions resulting from the RNCLexicon are based on Plutchik's research and his Wheel of Emotions. These eight primary emotions are grouped into opposites: joy-sadness, acceptance (trust)disgust, fear-anger, and surprise-anticipation. Figure 3 shows the basic emotional effects from COVID-19 and its vaccines. It appears that the highest frequency is trust, followed by fear. Despite many conspiracy theories and myths, there is an indication that people have more trust that the pandemic and its vaccines are real (not a hoax). On the other hand, the lowest frequency is anger, followed by disgust. Even though individuals are dealing with hardship, people can hold their anger to the situation. Feeling angry arises when people are mistreated. Because most individuals trust the issue, it is most likely that people can hold their anger when coping with hardships. People tend to be more disgusted than angry when dealing with the COVID-19 pandemic and its vaccines. If there is an option, people do not want to get COVID-19 vaccines. However, it seems individuals in societies can hold their anger because they realize the benefits of getting vaccinated to combat the pandemic. like stemming, lemmatization is a process of reducing the form of words into the root of their dictionary forms. The purpose of performing lemmatization is to remove inflectional endings and return them to their dictionary base forms. WordNetLemmatizer was used for the experiment. Studies have shown that this process could enhance text-based sentiment analysis [6]. The final step of text preprocessing is to remove words used to search in the tweet collection. Keeping a collection of words, such as COVID-19, vaccine, and corona, will increase the skewness of word frequency. Consequently, removing these words will help normalize the distribution of the dataset.

NRCLexicon
The NRC Emotion Lexicon is a directory of English phrases and their relations with eight basic emotions and two sentiments (positive and negative) [56]. A lexicon is an emotional analysis that contains a set of words that will classify texts according to Plutchik's Wheel of Emotion [57]. These eight basic emotions are joy, trust, fear, surprise, anticipation, anger, disgust, and sadness, which are popularly called Plutchik's Wheel of Emotions [58]. The NRCLexicon technique classifies tweets associated with the NRC Emotion Lexicon directory into ten classes: positive sentiment, negative sentiment, and eight basic emotions (joy, trust, fear, surprise, anticipation, anger, disgust, and sadness) [59]. The 8 basic emotions resulting from the RNCLexicon are based on Plutchik's research and his Wheel of Emotions. These eight primary emotions are grouped into opposites: joy-sadness, acceptance (trust)-disgust, fear-anger, and surprise-anticipation. Figure 3 shows the basic emotional effects from COVID-19 and its vaccines. It appears that the highest frequency is trust, followed by fear. Despite many conspiracy theories and myths, there is an indication that people have more trust that the pandemic and its vaccines are real (not a hoax). On the other hand, the lowest frequency is anger, followed by disgust. Even though individuals are dealing with hardship, people can hold their anger to the situation. Feeling angry arises when people are mistreated. Because most individuals trust the issue, it is most likely that people can hold their anger when coping with hardships. People tend to be more disgusted than angry when dealing with the COVID-19 pandemic and its vaccines. If there is an option, people do not want to get COVID-19 vaccines. However, it seems individuals in societies can hold their anger because they realize the benefits of getting vaccinated to combat the pandemic.  According to Plutchik's Wheel of Emotions, the eight basic emotions are in opposite pairs (fear-anger, joy-sadness, trust-disgust, and surprise-anticipation). Figure 4 shows the emotional impact of COVID-19. The graph shows that fear and anger are expressed in about the same frequency. There is a possibility that people feel fearful and angry at the situation. People might fear the novel virus because its medications have not been found yet. At the same time, communities might be angry with the situation because there was not much they could do to combat the disease. There is a possibility that individuals are slightly more fearful when dealing with the pandemic and getting COVID-19 vaccines than being angry with the situation.   The second group of emotions (joy and sadness) in Figure 5 shows that people tend to be sad when dealing with the pandemic and its vaccines than being joyful. There is a likelihood that vaccine conspiracy theories, myths, and other types of anti-vaccination campaigns negatively impact societies. In addition, the use of vaccines as an emergency medication to combat COVID-19 is still being debated, where some groups of people still resist getting vaccinated. It makes sense that people are more heartbreaking when dealing with a new disease than being joyful. According to Plutchik's Wheel of Emotions, the eight basic emotions are in opposite pairs (fear-anger, joy-sadness, trust-disgust, and surprise-anticipation). Figure 4 shows the emotional impact of COVID-19. The graph shows that fear and anger are expressed in about the same frequency. There is a possibility that people feel fearful and angry at the situation. People might fear the novel virus because its medications have not been found yet. At the same time, communities might be angry with the situation because there was not much they could do to combat the disease. There is a possibility that individuals are slightly more fearful when dealing with the pandemic and getting COVID-19 vaccines than being angry with the situation. The second group of emotions (joy and sadness) in Figure 5 shows that people tend to be sad when dealing with the pandemic and its vaccines than being joyful. There is a likelihood that vaccine conspiracy theories, myths, and other types of anti-vaccination campaigns negatively impact societies. In addition, the use of vaccines as an emergency medication to combat COVID-19 is still being debated, where some groups of people still resist getting vaccinated. It makes sense that people are more heartbreaking when dealing with a new disease than being joyful.  The next emotional impact is trust and disgust. According to Figure 6, there is a slight possibility that people tend to trust or be more accepting than being disgusted when dealing with the COVID-19 pandemic and its vaccines. Individuals in communities accept (trust) the reality that COVID-19 does exist, and not fake news (hoax). People tend to act based on what they believe, so accepting the reality that the pandemic is real might influence more people to accept COVID-19 vaccines as an emergency treatment to combat the pandemic.   The last emotional impact of COVID-19 is surprise-anticipation. Figure 7 shows that anticipation is more frequent than surprise. There is a probability that communities have been more anticipative (being predictable) than surprised by the news of the pandemic and its vaccines. Because the COVID-19 pandemic is something new to society, there is a possibility that people are more anticipated that there could be an emergency treatment to combat the disease. Individuals in societies are not surprised by any new medication or invention to quickly fight the COVID-19 pandemic.    shows that positive daily sentiments are more frequent than negative sentiments. The graph demonstrates that the number of positive tweets is consistently more than the number of negative sentiments. It indicates that people feel more positive about getting COVID-19 vaccines as an emergency prevention of getting the novel virus. Even though people are dealing with such hardship during the pandemic, people feel more optimistic than thinking negatively about the COVID-19 issue. Because people trust that the pandemic is real, not a hoax, communities feel optimistic that they can cope with the difficulties.  To test the pairs of daily emotions, we performed a t-test to check if the relations among the basic emotions are statistically significant or not. Table 2 shows that the p-values of the relations among the pairs of emotions resulting from COVID-19 are significant. According to the t-test results, we can confirm that people have been sad and more miserable than joyful when dealing with pandemic. Facing the COVID-19 pandemic, people have more trust that COVID-19 is a real issue than disgust that the novel virus is a conspiracy. Furthermore, individuals in societies have been more worried (fearful) when coping with the pandemic than angry with the situation. In addition, because more people trust that the novel virus is a real issue, the COVID-19 pandemic has been anticipated rather than being a surprising topic. Finally, the p-value of the negative-positive relation is also close to zero. It suggests that there is significant evidence that people feel optimistic about getting the COVID-19 vaccines, and they trust these vaccines as an emergency treatment to prevent the spread of the novel virus.

Complex Emotional Effects
Complex emotion [60] is the combination of two basic emotions (anger, fear, trust, disgust, surprise, anticipation, joy, and sadness) blending into a new formation of emotions. In addition, the American Psychological Association [60] suggests that the evolutions of new compositional emotions are seen more often in older adults than children. Aggressiveness is a mixture of anticipation and anger, while contempt is a fusion of disgust and anger. In addition, remorse is a blend of sadness and disgust, while disapproval is a mix of surprise and sadness. Next, awe is a formulation of fear and surprise, while submission is a mixture of trust and fear. The combination of joy and trust forms love, while the unification of joy and anticipation forms optimism. Figure 9 shows the complex emotions resulting from the tweets. The graph indicates that most people feel submissive to COVID-19 and its vaccines. Society members trust that the pandemic is real, but they fear the situation. Consequently, most people feel obedient to the rules and regulations related to COVID-19. The next larger area in the graph is love. Because people feel trust about an emergency solution to combat the spread of the novel virus and are slightly joyful with the benefits of the vaccines, many individuals in communities love to have COVID-19 vaccines. People feel more submissive than loving because they have more fear than joy. The smallest area of the complex emotional effects is contempt. According to the complex emotions, contempt has the lowest area because anger is the lowest emotion, and disgust is the second-lowest one. It indicates that people tend to be contempt of the pandemic and feel optimistic about dealing with COVID-19, especially with the benefits of getting COVID-19 vaccines.

Neural Network Models
The next stage in this project was building neural network modeling. We built three deep learning approaches: LSTM (Long Short-Term Memory), Multiple-Layer Perceptron, and BERT (Bidirectional Encoder Representations from Transformers). We tested the three methods and compared which method had the best performance.

1DCNN
In this neural network model, we embedded a maximum of twenty thousand words and applied a 1D convolutional layer with a kernel size of 1 and ReLu activation function. In addition, 1D MaxPooling used a pool size of 1. Next, we applied a dense layer with a unit size of thirteen, and the activation functions were ReLu. Lastly, we used another dense layer with a unit size of ten and applied sigmoid activation. Figure  10 shows the architecture of the proposed 1DCNN model.

Neural Network Models
The next stage in this project was building neural network modeling. We built three deep learning approaches: LSTM (Long Short-Term Memory), Multiple-Layer Perceptron, and BERT (Bidirectional Encoder Representations from Transformers). We tested the three methods and compared which method had the best performance.

1DCNN
In this neural network model, we embedded a maximum of twenty thousand words and applied a 1D convolutional layer with a kernel size of 1 and ReLu activation function. In addition, 1D MaxPooling used a pool size of 1. Next, we applied a dense layer with a unit size of thirteen, and the activation functions were ReLu. Lastly, we used another dense layer with a unit size of ten and applied sigmoid activation. Figure 10 shows the architecture of the proposed 1DCNN model.

1DCNN
In this neural network model, we embedded a maximum of twenty thousand words and applied a 1D convolutional layer with a kernel size of 1 and ReLu activation function. In addition, 1D MaxPooling used a pool size of 1. Next, we applied a dense layer with a unit size of thirteen, and the activation functions were ReLu. Lastly, we used another dense layer with a unit size of ten and applied sigmoid activation. Figure  10 shows the architecture of the proposed 1DCNN model.

LSTM (Long Short-Term Memory)
LSTM is a special type of advanced Recurrent Neural Network (RNN), a sequential network, that can handle long-term dependencies, which comprise the vanishing gradient encountered by an RNN. Similarly, an LSTM model remembers the previous information and uses it to process the current input, but it is designed to avoid the long-term dependency issue that is RNN's weakness. In this neural network model, we embedded a maximum of fifty thousand words with an embedding dimension of one hundred words, and a maximum sequence length of two hundred and fifty words. In addition, we applied a 1D spatial dropout of 0.2 before the LSTM layer. We applied one-hundred-unit size in the LSTM layer, with dropout of and recurrent dropout of 0.1. Before the last layer, we add another dropout of 0.3. Finally, we applied a dense layer with a size of ten and SoftMax activation function. Figure 11 shows the architecture of the proposed LSTM model. LSTM is a special type of advanced Recurrent Neural Network (RNN), a sequential network, that can handle long-term dependencies, which comprise the vanishing gradient encountered by an RNN. Similarly, an LSTM model remembers the previous information and uses it to process the current input, but it is designed to avoid the long-term dependency issue that is RNN's weakness. In this neural network model, we embedded a maximum of fifty thousand words with an embedding dimension of one hundred words, and a maximum sequence length of two hundred and fifty words. In addition, we applied a 1D spatial dropout of 0.2 before the LSTM layer. We applied one-hundredunit size in the LSTM layer, with dropout of and recurrent dropout of 0.1. Before the last layer, we add another dropout of 0.3. Finally, we applied a dense layer with a size of ten and SoftMax activation function. Figure 11 shows the architecture of the proposed LSTM model.

Multiple-Layer Perceptron
A Multiple-Layer Perceptron (MLP) in a neural network is a layer deeply connected with its previous layers. It means that the neurons of the layers are connected to each neuron of its preceding layer. Matrix-vector multiplication is applied where the row vector of the output from previous layers is equal to the column vector of the MLP. Figure  12 and Figure 13 show the MLP model.

Multiple-Layer Perceptron
A Multiple-Layer Perceptron (MLP) in a neural network is a layer deeply connected with its previous layers. It means that the neurons of the layers are connected to each neuron of its preceding layer. Matrix-vector multiplication is applied where the row vector of the output from previous layers is equal to the column vector of the MLP. Figures 12 and 13 show the MLP model.

Multiple-Layer Perceptron
A Multiple-Layer Perceptron (MLP) in a neural network is a laye with its previous layers. It means that the neurons of the layers are neuron of its preceding layer. Matrix-vector multiplication is applied tor of the output from previous layers is equal to the column vector o 12 and 13 show the MLP model.  We started with two layers using a sequential model in this n unit size in the first layer was sixteen with a ReLu activation function layer, we added a dropout of 0.43. Lastly, the final layer with the size ed with SoftMax. Secondly, we applied three MLP layers of neural ne with a unit size of 128 with a ReLu activation function. Next, we ap 0.61 before adding another MLP layer. The second MLP layer size w ReLu activation function. Before the last layer, we added a dropou added the last layer with the size of ten and an activation function of

BERT (Bidirectional Encoder Representations from Transformer
BERT (Bidirectional Encoder Representations from Transform project by the authors of Google AI Language. The primary key to th plying the bidirectional training of Transformers (a trendy attention ral network model, we applied seven hundred and sixty-eight sizes by a hidden layer with fifty units and an output layer. In this neural applied an Adam optimizer with a learning rate of 3 × 10 −5 and decay We started with two layers using a sequential model in this neural network. The unit size in the first layer was sixteen with a ReLu activation function. Before the second layer, we added a dropout of 0.43. Lastly, the final layer with the size of ten was activated with SoftMax. Secondly, we applied three MLP layers of neural networks. We started with a unit size of 128 with a ReLu activation function. Next, we applied a dropout of 0.61 before adding another MLP layer. The second MLP layer size was thirteen with a ReLu activation function. Before the last layer, we added a dropout of 0.1. Finally, we added the last layer with the size of ten and an activation function of SoftMax.

BERT (Bidirectional Encoder Representations from Transformers)
BERT (Bidirectional Encoder Representations from Transformers) is a published project by the authors of Google AI Language. The primary key to this innovation is applying the bidirectional training of Transformers (a trendy attention model). In this neural network model, we applied seven hundred and sixty-eight sizes of input, followed by a hidden layer with fifty units and an output layer. In this neural network model, we applied an Adam optimizer with a learning rate of 3 × 10 −5 and decay of 1 × 10 −8 .

Results
In this section, we show the results of the experiments. We examined the result by applying neural network experimentations, including 1DCNN, MLP, LSTM, and BERT, using ten classes (positive, negative, joy, sadness, trust, disgust, fear, anger, surprise, and anticipation).
This section discusses our neural network experimentations on COVID-19 tweets. We examined the performances of the 1DCNN, MLP, LSTM, and BERT models. The first experiment was when we applied the 1DCNN model. According to this experiment, we achieved the highest accuracy of 0.886133. Additionally, we achieved the highest performance with a training time of 1744 s. Figures 14 and 15 show the 1DCNN accuracy and loss graphs.      Figures 16 and 17, we notice that the model converges when the training is about 3.7 epochs.
The subsequent model experimentation was the MLP model. In this experiment, we applied two and three dense layers of artificial networks. When we applied the two layers of the MLP model, we achieved the highest accuracy of 0.8478 with a training time of 203 s. On the other hand, when we applied the MLP model with three layers, we accomplished an accuracy of 0.8117 with a training time of 1405 s. When we applied two MLP layers, the model accomplished faster than the model with three dense layers because the data had to go through another model layer. Figures 18 and 19 show the MLP accuracy and loss graphs with two dense layers, while Figures 20 and 21 represent MLP accuracy and loss graphs of the three dense layers. Our next model performance was the LSTM model. Our experimentation on this deep learning model achieved the highest accuracy of 0.8993. To accomplish this performance, it took 27,597 s for five epochs. From Figures 16 and 17, we notice that the model converges when the training is about 3.7 epochs.  The subsequent model experimentation was the MLP model. In this experiment, we applied two and three dense layers of artificial networks. When we applied the two layers of the MLP model, we achieved the highest accuracy of 0.8478 with a training time of 203 s. On the other hand, when we applied the MLP model with three layers, we accomplished an accuracy of 0.8117 with a training time of 1405 s. When we applied two MLP layers, the model accomplished faster than the model with three dense layers because the data had to go through another model layer. Figures 18 and 19 show the MLP accuracy and loss graphs with two dense layers, while Figures 20 and 21 represent MLP accuracy and loss graphs of the three dense layers.  The subsequent model experimentation was the MLP model. In this experiment, we applied two and three dense layers of artificial networks. When we applied the two layers of the MLP model, we achieved the highest accuracy of 0.8478 with a training time of 203 s. On the other hand, when we applied the MLP model with three layers, we accomplished an accuracy of 0.8117 with a training time of 1405 s. When we applied two MLP layers, the model accomplished faster than the model with three dense layers because the data had to go through another model layer. Figures 18 and 19 show the MLP accuracy and loss graphs with two dense layers, while Figures 20 and 21 represent MLP accuracy and loss graphs of the three dense layers.           Finally, the last model experimentation was the BERT model ( Figure 22). In this neural network model, we accomplished an accuracy of 0.9671 with a training time of 8429 s. Applying this model, we achieved the highest model performance overall. The training time was slightly longer than some other models, but the LSTM model had the longest training time (27,597 s), which was almost three times the BERT training period. Finally, the last model experimentation was the BERT model ( Figure 22). In this neural network model, we accomplished an accuracy of 0.9671 with a training time of 8429 s. Applying this model, we achieved the highest model performance overall. The training time was slightly longer than some other models, but the LSTM model had the longest training time (27,597 s), which was almost three times the BERT training period.

Discussions
For our first model experiment with 1DCNN, the result of this neural network model is higher than some previous deep learning models, with an area under the curve of 0.7600 for three classes (positive, negative, and neutral) [54]. In this experiment, we used ten classes (two sentiment and eight emotions), but our model was able to perform higher than some previous research studies.
In addition, for our second model experiment with LSTM, the model's performance is better than some previous studies: 0.7544 [49], 0.6500 [48], 0.8660 [54], and 0.7752 [43]. One model [55] has an LSTM accuracy of 0.9167. However, other studies only used two classes (positive and negative), while our study applied ten classes (two sentiments and eight emotions).
Our experiment showed that two layers of MLP achieved the fastest time (203 s) to reach the maximum result of 0.8478, while the three layers of MLP achieved its best performance (accuracy of 0.8117) within 1405 s. The other models took a long time to achieve their best model performances, such as the 1DCNN model took 1744 s, the BERT model took 8429 s, and the LSTM model took 27,597 s. In terms of time efficiency, the MLP model with two layers performed the most efficiently, and the LSTM model was the most inefficient or time-consuming.
According to our results, BERT was the best model classifier in our experiment. The BERT model's achievement is better than most of the previous BERT model accuracies: 0.9235 [45], 0.9160 [42], 0.9150 [52], and 0.8900 [44]. One BERT model [47] achieved a similar accuracy (97%) as our model, but this previous experiment used two classes only (positive and negative) when compared with our experiment (two sentiments and eight emotions).
On the other hand, the MLP model with three layers achieved the lowest model performance (0.8117). The second highest model performance was achieved using LSTM (0.8993), followed by 1DCNN (0.886133). This model's achievement is like a previous research experiment [54], but the previous study only applied two classes (positive and negative), while our experiment used ten classes (two sentiments and eight emotions).

Discussions
For our first model experiment with 1DCNN, the result of this neural network model is higher than some previous deep learning models, with an area under the curve of 0.7600 for three classes (positive, negative, and neutral) [54]. In this experiment, we used ten classes (two sentiment and eight emotions), but our model was able to perform higher than some previous research studies.
In addition, for our second model experiment with LSTM, the model's performance is better than some previous studies: 0.7544 [49], 0.6500 [48], 0.8660 [54], and 0.7752 [43]. One model [55] has an LSTM accuracy of 0.9167. However, other studies only used two classes (positive and negative), while our study applied ten classes (two sentiments and eight emotions).
Our experiment showed that two layers of MLP achieved the fastest time (203 s) to reach the maximum result of 0.8478, while the three layers of MLP achieved its best performance (accuracy of 0.8117) within 1405 s. The other models took a long time to achieve their best model performances, such as the 1DCNN model took 1744 s, the BERT model took 8429 s, and the LSTM model took 27,597 s. In terms of time efficiency, the MLP model with two layers performed the most efficiently, and the LSTM model was the most inefficient or time-consuming.
According to our results, BERT was the best model classifier in our experiment. The BERT model's achievement is better than most of the previous BERT model accuracies: 0.9235 [45], 0.9160 [42], 0.9150 [52], and 0.8900 [44]. One BERT model [47] achieved a similar accuracy (97%) as our model, but this previous experiment used two classes only (positive and negative) when compared with our experiment (two sentiments and eight emotions).
On the other hand, the MLP model with three layers achieved the lowest model performance (0.8117). The second highest model performance was achieved using LSTM (0.8993), followed by 1DCNN (0.886133). This model's achievement is like a previous research experiment [54], but the previous study only applied two classes (positive and negative), while our experiment used ten classes (two sentiments and eight emotions).
Comparing our experimentation on several neural networks, the BERT model accomplished the highest model accuracy of 0.9671 with ten classes (two sentiments and eight emotions). Our next highest model accuracy was achieved by the LSTM model (0.8993). However, the LSTM model's training time was time-consuming as it was three times longer than the BERT model's training period. We consider the BERT model as the best model experimentation based on our dataset, even with a relatively long training time.

Contributions
According to our experimental results, below are some of the contributions of this study:

•
We designed, developed, and evaluated deep neural network models with ten classes (positive, negative, joy, sadness, trust, disgust, fear, anger, surprise, and anticipation Our research study identified eight basic emotions from the tweets (joy, sadness, trust, disgust, fear, anger, surprise, and anticipation) in addition to the two sentiments (positive and negative). Furthermore, we extended the basic emotions into complex ones (aggressiveness, contempt, remorse, disapproval, awe, submission, love, and optimism). Most of the previous classes were based on the sentiments only (positive, negative, and neutral) [45,51,54,55]. Another study classified their dataset into four basic emotions (sad, joy, fear, and anger) [44]. Finally, a study used a combination of primary and complex emotions (optimistic, thankful, emphatic, pessimistic, anxious, sad, annoyed, denial, official, report, surprise, and joking) [40]. Our research is based on a ten-class solution using deep learning models of LSTM, BiLSTM, and BERT to classify COVID-19 vaccine hesitancy tweets into positive, negative, joy, sadness, trust, disgust, fear, anger, surprise, and anticipation.

Conclusions
The COVID-19 pandemic has impacted the global community tremendously. Various rules and regulations forced many business entities to change their strategies to survive while coping with the disruptions and hardships impacted by the coronavirus disease. Conspiracy theories, anti-vaccination campaigns, myths, or controversies regarding the safety issue of COVID-19 vaccines made some people hesitate to get vaccinated. Other groups supporting vaccination kept encouraging the use of vaccine to combat the coronavirus disease. Examining the level of COVID-19 vaccine hesitancy is an effort to combat this infectious virus. Since there are pros and cons of getting COVID-19 vaccines, examining vaccine hesitancy using social media dataset becomes an essential topic.
Our study indicates that people gradually feel positive about COVID-19 vaccine, shifting their hesitancy concerns and getting COVID-19 vaccines due to improved public trust in the benefits of getting vaccinated. Most people trust that the pandemic is a real issue, and not a hoax. Individuals in societies can hold their anger when dealing with COVID-19 vaccines. In addition, most people feel submissive when dealing with COVID-19 and its vaccines. Second, the majority of people feel loving when coping with the novel virus and its vaccines. Few people feel contempt because they are able to hold their anger when facing COVID-19 and its vaccinations. Daily tweet sentiment analysis shows significant evidence that communities have been optimistic about getting vaccinated as an emergency exit to combat COVID-19.
At the beginning of February 2022, there was an indication that people reduced their discussion about COVID-19 on social media platforms, such as Twitter. This could be attributed to various factors, such as people shifting their opinions as the pandemic felt less scary than before at the beginning of 2019. In addition, discussions on social media shifted to new hot topics, such as the war between Ukraine and Russia, after lingering and chatting about coronavirus for a couple of years.
Based on our experimental results, we successfully attained a considerably better performance in our sentiment analysis on COVID-19 vaccine hesitancy. Our findings show that compared to supervised learning, artificial neural network techniques have better computational performance, as demonstrated in the present study using sentiment analysis and model validation. Our 1DCNN model showed higher performance when compared to other deep learning models in related works. The application of LSTM also achieved a more comprehensive sentiment analysis of 10 classes of emotions, giving a broader outlook of classified sentiments, when compared to studies that had binary classification (positive and negative). Finally, our neural network experimentation showed that BERT performed better than the other neural network models. The BERT model with ten classes achieved the highest accuracies of 0.9671 which is higher than previous research studies (mostly using three classes). The training time to achieve its highest performance was 8429 s. Our second highest accuracy was achieved by the LSTM model (0.8993), which had an extremely long training time of 27,597 s.

Future Work
To improve the current research study, we would like to implement a time series analysis on companies related to COVID-19 vaccine stock prices in future studies. We would like to gather various COVID-19 vaccine companies' stock prices, and not only from the three major companies in the U.S. (Pfizer, Johnson and Johnson, and Moderna).
The worldwide COVID-19 pandemic has impacted the global community tremendously. Schools and business closures have caused a significant transformation in societies. Various rules and regulations have forced many business entities to change their strategies to survive while coping with the pandemic.