Research on the Emotional Polarity Classification of Barrage Texts

With the widespread application of multimedia technology, video barrage has become an important form of expressing opinions and emotions. The classification of Chinese barrage texts according to emotions is of great significance for mining users’ emotions. However, the large number of Internet buzzwords in the barrage texts brings certain difficulties to sentiment analysis. Therefore, in order to address the ambiguity problem of Internet buzzwords in the barrage texts and effectively classifying the video barrage texts, the BERT pre-training language model and Support Vector Machine are combined to classify the barrage texts of positive and negative emotions. In this paper, the word vectors of the barrage texts are firstly generated by the BERT model. The SVM model is used as a classifier, and Particle Swarm Optimization is introduced to optimize the parameters of the SVM model. This method is compared with the traditional methods on the barrage data sets of different video types. The experimental results show that the precision, recall and F1-score of the proposed method are higher than those of the traditional models.


Introduction
Barrage will scroll from right to left in the video screen, allowing people to see other people's comments on the video clip while watching the video, thereby creating a sense of sharing and interaction [1] . Barrage texts are generally short and contain personalized expressions such as Internet buzzwords. The emotional information contained in it will be more direct and concise than general long comments. Analyzing the sentiment of the barrage texts is more effective for video content operation and public opinion analysis. With the rapid development of digital media technology, related scholars have gradually paid attention to the analysis of barrage and have carried out some researches.
Zheng et al. [2] extracted sentiment words to calculate sentiment values by establishing a sentiment analysis model based on the sentiment dictionary, and combined with time series to analyze the sentiment data of the texts to visualize sentiment information. Hong et al. [3] established a dictionary of commonly used words in online barrage based on the colloquial characteristics of barrage texts. It can better recognize the network vocabulary that appears in the barrage texts. Zhuang et al. [4] proposed an LSTM sentiment analysis model based on the attention mechanism. By mining sentiment keywords and combining sentiment dependence, the sentiment information in the barrage video can be obtained more accurately. Qiu et al. [5] considered the influence of facial expressions and tones on the emotion of the texts, and at the same time they also constructed a comprehensive emotional dictionary based on BosonNLP emotion dictionary, expression dictionary, degree dictionary, negative dictionary, etc, to improve the effect of barrage texts emotion analysis.
Although the above related researches had achieved good results, the barrage texts contain lots of Internet buzzwords, which lead to the existence of polysemous words. It is difficult to distinguish the Since BERT [6] (Bidirectional Encoder Representations from Transformers) was proposed, it has made many impressive achievements and has attracted the attention of many researchers. Liu et al. [7] combined the word vectors generated by BERT with BLSTM and attention mechanism to classify texts according to different emotions, which improved the classification accuracy. Yan et al. [8] changed softmax to A-softmax, and combined with BERT to increase the accuracy of Chinese texts classification. At the same time, they also proposed the combination of BERT and SVM to explore the feasibility of combining semantic features and traditional features in deep learning.
From the actual classification effect of some related studies [9,10] , using Support Vector Machines to classify texts is not only simple and easy to implement, but it is also suitable for small sample learning and nonlinear data classification.
The BERT pre-training language model makes full use of the context information of the words, so that the same word in different contexts can be expressed by different word vectors. It is used in this paper to obtain the dynamic feature representation of the barrage texts. At the same time, this paper also uses SVM as the classifier, and uses particle swarm optimization (PSO) to optimize the model parameter [11] , which further improves the precision of the model in the sentiment classification of barrage texts.

Related research
BERT's model architecture is a multi-layer bidirectional Transformer encoder [12] , and can be trained by a multi-task method of predicting the target word and the next sentence. It can be fine-tuned according to the downstream task, and the embedding of BERT fusion full-text information can be directly used as word embedding for classification tasks [13] . The structure diagram of the BERT pre-training language model [6] is shown in Figure 1  The Encoder-Decoder structure is used in Transformer. The coding layer is composed of 6 Encoders, and each Encoder contains self-attention and feedforward neural network, which is conducive to the current node to obtain the semantics of the context. The decoding layer is composed of 6 Decoders. Each Decoder has a two-layer network of Encoder, and an encoding-decoding self-attention layer, so that the content that currently needs to be paid attention can be obtained. Therefore, using BERT to vectorize the texts can retain more contextual information, as well as the different meanings of the same word in different contexts. It can generate a more flexible feature vector representation, which can be used as the feature input of downstream classification models to improve the effect of classification according to emotional polarities.

Model describes
In order to solve the ambiguity of online buzzwords, and effectively classify the Chinese barrage texts according to different emotional polarities, this paper proposes to integrate the BERT and SVM model. The model flow chart of this paper is shown in Figure 2. The specific workflows are as follows: Step 1. We crawled the barrage data of the Bilibili website, and removed too long and meaningless barrage texts, such as check in, I am the first, 20200621. In addition, we converted the emoji expressions in the barrage into corresponding texts. In this experiment, the emotional polarities of the barrage texts were divided into positive and negative. In order to reduce the workload of manual data annotation for machine learning, this experiment called API which named analysis of the sentiment tendency of the text from Baidu to carry out sentiment annotation. The barrage text of positive emotion was marked as 1, and the barrage text of negative emotion was marked as -1. Through the above operations, the barrage corpus needed for the experiment in this paper was obtained.
(2) In expressions (1) and (2), the value of subscript n is greater than the value of subscript m. The in the expressions represents the barrage texts, and represents the emotional label of the texts (i = 1, 2, . . . , n, =1 or = −1).
Step 3. Inputting the barrage texts to the BERT layer for word vector training of sentences. In this experiment, we used Google's pre-training chinese_L-12_H-768_A-12 model to perform BERT encoding on each barrage, so that the word vector of each barrage text became 768 dimensions. And further extracted the feature vector of the meaning information of each word in the fusion text, that was, the vector of special characters marked as [CLS] at the beginning of each text.
Step 4. Then we imported the feature vectors generated by BERT and emotional polarity labels into the particle swarm optimization (PSO), and iteratively searched for the penalty-factor and kernel function parameters that were most suitable for the classification. The PSO algorithm is simple and easy to implement and does not have many parameters to adjust.
Step 5. Finally, we substituted the obtained penalty-factor and kernel function parameters into the SVM classifier for training, and classified the texts in the test set S 2 according to positive emotion and negative emotion. In this paper, the widely used RBF kernel function was selected as the classifier kernel function to construct a non-linear map to solve the linear inseparability problem of high-dimensional data.

Experimental data
Bilibili is the largest barrage video sharing website in China, attracting many video lovers [14] . This  Table 1.

Experimental environment and parameter settings
The programming tool used in this experiment was PyCharm 2020, the environment manager was Anaconda 4.9.2. The experimental environment settings are shown in Table 2. The experimental parameters mainly included the parameters of the BERT model and the SVM model. The Chinese pre-training model chinese_L-12_H-768_A-12 released by Google was used in the BERT model. Its parameters were as follows: the number of hidden layer neurons was 768 and the number of layers was 12, the number of attention heads in each hidden layer was 12, and the dropout probability was 0.1, using Gelu as the activation function of the model. At the same time, the pre-training model was fine-tuned in the process of model training to be more suitable for the sentiment classification of this paper. The PSO algorithm was used to search for the most suitable penalty-factor and kernel function parameters. The parameters of PSO were as follows: the number of particles in the particle swarm was 100, the particle dimension was 2, the maximum number of iterations was 50, the acceleration coefficients c1 and c2 were 2, the inertia factor was 0.8, the maximum value of the parameter was 15, and the minimum value was 0.001. This paper adopts the most widely used precision, recall and F1-score as the evaluation criteria of the model. The formulas are as follows:

Result analysis
The experimental results of the three barrage text data sets running on the comparison models are shown in Table 3. By comparing the results in Table 3, it can be found that compared to other models, the precision, recall and F1-score of the B-SVM model are all improved. The features of texts obtained by BERT can make full use of the context information of the words, and better distinguish the different meanings of the same word in different contexts, thereby improving the effect of the classification of the barrage texts according to emotional polarity. In addition, the B-SVM model has a better performance in the sentiment analysis of the barrage texts than the models using other classifiers, which also proves the efficiency and flexibility of the SVM model in texts classification. The results of partial classification are shown in Table 4.

Conclusions
At present, there is a little work on the analysis of barrage texts, but the barrage itself contains the audience's direct emotions and opinions on the video content, so the barrage analysis and sentiment