Text to Emotion Extraction Using Supervised Machine Learning Techniques

Proliferation of internet and social media has greatly increased the popularity of text communication. People convey their sentiment and emotion through text which promotes lively communication. Consequently, a tremendous amount of emotional text is generated on different social media and blogs in every moment. This has raised the necessity of automated tool for emotion mining from text. There are various rule based approaches of emotion extraction form text based on emotion intensity lexicon. However, creating emotion intensity lexicon is a time consuming and tedious process. Moreover, there is no hard and fast rule for assigning emotion intensity to words. To solve these difficulties, we propose a machine learning based approach of emotion extraction from text which relies on annotated example rather emotion intensity lexicon. We investigated Multinomial Naive Bayesian (MNB) Classifier, Artificial Neural Network (ANN) and Support Vector Machine (SVM) for mining emotion from text. In our setup, SVM outperformed other classifiers with promising accuracy.


Introduction
Emotion plays a very important role in communication as hundred of words can be told using an emotional expression.The internal state of human is expressed through emotion which is communicated in various forms such as verbal words, facial expressions as well as written text [1].Expansion of internet has made text communication popular with people of all ages.They are involved with different social media and blogs as those are great means of communication.They express their sentiments and opinions through messages, status, comments, reviews etc.A communication containing emotional content is livelier and also has a great impact on the mind of participants [1].Therefore, understanding the emotion conveyed through text is very important to ensure proper communication and avoid misunderstanding.
Additionally, text to emotion extraction has significant applications in different fields of Human Computer Interaction (HCI) including robotics.
Enormous amount of text are produced daily on different social media and blogs.Manually analyzing the emotion conveyed in these texts is near to impossible and hence automated tools are necessary for the task.The interdisciplinary field of computing that spans computer science, psychology and cognitive science dealing with the systems related to emotions is known as Affective Computing [2][3][4].Affective computing researchers have proposed various rule based approaches to extract the emotion of text automatically.Neviarouskaya et al. [3] developed a lexical rule based approach known as Affect Analysis Model (AAM) based on lexicon built from WordNet-Affect [5] where each word in the database is represented by a vector of intensities of nine emotion defined by Izard [6].They analyzed a sentence in word, phrase and sentence level then provided the emotion with maximum total intensity as output.L. Dey et al. [7] provided implementation details of real time chatting application that showed the emotion of the message by analyzing affective features of the words of the text.They used a modified version of AAM with their own lexicon.An active learning based system which took experts suggestion to continuously train the system and improve the accuracy along with text to emotion extraction was proposed by M.-U.Asad et al. [1].The Naïve Bayesian classifier was used for active learning phase in that system.The authors further 1395 augmented the exploratory data warehouse (DW) technology to their previous existing system for enhancing their dataset [8].All these systems relied on emotion intensity lexicon where each word is assigned with a vector of intensity values of different emotions.However, creating emotion intensity lexicon is very time consuming and tedious job.Moreover, there is no hard and fast rule for assigning emotion intensity values to a word.Thus, intensity values vary from person to person which introduces bias in the output.A Machine Learning based system would solve the problems regarding the rule based approcah.Machine Learning is an approach which is used to enable machine to learn from empirical data validated by experts [9].Machine learning task is divided into supervised learning and unsupervised learning.In supervised learning, also known as classification, data tuples and their corresponding classes are given to the system and a classification model is developed on the train data.The system predicts the class level of new data based on the classification model.On the other hand, in unsupervised learning known as clus tering, no class level is given and system forms different clusters from the data based on any similarity measure.The task of emotion extraction from text can be represented as supervised machine learning problem.
Supervised Machine learning based approaches for sentiment analysis are investigated by different researchers.Those works are confined to finding positivity, negativity and neutral nature of text.Pang et al. proposed machine learning based approach for finding only positivity and negativity in movie review domain [10].They used the standard Bag of Features framework with Naïve Bayesian classifier, Maximum Entropy and Support Vector Machines.Twitter [11] is popular social media all over the world and great source of emotional data.People express their status through short text form known as Tweets.Sentiment analysis of tweets on GSM services is experimented in [12] where the authors used Multinomial Naïve Bayes for classification.A methodology proposed in [13] is related to our work which proposed a system to identify the six emotion category proposed by Ekman et al. [14].However, extracting more emotion category provides more insight about data.To the best of our knowledge, there is no machine learning approach of extracting nine emotion categories defined by Izard [6] from text data.
Although machine learning based approach of emotion extraction from text which could eliminate the problems of rule based approach, it was not taken into account .In that connection, we proposed a machine leaning based approach for the task.Our system relied on text labeled with emotion category rather emotion intensity lexicon and hence eliminate the tedious job of creating emotion intensity lexicon.Labeling a text with an emotional marker is comparatively easier than labeling a word with emotion intensity values.In our experiment, we used social media data from Twitter [11].We collected 4000 tweets from twitter and labeled them with proper emotion category by human expert.We used some portion of them to train our system and the other to test.Among the various emotional states we used the nine basic emotion categories namely, 'anger', 'disgust', 'fear', 'guilt', 'interest', 'joy', 'sad', 'shame' and 'surprise' that can be identified by facial expression defined by Izard [6] along with 'neutral' category which stood for absence of emotional content.
The unique contributions of this work are: 1) We showed a machine learning based approach of extracting nine basic emotion categories from text which eliminated the necessity of emotion intensity lexicon; 2) We showed a comparative study on performance of three most common classifiers in the job of emotion extraction from tweets.This remainder of the paper is organized as follows.In the Section 2, we describe the methodology of this research.The outcomes of our experiment and the reasons behind them are discussed in Section 3. We conclude out work and also keep pointer to future directions in Section 4.

Research Method
In this work, we followed a supervised machine learning approach to identify the nine emotion categories proposed by Izard [6].In supervised machine learning system, the system is trained with examples for which class labels are known.After the training, the system is tested on a data set with unknown class labels.The architecture of our system is shown in Figure 1.At first, we prepare our dataset to make it suitable for training and testing.Preparation of data is performed in the Data Preparation Module of our system.Emotion Extraction M odule of our system is responsible for extracting emotion from processed data.

Data preparation module
In this module, data are processed so that they are ready to feed into the classifier algorithm.We collect data from social networking site Twitter because it is a great source of emotional text [11].In this site, people express their emotion and opinion as a short form of text known as Tweet.Twitter API is used to collect data from Twitter.Collected tweets contai ned different types of noise.For this reason, we perform cleaning operation on the tweets.We eliminate re-tweets along with the tweets containing links to other multimedia content.After that, we manually label the tweets to one of the nine emotion categories defined by Izard [6].Tweets containing no emotional content are added to 'neutral' category.Furthermore, all parts of data are not important for emotion extraction task.We apply preprocessing to eliminate uninteresting and trivial parts of the data.Preprocessing is very important step of classification because the accuracy of natural language processing system greatly depends on it [15].
Tokenization is the first step of preprocessing text.In tokenization, the text is chopped into a stream of words, phrases, symbols or other meaningful units called tokens.Along with decomposing text into tokens, tokenization may also eliminate some symbols such as punctuations.Stopwords are the token or term those occur frequently and are essentially meaningless in the sense that they contribute very less to the context and content of the text.Thus they are not important for classification of text and hence are eliminated.Normalization is a transformation process that transforms different representations of a word to one ideal form.For example, USA and U.S.A. are normalized to USA.Different variations of a word convey almost the same meaning and hence can be considered as the same word.We use stemming to convert variants of a word to the base form of that word.
Figure 1.Architecture of our system

Emotion extraction module
The main task of emotion extraction from data is done by this module.In this module, to extract the emotional category of text, a classification model is created based on the training data.For this purpose, we extract important features from the text.We apply language modeling technique for feature extraction.We use Vector Space Model (VSM) and Probabilistic Language Model (PLM) for language modeling.
In VSM, a tweet is represented as a vector of terms in a common vector space.A dictionary is formed by collecting all the terms in the domain of discourse or corpus.Each term represents a direction and hence a unit vector in the document vector.For example, if a corpus consists of two tweets, ='I am sick today' and ='I am angry', then the dictionary formed by VSM modeling of the tweets is shown in Table 1.The dictionary contains the count of each term in each of the tweets.The VSM representations of the tweets are as follows.

TELKOMNIKA
The number of occurrences or frequency of a term in document is called term frequency ( ).The of the terms can be viewed as a quantitative representation of the document.This model is known as Bag of Words model in the literature of language modeling.However, provides the importance of term to a particular document i.e., local weight.The problem of local weight is that it provides importance regarding a document rather the whole corpus.That's why we use global weight which provides importance of a term according to the whole corpus.Global weight is defined as, Where is called Inverse Document Frequency.If there are total N number of documents in the corpus and represents the number of documents in the corpus that contain the term then Inverse Document Frequency is defined as, We use probabilistic language modeling which provides the probability of a sequence of words.We use probabilistic N-gram modeling of language in our experiment.For a sequence of words, w 1 ,w 2 ,w 3 ,…..w i ,…..w n the Gram rule of language modeling is defined as,

= ∏
If the value of k = 1, then the language model is called bigram and for k = 2, the model is called trigram.Similarly, unigram model is defined as, P(w 1 w 2 w 3 …w n ) = P(w 1 )× P(w 2 )× P(w 3 )×..

...× P(w n )
Unigram model doesn't consider the conditioning context that's why bigram modeling is used in our experiment.To deal with Zero Probability problem, we use Add-1 smoothing technique which adds 1 to each count so that no probability becomes zero.Important features extracted from the train data are fed into the classifier algorithm to create the classification model.We experiment Multinomial Naïve Bayesian (MNB) classifier, Support Vector Machines (SVM) and Artificial Neural Network (ANN) for classification model creation.
The Multinomial Naïve Bayesian classifier is a probabilistic classifier that predicts the class label based on the probability of a document to be of a particular class using Bayes' theorem.It assumes that the effect of an attribute on the class label is independent of the effect of other attributes [16].The reason for using this classifier is that it found to faster that other classifiers [17].Let, denotes the training tweets and their associated class labels.Each tweet is represented by an n-dimensional attribute vector, Here; attribute value represents the measurement done on the attribute .We have a total of n attributes and m classes .The classifier will predict that a tweet belongs to the class for which the posterior probability, conditioned on is the highest i.e., the tweet belongs to the class iff, >  The value of the function is the emotional class of a tweet.The Artificial Neural Network is based on the modeling of neurons of human brain which can be considered as a weighted directed graph in which neurons represent the nodes and weighted directed edges can be considered as input and output connection of neuron [16].A Neural Network is necessarily a connected collection of simple units called perceptron.Each link has a weight which defines the properties of the link.It has a bias input with weight .Each perceptron computes the weighted sum of the inputs and then applies a sigmoid function to generate the output.The Neural Network learns by iteratively processing each tweet of the training set.It calculates the error by comparing the output of the network with given target value.It than back propagates the error so that weight of connection links are adjusted to reduce the mean-squared error between the network output and actual target value.
The Support Vector Machine is a supervised classification technique in which maps the original data into a higher dimension space using nonlinear mapping.It then searches the higher dimension for a decision boundary which separates the tuple of one class from another which is called optimal separating hyperplane.SVM finds this hyperplane using support vectors.Support vectors are also training tuple that are closest to the decision boundary and pushed up by hyperplane [16].The SVM is a binary classifier and its main function is to find a hyperplane, known as Maximum Marginal Hyperplane (MMH) which maximizes the margin between classes.A separating hyperplane can be written as,

Where
weight vector of the form, { } is the number of attributes and is a scalar quantity referred to as bias.The equation of the MMH can be written as a constrained convex optimization problem which can be solvable using Lagrangian formulation.After solving we find the following equation for MMH as decision boundary, Moreover, SVM possesses high-dimensional input space and it is less affected by the overfitting problem.As like other text categorization problem, in our problem there are only a few irrelevant features and almost all features contain some considerable information that are necessary for emotion extraction.The number of features of our dataset in Bag of Words and Bigram modeling are 2754 and 8831 respectively.Because of this high-dimensional input space, SVM can capture all the related features.
Another aspect that is noticeable from the both Table 2 and Table 3 is bigram modeling provides better accuracy than Bag of W ords modeling.This is because the Bag of Words modeling doesn't consider the context of a term in a tweet.However, the emotion of a term is also related to the previous terms of the tweet.In bigram modeling, the relation between two consecutive terms is also taken into account thus it provides better accuracy.
The reason of better accuracy of MNB classifier than that of ANN is that we used weight and MNB performs better with it.On the other hand, the accuracy of ANN is lower than expected.We used default setup for ANN classifier.The accuracy may be increased by performing grid search method for optimal learning rate and changing the different parameters such as the number of hidden layers, number of iteration etc. of the network.Furthermore, increasing the size of dataset may increase the performance of all three classifiers.

Conclusion
Extracting emotion from text has a lot of important applications, such as e-commerce, product review analysis, personal or group level communication, automated motive detection for robotics etc. Automatically extracting emotion from text can be done by various approaches.A lot of rule based methods are established there.The main problem with these methods is creating a reliable emotion intensity lexicon with less possible effort.We analyzed supervised machine learning based approach to solve the problem.We experimented the Multinomial Naïve Bayesian classifier, Support Vector Machine and Artific ial Neural Network in this research.In our setup, SVM outperformed other two classifiers with impressive accuracy for the task.Out work can be used to eliminate currently available tedious systems of creating emotion intensity lexicon.Although the text classification problem is very old, emotion extraction from text using machine learning techniques is comparatively new.There are many scopes of work in this sector.We experimented three supervised machine learning methods.Comparative study of others classifiers will be helpful for new researchers of this domain.
As a pointer to the future work, there is a lack of established dataset for 9 emotion categories proposed by Izard.Creating an established dataset will also be a great work in this domain.We used bigram model for semantic analysis.More complex semantic analysis model can be used with advanced feature extraction technique to increase the accuracy of the system.