Incorporating Textual Information on User Behavior for Personality Prediction

Several recent studies have shown that textual information of user posts and user behaviors such as liking and sharing the specific posts are useful for predicting the personality of social media users. However, less attention has been paid to the textual information derived from the user behaviors. In this paper, we investigate the effect of textual information on user behaviors for personality prediction. Our experiments on the personality prediction of Twitter users show that the textual information of user behaviors is more useful than the co-occurrence information of the user behaviors. They also show that taking user behaviors into account is crucial for predicting the personality of users who do not post frequently.


Introduction
Personality information of social media users can be used for various situations such as analyzing crowd behaviors (Guy et al., 2011) and building recommender systems (Wu et al., 2013).Many researchers have focused on developing techniques for predicting personalities and reported that models that use the textual information of target user's posts achieved relatively high performance (Luyckx and Daelemans, 2008;Iacobelli et al., 2011;Liu et al., 2017;Arnoux et al., 2017).However, some social media users frequently read others' posts but rarely post their own messages.Predicting the personalities of such users is generally difficult, but a substantial portion of them often express their opinion or preference through social media activities such as liking and sharing.
Figure 1 shows tweet examples related to Halloween.The upper tweet was posted by a user who is hosting a Halloween party and thus this user is considered to be extraverted.In contrast, the lower tweet is a post consisting of Halloween illustrations, which is considered to be posted by an introverted user.In this way, user personalities can be predicted from their posts.Moreover, users who like or share such tweets are expected to have a similar personality to the user who posted the tweet.Henceforth, we collectively refer to likes and shares as behaviors.
Several studies have leveraged the information derived from the user behaviors for personality prediction (Azucar et al., 2018).For example, Kosinski et al. (2013) and Youyou et al. (2015) proposed personality prediction models for Facebook users that leveraged a user-like matrix, the entries of which were set to 1 if there existed an association between a user and a like and 0 otherwise.Shen et al. (2015) considered the types of the posts (e.g., photos, videos, or status updates) that a target user likes or shares.However, these studies do not take into account the textual information related to user behaviors.We consider that the textual information of tweets that target users have liked/retweeted (shared) contains useful information for predicting their personalities.Therefore, in this paper, we investigate the effect of the textual information of the tweets that target users liked/retweeted.

Related Work
Many studies on personality prediction for social media users utilize the textual information derived from the user's posts.Luyckx and Daelemans (2008) extract syntactic features like part-ofspeech n-grams to predict personality of essay authors.Iacobelli et al. (2011) test different extraction settings with stop words and inverse document frequency for predicting personality in a large corpus of blogs using support vector machines (SVM) as a classifier.Liu et al. (2017) use Twitter user posts and propose a deep-learning-based model utilizing a character-level bi-directional recurrent neural network.Arnoux et al. (2017) build a personality prediction model for Twitter users that utilizes word embedding with Gaussian processes (Rasmussen and Williams, 2005).Reasonably good performance can be achieved by taking only 25 tweets into consideration.
Several studies have shown that user behaviors such as likes and shares are also useful to predict user personalities.Kosinski et al. (2013) and Youyou et al. (2015) used page likes on Facebook to create a user-like matrix and proposed personality prediction models based on the matrix.While Kosinski et al. (2013) and Youyou et al. (2015) only use the binary information related to user behaviors, Shen et al. (2015) proposed a personality prediction model that considers the number of likes and shares.Farnadi et al. (2013) focus on network properties such as network size, density, and transitivity, and time factors such as the frequency of status updates per day and the number of tweets per hour in addition to user posts.
For tasks other than personality prediction, several studies leverage the textual information derived from user behaviors in social media.Ding et al. (2017) applied texts that users liked and posted to predict substance users such as people who drink alcohol.They showed that the distributed bag-of-words (DBOW) models (Le and Mikolov, 2014) achieve good performance.Perdana and Pinandito (2018) used texts that users liked, shared, and posted for sentiment analysis.They convert them into weighted features using tf-idf and applied Naïve Bayes.They reported that texts posted by a user lead to a better performance than texts that the user liked/shared, but that the best performance can be realized by combining them.

Dataset
In this study, we predict the personalities of Twitter users.As the personality model, we use the Myers-Briggs Type Indicator (MBTI) (Myers et al., 1990), one of the most widely used personality models, as well as the Big Five (Goldberg, 1990).

Myers-Briggs Type Indicator
The MBTI recognizes 16 personality types spanned by four dimensions.Extraverted and Introverted (E/I) describe the preference of approaching the outer world of people and things vs. the inner world of ideas; iNtuition and Sensing (N/S) describe the preference of the intuition and the possibilities in the future vs. the perception of things of the present moment; Thinking and Feeling (T/F) describe the preference of rational decision making based on logic vs. subjective values; and Judging and Perceiving (J/P) describe the preference for the control of external events vs. the observation of these events.
The MBTI is often identified through a personality analysis test that consists of selective questions.Several Web sites offer such personality analysis tests, such as 16Personalities1 is one of such websites, where users can determine their MBTI type by answering 60 questions.The results are represented by 16 roles, such as Mediator for INFP and Executive for ESTJ-one for each combination of the four MBTI dimensions (e.g., I, N, F, and P).The Web site has a function that lets users post their results to Twitter with the hashtag #16Personalities.Figure 2 shows an example of such tweets.In this example, the user is analyzed to be "Protagonist", which corresponds to ENFJ in the MBTI.We collected the tweets that contain the hashtag #16Personalities and use them in the experiments.

Data Collection from Twitter
We collected tweets written in Japanese.Twitter Premium search APIs 2 were used to find the tweets containing the hashtag #16personalities and listed 72,847 users who posted such tweets in 2017 and 2018.We refer to a tweet with #16personalities as the gold standard tweet.Next, we collected the latest 3,200 tweets for each user and then discarded the tweets that were posted after the gold standard tweet.Only the users with 1,024 or more tweets were used in this study.The number of such users was 20,364.Table 1 the statistics of users for each personality dimension.We can confirm that there are biases in the number of users for all dimensions and that the bias for the E/I dimension is particularly noticeable.
To build a model based on the text related to user behaviors such as like and retweet, we collected up to 1,024 liked tweets and 1,024 retweeted tweets for each user.Table 2 shows the distribution of users based on the number of likes or retweets.14,735 out of 20,364 users liked more than 1,023 tweets and 157 users liked no tweets.Only 3,137 users retweeted more than 1,023 tweets and 162 users retweeted no tweets.

Personality Prediction Models
We treat personality prediction as a set of binary classification tasks and build four binary classifiers independently for each dimension of the MBTI.We regard the personality of the users shown in the tweets with #16personalities as the gold standard personality and attempt to predict it using the SVM classifier.Figure 3 shows an overview of the model.Specifically, we use linear SVM for classification with two types of features: those derived from the tweets that the target user likes or retweets and those derived from the tweets that the target user posts.

Features derived from User Behaviors
Use of co-occurrence information We build a model similar to Kosinski et al. (2013).They leveraged a co-occurrence matrix of users and likes, the entries of which were set to 1 if there existed an association between a user and a like and 0 otherwise.Similarly, we create the binary matrix of users and behaviors, the entries of which were set to 1 if the user liked/retweeted a tweet, 0 otherwise.For the sake of computational efficiency, we consider tweets that are liked or retweeted by at least ten users.Then, we apply singular value decomposition (SVD) to the matrix and use the dimension-reduced vectors as the features of the SVM classifier.

Use of textual information
We propose three models that consider the textual information on user behavior.All three models use MeCab 3 with the IPA dictionary 4 to perform morpholog-ical analysis.The first and second models use the 10,000 most frequent words.The first model uses them as BOW features of the SVM classifier and the second model further applies SVD.The third model is a model using DBOW proposed by Ding et al. (2017).This model uses words that have appeared ten or more times.Henceforth, we refer to these models as BOW, BOW w/ SVD, and DBOW, respectively.

Features derived from User Posts
We apply a similar procedure to generate features derived from user's posts as BOW w/ SVD.We first extract the 10,000 most frequent words and make a user-word matrix.We then apply SVD to the matrix and use the dimension-reduced vectors as the features of SVM.

Experimental Settings
We randomly split the users in our Twitter dataset into three parts: training, development, and test sets.Specifically, we used 5,000 users as the test set, 5,000 users as the development set, and the other 10,364 users as the training set.We adopted the area under the curve (AUC) of the receiver operating characteristic (ROC) to evaluate each model.

Textual vs. Co-occurrence Information of User Behaviors
We first compared the performance of the models using the textual information of user behaviors and the performance of the models using the co-occurrence information of the user behaviors.
We built the BOW models, BOW w/ SVD, and DBOW as the models using the textual information.We report results on three settings: 1) considering only likes, 2) considering only retweets, and 3) considering both likes and retweets for each model.We varied the number of dimensions in the reduced space of SVD with 50, 100, 200, 300, and 500, and the vector sizes for DBOW with 50, 100, 200, 300, and 500 and tuned them on the development set.We also optimized SVM parameter C on the development set.
Table 3 shows the experimental results.We found that the models using the textual information of user behaviors performed better than the models using the co-occurrence information of user behaviors.Among the textual informationbased models, BOW w/ SVD achieved the best AUC scores.We thus adopt the BOW w/ SVD model as the textual information model in the following subsections.
As for the types of behavior, the models based on likes and the models based on retweets achieved almost the same performance, and the models that combine both of the features achieved the best performance.

Effect of the Number of User Behaviors
We are interested in the relation between the performance of the personality prediction and the number of behaviors that the model takes into account.Thus, we performed experiments with various sizes of user behaviors.We used the BOW w/ SVD model for this experiment.
Table 4 shows the experimental results for each dimension.We can see that there is a strong correlation between the performance and the number of user behaviors taken into account.However, because the performance improvement between 256 and 1,024 was considerably small, we assume that the performance of the models will not be largely improved even if the models consider more behaviors.For each feature, as in the previous experiment, the models of likes and the models of retweets had almost the same performance, and the models that combine both features achieved the best performance.

Incorporating Textual Information of User Posts and Behaviors
We compare the performance of the models based only on the textual information of user posts and the models that also leverage the textual information of user behaviors.Specifically, we examined the effect of the textual information derived from user behaviors by changing the number of user posts.The number of texts varied from 1 to 1,024 in multiples of four.We selected the same SVD dimension for posts, likes, and retweets from 50,  100, 200, 300, and 500 and tuned the vector dimensions and SVM parameter C on the development set.Note that we used all 1,024 behaviors to make the features derived from user behaviors in this experiment.
Table 5 shows the experimental results.We can confirm that there is a strong correlation between performance and the number of user posts taken into account.When we used only a small amount of a user's posts, the performance was significantly improved by taking the user behaviors into account.However, when we used 1,024 of user's posts, we could not confirm any improvement by taking the user behaviors into account.Therefore, we conclude that utilizing user behavior is crucial for predicting the personality of users who do not post frequently-say, users who posted fewer than 256 tweets-but it is not useful when we can collect a large number of tweets posted by the target user.
When we focus on the performance of each dimension, we can find that the importance of the information derived from user behaviors, especially likes, is relatively large for the N/S dimension.For example, in the case of the N/S dimension, the AUC score taking 1,024 liked tweets into account (0.6786) was higher than that achieved by taking 256 user tweets into account (0.6765), unlike the other dimensions.

Conclusion and Future Work
In this paper, we investigated the effects of considering user behaviors such as likes and retweets for personality prediction.Through experiments using Twitter data, we found that the textual information of user behaviors is beneficial to predict the user's personality and that utilizing user be-haviors is crucial for predicting the personality of users who do not post many tweets, e.g., less than 256, but that the effect of taking user behaviors into account is very limited when we can collect many tweets posted by the target user.
In the future, we plan to explore other useful textual information for personality prediction, such as text in a web page to which the target user linked and public comments directed to the user (as reported by Jurgens et al. (2017)).We can also include replies to the target user's tweets to see if we can improve personality prediction.

Figure 1 :
Figure 1: Tweet examples.The upper tweet is about a Halloween party and the lower tweet is about Halloween illustrations.

Figure 3 :
Figure 3: Overview of the personality prediction model.

Table 1 :
The number of users in each dimension.

Table 2 :
Distribution of users based on number of likes or retweets.

Table 3 :
Average AUC scores of user behavior-based models.

Table 4 :
AUC scores of user behavior-based model (BOW w/ SVD) for different number of user behaviors.

Table 5 :
AUC scores of models with features derived from user posts with different number of user posts with/without behavior-based features.