Emoji Emotional States: Classification on the Valence and Arousal Axes


 Emojis are frequently used by people worldwide as a tool to express one’s emotional states, and have recently been applied to the research field to assess the same. However, the details of how they correspond to the human emotional states remain unidentified. Thus, this study aimed to understand how emojis are classified on the valence and arousal axes, and to examine the relationship between the former and the human emotional states. In an online survey of 1,082 participants, a nine-point scale was employed to evaluate the valence and arousal levels of 74 facial emojis. The cluster analysis revealed these emojis to be categorized into six different clusters on the two axes of valence and arousal. Further, the one-way analysis of variance indicated these clusters as having six valence and three arousal levels. From these results, each cluster was interpreted as (1) a strong negative sentiment, (2) a moderately negative sentiment, (3) a neutral sentiment with negative bias, (4) a neutral sentiment with positive bias, (5) a moderately positive sentiment, and (6) a strong positive sentiment. Therefore, facial emojis were found to comprehensively express the human emotional states.


Introduction
Emoji, a powerful non-verbal communication tool 1 , is frequently used by individuals globally as a tool to express their emotional states during daily communication 2 . Recently, it has also been applied to research elds, such as consumer studies, to assess users' emotional states. For example, a set of human facial emojis has been employed for evaluating food products to understand participants' preferences 3 . As compared to the traditional text-based methods, the use of emojis in such studies has been advantageous, such as the ease of answering questions and targeting people with different languages. Therefore, it is essential to understand how each emoji is associated with the human emotional states to accelerate its applicability.
The human emotional states can be plotted on two independent axes: arousal and valence (i.e., the core affect 4 ). Jaeger et al. 5 reported that 33 types of facial emojis (e.g., ) can be classi ed into 3 clusters (i.e., positive, neutral, and negative) on the latter. However, they did not consider both the axes. Therefore, how facial emojis are classi ed on these two axes has not been identi ed thus far. Clarifying how emojis are categorized on both the arousal and value axes using cluster analysis will allow us to understand common features of emoji that represent similar emotional state and to better interpret the results of studies that use facial emojis to assess emotional states.
There are various kinds of facial emojis that are considered to be able to assess emotional states.
According to Emojipedia 6 , the Apple platform (ver. iOS 14.6) has implemented 94 kinds of human facial emojis thus far. However, previous studies have employed insu cient facial emojis. For example, Jaeger et al 5 and Phan et al. 7 used only a set of 33 and 6 facial emotions to assess emotional states, respectively. Therefore, it remains unclear how emojis not utilized in previous studies are categorized on the arousal and valence axes. Clarifying this would increase the number of emojis employed in research to evaluate the human emotional states; furthermore, it would allow more detailed analyses to be conducted.
Therefore, this study purported to understand how various facial emojis used by smartphone systems (Unicode 13.0) are classi ed into the valence and arousal axes. Additionally, it aimed to explore the relationship between these emojis and the human emotional states based on the core affect theory. As mentioned above, previous studies categorized human facial emojis into three clusters on the valence axis. However, they can be plotted on the two independent arousal and valence axes. Thus, we hypothesized that facial emotions could be classi ed into multiple clusters on these axes.

Analyzed data
The responses to the two dummy questions were examined and 122 participants who answered both of them incorrectly were excluded from the remaining analyses. Consequently, the data of 960 participants could be employed for further analyses (M age= 30.55, SD = 5.32; 455 males, 505 females). The mean and standard deviation of the valence and arousal levels for each emoji were plotted on a scatter plot ( Figure 1). In this plot, the horizontal and the vertical axes denoted the valence and arousal levels, respectively.
Clusters of emojis on the valence and arousal axes A hierarchical cluster analysis was performed on the valence and arousal levels of each emoji to classify similar emojis into several clusters. The Euclidean distance and the Ward aggregation criterion were considered in the analysis (the Z-scores were calculated for each rating and used). The optimum number of clusters was obtained from the dendrogram and the Calinski and Harabasz index 8 , as performed in a previous study 5 . Consequently, a six-cluster solution was retained. Table 1 indicates the detailed   information for each cluster, and Tables 2-4 display the facial emojis classi ed into each cluster. clusters on the former (ps<.05), thus indicating that they were independent of each other. Furthermore, signi cant differences (ps<.05) were found among the following three classes with respect to the arousal levels: high (clusters 1 and 6), moderate (clusters 2 and 5), and low (clusters 3 and 4) (See Table 1 for further details)

Discussion
This study aimed to assess how various facial emojis are classi ed in the valence and arousal axes, and to examine the relationship between these emojis and the human emotional states using the core affect theory. Based on the data of 1,082 participants, we analyzed the valence and arousal levels indicated by each of the 74 emojis. The emojis used in this experiment tended to be distributed in a U-shape on the two axes ( Fig. 1), which was similar to a previous study's result that employed only 33 facial emojis 5 . The present research's emojis were classi ed into six different clusters on the two axes, as we hypothesized initially. Further, these clusters had six valence and three arousal levels (Table 1). Thus, each cluster was interpreted as: (1) a strong negative sentiment, (2) a moderately negative sentiment, (3) a neutral sentiment with a negative bias, (4) a neutral sentiment with a positive bias, (5) a moderately positive sentiment, and (6) a strong positive sentiment. A previous study categorized emojis into only three levels on the valence axis, possibly because it utilized limited emojis and focused only on one axis constituting the human emotional states. However, the present study con rmed that the emojis could be classi ed into six clusters on the valence and arousal axes by using plenty of them and that both axes constituted the human emotional state. Therefore, it was concluded that emojis could display the human emotional states in a greater detail than that reported previously.
The human emotional states (the core affect) are circularly aligned on the valence and arousal axes 4 . Because this study acquired the valence and arousal levels indicated by each emoji, all six clusters could be corresponded with the emotional states described in the core affect (Fig. 2), which has been discussed in the following sections.
In the "strong negative sentiment" cluster, the following 12 emojis were classi ed: , , , , , , , , , , , . This cluster was characterized by extremely low valence and high arousal levels (valence: 2.74 ± 0.40, arousal: 6.91 ± 0.37). The human emotional states based on the core affect 4 that are considered to correspond with such levels would be "Nervous," "Stressed," and "Upset." In present study, emojis named "Confounded face ," "Tired face " and "Angry face " were classi ed into this cluster. These emojis have also been categorized in the "negative sentiment" cluster, interpreted with the words "Nervous / anxious / worried", "Stressed", and "Angry / annoyed," in previous study 5 . However, a prior research included the emoji named "face screaming in fear " in the "neutral/dispersed sentiment" cluster. Additionally, it may be interesting to note that this cluster did not have an emoji with the corners of the mouth raised (i.e., smiling). These results indicated that the emojis classi ed in this cluster corresponded appropriately to the emotional state of the core affect in general; however, some emojis' interpretations require caution because they are inconsistent across studies.
Its characteristics were low valence and moderate arousal levels (valence: 3.59 ± 0.37, arousal: 5.84 ± 0.30). The human emotional states that were considered to correspond with these levels were "sad" and "depressed." The present study evidently classi ed the emojis named "crying face " and "Pensive face " in this cluster. These emojis have also been categorized in the "negative sentiment" cluster, and both of them interpreted with the words "Sad / unhappy" and "depressed," in previous study 5 . Furthermore, none of this cluster's emojis were grouped into the more positive clusters (i.e. "neutral/dispersed sentiment" and "positive sentiment") in previous research. Additionally, this cluster did not include the emojis with a raised corner of the mouth (i.e., smiling), similar to those classi ed into the "strong negative sentiment" cluster. Therefore, it was reasonable to infer that the emojis grouped into this cluster corresponded exceedingly well to the emotional state of the core affect.
In the "neutral sentiment with a negative bias" cluster, the following 19 emojis were included: , , , , , , , , , , , , , , , , , , . Its characteristics were a low to moderate valence and a low arousal level (valence: 4.27 ± 0.38, arousal: 4.83 ± 0.27). The human emotional states that were considered to resemble these levels were "lethargic" and "fatigue." The emojis named "expressionless face ," "neutral face ," and "grimacing face " were grouped into this cluster; they were also categorized in the "neutral/dispersed sentiment" cluster. In a previous research, the former two emojis were interpreted using the words "neutral/indifferent," "no comments/opinion," and "confused/unsure" and were considered to be close to the emotional states represented by this cluster. However, the interpretation of the "grimacing face " emoji in a previous study included both positive (i.e. "happy" and "excited") and negative emotions (i.e. "nervous/anxious," "worried," and "stressed"). Therefore, it was rational to infer that the emojis classi ed in this cluster corresponded appropriately to the emotional state of the core affect in general; however, some emojis require caution because their interpretations can vary.
In the "neutral sentiment with a positive bias" cluster, the following 12 emojis were classi ed: , , , , , , , , , , , . Its characteristics comprised a moderate to high valence and a low arousal level (valence: 5.49 ± 0.38, arousal: 5.19 ± 0.29). The human emotional states that were considered to correspond with these levels were "calm," "relaxed," and "serene." The "relieved face " emoji was classi ed in this cluster.
However, none of the emojis grouped into this cluster in this study were categorized into the "neutral/dispersed sentiment" cluster in a previous one. The "smiling face with sunglasses ," "relieved face ," and "face with tongue " emojis were grouped into the "positive sentiment" cluster; additionally, the previous study's participants interpreted them using the words "be/act cool," "happy," "naughty/playful," "exited," and "content/satis ed." Hence, it can be considered that this cluster's emojis represented a slightly greater positive emotional state than the valence and arousal levels.
The human emotional states that were considered to resemble with these levels were "contented" and "happy." Interestingly, all emojis in this cluster had a raised corner of the mouth or smiling eyes (i.e., they were smiling). None of them were classi ed into the more negative clusters (i.e. "neutral/dispersed sentiment" and "negative sentiment") in previous studies. Therefore, this cluster's emojis corresponded well with the emotional states in the core affect.
The human emotional states that were considered to correspond to these levels were "elated" and "excited." All emojis in this cluster had a raised corner of the mouth or smiling eyes (i.e., they were smiling), similar to the those classi ed into the "moderately positive sentiment" cluster. Additionally, many of them indicated richer expressions, such as heart (star)-shaped eyes, pink cheeks, or throwing hearts. The "smiling face with smiling eyes " emoji was interpreted using the words "happy," "feeling good," and "excited" in present study 5 . Further, similar to the "moderately positive sentiment" cluster, none of the emojis classi ed into this one in the present study were grouped into the more negative clusters (i.e. "neutral/dispersed sentiment" and "negative sentiment") in previous research. Therefore, it was acceptable to deduce that the emojis classi ed into this cluster corresponded extremely well to the emotional state of the core affect.
The emojis used in this experiment and a previous study 5 were distributed in a U-shape on the two axes of valence and arousal, while the human emotional states (the core affect) were circularly aligned on these axes 4 . Therefore, emojis at least used in present and previous study may not be able to capture the human emotional states with neutral valence and high arousal level (i.e., "tense" and "alert"). Jaeger et al. 5 noted that "open-mouthed facial emoticons (e.g., , )," which indicate surprise expressions, may indicate a higher level of arousal. In fact, arousal level for these emojis in present study was somewhat higher than the mean of clusters 4 in which these emojis were included (5.52 and 5.24 respectively, while the mean for cluster 4 was 5.19). However, these emojis could not reach su cient levels of arousal to capture "tense" and "alert". Because current and previous study used major facial emojis, it may be di cult to indicate these emotional states by current set of emojis. We encourage the emoji designers to produce new emojis that can express these human emotional states.
The fact that emojis can display the human emotional states in a considerably greater detail than reported previously will accelerate their use in research elds such as consumer studies that need to evaluate these states. This is because the traditional text-based methods taxing for the participants 9 , and emojis are considered an easier way to examine the human emotional states. In addition, the latter may have the advantage of being less sensitive to the participants' native language than the former. However, this study, as well as the other ones on the human emotional states expressed by emojis, have many limitations, which are discussed in the following section. Further research is warranted to deepen our understanding of the relationship between emojis and human emotional states; nevertheless, this would further increase the emoji use in various research areas where the human emotional states need to be assessed.
This study has several limitations that must be acknowledged when interpreting the results. Firstly, all the participants of this study were young Japanese adults and individuals with other demographics were not included (i.e., age, sex, and culture). Although the "use" of emojis has been reported to be signi cantly in uenced by demographic characteristics such as age, gender, and culture 2 , their "interpretation" has been indicated NOT to be signi cantly impacted by these characteristics 5,10,11 . Therefore, we believe that the results of this study are consistent with other demographics.
Secondly, we cannot deny the possibility that slight differences in the emoji design may have affected this research's ndings because we only employed the emojis displayed on the twitter. Even with the same code, the emoji designs displayed on different devices, such as PC, mac, Android, and iPhone, vary slightly. Since researches using emoji are conducted with various types of devices, it may be necessary to understand how minor discrepancies in the emoji designs displayed on different types of devices affect the interpretation of the human emotional states.
Finally, we associated emojis with the human emotional states INDIRECTLY based on the valence and arousal axes and the theory of core affect 4 . This is because the primary purpose of this study was to understand how various facial emojis are classi ed on these axes. Further research is warranted to directly relate the human emotional states with emojis. However, we believe that the results of this study are su ciently reliable because the interpretation of emojis in each cluster was consistent with that reported by a previous study, which directly linked the human emotional states with emojis using openended questions.
This study purported to understand how various facial emojis are categorized on the valence and arousal axes, and to assess the relationship between these emojis and the human emotional states. It provided evidence that the emojis could be grouped into the following six clusters on the two axes of valence and arousal: (1) a strong negative sentiment, (2) a moderately negative sentiment, (3) a neutral sentiment with a negative bias, (4) a neutral sentiment with a positive bias, (5) a moderately positive sentiment, and (6) a strong positive sentiment. Further, we corresponded each of these clusters with the emotional states described in the core affect theory. Thus, we concluded that the emojis display the human emotional states in a considerably greater detail than that reported previously. This would accelerate their use in research elds that need to evaluate the human emotional states.

Participants
Overall, 1082 participants aged between 20 and 39 years (M age = 30.53, SD = 5.30; 532 males and 550 females) living in the capital region of Japan partook in an online survey. They were uent in Japanese (the language of the survey implementation). They were registered in an online panel maintained by a marketing research rm (https://www.myvoice.co.jp/voice/). In accordance with the ethical approval obtained prior to commencing data collection, the eligible participants were assured that their responses would remain con dential. This study was conducted in accordance with the World Medical Association's Declaration of Helsinki, and All the protocol of this study was reviewed and con rmed by local institutional review board (Committee on Ergonomic Experiments of National Institute of Advanced Industrial Science and Technology). All participants provided informed consent before they participated in the study.

Emojis used in this study
The present research employed human facial emojis, as in a previous study 5

Questionnaire
The online questionnaire consisted of two parts. The rst one examined the participants' sociodemographic and background characteristics. The second part evaluated the valence and arousal levels of each emoji using a nine-point scale (1: displeasure to 9: pleasure for valence, and 1: weak to 9: strong for arousal), as in a previous study 5 . Considering the workload of the participants, we requested them to rate only 30 of the 74 emojis individually. There were 16 different pre-de ned patterns of the order in which the emojis were presented; furthermore, each participant was randomly assigned to one of them.
Consequently, each emoji was assessed by a minimum of 420 participants.
To check whether the participants complete the questions properly or not, two additional dummy questions were included in the questionnaire. Those questions can be answered easily, if they properly read the instructions (e.g., "What is the subject of this questionnaire that you are being asked to answer?"). Dummy questions were displayed every 10 questions (i.e., on questions 11 and 21). If the participants who could not answer these questions correctly, they were excluded from the further analyses.

Data analysis
To understand how various facial emojis are classi ed on the valence and arousal axes, we conducted a hierarchical cluster analysis and a one-way ANOVA. The Tukey's test was used for a post-hoc analysis when the main effect was obtained. The results of the ANOVAs were considered statistically signi cant if the p values were less than 0.05, and if the effect size (η2) was greater than 0.06. All statistical analyses were performed using the SPSS statistical software package (IBM SPSS Statistics Version 26, SPSS Inc., Chicago, IL, USA) and the R software 12 . Figure 1 The scatter plot of the mean and standard deviation of the valence and arousal scores for 74 facial emoji evaluated by the participants. The vertical and horizontal axes represent the valence and arousal levels, respectively. The error bars indicate one standard deviation for each variable (i.e., valence and arousal)