Emotion Embodied: Unveiling the Expressive Potential of Single-Hand Gestures

3.1.1 Image Stimuli Selection.

  Prior to eliciting gestures for different emotion expressions, our first step is to elicit these 12 emotions. We used a set of images as emotional stimuli, which is a commonly used approach in social psychology studies [24]. To determine the appropriate stimuli for the 12 emotions, we searched through 900 images from the OASIS database, an open-access image source that was published in 2017 and widely used in numerous emotion elicitation studies [41]. The 900 images were randomly divided into six parts (each containing 150 images); each part was reviewed by one of the six researchers (three male and three female) independently to select the most representative image for each emotion, with notes explaining their choices. Next, the entire research group reviewed all the selected images and voted for the best one for each emotion. Ultimately, we selected 12 image stimuli to be presented in the survey (see Figure 2). Note that we also compared the valence and arousal rating of each image from the data provided by OASIS, which consistently aligned with the emotion words that they were chosen to represent.

3.1.2 Survey Design.

The survey started with a consent form describing the survey procedure and potential risks (e.g., emotional fluctuation during the survey). We also appended a video illustrating the process of capturing single-hand gestures from different angles and our requirements for the photos and videos regarding clarity and resolution. The instructions explicitly stated that all the photos and videos collected in the survey would be used exclusively for research purposes and would not be linked to participants’ personal identities. Participants were strongly encouraged not to include any personally identifiable information, such as their faces, in the captured media. For each emotion, participants were first asked to select an option from the list of 12 words that best describes the emotion they felt, and to rate the valence and arousal of that emotion on a scale from 1 to 7. To ensure that participants choose the word solely based on their perceived emotions, we did not require them to choose different words for different image stimuli. The rating questions were designed to assess participants’ understanding of the emotions portrayed by the image stimuli and to identify potential careless respondents for later exclusion. Next, participants were requested to upload a photo and a short video (2 to 5 seconds) to capture the static gesture and their gesture-forming process. The 12 image stimuli were presented in a random order to counterbalance any bias or influence that may arise from the specific sequence in which the images are shown.

For quality control, a test question was inserted randomly in the survey, asking participants for their year of birth. Inconsistent responses between the test question and the provided age would result in exclusion from the analysis. At the end of the survey, participants were asked basic demographic questions, including gender, age, occupation, and regions they live. On average, the survey took each participant 30 minutes to complete. To thank participants for their time, each of those who completed the survey received a USD 5 Amazon Gift Card.

3.1.3 Participants.

We disseminated the survey through flyers on campus and online posts on Reddit and Facebook. Among the 457 participants who filled out the survey, 86 completed all the questions. After a thorough review of the uploaded photos and videos, we excluded the data from 23 participants due to irrelevant uploads (e.g., photos or videos not capturing any hand gestures), unclear visuals (e.g., difficulty in discerning gesture shape and position), repetitive uploads (e.g., we did not enforce participants to capture different gestures for each emotion, but we excluded those who uploaded multiple instances of the exact same content), contrast responses in valence/arousal choice compared to the associated emotion word (e.g., choosing “angry” with a high-valence rating, or choosing “happy” for a clearly negative image stimulus), or failure to pass the test question. As a result, we included data from 63 participants (39 females, 23 males, and one non-binary), whose ages ranged from 18 to 54 (M = 26, SD = 7.67). Our participants were from diverse regions including US (38), UK (11), Hong Kong (5), Kenya (3), Indian (3), Nigeria (1), Lithuania (1), and Mexico (1).

3.3.1 Gesture Coding.

Before mapping the gestures to different emotions, we first aimed to characterize the prominent gestural features from the collected photos and videos. This approach to manually characterize the gestures has also been widely used in prior work to ensure that subtleties and nuances in gestures are not overlooked [35, 69, 88]. Our gesture coding procedure involved the following steps:

Upon finalizing the codebook, we devised two parts of gestural features—static features based on the photos and motion features based on the videos. The static features included the following.

The motion features include the following.

3.3.2 Statistical Tests.

We used the words chosen by participants to represent their emotions in response to each stimulus. Rather than relying on the emotions that the original stimuli were intended to evoke, we considered the participants’ own choice of their emotions as the basis for our analysis. In accordance with the Russell Emotion Circumplex model [75], we categorized the 12 elicited emotions based on the level of valence (positive or negative) and arousal (high or low). To examine the correlations between gesture features and the two dimensions of emotions (valence and arousal), we performed a Chi-square test [81], by treating emotion valence and arousal as independent variables and each gesture feature as a dependent variable. We then conducted a residual analysis for each significant test. Noting that participants sometimes formed similar gestures to express different emotions, we are interested in investigating if any individual traits contribute to the diversity of gestures they used. Thus, we conducted correlation analysis  [55] to examine the relationships between the number of gestures that participants uploaded and their demographics such as age and gender. For participants whose gestures are labeled with multiple others, we specifically examined whether these gestures are in a similar shape; if not, they were further categorized with distinct labels (e.g., other 1, other 2) to facilitate the correlation analysis.

Figure 3:

View Figure

3.3.3 Interview Data Analysis.

All the interviews are audio recorded and transcribed into text. To derive insights into participants’ gesture-forming rationales, we analyzed the interview transcripts alongside the captured photos and videos of the gestures. Four researchers (two senior researchers, one graduate RA, and one undergraduate RA) first independently read and familiarized themselves with three out of the 11 transcripts and noted down the recurring patterns. Through multiple rounds of discussions, we identified the major themes from the interviews that are centered around the connections between image stimuli, emotion understanding, and gesture expression. Then, the two RAs collaboratively completed the analysis of the remaining transcripts. Next, the entire research team worked together to merge similar codes, address discrepancies, and refine the themes identified. Note that interrater reliability check was not performed for the interview analysis as we aimed to understand participants’ rationale for their gesture formation in this particular study rather than to generalize the rationale patterns [54]. Instead, the analysis reliability was ensured through independent coding and cross-checking among the four coders. To provide a visual representation of the insights gained from the interviews, the graduate RA sketched all the gestures formed by the 11 participants who took part in the interviews.

4.1.1 Emotion Distribution.

Our original image stimuli were selected to evoke 12 different emotions (as described in Section 3.1.1). However, participants sometimes chose the same word in response to different stimuli. As a result, each of them chose five to 12 distinct emotion words (M = 8.75, SD = 1.54). We found that participants tended to choose words representing high-arousal emotions such as excited, happy, angry, and frustrated, compared with those representing low-arousal emotions such as content, calm, tired (see Table 1). Thus, in our subsequent analysis to quantify gestural features across emotions, we focused on the two dimensions of emotions (valence and arousal), as their sample distributions are more balanced than those within individual emotion words.

Figure 4:

View Figure

4.1.2 Gesture Overview.

In response to each of the 12 image stimuli, each participant captured gestures in a photo and a short video; therefore, we collected 756 single-hand gestures from the 63 participants. Our participants used 4 to 12 unique gestures to express the emotions they experienced in the survey (M = 7.71, SD = 1.56). Interestingly, we found a moderate negative correlation between participants’ age and the number of unique gestures they formed (cor: -.25, p <.05): the younger participants tended to use more diverse gestures. Most participants used the same hand (often the left one) to form all the gestures, while using the other hand to take photos and record videos; only two participants alternated two hands throughout the survey. Five participants used a front-facing camera to capture their gestures, and others all used a back-facing camera.

4.1.3 Gesture Names Across Emotions.

Among the collected gestures, 667 (88.23%) were assigned with one of the 20 gesture names, while the remaining gestures (89, 11.77%) were labeled with others. The most commonly appeared gesture names were illustrated in Figure 4, which include “closed fist” (104, 13.76%), “open palm with spread fingers” (92, 12.17%), “thumbs up” (73, 9.66%), “ok” (60, 7.94%), “victory” (58, 7.67%), “open palm with fingers pressed together” (43, 5.69%), “thumbs down” (41, 5.42%), “scratch” (40, 5.29%), “scoop (27, 3.57%), “index finger one” (23, 3.04%), “finger touch” (16, 2.12%), “grab” (15, 1.98%), “horn” (11, 1.85%), “gun” (11, 1.46%), and “number six” (11, 1.46%).

As shown in Figure 3, “closed fist,” “scratch,” and “thumbs down” are often used for expressing negative emotions, while “closed fist” appeared more frequently in high-arousal emotions (tense, angry, frustrated), “thumbs down” appeared more frequent in low-arousal emotions (depressed, bored, tired), and “scratch” only appeared in high-arousal emotions. On the other hand, “thumbs up,” “ok,” and “victory” are commonly used for expressing positive emotions no matter whether the arousal level is low or high. An open palm, either fingers pressed or spread, is seen in all types of emotions. In addition, some gestures, although not widely used by participants, seemed to be used more often in certain emotions. For instance, most “number six” gestures are used for expressing positive emotions with high arousal, most “grab” gestures are used for expressing negative emotions with low arousal, and most “scoop” gestures are used for expressing low arousal emotions.

4.1.4 Gesture Motions.

Of the 756 gestures, 131 (17.32 %) were labeled with “none” as they did not involve any motions, 68 (8.99%) were labeled with “others,” and 114 (15.08%) were labeled with “multiple others” which involved more than one motions. Among the remaining 443 gestures, 195 (44.02%) involved repeated motions, meaning that their ending status as “moving”; 248 (55.98%) were one-time motion without repetition, meaning that their ending status is “static” (we still coded the motion frequency of this group depending on how fast the hand shape was formed). The most commonly observed repeated motions included “finger extension and flexion” (53, 11.96%), “arm extension and flexion” (41, 9.26%), “shaking” (34, 7.67%), “palm flipping” (33, 7.45%), “radial and ulnar deviation” (19, 4.29%), and “knocking” (18, 4.06%). The most commonly observed one-time motions included “finger flexion” (124, 28.00%), “finger extension” (37, 8.35%), “wrist flexion” (13, 2.93%), “palm supination” (11, 2.48%), “palm pronation” (10, 2.26%), and “squeeze” (9, 2.03%). Note that a continuous repeated motion can consist of two types of one-time motions (e.g., “finger extension and flexion” consists of repeated “finger flexion” and “finger extension;” “palm flipping” consists of repeated “palm supination” and “palm pronation”). We illustrated these motions in Figure 5.

Figure 5:

View Figure

4.1.5 Gestural Features × Emotion Dimensions.

The Chi-square test showed that the finger-pointing direction (χ² = 285.51, p <.001) is significantly correlated with the valence and arousal levels of the emotions. Through residual analysis ¹, we found that when expressing negative emotions with low arousal, participants tended to point downward with their fingers (residual = 6.33); and when expressing positive emotions with high arousal, participants tended to point upward (residual = 5.87). Note that apart from “thumbs up” and “thumbs down” which have a clear indication of the finger-pointing direction, other gestures with the same name may have different finger-pointing directions (e.g., we observed downward pointing gestures a shape of “ok,” “victory,” and “number six”).

Gesture strength (χ² = 107.44, p <.001) is also significantly correlated with the valence and arousal levels of the emotions. The residual analysis showed that when expressing negative emotions with high arousal, the gesture strength is more likely to be tight (residual = 6.23), and when expressing negative emotions with low arousal, the gesture strength is more likely to be loose (residual = 4.54). However, no significant trend of gesture strength was observed within the positive emotions.

The Chi-square test did not yield significant results for other gestural features (e.g., palm direction, ending status, motion frequency), suggesting that in our study, these features are not correlated with the valence or arousal levels of the expressed emotions.

4.2.1 Emotion Activation With Image Stimuli.

When recalling their survey experience, participants shared four ways that the stimuli activated their emotions. First, participants often relied on the interpretation of how the people in the image felt (P2, P3, P8, P6, P10, P11). They mentioned analyzing the facial expressions and body language of the people in the images: “she (the woman in Figure 2 (5)) is getting a massage. She’s sleeping and smiling, and I was like, oh she’s in a very relaxed state” (P2). Second, participants tended to project themselves in the depicted scenario (P1, P2, P4, P7, P9, P10), as P1 recalled: “The mountain and the lakes (Figure 2 (6)) were just so beautiful, it’s like if I am in such a good view, maybe in New Zealand, I would feel peaceful and calm.” Third, participants engaged in imagining themselves interacting with the people in the image (P1, P2, P5, P8). This not only involves projecting oneself in the depicted scenario but also considering how one would respond to the people and objects in that scenario. For example, seeing an old lady tired and upset, P9 imagined themselves talking to the lady: “The old lady in the picture (Figure 2 (12)) kinda looks tired, and worn out like someone who has maybe spent all day with negative things. I felt like maybe I was talking to her and also felt depressed.” Fourth, some participants were invoked with related personal memory (P8, P10): “It reminds me of my sister. She just gave birth to a baby, so seeing this picture (Figure 2 (2)), I could totally feel how happy it is” (P8).

4.2.2 Emotion Externalization Through Gestures.

  Upon understanding or experiencing the emotion, participants externalized that emotion by forming different gestures using a single hand. In particular, we identified four channels through which they made a connection between the emotion and their gestures. Depending on the nuances of the emotions that participants experienced, it is common to express an emotion through multiple channels.

Figure 6:

View Figure

Communication Norms. Most commonly, participants relied on shared understandings and conventions regarding nonverbal communication to form a gesture (P1, P3, P5, P6, P7, P8, P9, P10, P11). The most representative examples are using “thumbs up,” “ok,” and “victory” to convey positive emotions, as P5 mentioned: “I didn’t think too much about it. The gesture (victory) for me basically means cool, that’s what you show your friends when you feel awesome. so I just went for it.” P3 also added: “honestly these gestures (ok) are kind of like they came from everyday life.” Similarly, a “closed fist” and “thumbs down” were considered as a conventional way to convey negative emotions, as P4 explained that “thumbs down shows disagreement and I don’t like it;” and P11 believed that “I think most people know, it (a closed fist) means frustrated or something.”

Sometimes, the ways that participants formed the hand gestures were influenced by the communication norms in certain cultures (P1, P4, P5, P10). For instance, P1, a fan of street dance, used “number six” to express the feeling of excitement (Figure 4 (d)): “my inspiration is from the street dance because when we see amazing poses, we respond just like this one (forming a “number six” gesture) with our hands.” Likewise, P4 was inspired by the rock culture and used the “rock” gesture to express happiness (Figure 1 (b)): “I like rock music and this just means let’s rock or it is cool.”

When asked about how to choose between existing gestures with similar meanings (e.g., “thumbs up” versus “number six”), participants found themselves being influenced by the elements in the image stimuli. For example, when presented with an image depicting a skydiving person and an image with two kids laughing, P1 felt happy (positive valence, high arousal) in both scenarios. However, they responded with “number six” to the former stimulus and “thumbs up” to the latter stimulus, because they considered “thumbs up” is more appropriate for kids.

Creative Embodiment. Beyond communication norms, participants also self-created gestures in non-conventional forms. They assigned meanings and narratives behind each gesture, forming a more personalized mode of expression (P1, P2, P3, P6, P7, P10). For example, when presented with the image stimulus of a person skydiving, P2 expressed the excited emotion (positive valence, high arousal) by quickly opening their palm to mimic the procedure of a parachute opening (Figure 5 (a2)). They explained: “just the thought of skydiving also seems very adventurous so it’s like Like there like a bird or letting go of stuff. I remember I did it (opening the palm) very fast because I think this is how fast it happens.” P1 also created a narrative related to “letting it go” with an open palm (Figure 1 (j)), but in a different emotional state—depressed (negative valence, low arousal): “When you feel depressed, usually that’s because you lost something. So I’m opening my hand to let them go.”

In another example, P7 expressed the bored emotion (negative valence, low arousal) with an open palm by repeatedly extending and flexing all the fingers to create an action of “pushing away” (Figure 5 (a)) as they explained: “I used that gesture to get away the boredom that is making you feel bad, so like I am pushing it away.” P6 also created a gesture of “pushing away” to express a feeling of tense (negative valence, high arousal) with a similar rationale, while they used a closed fist instead of an open palm: “When you feel tense, you are like confronting whatever in front of you, so I’m trying to push it away a bit harder.”

Furthermore, we found the elements in the image stimuli often served as a source to inspire participants’ creation of gestures. For example, when presented with a scene featuring mountains and a lake (see Figure 2 (6)), several participants (P1, P2, P3, P11) tried to mimic the gentle wave of the water to convey the feeling of calmness (Figure 1 (g)), as P2 explained: “This gesture allowed me to visually represent the serene and tranquil atmosphere depicted in the image.” Sometimes, participants used their hands to mimic the posture of the person in the image. This was commonly observed in expressing tired and depressed, where the image stimuli included people who sat in corners alone, lowering their heads. In response to the stimuli, participants formed a loose “open palm”, slowly “falling down” (Figure 5 (g)) to depict “a person with no power” (P1), “a downhearted person” (P7), and “a depressed person who is going down and down” (P11).

Physical Expression. Physical expression, differing from communication norms or creative embodiment, refers to a process of directly externalizing emotion based on one’s physical instinct, and oftentimes, as a way to vent out the emotion. All the participants mentioned such physical expressions during the interviews. For example, in an attempt to express an emotion of negative valence and high arousal, we found several instances with a closed fist heavily shaking (Figure 5 (c)) or punching forward (Figure 5 (j)). As participants explained, this gesture serves as a physical manifestation of the intense emotions when they felt angry or frustrated (P1, P2, P3, P5, P6, P7, P8, P9, P10, P11): “Personally, when I’m angry, I like to just hit some stuff like no point” (P9). Another example is the use of a “scratch” gesture (Figure 4 (j)) to express frustration (P1, P3, P6, P7, P10, P11), as P3 elaborated: “When you feel tense, lost, and don’t know what to do, you just keep scratching your head, your leg, or somewhere else. So yeah, this is what I would do if I’m frustrated” (P3). The “squeeze” motion (Figure 5 (h)) we found from the survey data was also used to convey similar emotions.

Abstract Expression. Occasionally, participants may resort to abstract expressions, where they could not articulate why and how they formed the gestures to express certain emotions (P2, P8, P10, P11). For example, P8 formed a “finger touch” gesture (Figure 4 (m)) to express bored emotion but found it difficult to provide a clear explanation: “there’s not much meaning. I thought actually, I was trying to mimic something that feels bored on my own.” One reason for this abstract expression was “running out of gestures” at the end of the survey (P2, P10), leading participants to create gestures that might not have a direct connection to the specific emotions they were trying to express. When P10 used a “finger touch” gesture to express the feeling of anger, they acknowledged that the gesture was not that intuitive, and partly it was due to the lack of sources: “I wanted to show something strong but did not know how to do it. Also, I don’t want to repeat my gestures.”

4.3.1 Expressing Positive versus Negative Emotions.

  Reflecting on their own emotion expressions with single-hand gestures, participants realized that when expressing positive emotions, besides “upward” gestures such as thumbs up and victory, they also preferred “more open and large” gestures such as an open palm (P1, P2, P7). P2 explained this was because they would like to “share the happiness and let others know” and P7 believed that “positive feelings are inherently open and welcoming.” Negative emotions, on the contrary, were considered “less open” (P1, P2, P3) and “personal” (P7). Thus, participants tended to use a closed fist or scratch gesture to “close” themselves: “I don’t think I want to share anybody about my negative feelings.” (P1).

4.3.2 Expressing High-Arousal versus Low-Arousal Emotions.

Although our statistical test did not yield any significant results with motion-related features, several participants mentioned that they liked to add movements for emotions with high arousal levels (P1, P2, P3, P7, P8, P10). During the interviews, P3 explained the reasons for quickly shaking their fist to express the angry emotion (Figure 5 (c)): “Because it is so intense, I just wanna shake my fist as fast as I could to express such intensity.” P7 and P10, chose to repeat the action of downward pointing when they used “thumbs down” to convey a feeling of anger or frustration (Figure 5 (i)): “compared with tired and depressed (emotions) where I did not move my hand a lot, this time I kept pointing down to express the anger” (P10).

In expressing positive emotions, similarly, participants mentioned adding movements when they felt extremely excited or happy. A typical example is the “repeated upward pointing” with “thumbs up” (Figure 5 (k)), as P8 noted: “I would associate something slow with being tired or sad or lazy, but if it’s like fast, it’s more positive or more intense.” In another example, P1 kept rotating their hand while making a “number six” gesture (Figure 1 (c)) to express the high arousal emotion (e.g., happy). The observation of repeated movements, in our analysis, was attributed to physical expression, as participants made the movements largely driven by their physical instincts.

4.3.3 Using Same Gestures to Express Different Emotions.

We found several instances where participants used the same gestures, mostly commonly—an “open palm” to express different emotions. As we mentioned earlier, some participants felt an open palm conveys openness and positivity (P1, P2, P7), whereas others used an open palm to express a negative feeling (e.g., waving the palm to “let it go”). In part, participants acknowledged that the distinctions between emotions within the same valence-arousal quadrant were not always obvious (P5, P9, P11). For example, P5 used the same “open palm” gesture to express the feeling of calm and explained: “But you know, being relaxed and being calm, is more of the same thing. I cannot really get a difference between them.” Likewise, P9 used the “victory” gesture for both happiness and excitement, noting that “maybe feeling excited is more intense but to me they are basically the same, which is that you feel good about something.”

5.2.1 Emotion Characterization.

Our survey results showed that participants tended to choose words representing high-arousal emotions rather than low-arousal emotions. One explanation could be that high-arousal emotions are more intense and captivating in nature [31], and therefore are more noticeable. Also, the word choice may be influenced by participants’ personal experiences—the frequency of encountering certain emotions in daily life. Words such as happy, excited, and angry are more commonly used and discussed in daily life, compared with those such as content, calm, and tired. As a result, participants might be more familiar with these words and find them readily available for expressing their emotions [61]. During the interviews, participants mentioned not being able to discern the difference between certain emotions (e.g., relaxed versus calm). This could be due to the inherent similarity of emotions in the same valence-arousal quadrant [75], or participants’ infrequent exposure to the words in life.

While not all participants ended up choosing 12 different emotions, the average and minimum number of emotion words chosen were 8 and 5, respectively. This finding suggests that the spectrum of emotional responses extends beyond the simplistic categorization of emotions by valence (positive, neutral, and negative), which are commonly employed by commercial emotion tracking systems [11] or questionnaires [46]. Future designs for emotion tracking should take into account this diversity in framing emotions.

Taken together, the nuances in participants’ word choice in our survey led us to ask: how to better characterize the captured emotions in a way that matches individuals’ mental models? Although valence and arousal are commonly used in academic research to describe emotions, they may not necessarily align with individuals’ everyday expressions of emotion. That being said, while these two concepts are theoretically able to reflect on any emotions in a joint manner, it is hardly possible to accurately locate one’s everyday emotions that vary in a subtle and instant manner in real-world scenarios. One opportunity is to gather their preferences and prior experiences and provide personalized options. This process can be facilitated by starting with high-frequency words from each valence-arousal quadrant (e.g., more words representing high-arousal emotions) and gradually guiding individuals to customize their preferences thereafter. Due to the intricate and complex nature of emotions, further research is needed to explore how individuals express emotions associated with additional meanings from other emotion models (e.g., the six emotions specified in the Basic Emotion Theory involving sadness and disgust [22]).

5.2.2 Emotion Contextualization with Gestures.

As the source of emotion elicitation, the presented image stimuli inevitably added nuances to the emotions that participants experienced and to some extent, influenced their emotional responses and subsequent expressions through gestures. This influence was observed in all the participants, highlighting the contextual nature of emotions that are affected by various factors, including external stimuli, personal experiences, and individual interpretations. Therefore, when designing emotion tracking systems, capturing only the words or numeric values of the experienced emotions is not sufficient, and the importance of capturing what individuals were experiencing when certain emotions arose has been emphasized in prior work [11, 72, 78]. This emotional experience involves different contextual information, which can be automatically tracked (e.g., date, weather, physical activities, and location) or requires manual annotation (e.g., events, people, and elaboration on other details).

During the interviews, we found that even with a single hand, participants were able to convey rich meanings alongside the image stimuli, which pinpoints how representational gestures provide an implicit way to understand the underlying mechanism of semantic-driven information carried by hand gestures. This brought up a new possibility of contextualizing emotions with gestures. For example, a slowly waving “open palm” mimicking the gentle water waves can refer to an unspoken yet personalized record to depict a relaxed vacation; a hand gradually falling down may reflect a tiring person grappling with work stress; and a rotating “number six” could resemble an exhilarating concert memory, etc. As these gestures are easy to perform, they can serve as situated references that provide contextual cues of the experienced emotions. Through creative embodiment, individuals can incorporate their personal experience into a simple hand gesture, which can accommodate the diverse ways in which individuals express their emotions and potentially foster self-awareness as a way to enhance mental wellbeing [44]. It is also worth pointing out, that younger participants were more likely to employ more diverse gestures. This observation suggested that younger individuals may find it more natural and accessible to engage in gesture-based emotion contextualization. Therefore, the age of target users and their preferences for and behavioral patterns of gestures should be fully understood to inform the design of user-centered gesture-based emotion-tracking systems.

5.3.1 Enhancing Multimodal Emotion Tracking.

Although many wearable and sensing technologies can accurately detect emotion arousal, capturing the valence has been more challenging [21, 64, 70, 86, 94]. For instance, heart rate (HR) or heart rate variability (HRV) captured by a smart watch can suggest when a person feels aroused, but it is unclear whether the arousal is from a feeling of happiness or anger [21, 70]. As such, some mobile health apps (e.g., Cardiogram [14]) allow users to manually label their emotions alongside their recorded heart rate data, which provides a better understanding of how one’s HR is correlated with different emotional states, but imposes a burden on recollecting their emotional experiences. In this case, a single-hand gesture can come into play. When an abnormal HR or HRV is detected, the watch can alert the wearer through a vibrate notification with a short message asking how they are doing at the moment. To respond, the wearer could simply perform a “thumbs up,” “thumbs down,” or “open palm” to indicate if they are in a positive, negative, or neutral mood. This effortless interaction enables quick integration of one’s subjective feelings into the emotion recognition system in real-time, lowering the labeling effort and reducing recall bias.Although some gesture-based expressions can be abstract and hard to interpret, they still offer an avenue for individuals to convey their emotional state without explicitly disclosing sensitive information, which to some extent preserves individuals’ privacy. To better inform the recognition of individual emotions with gesture input, future systems can provide an entry for users to revisit and edit the meanings of their gestures through text or speech input [50, 51], as a way to reinforce personalized learning.

Gesture input can also be integrated into multimodal systems to enrich the meaning of emotions. Drawing from our findings on the relationships between emotion dimensions and gestural features, the nuances in finger-pointing direction and gesture strength can be used to infer the valence and arousal of the corresponding emotion. Besides, several participants preferred using “larger and open” gestures to express positivity and willingness for sharing. Interestingly, a recent work by Asaliouglu et al. also analyzed the relationships between gesture size and emotional states and showed that narrow gestures were connected to a higher level of arousal [5]. Taking this work together with ours points to opportunities for examining the connections between emotional dimensions and gestural features that were not characterized in this study. In addition to static features such as gesture size, motion features such as moving range and fluidity can add insights into the emotion externalization process, which warrants lab experiments equipped with precise motion and wearable sensors.

5.3.2 Beyond Emotion Tracking.

As Koh et al. noted in their mapping between symbolic gestures and commonly used emojis [36, 37], gesture input has great potential to enhance the communication experience in instant messaging. Our work offers the possibility to expand such communication on multiple platforms and devices such as wearable devices (similar to emotion labeling mentioned above) and motion sensors at a short distance [83], as well as in a wide range of contexts especially when typing or speaking is not convenient (e.g., public spaces). Our participants also brought up the idea of using single-hand gestures to facilitate communication when one of their hands is not available. Moreover, the insights gained from this study can be applied in human-robot communication. As Zhou et al. found in their work, humans perceive and interpret robots’ emotions through tactile gestures (e.g., shaking, pushing, patting) in a similar manner to how they interact with other humans [95, 96]. While our work focused on nontactile gesture interaction, we see the opportunities to incorporate tactile stimulation in the future and empower individuals to establish a more intuitive and personalized means of communicating with robots. For example, one could define their own gesture sets as commands that instruct a robot to perform certain tasks, and the robot could simulate its user’s gesture patterns as part of responding actions, further enhancing the engagement of the interaction.