Motion Capturing Emotions

Abstract The paper explores the activities conducted as part of WhoLoDancE: Whole Body Interaction Learning for Dance Education which is an EU-funded Horizon 2020 project. In particular, we discuss the motion capture sessions that took place at Motek, Amsterdam as well as the dancers’ experience of being captured and watching themselves or others as varying visual representations through the HoloLens. HoloLens is Microsoft’s first holographic computer that you wear as you would a pair of glasses. The study embraced four dance genres: Ballet, Contemporary, Flamenco and Greek Folk dance. We are specifically interested in the kinesthetic and emotional engagement with the moving body and what new corporeal awareness may be experienced. Positioning the moving, dancing body as fundamental to technological advancements, we discuss the importance of considering the dancer’s experience in the real and virtual space. Some of the artists involved in the project have offered their experiences, which are included, and they form the basis of the discussion. In addition, we discuss the affect of immersive environments, how these environments expand reality and what effect (emotionally and otherwise) that has on the body. The research reveals insights into relationships between emotion, movement and technology and what new sensorial knowledge this evokes for the dancer.

feedback for the dancer (audio, visual, audio-visual, verbal etc.) to test the effectiveness of responsive and interactive environments for dancers.

Context of Project
WhoLoDancE is working on introducing innovative technologies for dance learning and creation, based on the underpinning of interoperability standards and on the integration of different components into a smart learning environment. The aim of the WhoLoDancE digital ecosystem is to include tools for both formal and informal learning, developing and integrating these tools and systems that apply game-based learning and virtual environments to real-life learning situations, to encourage greater take-up of technology in the context of dance learning and creation. Moreover, the tools in development are indicating that they will be of particular benefit in supporting creative and choreographic processes by stimulating new movement pathways and sequences, within and across different dance genres.
Dance is a diverse and heterogeneous practice. Introducing tools to the embodied space of dance learning and dance making is an exciting challenge; the art form has developed training systems that are genre specific and that develop between different communities of practice. The WhoLoDancE project team has taken account of this variety of approaches by developing a protocol for the creation and/or selection of dance sequences drawn from different dance styles for different teaching, learning and creative modalities. As dance learning practices can be highly idiosyncratic and generally lack standardisation beyond dance genres and specific schools and techniques, one of the first project challenges has been to establish some parameters for the project. A methodology was agreed in the early stages of the project, which took account of different dance genres and teaching practices, and which generated a range of typical movement sequences for each of the four genres we are focusing on; Classical Ballet, Contemporary, Flamenco and Greek Folk dance.
Classical ballet and Greek folk dance have codified positions and sequences of movement that have stabilised these practices to a greater extent than many other dance genres, thus making them more amenable to categorise and "capture." Contemporary dance is more diverse and is less easy to regularise and categorise. Capturing contemporary dance practice thereby requires a process of identifying repeatable sequences of movement that can be considered broadly representative of the genre. By contrast, Flamenco includes codification of some sequences but also relies on improvisation. Codified movement phrases in Flamenco have developed over time and are an integral part of the dance, yet there is always interpretation and room for flexibility within the form. Taken together, the genres offer a range of different teaching methods, aesthetics and compositional practices and thus provide a valuable range of use cases. The sequences selected from each genre were then motion captured using a Vicon camera setup with at least 24 cameras, for the purpose of acquiring kinetic material that has built a diverse library of dance movement sequences and which are the building blocks for the tool development.
Building a repository of dance movements has depended on being able to draw on the expertise and experience of dancers, teachers and choreographers from this range of dance genres. In addition, we are also utilising the knowledge of technical experts to assist with the design and application of digital tools.1 The project team, drawing on the principles of Laban Movement Analysis (Preston-Dunlop 1983, Sanchez-Colberg & Preston-Dunlop 2002, which originates from research into human movement, designed a set of Movement Principles-Symmetry, Directionality, Balance, Alignment, Weight-bearing vs Gesturing, Gross vs Fine Motorics, Coordination, Motion Through Space, Rhythm and Phrasing and Stillness. They were established early on to provide a baseline of movement that is common to all dance forms, and which were fundamental to the design of the project's software. The Principles drew on the collective knowledge and experience of the European-wide consortium, which includes both dance and technology experts with different artistic, pedagogical and design experiences. The validity of these Movement Principles was tested in 2016 and questions were asked about their application by gathering the views of, and feedback from, the wider dance community and particularly from those working with new technologies. The general consensus was that these Principles, in practice, are widely used. We are using the Movement Principles as a foundation when discussing the entanglement of movement, technology and emotion. Maxine Sheets-Johnstone states that emotion and movement are "dynamically congruent" (7). She explains this through describing how we restrain our emotions through movement or how we feign them. This highlights the naturally occurring dynamic relationship of emotion and movement and why it is meaningful. In WhoLoDancE, we are interested in how, or whether, movement is expressed and "read" differently when rendered through various digital methods, and whether this naturally occurring relationship is disturbed by digital processing. Technology often sees the body as an object; we want to challenge this assumption and explore what new information can be obtained when the dancing body and technology work together.
We ground our discussion using the notion of embodiment, which generally refers to the corporeal knowledge that we have that enables us to sensorially experience the world. In other words, when we participate or watch dance, we absorb information sensorially and make sense of this information from our cultural histories and perceptive abilities that enable us to live this information through our body and consciousness. Sobchack (4) defines embodiment as a "radically material condition of human being that necessarily entails both the body and consciousness, objectivity and subjectivity, in an irreducible ensemble." In addition, we refer to the kinesthetic sense; the word kinesthesis is derived from the Greek word kin-movement-and aesthesis-sensation. Combining kinesthesis with empathy, this concept emerges as an empathetic interaction between the dancer/student and the visual representation, in this case, an avatar. This interaction is a sensory experience, perhaps facilitated or augmented by emotion, memory and imagination.
Fundamental to the work in WhoLoDancE is motion capture. Motion capture creates data that can be processed, stored and categorised for multiple applications (for example, biomechanical analyses, movement training as well as artistic renderings). Behavioral investigations of body and action perception have profited from the use of motion capture technology, which allows the researcher to record the movement trajectories of markers that are attached to an individual's body (Hove & Keller 2010). Whatley (2012) discusses artists Ruth Gibson and Susan Kozel's work with motion capture and how they are interested in the potential of motion capture in capturing the humanness of the moving body and the possibilities this gives in the transmission of corporeality through the avatar. Whatley concludes that Gibson's work engages the viewer kinesthetically and emotionally through the use of motion capture (268).
Furthermore, we consider sensorial destabilisation in the context of immersive environments and visual representations, such as avatars. Isabelle Choinière (2017) discusses how technology recalibrates sensorial and proprioceptive awareness and how technology may produce a dilated reality. We are interested in examining this concept of dilated reality and its relationship with emotion in dance environments, and how that affects the corporeal experience of the moving body.

Background
Dance, as an art form, can be broadly described as a series of motions with an organised intention that has an aesthetic and inherent value. Dance is also a social, recreational and cultural practice and thus is also an empirical indicator of the body's history that has a meaning and can create and nurture traditions while serving as a creative outlet. In the mid-1990s when dance motion tracking systems were relatively new, performers learned how to play the motion tracking instrument (Dixon 205). Since then techniques and the technology itself have developed, which has offered the dance community new ways of looking at and thinking about its practice. While the development of technology is a key factor of the WhoLoDancE Project, thinking about how the technology functions is not the primary focus of this paper. More particularly, exploring how the technology enhances the dancer's experience and offers new ways for dancers to think about their bodies in real space and also within a virtual space, is at the crux of this discussion.
Dance practices require that the dancer combine different sensory and motoric modalities ranging from proprioceptors, to kinesthetic and embodied memory. When these elements are taken into a digital environment what emerges are new insights to how we think about and look at the body, and particularly the dancing body. Within WhoLoDancE, the experiments with the HoloLens and the avatars2 created through the rendering of motion capture data emphasise different properties of the movement; some are more figurative and others more qualitative and they serve as transmissions of emotional states. The technology is an extension of the dancing body and allows the dancer to develop an awareness of the body that is directly connected to emotional states. The avatars and the wearing of the HoloLens ask the user to think about "space" in a different way and raise questions around if the avatar is an "alternative mirror" and starts to blur lines of perception and reality. One participant notes: "[i]t was great playing with the traces and the trails. Usually in the dance studio you get instructions of trying to imagine the shape of your movement traces, the trajectories, but with this tool I could actually see them!"3 The "traces and trails" that this participant discusses are the visual representations of the movement pathway. Through the HoloLens, these are seen in the studio space in front of them and affords the dancer a way to perceive what has previously only been available through imagery.
Dance is an art form that will always be dependent on body to body transmission. Digital technologies, such as those being developed within WhoLoDancE are not intended to replace the live experience. Rather, digital technologies are encouraging choreographers, artists and dance learners to think about their work in different ways and this may have an influence on how dance is taught and made, and may gradually change practices. What we are interested in is how these tools become a testing ground for rethinking prior practices and assumptions. The artistic contributions of the dancers in the project are fundamental to our work, not least to ensure that the project outcomes can be valuable to the wider dance community.
A unique aspect of the project is the conversation between very different dance genres. Each genre has a tradition associated with how it is taught, learnt and performed, whether in a professional or recreational context. These traditions, which are culturally inscribed, are calibrated carefully to follow particular methods and so can be resistant to digital intervention. Moreover, in these early stages of the project, the technologies demand specific equipment, technical expertise and facilities that are expensive and relatively inaccessible to most dance practitioners and teachers.
The HoloLens, which is proving such an interesting tool for eliciting new experiences about the body in motion, is itself a rather cumbersome headset that can inhibit a dancer's natural movement, wearing a machine on the face continuously is rather prohibitive, so is potentially both limiting and revelatory transformative technology for dance. It requires the full concentration of the eye, and when wearing, it feels like a new periphery becomes available and shifts the understanding of self-reflection. Being in your body and out of your body simultaneously, re-orientates the body in space and primes the dancer to see and experience what they might have missed within a traditional 2D dance studio space.
The design and structural features of the WhoLodancE environment are exploring the potential for the dancer to step into the lived experience of another body, the avatar, yet invites them to be in their own body and experience movement through their own lived body. The paradoxical nature of this relationship may begin to unlock a new creative potential that is yet unknown but may become equally important to the conversation about embodiment and creativity. What has emerged thus far is that the more figurative avatars encourage the dancer to critically examine her movement accuracy whilst the qualitative avatar encourages the dancer to engage more with her feeling state or mood, as one participant states: "so seeing myself as a bulky avatar with big volume, was an interesting experience and triggered me to move in new ways." This data is then offered to the user to dance with, in, or alongside, creating unique relationships, opening up new ways of thinking about co-creation and embodied movement making, and meaning.
When wearing the HoloLens and watching the moving visual representation in front of her, the dancer experiences a haptic necessity to touch or interact with the avatar. The dancer identifies with the inner mechanisms of the perceived movement and the tactile exterior of the body, using the proximal sense of touch. This experience seems to chime with what Laura Marks (2000) refers to as haptic visuality. Haptic visuality values the proximal senses of touch, feel and smell over the distant senses of seeing and hearing and evokes the sensory network. The experience stimulates the tactile sense which heightens the emotional interaction. The dancer's sensate body becomes enlivened when faced with the avatar, and the emotional interaction is at first, one of curiosity but develops into a deeper sense of connection. The connection between the avatar and the dancer is thereby affected by our embodied knowledge, facilitated by emotion and the haptic visuality of the representation in the immersive environment.

Immersive Environments
Immersive environments surround the participant in a natural setting like a room in one's house. What constitutes the immersion can be a variety of elements from the presence of large screens that alter direct and peripheral vision, surround sound that envelops the participant (Fisher 1990) and design structures that are navigated in the space. Sherman and Craig (9) separate mental and physical immersion and describe the latter as "bodily entering into a medium." Holography requires physical immersion and can give the illusion of existing in another space, a virtual space within a particular environment.
When considering if the virtual space, through the use of a HoloLens, primes the user to consider new ways of seeing the dancing body, we suggest that indeed there is a heightened way of viewing the body and also dancing with the avatar that lives within a digital space. Through transforming the way, the image of the dancing body is experienced the HoloLens is an interactive mirror that asks the dancer to split their attention between what is lived inside their body and what is reflected on screen. Using Heidegger's (1962) idea of "dwelling" and Polanyi's (1964) concept of "indwelling" we see that through the use of the holographic images and the immersive environments these two notions exist and open up to the user a specific intimate connection with their own body and that of the avatars. In other words, the dancer is confronted with herself and a digital identity, dealing at once with her own corporeal in-body sensation whilst viewing her own avatar; a projection of herself, divided, separated yet with the potential to be merged, thus complicating the proposition, "what is me and what is not me." This experience design plays with perception through the blurring of lines which allows the dancer to exist within two realities.
The consortium's dance partner, K-Danse Company, has experience with immersive virtual semiholographic projections. One particular example comes from the interactive choreographic digital installation A-METAPHORÁ.4 In this example,5 K-Danse Company dancers were in dialogue with semiautonomous avatars which acted as distributed sonic and visual doubles. The movement generated from this dialogue was extracted and captured for WhoLoDancE. Both the danced movement approach and the design of the environment (music, lights, colours, graphics and behaviour) were based on the same concept of a body made of multiple articulated and interconnected limbs. While some responses of the avatar were automatic, such as the response to the entire body of the dancer pausing for a certain duration, other responses varied depending, for example, on the level of fragmentation of the movement, the changes in energy and the spatial occupation. In A-METAPHORÁ, the dancer was asked to improvise while integrating various choreographic requirements and maintaining a constant relationship with her avatar double. All this led to the experience that the dancer was inside the body of her own but also of another.
A different example emerges from within the practice of Flamenco. Flamenco gestures exist within the action and the interpretation of different emotional states. Those can range from sorrow, pain, anger to joy and love and these emotions become indicators which give shape to the movement. For the Flamenco dancer who was captured in WhoLoDancE, dancing with her own avatar elicited an experience that was similar to that of the K-Danse Company mentioned above. However, in this dialogue, the avatar clearly reflected the different emotional states associated with Flamenco. Dancing with a 3D version of herself layered the dancer's experience and offered a new kind of intimacy with, and a way of relating to, her own body. The capacity to experience emotion and gestures empathetically with the hologram of the dancer and to experience emotion through (Flamenco) gestures, suggests that there is a heightened perception leading to action. Neurophysiologist Alain Berthoz and Jean Luc Petit (17) state that perception, "comes down to nothing more than the construction of a visual, tactile representation . . . a multi-modal representation, or even a supra-modal representation in the mind of the perceiving subject." Making decisions about future action is an important part of perception and Berthoz (10) makes the two synonymous with each other: "perception is simulated action." The perceptual system is integrated with the ocular musculature and the vestibular system, which allows for information from other sensory inputs to be important in perceiving (Gibson 1979). John J. Gibson was an early researcher of perception and pioneered the idea that perception involved information from the external environment, and thus from other senses. He states that the perception of one's own movement is deduced from the perception of space in which the movement occurs (1994).
Gibson's theory of perception is in the context of participation of the efferent and afferent systems in perceiving our environments. The proprioceptive sense informs our own body of its positioning in space, posture and balance and will also inform our accuracy of perception of others' bodies in space. Susan Foster suggests that all of our senses are channels that extract information from the environment and result in kinesthetic empathy experienced as "movement contagion" (57), a result of the intertwining of sensorial information. Misi and Pimentel (2016) employ Merleau-Ponty's body schema and Schiller's kinesfield to discuss what they term mediadance6. They state that technology disrupts and restructures the body schema through a continual process of adaptation and this creates the basis for perception in mediadance.
With mediadance, and in particular immersive performance, reality is dilated through the recalibration of proprioception and sensorial awareness (Choiniere 2017). The dilation happens through the efferent and afferent systems becoming destabilised by exteroceptive stimuli created by the technology. The dancers of the project who viewed an avatar through the HoloLens felt an ontological shift in their sensory awareness, a kinesthetically empathic connection with the avatar and felt compelled to move with it and invited to participate in the dance. The different way of viewing the performer with more focused and detailed proprioceptive and sensorial information available to them expanded their corporeal knowledge and emotional understanding of the dancing body.
In moving through our worlds, we experience and express ourselves as embodied. In particular, in dancing, we can express an aesthetics of embodiment (Barbour & Hitchmough 2014). The question that follows is then how is this challenged when there is an avatar that can be embodied (by our physical bodies) through augmented reality? In our work on the WholoDancE project, we found that through the immersive dance environments the dancer can experience an increase in their perception of fluency and quality of their movements. One participant notes: "I'm a small shaped dancer, so seeing myself as a bulky avatar with big volume, was an interesting experience and triggered me to move in new ways." The avatar that has been referred to as a moving "blue blob" follows the movement and kinesphere of the dancer to give this perception of volume to the body and movement. This avatar's ability to swell and shrink offers the user agency as they can respond through mimicry or countering what is displayed. We found that the dancer's perception of the dancing self changes as a consequence of the information that is fed through the screen.
Kinematics information, that is the motion of bodies or the geometries of motion-fed back through the avatars, introduces a deeper level of learning and seeing the body. Dancers respond to these virtual and immersive environments in a similar way that they do to a mirror in a dance studio. The mirror is one of the first (and continuing) technologies that dancers use to feedback information about their dancing. But the mirror prioritises a frontal orientation and prevents the dancer gaining a sense of her three-dimensionality as a body moving through and inhabiting space. Thus, the ability to inhabit a virtual volumetric avatar with the HoloLens takes the dancer/learner one step further, as this participant shares: "[s]eeing myself dancing as a female avatar was an interesting and strange feeling. It was amazing! I like the fact that most of the avatars were gender neutral. I was mostly intrigued by some avatars where the human shape was distorted; I would like to play more with this aspect." It allows the dancer to have an in-body-out-of-body experience where the permission to explore the dancing avatar and "to step into" someone else's body while still in their own body is facilitated.
As Sevdalis & Keller (2005) noted, memory for observed dance movements may be enhanced when the participant has previously performed the movements (e.g., in ballet or modern dance) relative to when he or she has only observed someone else perform them (Foley, Bouffard, Raag, & Disantorose 1991). This is relevant to our discussion as we see that the avatar can support the learning and acquisition of new dance information and knowledge. The dancer becomes an active observer, and simultaneous mover, processing information about what is viewed at the same time as moving with and in sync with the avatar. In other words, the relationship between dancer and avatar is conducive to learning, memorizing and performing dance movement. There is compelling evidence that the kinematics of body and body-part movements are at least sufficient, and may often be important, in furnishing cues for the perception of emotional expressions (Atkinson et al. 2007).

Participation Eliciting Emotion
Immersive environments provide sensory feedback (Sherman and Craig 2003) and reinforces emotional engagement and affect. Affect has become an important area of study for the dance community and Thien's (451) work on affect and its ability to describe the motion of emotion, both in a communicative and literal sense, is relevant to our work within WhoLodancE. Dance often requires highly refined motor skills and competency, as well as augmented knowledge of how one's own body moves (Sevdalis & Keller 2005). Within the immersive environment, there are shared embodied experiences of affect and emotion that come out within this virtual space that might not be possible in other environments. Our sensate body provides the information that, through cognitive processing, produces an emotional response (Damasio 2000).
The dancers who experienced the immersive environment shared what could be interpreted as a dilated reality. Their interaction with the avatar heightened their sense of embodied knowledge, learning more about their own movement in relation to their own physical body and in relation to the projected avatar, expanding the dancing field in which they are both occupier and witness. The dancer is confronted by a disturbed and expanded reality, which can produce emotional responses ranging from surprise to elation, from anticipation to confusion, even fear or ecstasy. The experience of being in the virtual environment highlights these phenomenological tensions and how bodily representations can expand corporeal knowledge and emotional engagement. The tool develops an emotional bodily awareness through this double existence. The ability to step into this other self enhances the dancer's ability to engage with the emotional self and offers a new potentiality of experiencing different emotional states.
The tracking of basic emotions, such as happiness, sadness, anger, or fear, expressed in dance or other forms of whole body movement, have been the focus for several projects that claim that emotions can be communicated accurately in dynamic full and point-light displays (Camurri, Lagerlöf & Volpe 2003;de Meijer 1989;Dittrich, Troscianko, Lea, & Morgan 1996;Walk & Homan 1984). The body becomes a form of "digital object" (El Raheb and Ioannidis 2013) which then exists within a digital environment. Claiming the body as object is problematic as, in our view, the body has a more central role to play-a "necessarily embodied self" (Edwards 1998). Therefore, a virtual space enables the user to experience a new sense of embodiment offering the dancer or learner a new entry point into viewing the dance work.
Berthoz states that action that happens in perception is a construction of a representation so when the dancer is faced with an avatar, itself a representation of a dancer, our perception is fed with rich and immediate information of the body. With the HoloLens there is a blending of the digital and real world environments which blurs what is real and allows the dancer to engage with the body in an in-body-outof-body experience. The blurring of boundaries in the immersive environment is where the body becomes more open and fragile, destabilising what we know as embodied knowledge to create new knowledge. The dancer can live or relive a moment through the avatar which frames the interactivity in an exciting way. Consequently, what emerges through these experiments reveals that the task of tracking dancers' emotions is not so straightforward. On the one hand, a dancer may be attempting to express an emotion as part of her performance. Her intention is to communicate a particular emotional state for the audience. But as with any performance, the dancer may be able to separate herself from the "expression" whilst her own emotional "felt-sense" in the dance may be rather different. Accordingly, the tracking of emotion may not be able to distinguish between the performance of emotion (as a form of theatricality and artificiality) and the actuality of the dancer's emotional world.
Another discovery thus far is that embodied experiences become shared experiences through the use of the WhoLoDancE tools. When thinking about the many Flamenco dance styles that were motion captured the dancer prepared materials that covered a full spectrum of emotions that can be divided into Cante jondo, 7 cante intermedio, 8 and cante chico. 9 Each variant conveys a mood that is specific to that category and within that category there are branches, called palos. Cante jondo is the oldest form of Flamenco and the intense and sad form of cante which deals with anguish, pain, suffering, death, and religious sentiment, was included and the martinete style was captured. Take for example when the dancer embodies the Flamenco style of dance that is considered sad. The dancer is dancing in close contact with the hologram so experiences the avatar's emotional state. Whilst not a real body, the dancer may not distinguish between the real and unreal when an emotion is effectively read and experienced. The avatar is a representation that is embodied by the dancer. The tool and its platform then serves as a stimulus that elicits heightened emotional states, whether or not the dancer and avatar are actually the same dancing body. The manner by which one performer's dance exists and is embodied through the virtual space by another, thus opens up a creative possibility that might not have been possible without the digital tool.

Conclusion
The WhoLoDancE project is investigating fundamental questions about what happens when dancing bodies become data through motion capture processes, and how that data can be a source for dancers to gain more insights to how they learn, teach and make dance. The processes are technologically demanding and can appear to remove the dancer from the body-to-body corporeal experience of dancing, that has always been a primary condition of what it means to dance. By bringing together several different dance practices, and establishing some common ground from which the project can develop, our work is revealing some fascinating insights to how the relationship between a dancer and a dancing avatar-whether or not of the same-can heighten the dancer's kinaesthetic awareness and support a tuning of perception, so the dancer develops a more sensorial engagement with the avatar projection as her dancing partner, which reflects back on her sense of her own body in motion.
Whilst the technologies are still in development, they have the potential to augment traditional practices for teaching, learning and creating dance. The aim is for WhoLoDancE to develop a number of tools that are modular and complementary, ranging from a blending engine that will allow users to create new movement sequences drawn from the library of motion capture dance sequences, to low-level motion tracking that will enable users to search for similar movement patterns in the repository, to the more sophisticated HoloLens avatar projection tool as we have discussed here. Underpinning the tools is a question about how the human aspects of motion capture enter the research. Emotions play a role in guiding not only how the dancer moves but in how she perceives her movement. Motion capture data is replete with the dancer's signature that is much more than the angles of the joints and the trajectories of movement created by the ocular camera system's reading of the body. Emotions are thus never far from the motion capture process and the data that emerges. What this offers is therefore the potential for dancers to expand their own understanding of how they move, why they move, and how that movement reflects back to them information about their own aesthetic proclivities, technical goals and creative impulses, thus enriching their own and others' experience of dance.