Virtual Reality and Choreographic Practice: The Potential for New Creative Methods

Virtual reality (VR) is becoming an increasingly intriguing space for dancers and choreographers. Choreographers may find new possibility emerging in using virtual reality to create movement and the WhoLoDancE: Whole-Body Interaction Learning for Dance Education project is developing tools to assist in this process. The interdisciplinary team which includes dancers, choreographers, educators, artists, coders, technologists and system architects have collaborated in engaging, discussing, analysing, testing and working with end-users to help with thinking about the issues that emerge in the creation of these tools. The paper sets out to explore the creative potential of VR in the context of WhoLoDancE and how this may offer new insights for the choreographer and dancer. We pay attention to the virtual environment, the virtual performance and the virtual dancer as some of the key components for equipping the choreographer to use in the creating process and to inform the dancing body. The cyclical process of live body to virtual, back to the dancing body as a choreographic device is an innovative way to approach practice. This approach may lead to new insights and innovations in choreographic methods that may extend beyond the project and ultimately take dance performance in a new direction.


Introduction
Virtual worlds have, since the early 1990s, been explored by technologists, artists, researchers and niche communities interested in interdisciplinary working environments. There have also been efforts to try and create a discourse and shared Cisneros et al: Virtual Reality and Choreographic Practice 2 vocabulary for these interdisciplinary projects. For example, VR expert Raph Koster, in the early 1990s stated that, ' a virtual world is a spatially based depiction of a persistent virtual environment, which can be experienced by numerous participants at once, who are represented within the space by avatars' (Koster 2004). Edward Castronova, expert in synthetic worlds, suggested that a 'virtual world' is a ' crafted place(s) inside computers that are designed to accommodate large numbers of people' (Castronova 2005), while Bell (2008) claims that the virtual world must consider the individual within that virtual world. The WhoLoDancE: Whole-Body Interaction Learning for Dance Education 1 project is exploring ways in which the virtual environment can augment and contribute to dance learning and expand choreographic techniques.
An intention of working with Virtual Reality (VR) technologies is to create new kinds of immersive experiences whereby the dancer can learn independently with a virtual teacher, by being able to dance with a dance expert or even the learner's own projected image. The dancer may find herself dancing closely with her own avatar, discovering more about the spatial or dynamic properties of her dance by receiving immediate 3-D feedback. Whilst VR is thus proving to be a valuable tool for dance learning, teaching and creating, questions arise about the accessibility and portability of the technology.
To integrate expensive and technically demanding equipment into artists' working spaces is not always possible. The challenge for the project, and hence the discussion here, is to offer the choreographers and dance students tools that are meaningful, accessible, useful, intuitive and provide them with tools that can advance the field of dance whilst genuinely supporting the needs of dance artists, teachers and learners. Albeit nearly two decades ago, Sparacino et al. (1999) wanted to augment the expressive range of possibilities for performers and stretch the grammar of the traditional arts rather than suggest ways and contexts to replace the embodied performer with a virtual one. Within WhoLoDancE, a similar approach is being taken whereby the tools being developed are not intended to replace the 1 See www.wholodance.eu. live experience or teacher but rather to enhance and expand current practices and to push the boundaries of the dance art form. The virtual environment and digital tools will thus provide an expanded range of ways to create dance material, which in turn may build on knowledge in the broad area of VR whilst influencing how dance is taught and made, and may gradually change methods of working and collaborating.
In short, what we are interested in is how these tools can be used to explore new ground and rethink current practices in the domain of dance.
VR has received a great deal of attention in the last two decades, with the most important information technology and communication (ITC) companies leading the technical evolution of head-mounted displays to offer experiences in Cross Reality (XR), which includes Virtual, Mixed or Augmented Reality. In 2014, Google started releasing the design of a cardboard-made VR visor, which could be used with a smartphone to enjoy a low-cost VR experience. In the same year, Facebook acquired the company Oculus VR. One year later, Microsoft presented the HoloLens device for Augmented/Mixed Reality. In 2016 the Mozilla VR team and Google Chrome team announced the version 1.0 of the WebVR Api, a set of tools to develop VR tools within web pages, and the following year Microsoft updated Windows 10 to support third-party Mixed-Reality devices within the Windows Mixed Reality platform. Apple introduced ARKit in iOS11 to natively support Augmented-Reality applications. Beside these ITC leaders, other companies produced their devices for XR, with the notable mention of the state-of -the-art VR visor, the HTC Vive. Despite the large interest from companies, a clear application to exploit the full potential of VR has not emerged yet. For this reason, researchers, developers and artists are still investigating and experimenting, assisted by the current high level of technological maturity of devices. VR headsets have been used for watching 360 degree images and videos (see Figure 1), which are videos or images shot with a spherical fashion to allow spectators to choose their viewing angle, as if they were in the centre of the scene. This application provides a low degree of interaction, since it only applies to watching pre-recorded videos or live streams, with some marginal possibility of hypermediality for receiving Cisneros et al: Virtual Reality and Choreographic Practice 4 information on what appears in the screen. An example is the Dutch Opera Ballet's VR ballet. 2 The company teamed up with Samhoud Media and produced a VR performance that employed 360 degree stereoscopic cameras and quadraphonic audio. A higher degree of flexibility is provided by the combination of motion capture techniques, animation and computer graphics, that allow the user to digitally acquire the kinematics behind motion and instantaneously render it through a number of scenes and characters. For this reason, in our project we focus on the latter kind of application.
VR devices can be roughly divided into two categories: desktop-based and mobile-based. The former typology requires a desktop computer to perform the computation of the graphics, and may need a setup for the room in order to track the headset position. The latter are stand-alone viewers usually supported by smartphone devices that are more portable and less expensive, but at the usual cost of lower computational power. Moro et al. (2017) conducted an investigation on the different degree of engagement and effectiveness in student learning when using desktop-based or mobile-based devices, and discussed that the latter are more 2 See https://www.operaballet.nl/sites/default/files/documents/opera/VR_nationaalballetV1.pdf. prone to produce a sense of cybersickness in the viewer. Nevertheless, given the rapid evolution of VR devices, it is likely that the next generation of devices will be able to combine the advantages of both categories. For this reason, in this paper we discuss the elements and questions raised by the applications of VR in choreography and teaching scenarios, regardless of the current VR devices, whilst mentioning the level of computational power or generic specifications of devices where appropriate.

Background and Context to WhoLoDancE Project
In the mid-1990s when dance motiontracking systems were relatively new, performers learned how to play the motiontracking instrument (Dixon 2007 From inception, WhoLoDancE set out to work with professional dancers interested in the intersections of dance and technology, dance learners, and technologists keen to deepen the discourse on mediated performances. While the development of technology is a key factor of the the project, thinking about how the technology functions is not the primary focus for this paper. More particularly, exploring how the technology enhances the learners' experiences, the choreographer's practice and offers new ways for choreographers to think about bodies in real space and also within a virtual space, is at the crux of this discussion, which is grounded in working directly across disciplinary boundaries. The paper also considers the implications for a choreographer when working and within what we are calling a virtual studio.
The collaborative relationships in the WhoLoDancE project has been one based on curiosity in each other's disciplines and enquiry into what each discipline can offer the other. Identifying terminology used by each of the dance genres that are the focus in the project, which include Greek folk, ballet, contemporary and flamenco, and finding a shared language among the dance partners, was a fundamental component of the first phase of the project. This agreed language was developed from Laban's Movement Principles and allowed the dance partners to conduct a questionnaire with professional dancers, teachers, practitioners and young dance learners who may or may not have an interest in dance and technology. The aim was to see if the project consortium's agreed definitions and terminology resonated with practitioners and choreographers from each of the identified genres.
The design of the tools has thus been developed through a dialogic process, taking account of requirements provided by users, partners from the dance community, and the technical possibilities and limitations highlighted by the scientific partners. The tools have since been tested in several demonstration events in several European countries as well as the UK, in order to collect feedback for improving usability. Whilst the tools are expected to be released to the dance community for pedagogical, choreographic and artistic purposes by January 2019, what has emerged thus far, and hence forms the basis for this discussion, are the questions raised by the use of VR in choreography.
As noted earlier, WhoLoDancE is working with four dance genres -flamenco, Greek folk dance, contemporary and ballet. Dance partners who are professional practitioners have served as representatives of specific instances of each genre and were captured using sophisticated motion capture equipment and a large repository of movement has been created. For some, dancing in the motion capture studio was a new experience, finding out about how it felt to wear tight suits with optical markers, performing in a studio surrounded by cameras (see Figures 2, 3 and 4). The cameras captured the markers from several angles. Software processed the information on the location of markers to compute their positions in the 3D space, from which it was possible to obtain the 3D positions and orientations of the limbs of the dancers, such as head, spine, limbs, arms and legs. Motion capture work tends to accurately capture larger sections of the body but it was important in WhoLoDancE to focus on, and document, the smaller gestural motions that are particular to the genres, for example the intricate hand/finger movement in flamenco dance, as in Figure 5.
The aforementioned round of motion-capture sessions generated a significant database and led to the development of the Movement Library, a web-service that users (choreographers, teachers, dancers, students, etc.) are able to navigate to find recordings of different movements across the four dance genres. The Movement    Library allows users to annotate each recording with qualities, properties, or actions, through an easy-to-use interface; such annotations are then used to retrieve items in the library. Other tools created to easily navigate the Movement Library are the search by similarity and by movement sketching. The former allows users to choose an excerpt of a recording (a query) and quickly retrieve all the recordings in the Library that are similar to that query according to a user-defined criterion, for instance, similar path through the stage, or similar acceleration of one limb. The latter allows the user to record the query movement, hence sketching a movement, and retrieve the most similar ones in the Library, according to different criteria. While the latter requires users to have some kind of motion-capture device of their own, the former requires users to start with a movement before retrieving the others.
The Movement Library and the Sketching tools may be useful for providing dance students or choreographers with examples of movements. The set of movements also lend themselves to research scenarios as they can enable researchers to investigate similarities among steps across different genres. Through the Blending Engine, a creator tool for choreography, with strategies to combine different movements together both in time and in the virtual space, also allows the user to mix and match and build an avatar that can perform several combinations of 'steps' and movement phrases. With regards to the time dimension, the engine blends two movements in order to make the transition between them as smooth as possible. In terms of the space dimension, two or more recordings can be combined in the same moment starting from different limbs to create new steps with, for example, the leg movements of a Greek folk dance and the torso, arms, and head movement from flamenco. Using the Blending Engine, a choreographer may use search by similarity and by movement sketching to retrieve recordings from the Movement Library, and combine them to make a whole new performance. This ability to construct and deconstruct a sequence of steps and movement phrases affords the user an opportunity to choreograph and to work creatively with a body that exists within a virtual environment.
Alongside these tools that are immediately usable for the dance community, other issues have been investigated in WhoLoDancE that are speculative yet pertinent to this discussion. Through formal conversations 4 between technologists and artists, we aimed at identifying those qualities that are more effective to describe a dance performance and then develop techniques to automatically extract those qualities from a dance recording. These strategies are either manually designed by a technician (rule-based) or developed using machine learning algorithms, which needs to process annotations collected in the Movement Library (data-driven).
Most of the tools in the WhoLoDancE project have been developed for traditional devices, such as computers or mobile phones, on flat screen, but with the intention to bring them eventually to an immersive environment. To turn now to our work specifically with VR technologies, we designed a framework for immersive visualisations of dance performances using low-end head mounted devices, and we developed a web-based tool as a prototype for visualising recordings from the Movement Library or performances created with the Blending Engine. During one of the recording sessions, we experimented with the use of Microsoft HoloLens to watch a recording session as a projection in the physical space (Mixed Reality). Doing this, artists had perspectives of their own body that they may not have experienced before, re-watching their own recorded movements through a different virtual body (an avatar), from every angle.
The artists discovered that the visualisations encouraged them to think differently about the space, the bodies (the avatar and the live body) in space and their creative practice in that space. Seeing different types of avatars in the virtual environment has the potential to prime the user and ask them to think differently about how they moved. When the dancer moved with one of the avatars used within the project, its ability to swell and shrink in response to the dancer was intriguing 4 Ethics for WhoLoDancE Project: It is important to note that the research team worked directly with subjects and collected data from key stakeholders throughout the life of the project. As mentioned above, the dance experts, researchers, practitioners, choreographers and educators were regularly consulted. Prior to WhoLoDancE beginning ethical clearance was gained for the project and any individual interviewed was asked to read and sign a project information sheet and consent form.
Data collection and analysis then took place and the research team relied on those results to direct the project. and allowed for a playful quality and creative exploration of the live and virtual environment (see Figures 6 and 7). In summary, the general 5 feeling was that the visualisations are an integral part of the creative process. These experiences have 5 WhoLoDancE Film-July Edit: https://vimeo.com/179879432.  been particularly important for opening up debate in the project about the body in VR environments and performance (see Figure 8).

Thinking of a VR Performance
VR has the ability to shape choreographic formations and patterns that have not been fully explored or even considered before, broadening the possibilities  using VR devices. In this sense, VR is not only a creative tool, but also an artistic medium.
When considering how the VR technology can assist choreographers in their creative process, from the inspiration of the idea, to the production and the final refinements, there are four key elements to be modelled: the virtual performer (or virtual dancer), the virtual environment, the virtual performance and the spectator. The virtual dancer emulates the real dancer (in fact, it can be a 3D recording of a real performer) and it is composed of a body, the graphical representation of the dancer, namely the avatar. The virtual dancer acts in a virtual world, which may be a blank 3D space, a 3D model of a real space or any other virtual environment.
Using a virtual environment, it is even possible to overcome the limitations of the physical space, which is usually static (it cannot change) or slowly dynamic (it needs a certain amount of time to evolve). An example of the dynamicity introduced by the virtual environment is the possibility of changing the perception of the size of a room from a few square meters to a vast space within milliseconds or to change some properties, such as the shape and the colours according to the performance.
This dynamicity is one of the possibilities provided by the digital technology, and it is a part of the virtual performances. Performances that take place in physical locations

Virtual dancer
Steve Dixon (2007) suggests 'when the body is "transformed," … into digital environments, it is not an actual transformation of the body, but of the pixilated composition of its recorded or computer generated image ' (2007, 212). As a result of the motion capture process, which produces a transformation of the body as Dixon suggests, we acquire a set of digitalized dancers, which 'perform' movements containing information about the different limb position and orientation at each instant, or frame. This is organized in a hierarchical structure that replicates the kinematics of body, as in the body skeleton. This information is then linked to a graphical representation, which is commonly defined as avatar or virtual dancer.
The easiest implementation of an avatar involves reproducing the skeleton structure of the motion-capture recording, where each joint is usually represented by a sphere and a line connects joints to create limbs. This visualisation, similar to a stick man, has been used in the first versions of the WhoLoDancE tools for their simplicity.
In some scenarios we have increased the size of the head joint to make it more realistic, as shown in Figure 10. Consortium dance partners in the project found the stick-man avatar useful to reduce the dance to its primitive movements, deprived of the properties of the physical body or the facial expression. An evolution of the stickman avatar in the project is the arrow-man, with arrows indicating the orientation of the main limbs, with the aim of highlighting the directionality movement principle, as shown in Figure 11. In later versions of our tools we employed two humanoid avatars called snowman and android, the former developed within the project and the latter from a repository of avatars that technologists within the project built and can access. The two avatars are shown in Figures 12 and 13, respectively. Beyond   particular things that extend the body/self to serve as a vehicle for expanded sensory means, to also enable a kind of informative sensory loop, and to creatively express one's self in that reality (e.g., through text, drawings, music, movement)' (2017,268).
We argue that in combining the virtual and the live body, the result is a virtual studio that offers the user multisensory feedback that is directly born from the moving body whilst experiencing the virtual body.
Virtual environments are composed of 3D objects and lights, which affect objects, shadows, colours and textures. Real-time rendering of all details is computationally demanding, requiring once again choreographers to adapt their ideas to the available resources. Nevertheless, as for the virtual dancer's considerations, choreographers may experiment to find a feasible balance between creation and available resources. It is worth mentioning the potential of using rendering styles for virtual environments as artistic styles. Masuch and Röber (2004) present a brief history of videogames rendering styles and underline that photorealism is not 'the only style desirable'. They analyse and discuss four styles: pen-and-ink drawing, oil painting, pencil/coloured crayon drawing, and cel-shading. The former three aim to mimic a certain drawing style, from comics, to impressionism, to books for children.
Cel-shading rendering, instead, produces 3D objects that are similar to cartoons by outlining edges, using flat colours for surfaces and applying flat-coloured shadows (Decaudin 1996). Cel-shading rendering provides a realistic, yet familiar, style that is especially helpful due to its low computational requirements with respect to photorealistic rendering. Using different rendering styles such as these may allow choreographers to experiment with how their performances visually look, providing yet more options as part of the creative dance making process.
However, in the current implementation and tools of WhoLoDancE, we are yet to investigate the use of different virtual environments or rendering styles for artistic purposes even though dance practitioners have suggested that this would interest them and be useful in thinking about their choreographic process. Thus far, we use both empty rooms (with dark or bright sky) and virtual Laban cubes (Laban 1966) to offer a reference in the navigation. The web-based VR tool also includes a virtual rehearsal space with wooden floor for a more realistic scenario.

Virtual performance
When considering what comprises a virtual performance, Brockhoeft et al. (2016) look closely at the history of interactive sets in dance and suggest that as 'new technology is developed, dancers have explored how to utilize it to enhance their artistic expression and movement invention ' (2016, 398). Digital technologies may improve the process, drawing direct links between measurable causes and controllable effects. A recent example is Stocos' piece Piano and Dance (Palacio and Bisig 2017). In Piano and Dance, a dancer's bodily movements, measured by means of a set of inertial measurement units (IMU) devices worn by the dancer, are used to control an electronic music composition executed with a mechanical acoustic piano. With the use of VR tools, the interactive paradigms between the performer and the environment can be enhanced. In order to do that, we need to identify novel measurable causes and novel controllable effects and define how to directly link them.
One of the tasks of the WhoLoDancE project is the modelling and direct computation of movement qualities, which will provide choreographers several causes to enhance their VR performances. Controllable effects may include properties regarding the avatar or the virtual environment. We mentioned that a virtual environment can be abstract, or related to a physical place, or an external location. A dynamic virtual environment has an infinite amount of properties that can change, including location, colours, lights, shapes and properties of the objects.
For example, in Choreomorphy (El Raheb et al. 2018) the authors exploit a simple interaction between a virtual avatar, controlled by motion-captured users, and a composition of 3D shapes, where a collision system is implemented and therefore the objects move when touched by the avatar. This is a simple example of interaction and enhancement of a performance within the virtual environment.

The Spectator
We have discussed the design of the virtual space and of the virtual dancers who are going to use this space. In this section we provide an overview of the choices related to how the audience should enjoy, or be part of, the virtual space.
Commonly, dance performances are thought to have an audience watching or participating in them as collective experiences. VR devices are designed to be used by a single user, offering a first-person experience. Hence, in the virtual scenario, the performance could be thought through from an individualistic perspective. In this basic scenario, the user wears a VR headset and attends to the performance unaware of any other people in the audience. This may have the effect of increasing the spectator's individual engagement with the artistic piece, while maintaining the 'fourth wall' between the spectator and the performance. One solution for breaking the wall is to assign a virtual entity to each spectator and make therefore the audience part of the virtual environment. We refer to this entity as the virtual spectator, i.e., an avatar that represents the spectator in whichever form is meaningful in the performance. The use of virtual audience may reduce the intimacy, since each spectator would be aware of the rest of the audience, but conversely increases the engagement of audience -as a collective entity-with the performance.
When a user wears a VR device, he/she is immersed in a virtual, alternative reality yet still living and existing within a real architectural space. As basic features, The audience may also have an active/interactive role in the performance.
We described how choreographers can link dance properties to avatars or the environment to enhance the performance, and that audiences may be part of the

VR and the Choreographic Body
The VR tool in WhoLoDancE allows the choreographer time to experiment with new material either from a pre-recorded source or to experiment in real-time using motion capture. Some of the tools in the project have been used by a small group of dancers and choreographers so far who have commented on the unexpected richness and fullness of the information they received when experimenting with the technology.
When using the tool, users described being immersed in the environment and becoming aware of the peripheral objects. Sherman and Craig separate mental and physical immersion and describe the latter as 'bodily entering into a medium ' (2003, 9). However, our work thus far points to how entering a VR environment is both mental and physical immersion. One can mentally engage with the surroundings but also physically respond to positioning and peripheral vision. Moreover, this is similar to the term ' dilated reality' coined by Isabelle Choiniere (2017) whereby our sensory perception is recalibrated by being immersed in a VR environment. It opens up space around us and within us.
There is a paradox that surfaces when VR and dance are combined as there remain inherent tensions between the physical and the virtual. Virtual worlds are indeed unreal as they are artificial, fictitious, imaginary, intangible, and sometimes invented. Yet virtual environments are real, as well, as they include the human, live body navigating through the space. The dancer has the ability to move through spatial representations and the physical, embodied reality is part of a faux representation of an environment. VR allows for the unreal and real to be blended together, which fosters a social interaction between the technology and dancing body (see Figure 16). The two can meet in this virtual space and the option of dancing with the avatar allows for the bodies to be suspended in time and space and to co-exist with the real world.

Conclusion
Dance is an art form that will always be dependent on body to body transmission.
However, there are now many examples of how the transmission of dance knowledge is mediated through technology (see, for example, Bleeker 2017). It is plausible that VR offers a different realm of possibility to create choreography and inform the body. Susan Broadhurst (2006) suggests that at the threshold of the physical and the virtual, tension exists and she proposes a 'liminal space' where there is 'potential for a reconfiguration of creativity and experimentation' (Broadhurst and Machon 2006, xix (Marin, Blanco and Marin 2017). This is beginning to allow more dancers and choreographers to experiment with these technologies.
Mindful of the need to find ways for the WhoLoDancE tools to be more accessible, our aim is to extend the tools to VR and MR devices. Two of the tools that have more potential in VR are the Blending Engine and the Movement Library. The Blending Engine would allow choreographers to create and rehearse with an avatar in a virtual space, or in their studio using MR, similarly to composers working at their piano. The Movement Library is already a useful resource to access a large database of movements, but with VR it will become an important support for engaging with digital records of dance forms such as Greek folk or flamenco that are otherwise hard to find.
Our work in WhoLoDancE aims to show how dance can benefit from the spread of VR and MR devices. However, there is much work to do before a 3D immersive environment is commonplace for dance learners, teachers, performers and choreographers. But our work in the project is showing us that the dialogues that take place between technologists, dance artists, educators and scholars are very rich and lead to the development of tools that provide new insights to how we choreograph, perform and encounter dancer.

Funding Information
The work was supported by the Research and Innovation Action funded under the EU Horizon 2020 programme.

Competing Interests
The authors have no competing interests to declare.

Author Informations
Karen