Metaverse film. The rising of immersive audiovisual through disrupting traditional moving image and its conventions

The development of the metaverse and its immersive technologies has had an impact on contemporary filmmaking. Associated with the concept of metaverse are the terms “VR film”, “VR cinema”, “cinematic virtual reality”, and “metaverse film”. As new expressions of digital media, virtual reality (VR) and augmented reality (AR) have become hotbeds of groundbreaking experiments with immersive audiovisual languages that have led to the production of numerous narrative works experiences that combine the real and the virtual using immersive devices. The term ‘metaverse film” could encompass all cinematic practices related to the metaverse, including videos shot in virtual worlds such as VRChat. In a sense, machinima, or films made in the last 15 years using game engines, could therefore be said to be the origin of metaverse films. Similarly, immersive VR audiovisual storytelling is part of the cinematic practices in this emerging metamedium that immerses the viewer in parallel virtual universes and makes them part of the story. Metaverse has a disrupting potential that may add new insights to the way we think of film and storytelling. Following these ideas, we consider relevant not only for the academic field, but also for the industry to frame the “metaverse film”. Using a bibliographic and different catalogs and databases review as well as major XR sections of major film festivals as main methodology, we conducted a research that goes through the immersive audiovisual fiction produced during the last decade, with the aim to set its current state of the art that might help conceptualize the term and all that surrounds fiction and non-fiction experiences of the metamedium.


Introduction
The emergence of the metaverse in the current context has believers and skeptics in equal measure.It is very difficult to foresee how it will finally be configured and implemented in our daily lives, but this question could be considered more as an advantage than a defect, because in this lack of definition lies the true disruptive scope of the metamedium.The term metaverse permeates in its definition different areas of knowledge such as computer science, technological development, audiovisual content production, artistic practice, education, business and finance.Although, it is in the field of entertainment where it is making steady progress, Mystakidis suggests a big impact of the metaverse on education and defines it as: "a post-reality universe, a perpetual and persistent multiuser environment merging physical reality with digital virtuality " (2022: 486).Another definition of the metaverse is the one proposed by Mathew Ball: In the realm of moving images, the metaverse could be considered "a distributing channel for narratives characterized by their degree of customization and the level of interaction with other users" (Acevedo Nieto, 2022: 48).Despite sounding like a modern neologism, Neal Stephenson first used the term "metaverse" in his 1992 novel Snow Crash (1992), where "some of the action takes place in the real world, but much of it takes place in a mass-visited communal virtual world called the Metaverse" (Taylor, 1997: 177).The Naked Sun (1956) by Isaac Asimov, Neuromancer (1984) by William Gibson, and Ready Player One (2011) by Ernest Cline are other works in the metaverse literary canon.Video games are a medium that more naturally fits the idea of the metaverse.The idea of virtual worlds can be found in text-based MUDs (Multi-User Dungeons) from the 1970s, which were followed by MUSHes (Multi-User Shared Hallucinations) and also inspired video games like Habitat (LucasFilm, 1986), OnLive!Traveler (Activeworlds, 1998), and Second Life (Linden Lab, 2003), followed by Roblox (Roblox Corporation, 2006) and Minecraft (Mojang Studios, 2011).Finally, the connection between films and the metaverse can be found already in The Wachowskis' version of Gibson's Neuromancer's cyberspace in The Matrix (1999)(2000)(2001)(2002)(2003) (Verhoeven, 1990), based on the novella We Can Remember It for You Wholesale (Dick, 1966), and The Lawnmower Man (Leonard, 1992), based on Stephen King's short story.
The connections between metaverse, cinema and video games are linked and starts far before real-time rendering engines such as Unreal or Unity and their application to audiovisual production in films such as The Lion King (2019), for which Favreau began to build the first film sets with StageCraft virtual sets, later used in the production and filming of series such as Mandalorian (Favreau, 2021) (Martínez-Cano, 2021).Some new terms for cinematographic practices in the metamedium arise as "VR film" (Chang, 2016), "VR cinema" (Zarka & Shah, 2016), "cinematic virtual reality" (Mateer, 2017), and "metaverse film" (Zhu, 2022, cited in Linuo, 2022, p. 2).All these new concepts are based on experimenting with VR and AR for the creation of immersive audiovisual experiences that challenge the traditional linear narratives through the hybridization of 360-degree video, 3D CGI, and volumetric capture (Martínez-Cano & Roselló-Tormo, 2021).
Although haptic devices, VR, and AR headsets can be used in conjunction to improve user experience, this is not a need for the metaverse (Smart et al., 2007;Dwivedi et al., 2022).When these factors are combined, new experiences may be created in which the user actively participates in the virtual environment rather than simply seeing it from a distance.In order to develop the new audiovisual languages of the metaverse, filmmakers and video game artists are encouraged to experiment with various mediums.This is why a strong understanding of how new technology and the development of audiovisual image are changing traditional filmmaking approaches as a result of new technologies and the evolution of audiovisual imaging itself, as well as the discursive and expressive strategies used to engage the viewer with the story.
The creation of the metaverse, a networked virtual reality world, has enabled new advancements in audiovisual storytelling.This novel format immerses spectators in a completely participatory and immersive story while disrupting conventional moving image assumptions.The metaverse film creates new opportunities for narrative, audience participation, and immersive audiovisual experiences by leveraging the capabilities of virtual reality, augmented reality, and artificial intelligence.

Brief approach to the evolution of moving image conventions
Since the turn of the 20th century, moving images have captured viewers' attention, progressing from silent films shown on screens to the immersive audiovisual experiences we enjoy today.The norms of moving picture narrative have undergone constant change as a result of technological advancements, societal changes, artistic creativity and experimentation.The majority of movies were silent when cinema first began.These early movies only used visual storytelling techniques, relying on the ability of visuals to tell stories and generate feelings.Directors like Georges Méliès explored the possibilities of the medium by experimenting with analogical techniques and procedures that lead to spectacular effects and fanciful stories.However, the expressive potential of early moving pictures was somehow constrained by the lack of synced sound, an important turning point in the development of moving image standards introduced in the late 1920s.The ability to use speech, music, and sound effects to enhance storytelling was made possible by the merging of sound and pictures, revolutionizing the cinematic experience.Filmmakers were forced to think about how sound and image interacted, which gave rise to innovative methods including the expressive use of music, dialogue-driven stories, and synced sound effects.The switch from black and white to color movies marked the next significant advancement in moving picture traditions.The advancement of color technology broadened the visual options available to filmmakers, enabling more vivid and lifelike depictions of the outside world.Through the thoughtful selection and use of colors, color evolved into a potent tool for narrative, allowing filmmakers to provoke feelings, create atmosphere, and express meaning.
Following these innovations, widescreen formats like Cinemascope and IMAX were introduced, which further altered moving-image norms as technology developed.Widescreen formats provided greater aspect ratios, immersing audiences in expansive and visually stunning cinematic experiences.Directors may use the bigger picture to make aesthetically powerful compositions, which would increase the spectacle and impact of the story.Another paradigm in moving picture conventions was brought about by the digital revolution of the late 20th century.Filmmakers now have more control, flexibility, and accessibility thanks to digital technology.By enabling picture alteration, the incorporation of computer-generated visual effects, and the investigation of non-linear tales, it opened up new horizons for experimentation.
The development of digital cinema democratized the filmmaking industry and made it simpler for aspiring filmmakers to produce and share their work.The possibilities for moving image conventions have been significantly increased by the recent development of virtual reality (VR), augmented reality (AR), and artificial intelligence (AI).The lines between the audience and the narrative are now more ambiguous thanks to these technologies' unprecedented levels of immersion and involvement.Through VR experiences, viewers may enter the world of cinema and experience tales firsthand.By superimposing digital features over the actual world, AR turns mundane situations into fascinating worlds.As a result of user interactions, AI-driven technologies provide personalized and dynamic storytelling experiences, and also AI applications could be involved in the metaverse film industry production pipeline, for instance allowing new storytelling strategies, as dialogue interaction in immersive experiences with NPCs.The future of moving image standards presents tremendous chances for even more immersive and transformational narrative experiences as we embrace the potential of current emerging technologies and the expansion of the metaverse.

Defining the Metaverse Film
We use the term 'metaverse film" to encompass all cinematic practices related to the metaverse, including videos shot in virtual worlds such as VRChat.In a sense, machinima, or films made in the last 15 years using game engines, could therefore be said to be the origin of metaverse films.Similarly, immersive VR audiovisual storytelling is part of the cinematic practices in this emerging metamedium that immerses the viewer in parallel virtual universes and makes them part of the story, within the set itself.The term "metaverse film" could therefore be used for immersive audiovisual productions of this kind, as they create new virtual universes and participate in the construction of the metaverse.The use of VR and AR devices and headsets are a paramount element for the metaverse film, as it is the way to put the viewer inside the mise en scène or to overlay the virtual elements on the real-world fiction setting.VR films and AR films are therefor considered as two types of metaverse film practice, that constitutes different possibilities based on the different features of the technology.In this study, we will focus only on VR films, with a major interest in those works that hybridize several types of media in both: fiction and non-fiction narratives.
The cinematographic practice in the metaverse is obviously shaped by its major features: interconnectedness, immersion, user agency, persistence and scalability.Not only will the production of film experiences be transformed but also the way they are consumed by the audience and how they interact with the viewers, that go from passive to active participants.It will also have an impact on the film industry and will bring new socio-cultural and ethical considerations, due to the key elements of metaverse film, from which immersive experiences and presence designed together with the interaction of the audience and the construction of interactive narrative structures play a major role.

Main Characteristics of MetaVerse Film: Presence and Embodiment
Immersivity plays a significant role in creating visual environments within the metaverse.Immersive experiences are also further influenced by the factors such as high-quality graphics, reality sound design, haptic feedback, and interactive storytelling.In a framework for immersive virtual environments "Presence is defined as the subjective experience of being in one place or environment, even when one is physically situated in another" (Witmer & Singer, 1998: 225).Slater and Wilbur propose the sense of presence as "a state of consciousness, the (psychological) sense of being in the virtual environment" (1997: 605), which identifies three essential components: "place illusion, plausibility illusion, and body ownership illusion".Immersion as an outcome of presence connects with visual realism and suspension of disbelief of the audience.
Another key factor for the metaverse film experience is embodiment, defined as the perceptual and psychological phenomena in VR experiences where users perceive a feeling of presence and ownership over a virtual body, increasing their sense of immersion and involvement in the virtual world.To generate a realistic illusion of the user's body being there and active in the virtual realm, it includes the integration of sensory inputs, motor control, and cognitive processes (Sanchez-Vives & Slater, 2005;Riva et al., 2019).As Carpio et al. define: This phenomenon can be defined as our biological bodies' natural embodiment in relation to the sensations of existing within a virtual world (sense of self-location), being virtually incarnated there (sense of body ownership) an shaving the ability to control that body (sense of agency) within the virtual space (Jeannerod, 2003;Kilteni, Groten and Slater, 2012) (2023: 3).
Visual perception, auditory cues, haptic input, and motor control are all examples of multimodal integration.Together, these sensory inputs produce a compelling illusion of the user's body within the virtual world that is both cohesive and realistic.The establishment of a sensation of presence, where users believe they are physically there in the virtual environment, depends mostly on the integration of sensory data.Users are given a feeling of agency through embodiment, enabling them to engage with and manage their virtual bodies which supports the creation of the viewers' identity in the immersive storytelling (Brillhart, cited in Bucher, 2017).Users may control things, move about the area, and interact virtually through gestures, motions, and activities.The sensation of agency contributes even more to the embodiment experience by heightening the immersion and realism.
Furthermore, emotional involvement and cognitive functions have been connected to embodiment in VR encounters.It has been demonstrated to have an impact on social interactions, empathy, and emotional reactions in virtual settings.Users may emotionally relate to virtual characters, stories, and settings through the embodied experience, which makes for a more powerful and lasting encounter.Overall, the perceptual, cognitive, and emotional components of feeling present and attached to a virtual body are all included in the concept of embodiment in VR experiences.The sensation of immersion, agency, and engagement are increased, making VR an effective tool for audiovisual storytelling.

Main Characteristics of MetaVerse Film: Interactive Narrative Structures
From the combination of video games and movies comes out the interactive layer that is being added to the film consumption.In the concept of embodied virtual reality (EVR) films (Carpio et al., 2023: 1) understood as the next generation of cinema experiences we set the definition of the metaverse film, where the "interaction with virtual items and people also aids in the user's embodiment" (Carpio et al., 2023: 2).Interactivity also guides to a deeper sense of embodiment (Tricart, 2018) and could be determined by different strategies along the storytelling process.For instance, it could be developed as a branched structure that works through a voice interaction system, where the user chooses one answer out of the options displayed for every question of the NPC character, as in Terminal 3 (Malik, 2018).
The storytelling through interactive participation of the viewer could take several forms and strategies.Non-linear approaches offer a wide range of multiple options that evolves the story as the user is getting a sense of control and agency, feeling that their choices and actions have an impact on the narrative.Another possible way of creating an interactive immersive experience is through collaboration.In this model multiple users are experiencing and influencing the storytelling together in a sense of shared experience.Sometimes the interactive layer of the narrative is not altering the plot, just being a part of the experience that only seeks to add an emotional connection with the storyline as in the case of Queerskins: A Love Story (Szilak & Tsiboulski, 2019), where the viewer can interact with objects belonging to the main character, or in the case of Goliath: Playing with Reality (Murphy & Abdalla, 2021), where at some points users have to play a video game they are inside of and its part of what the authors want to tell the audience.We could define these types of linear interaction inside the storytelling as diegetic.Other types of interactive construction may seem even more simple, as it is the spatial ways of interacting to explore a space, but this way of giving agency to the viewers also might help to guide them through the storytelling.We could define them as spatial-diegetic ways of creating interactivity, but there are also spatial-non-diegetic ways which would only have the aim to give the chance to move the space establishing no connection to the storytelling, as the way we roam the space in The Key (Tricart, 2019), where we also find diegetic interactive participation as in the game with the jewels.
For storytellers and filmmakers, the use of interactive narrative in the creation of metaverse films through VR immersive technologies brings new potential and challenges.When viewers have choice and can deviate from a predetermined course, it is considerably more difficult to create an interesting and coherent story.To ensure coherence and emotional connection, authors must carefully construct the interactive aspects and balance them with the overarching story.Additionally, interactive storytelling in VR films rates opportunities for investigating various viewpoints.Viewers can develop a stronger sense of empathy and comprehension of the narrative by taking on the roles of characters in the fiction and experiencing the story from their point of view, strategy defined as perspective-taking experience, that come from psychology perspective-taking tasks (Piaget & Inhelder, 1956;2013).The potential for participatory narrative in VR films will only increase as VR technology develops, opening up fascinating new opportunities for both creators and viewers.

Methodology
Based on a bibliographic and different catalogs and databases review as _docubase MIT Open Documentary Lab (https://docubase.mit.edu),Meta VR for Good (https://about.meta.com/community/vr-for-good), VR Documentary Encounters (https:// vrdocumentaryencounters.co.uk),ArteTV (https:// arte.tv/digitalproductions/en/categories/vr-ar-360_en/) and XR sections in major film festivals as Venice VR Expanded and Venice Immersive -Cinema within La Biennale di Venezia, Sundance Film Festival New Frontier and Cannes XR among others, our aim is to build a chronological review on immersive cinema productions whose contributions from the field of experimentation and the hybridization of technologies and techniques, within the fields of fiction and non-fiction, have contributed to the development of the audiovisual language of metaverse film practice.
Due to the large number of productions from 2015 to 2022, the period on which we focus this study, we have chosen those immersive cinematic works developed for VR devices, and which have had a prominent impact on international circuits, setting a milestone in the evolution of immersive storytelling.This compilation constitutes a first glimpse into the topic, that continues to be developed, in order to create an extensive review of these audiovisual productions since their inception in the middle of the second decade of the 21st century.

Chronological Review of Metaverse film Productions 2015-2022
Metaverse films have the ability to produce memorable and meaningful experiences because of its immersive and sympathetic qualities that run along with its main features.One of the very first VR short films was Henry (Oculus Studio, 2015), a 3D animation about a hedgehog that served as an experiment with VR filmmaking, working on emotions with this first virtual character with whom the audience experiences this first immersive narrative.Another of these first productions of immersive VR film was Allumette (Chung, 2016), framed as the first VR film masterpiece by Wired.It is a 3D animation with a simple but touching strategy for the point of view as a puppet theater in front of the viewer.Another major VR experience from 2016 is Notes on Blindness: Into Darkness (Colinart, La Burthe, Middleton & Spinney), an immersive animation based on the short film Notes on Blindness (Middleton & Spinney, 2014), an interactive non-fiction experience where the audience gets access to the audio cassettes that John Hull recorded while experimenting how his vision was deteriorating until becoming blind.The imaginary is built up with 3D animated figures of light, combined with binaural audio to place the viewer in the different parts that go through John's memories.
A year after, Alejandro González Iñárritu tried out immersive media in his film Flesh and Sand (Carne y Arena, 2017), which put spectators in the shoes of migrants seeking to cross the Sonora Desert in pursuit of the so-called American Dream.This pioneering immersive storytelling combines VR content with the art related strategy of the installation, as the audiences get barefoot in a room full of sand that recreates the desert feeling in their feet roaming the physical space while they are inside the VR experience.2017 was also the year of production for the longest VR films to date, The Deserted (2017) by Ming-liang Tsai.A metaverse film that tackles the subject of dying.Working on the same topic, premiered at Sundance 2017 in a 2D animation style we found Dear Angelica (Unseld, 2017), a heartfelt tale of love from a daughter and memories, it makes the most of VR's immersive capabilities by leveraging the entire environment to create ethereal color paints that float in the virtual space while depict a fiction story.
The next year Laurie Anderson and Hsin-Chien Huang finished the VR trilogy in their collaboration with To the Moon (Anderson & Huang, 2018), "commissioned by the Louisiana Museum of Modern Art in Denmark, its US premiere at the Museum of Natural History marked the fiftieth anniversary of the Apollo 1 moon landing" (Miller, 2019).The lunar body, which is connected to cycles of rebirth, fertility, and death, presents a location where we could experience our own phase-shift in To the Moon immersive atmosphere, that also connects other film productions in the history of cinematography, from Le Voyage Dans la Lune (Méliès, 1902) to Apollo 13 (Howard, 1995), following the human fascination to reach the moon, an idea that it's been serving as a benchmark for human technological development and registered many times in film history.Also, in 2018 was created and released Home After War (Parameswaran, 2018), one of the first VR works that used Volumetric techniques with a very high-quality space recreation in a non-fiction project.A Nowhere Media production with the motto: "Returning to fear in Fallujah" that is included in Oculus VR for Good catalog, in which volumetric video recordings of victims' testimonies from the aftermath of the Iraq war are milestones of the documentary storyline.
The VR audiovisual fiction series Queerskins (2019present), created by writers and filmmakers such Illya Szilak and Cyril Tsiboulski, is an example of the implementation of interactivity in the plot, not as part of a branched narrative structure, but as medium for the construction of more emotional connections of the story with the audience.The same year was premiered Travelling while Black (Ross Williams, 2019).Shooted in 360-video, it shows new ways of mixing the timelines of the story through innovative editing techniques and strategies, where different time layers are permeated in the same immersive space-time.Although it has no human embodiment nor interactivity layer, this could be considered one of the masterpieces of metaverse documentary, and it combines stereoscopic 360 live action with non-diegetic graphical annotations, archive footage and photos.Gaze manipulation is one of its contributions to the immersive audiovisual language formulation, while the actor is within the scene as a guide, content is placed in the rear 180 degrees of view, and the point of view is omniscient.Also in 2019 it was released Wolves in the Walls (Fable Studio), winner of the 2019 Primetime Emmy for Outstanding Innovation in Interactive Media and one of the first VR film adaptation of a book (Wolves in the Walls, Gaiman and McKean, 2003), a 3D VR animation in which the interactivity layer evolves and keeps exploring narrative strategies, in this case allowing the audience to participate in the storytelling and help the main character, that also breaks the 4th wall in some parts of the storyline.One of the major innovations of Wolves in the Walls is the use and experimentation of artificial intelligence (AI) to create what they call "virtual beings", in this case for the main character and the creation of natural conversations with the user.
Another illustration of this interactivity evolution in metaverse films is The Hangman at Home (Michelle Kranot & Uri Kranot, 2020), which transforms the viewer into an actor who interacts with others in the scenes and earned the Grand Jury Prize for Best VR Immersive Work at the 77th Venice Film Festival in 2020 -Venice VR Expanded.In 2020 another non-fiction VR Film was created by Rose Troche, titled We Live Here (Troche, 2020), this 360-video documentary combines within its storyline some scenes with 3D animation and others with volumetrics and layers of interactivity that try to place the viewer in the situation depicted.Part of Oculus VR for Good catalog, it could be considered one of the hybrid productions that mix different media and techniques to create a whole immersive experience.The artist Matt Huynh brought their recollections to life in stark, evocative pen-and-ink drawings, which were then assembled into three-dimensional spaces" (Wolson, 2021).
In 2022 was premiered On the Morning You Wake (to the End of the World) (Colinart, Zandrowicz, Heolimeleikalani Osorio, Jamison & Brett), part of the Sundance 2022 official selection, a metaverse film by producers of Note on Blindness (2016), it shows how technically the evolution of Immersive VR film mediums, hybridizing real actors through volumetrics placed in 3D CGI environments, 360º video, animation, and interactivity.The examination of memory is one of its topics, represented as dispersed fragments around the setting that the spectator must piece together through a non-linear narrative form enhancing the audience's sense of agency and involvement.The seamless blending of live-action through the use of volumetrics and computer-generated imagery creates an aesthetically attractive atmosphere.The placement of focal points and the point of view guiding the viewer perspective builds a compelling storytelling experience.

Funding
This study is part of the research project: The role of virtual reality audiovisual narratives in social inclusion and the perspective of prosocial models: Analysis of their characteristics, effects and impact on young university students.
More recently, Montreal's 2022 Phi Centre exhibition Horizons VR featured Benjamin Steiger Levine's Marco & Polo (2021), as well as other VR movies like Kusunda (Felix Gaedtke & Gayatri Parameswaran, 2021), Goliath: Playing with Reality (Barry Gene Murphy & May Abdalla, 2021) which includes in the diegesis of the work some interactive moments in a video game style, and Re-Educated (Sam Wolson, 2021), selected in Venice VR Expanded 2021, "brings viewers inside a Xinjiang prison camp, reconstructed from the memories of three former detainees.[...] Supported and funded by a grant from the Spanish Ministry of Universities under the State Program for the Promotion of Talent and its Employability in R+D+I, State Mobility Subprogram, of the State Plan for Scientific and Technical Research and Innovation 2017-2020.It also forms part of the project: Immersive prosocial audiovisual narratives: Measuring their impact on society and analyzing their formal and technological characteristics.Regional Ministry for Innovation, Universities, Science and the Digital Society under the Program for the Promotion of Scientific Research, Technological Development and Innovation in the Valencian Community -AICO 2022.
, Spielberg's 2018 adaptation of Ready Player One, and Bigelow's 1995 film Strange Days, all refer to parallel virtual universes.Other examples are Total Recall