Robin Neuhaus
Marc Hassenzahl
Abstract
While realism is a common design goal for virtual reality (VR), VR also offers opportunities that are impossible in the real world (e.g., telekinesis). So far, there is no design support to exploit the potential of such “impossible” augmentations, especially for serious applications. We developed a card set and a workshop format, which features 15 opportunities to facilitate the ideation of augmentation-oriented VR. We piloted the method in five workshops with people in the early stages of developing a VR application (N=35). Participants found the cards easy to use and to inspire viable new concepts that differed from earlier ideas. Analysis of the concepts with interaction criticism identified two strategies: (1) augmentations that are only loosely related to the purpose of the application, simply to increase “fun”, and (2) augmentations that are closely related to the core purpose and thereby subtly facilitate its fulfillment. The latter has the greater potential.
1 INTRODUCTION
In recent years, the availability of Virtual Reality (VR) has increased tremendously. Through low-cost head-mounted displays, VR found its way into many domains, not only as games and entertainment, but also as serious applications in contexts such as medicine, elder care, vocational training, and other forms of work (e.g., [23,43,49,71]). Current VR provides highly immersive experiences [68,77]. In serious applications, developers use this mostly to create particularly realistic simulation-like virtual environments, which should closely resemble the physical world (i.e., virtual reality) [30:29,60] and be, at best, indistinguishable from real environments [54]. Consequently, real places, objects, movements, and gestures inspire their interaction design [29,40]. While pursuing realism may be a valuable design goal [27,54,82], a too narrow focus on it can also lead to mixed results [30:49,46,51]. For example, failed attempts at realism in interaction or virtual faces can lead to an “uncanny valley”-phenomenon [46,75], where acceptance is low because the system promises a form of realism it cannot provide. Moreover, technological limitations will most likely keep VR applications from reaching full realism for quite some time, if ever. Finally, while less realistic, unreal abilities, such as super-fast movement with seven league boots [28], can lead to better performance and more fun in VR, depending on the design [51].
A promising design approach to complement widespread approaches with their emphasis on realism, is to explicitly focus on what VR can enable users to do beyond what is possible in the real world. For example, a qualitative study of the user experience in VR showed that certain user groups find experiences different from reality especially enticing and interesting [25]. In this sense, VR can provide "magical" or "hyper-natural" interactions [30:290,51]. Various papers already highlight the potential of designing VR applications that offer “impossible experiences” [80], “beyond-real interactions” [1], or “superpowers” [67]. Inspired by the notion of human augmentation, a growing body of work uses the “unrealistic” side of VR to augment vision (e.g., [34,41]) or movement (e.g., [57,70]). Sadeghian and Hassenzahl [67:180] summarize this potential as follows: “[…] the real benefit of using VR may not necessarily lie in improving an imitation of reality, but in cultivating the unique possibilities of VR to augment human capabilities.” Furthermore, a recent paper [53] shows that the experience of feeling augmented by unrealistic abilities in VR is positively related to several experiential variables, such as hedonic quality, need fulfilment, and positive affect.
While the potential of “virtual unreality” seems increasingly evident, it is not yet clear how to design VR applications for augmenting experiences. While preliminary work [67] showed that equipping designers with the notion of “superpowers” helps them to create augmenting concepts for VR, there are no tools or methods available for a more systematic support of design. This paper presents a design tool, a set of design cards, and reports on their development based on a synthesis of literature and interviews (N=8). The card set features 15 augmentation opportunities in five categories to inspire and guide ideation. We embedded the cards into a workshop format to support the early conceptual design of VR applications. We ran five workshops with various teams (N=35) and conducted feedback rounds and single interviews with participants to evaluate the workshop format as well as the card set. In addition, we analyzed the resulting concepts using interaction criticism [6].
2 RELATED WORK
2.1 Virtual Augmentation
Since the early days of the computer, one goal has been to augment human beings [19]. In the real world, augmentation is limited by, for example, the law of physics. In virtual worlds these laws do not necessarily exist, while actions performed in virtual environments can still feel ‘real’ and can have real-world consequences [5,58,64]. Raisamo and colleagues [61] identified three general forms of augmentation which can be applied to VR: augmented perception, augmented action, and augmented cognition.
To augment perception, various senses can be virtually enhanced or altered. VR can augment vision by broadening the field of view [41,62], granting the ability to look behind and ahead at the same time [20], or zooming in and out [16,34]. Another possibility is to enhance hearing [24] or to even provide echolocation [3,4].
VR further allows for the virtual augmentation of action. New modes of movement are possible, such as super leaping [70], teleportation [12], flying [36,64], or using “flying carpets” [57]. Other augmented actions include slowing down time [33], telekinetic abilities [78], or novel ways of haptic feedback [2,50,69]. Having countless extra virtual hands can make grasping virtual objects faster [72].
VR can augment cognition by, for example, offering relevant information at the right time or extending memory through stored information [73]. Both augmented reality (AR) and VR help with the visualization of big data [56] and can support learning [74]. Specifically in the medical field, AR can support surgeons through additional visualizations [31] or by additional help and information acquired in VR simulators [59].
Further examples of augmentation do not easily fit the scheme of perception, action and cognition. For instance, changing one's body can lead to strong feelings of body ownership in the virtual world [32]. The experience of augmented capabilities or feeling ownership of another body can even extend into the real world and impact real behavior [58,64]. Going beyond illusions of body ownership of different human bodies, towards, e.g., animals, can increase enjoyment [36]. Other augmented abilities can help users with special needs [44], for example, by supporting the recognition of emotions on other people's faces [81]. Enhancing human abilities in VR presents exciting potentials, but there are pitfalls to bear in mind. Adding perceptive abilities can run the risk to become overwhelming and distracting [20]. Further, new abilities may require users to practice extensively to become beneficial [3].
While examples of virtual augmentations exist, the majority are early explorations or proofs of concept. The way augmentations are conceptualized is neither systematic nor methodologically supported. A first step are methods and tools which help and guide the design of applications with a focus on virtual augmentation. In addition, more work to explore the potentials and drawbacks of augmentation in specific application domains is required.
2.2 Developing Conceptual Designs for Augmenting VR Applications
Designers and developers often approach the conceptual design of VR with established design methods that are not specific to VR, such as human-centered design [22,30:373] or design thinking [13,47]. These approaches involve users in the ideation through participatory design processes. For example, designers teamed up with students to develop a VR application for fear exposure therapy [21] or mixed teams developed an application for osteoarthritis patients [39] or an application for users with cognitive impairments to exercise in VR [10].
While this sets a general direction, e.g., for the direct involvement of users, these approaches do not support designers to make use of specific qualities and challenges of VR at an early stage of design. Often, it is not easy to find users who have ample experience with VR and know about the potential strengths and weaknesses of the medium. Common strategies, such as brief introductions to VR with demo applications or 360° videos for demonstrations and early prototypes [21], run the risk of repeating what is already there, especially mimicking reality while omitting other, less obvious ideas. Developers look at existing applications in app stores (e.g., Steam or Oculus Store) to try to identify best practices of design principles and potentials themselves [35]. Therefore, several authors identified the need for guides, frameworks and methods which not only suggest a participative process, but also focus on the particular characteristics of VR [7,79] in a more prescriptive way.
While many acknowledged the specific augmenting potential of VR [1,66,67,80], so far, no tools or methods exist to guide designers in making use of it in a design process. An exception is Sadeghian and Hassenzahl [67], who explored the influence of “superpowers” on the development of VR concepts. In an ideation workshop, they asked designers to develop concepts for VR applications for relaxation in a particular scenario. First, they let participants develop concepts without further guidance. In a second phase, new ideas were based on “superpowers” (e.g., x-ray-vision, shapeshifting, super strength). In a comparison, they found that a focus on superpowers led to a stronger conceptual focus on interactions. Further, they described the emerging concepts from the second phase as “[…] more surprising, unconventional, and imaginative.” [67:184] While the results are promising and a first step towards realizing the potential of augmentation, we believe that more guidance is needed to properly utilize this approach in different application domains and in participatory processes. More than simply mentioning possible augmentations to designers, corresponding design tools are needed that can be used by people from different backgrounds and disciplines. Furthermore, for this approach to be successful, it needs to be used in conjunction with other methods to ensure that the augmentation capabilities add value in respective application domains and are consistent with the overall design goals.
2.3 Inspiring with design cards
Design cards are an established tool to summarize frameworks and methods and to present this information in an accessible and applicable way for designers and others in a design process (e.g., [26,38,48]). In their overview of publications on the topic, Roy and Warren [65] point out several strengths of card-based design tools, such as being easy to understand, to facilitate discussions, to present a suitable common reference for diverse teams of designers and other involved stakeholders, and to expand the design space. Further, design cards can help to make designers address problems playfully and change how they think about an issue [52]. Depending on their design, cards can be versatile and highly adaptable to different domains [26]. In addition to inspiration, design cards can be helpful to explain and articulate ideas and increase engagement in the process [38]. Most sources report almost exclusively positively on design cards; however, almost all evaluations are carried out by their respective makers [65]. Typical reported drawbacks are incomprehensible cards with too much or too little information [65].
Wölfel and Merritt [84] identify three types of design cards: cards to stimulate inspiration without specific instruction for use, customizable cards with instructions for use at a defined stage in the design process, and context specific cards. Currently, only a few card sets are published specifically intended for VR or mixed reality applications, but findings suggest that benefits and challenges are similar to other domains. The “Mixed Reality Game Cards” [83] support ideation for mixed reality games and were applied and tested over a series of studies. The authors showed that their cards strongly influence generated ideas, and that different modes of using their cards can benefit different target groups (e.g., a rule to choose and combine cards from a random selection can make it easy for inexperienced participants to quickly come up with an idea) and produce different types of concepts (e.g., forcing participants to combine specific cards can lead to novel but weird ideas). A study of the development and use of a card set for the design of exergames [48] showed that design cards facilitate the articulation of ideas and support the development of a shared language. There is a danger, however, that when cards are difficult to comprehend, focus shifts from creative work to understanding the cards.
Based on this, we set out to fill the research gap described above by providing a card set that inspires designers to apply augmentation capabilities beyond realism in their VR applications. While we target early stages of the design process with our card set (i.e., conceptual design), the openness of cards should allow designers to use them in other stages as well (e.g., to evaluate existing ideas). In the following, we present the virtual unreality design card set and its development.
3 DEVELOPING A DESIGN TOOL FOR AUGMENTATION-ORIENTED DESIGN
A first step to develop the design cards was to identify a selection of inspiring augmentations, which fit VR. In addition to existing work on augmentations that have already been implemented (see section 2.1 and table 1), we conducted interviews with people with extensive VR experience to explore augmenting interaction possibilities. We chose this approach to add experientially focused accounts of virtual augmentations and their consequences. This provided further inspiration for the cards and validated categories found in literature. In the following, we describe how we created the collection of augmentations on this basis. We then present the resulting set of design cards.
| Augmentation category | Description | References |
|---|---|---|
| Perception | Augmentations of perceptual abilities and capacities, e.g., expanding the field of view, having several perspectives at once, controlling and visualizing where one can hear | |
| Action | Augmentations of movement or other physical actions, e.g., superhuman jumping and flying, new modes of movement like teleportation, telekinetic abilities | |
| Cognition | Augmentations to cognitive abilities, e.g., added visualizations to make sense of data or other complex information, adding contextual information | |
| Other: Body/Identity | Changing one's own virtual body, e.g., allowing users to slip into other roles through swapping virtual bodies, altering one's own representation/avatar |
Table 1: Examples of virtual augmentations from literature, following the categorization by Raisamo and colleagues [61].
3.1 Interviews: Positive and Augmenting Aspects of VR
We conducted eight interviews, not only to expand our possible virtual augmentations but also to comprehend the positive and negative consequences of such augmentations. We found it crucial to go beyond the initial technological fascination that is characteristic for first-time users of VR. Consequently, we sought out experienced participants who have already used numerous applications and interact in VR on a regular basis. We recruited through social media and screened for extensive experience with VR (i.e., except one participant, all used VR both professionally and casually). In total, eight participants (participants stated their gender and age as part of the interview: 3 female, 5 male, none other, age 26-37, M=30, Median=29) took part in the study (see table 2). The participants received 30€ as renumeration. Including a pre-task of 15 minutes, each semi-structured interview lasted about 60 minutes.
| P. | Gender | Age | Occupation | Experience with VR |
|---|---|---|---|---|
| 1 | M | 37 | Researcher | Research applications, Personal experience |
| 2 | F | 28 | Game artist | VR game development & design, personal experience |
| 3 | M | 29 | IT specialist | Workplace applications, personal experience |
| 4 | F | 29 | Designer | personal experience |
| 5 | F | 26 | Designer | Design of commercial VR applications (planning), personal experience |
| 6 | M | 27 | Product manager | Commercial VR applications (training), personal experience, VR application development (past) |
| 7 | M | 34 | Developer | Development of commercial VR applications (medical), personal experience, VR game development (past) |
| 8 | M | 30 | Developer/Researcher | Development of research applications, personal experience |
Table 2: Participants in the exploratory interview study.
To ensure that participants' memories of VR and its experiential qualities were fresh, each participant used their own favorite VR application for 15 minutes right before the interview. After an introduction, the interview consisted of four parts. First, participants introduced themselves and described their experiences with VR. Second, participants described their favorite application and typical interactions with it. The interviewer asked questions about the application and the experience (e.g., Why is this your favorite application? How do you feel when using it?). Third, the interviewer asked about positive experiences with other applications. If the interviewer found that answers were exclusively about gaming in VR, they asked participants about experiences with serious applications. Fourth, the interviewer asked participants to speculate about future VR applications and interaction opportunities they would like to see. Here, participants had to imagine an ideal future application and describe it. Finally, we concluded the interview and gave participants the opportunity to ask questions. We conducted all interviews remotely via video call and obtained permission to record audio and transcribe it. Following an inductive approach to complement knowledge from literature, we performed a thematic analysis [8] to identify augmentations and their experiential consequences. The first two and the fourth author did this separately using MAXQDA1 before consolidating the results together.
The interviews led to a substantial collection of 28 different augmentations. This included detailed descriptions of what makes these augmentations feel special and positive, and examples of applications from which they came. As expected, mainly games were mentioned, while some participants also mentioned non-game applications (e.g., creative tools, a medical application). Altogether, we identified three broad categories of augmentations: (1) augmented physical abilities, (2) the virtual world as an augmentation by being explicitly differently from reality, and (3) taking on the role of someone else, roleplaying, and experiencing activities from a different perspective.
Physical abilities are a major category of augmentation. Participants reported that lack of strength is no obstacle in the virtual world, so one can lift and toss heavy objects. This does not feel similar to reality, as it lacks resistance and requires no training, but it can provide a fun moment of surprise (P5). Other participants (P2, P6) mentioned increased freedom of movement (e.g., through flying or extreme jumping). While they described it positively, they also acknowledged that especially movement abilities bear a risk of simulator sickness and general feelings of uneasiness when not implemented well. In addition, the feeling of accomplishment is lessened when there are fewer difficulties to overcome, and no risk involved. Finally, a common but unique ability in VR is to teleport oneself. However, participants almost exclusively described this for the potential of where one could go (P1, P3, P4). The teleportation act itself is rather a conventional means to an end, prevalent in numerous VR applications.
In line with literature, the participants mentioned several ways to augment perception as an interesting and unique way to explore the virtual world. Frequently mentioned was the ability to freely adjust levels of detail, either being able to zoom in to see things so small you could usually not see, or to zoom out to allow for a much more removed perspective (P1, P4-8). In two specific examples, one could observe a medical process at bacterial level, or zoom out to a planetary level (P8). Participants described these possibilities as stimulating and fascinating, especially when new perspectives lead to new insights. If the interaction feels satisfying, operating freely with such a “god-view” (P1) can feel almost “magic” (P4). Another possibility was to filter perception. One can switch off senses in VR, to become more focused (P5).
Beyond, we identified several experiences which were not clearly tied to a particular ability. Here, the virtual world itself, or the way in which it works (e.g., the laws of physics) felt augmenting. Fundamentally, the virtual world does not have to work logically and can be surprising, e.g., by mixing different (and impossible) biomes (e.g., P7 mentioned a forest of colorful mushrooms on a floating island) or tailoring environments exactly to a suspense arc (P5, P7). P1 described how an application which links user movement to the passage of time led to a different way of planning actions, allowing them to carefully consider each move: “It's funny, I'm in the situation, but I'm also somehow above it. […] I can turn my head slowly and observe the scene completely and think. Sometimes it's like this, you shoot somewhere, […] you move back somewhere else. […] And all that in one moment.” Additionally, the virtual world can be an exploration space with low stakes in which one can try actions one would not normally perform (P2, P5, P6). As one participant explained: “I also had this feeling of security, that in VR you can try things that are not possible in real life without worrying about doing something wrong” (P6). Ultimately, the virtual world allows various forms of customization and creation. Creative tools can facilitate unique forms of creative expression (P5). Planning tools, e.g., for interior design, can provide a sense of spatiality that is difficult to achieve in other media and add the ability to change things quickly (P4, P5, P6, P8). This can evoke feelings of control and competence.
Finally, some participants mentioned possibilities to slip into different “bodies”. VR allows users to alter one's own representation and to live experiences from this altered view (P1). There are various methods for this: through role-playing in games, through immersive first-person perspectives, or by taking on an abstract role like a god looking down on and controlling an environment. In the case of roleplaying, P1 and P2 explained that the way the character is conveyed has a strong influence on their actions and decisions. Some games utilize a bird's-eye view of the virtual world and allow for interactive control from above (resembling an interactive diorama). For one of these games, P7 mentioned that they simultaneously were part of the action and controlled it, which resulted in a double perspective: “There is a ship with your head in the game. So, it's like you're floating, looking at a robot and you move the robot […]. And also the controller has a physical appearance in the game, you can see it there and push the robot with it. […] They don't use the VR only for a point of view, but also like you are there, like a physical object there.” It seems beneficial to distinguish between visual augmentations (e.g., being able to zoom in) and taking a different perspective through altered bodily representations. Taking on a different body has an impact on your sense of self in the virtual world. Similarly, we found a distinction between having new virtual abilities (which are part of me) and being able to use magic devices in the virtual world with impossible functions.
To summarize, the interviews yielded a rich collection of augmentations that confirmed and extended what we found in literature. Moreover, the interviews not only helped to identify additional augmentations, but also provided subjective accounts of how these augmentations were experienced. Notably, we rediscovered many of the categories from literature in our interview study. However, many participants place greater emphasis not only on how their own abilities are expanded in the virtual world, but also how the virtual world itself or their bodies can be different.
3.2 The virtual unreality design cards
To form the basis for the design tool, we merged the results from our categorization of augmentations from literature and the results of the interview study. We developed a set of five overarching categories of augmentation in VR, which contained a total of 15 detailed augmentation opportunities (see table 3).
| Augmentation category | Augmentation opportunity | Brief description |
|---|---|---|
| Augmented worlds | Impossible and surprising worlds | The possibility of depicting worlds or circumstances differently instead of replicating reality. The virtual world can follow other rules and create surprising moments. Despite its otherness (e.g., through simplified geometric shapes), it is coherent in itself and makes sense. |
| Create and transform worlds | The possibility to actively design and customize the virtual environment yourself. Requirements and wishes of the user can be implemented spatially by adding elements or modifying the environment. The environment can be designed and adapted to one's own preferences. | |
| Augmenting and impossible objects | The possibility to design objects and elements that do not exist in reality. This creates completely new interaction possibilities. New objects can have unknown properties and offer new capabilities that are not possible in reality. | |
| Testing and experimenting | The possibility to create a safe space where actions do not have negative real-world consequences. This invites experimentation and trial and error. Simplified simulations can be used to playfully test complex or difficult processes. | |
| An Extended Self | Take on other roles | The possibility of being someone else and slipping into other roles. In this way, experiences can be felt from the point of view of others and completely new perspectives can be opened up. One can assume other identities, act accordingly, feel and gain new insights. |
| Change my shape | The possibility to change and freely model one’ s own shape. Your own appearance can be changed and modified at will. It can be completely different from the appearance in the real world and can have other human features, or even completely deviate from the natural human. | |
| Be several things at once | The possibility of being several things at once. One is not limited to the state of being or controlling only one entity. You can do several things in parallel or take part in the action at the same time and additionally observe from the outside. | |
| Augmented Actions | Teleport to other places | The possibility to teleport virtually to any imaginable place, fictional or real. Without having to pay attention to restrictions such as time, distance or feasibility, it is possible to immerse oneself in new places and worlds and gather new impressions. |
| Supernatural powers | The possibility to have superhuman strength and even move objects at a distance with telekinesis. The actual strength does not matter, so new possibilities for action arise. Moving huge and heavy objects, not getting tired or even interacting magically without touching becomes possible. | |
| Move freely in a space | The ability to move freely in a space. One can fly, hover and even move through seemingly solid objects. The environment can be perceived from new perspectives. Unusual viewing angles and flexibility result in new insights. | |
| Manipulate the passage of time | The possibility to control and manipulate the passing of time. Processes can be changed in their course and events and results can thus be influenced. One can slow down, stop, speed up or even go back in time. | |
| Enhanced Perception | Perceive more and better | The ability to enhance and expand perception beyond one's natural abilities. One can see magnified or miniaturized and allow other perspectives at the same time. One can see or hear into or through things and perceive previously inaudible frequencies. |
| Focused and precise perception | The ability to selectively influence perception. One can focus on certain stimuli and sources, filter out other sources, and even turn off individual senses altogether. An individual focus makes it possible to hear, see and feel differently and more accurately. | |
| Augmented Thinking | Infinite memory | The possibility to store and securely fi le unlimited knowledge and information. This results in the possibility of externalizing part of one's knowledge and memory and accessing it at any time. |
| Additional knowledge | The possibility of retrieving additional knowledge in suitable situations and presenting it individually as if it were internalized. The opportunity arises to expand available knowledge depending on the situation and to understand more. |
Table 3: Collection of categorized augmentation opportunities with descriptions.
To make this collection usable in design processes, we created a set of 15 design cards with augmentation opportunities based on it (see figure 1). We chose design cards as the specific implementation, for reasons outlined in section 2.3.
Figure 1: Cards of the design card set laid out on a table.
Each card follows a given structure (see figure 2). It starts with a comprehensive title and brief description to make the augmentation opportunity quickly understandable. Then, it describes the augmentation opportunities in further detail along four points: possibilities, design prompts, feelings, and quotes. Possibilities describe what this augmentation allows users to do. Design prompts offer specific questions to support designers or developers when creating ideas. Feelings describe emotions that the augmentation can trigger. Finally, the quotes extend a more human feeling of what the augmentation can lead to. Both the feelings and quotes are directly informed by the interviews. In addition, each card contains symbolic three-dimensional illustrations to help spark creativity without being too leading. The full set of cards in the appendix and as supplemental material.
Figure 2: Details of the design cards (full set available as supplemental material).
The design of the cards was informed by insights from literature about card-based design. We deliberately kept the cards open so that they can address different domains and different types of applications [26]. To allow for different uses [83], each card includes possibilities, prompts and resulting feelings and quotes. Vague illustrations in addition to textual descriptions of the augmentation opportunities were added to offer further avenues for inspiration and association [26,65].
Of course, the proposed augmentations only lead to positive interactions and experiences if they fit to a specific application domain. Therefore, the use of the cards must be linked to other methods for understanding users and stakeholders and their needs. Furthermore, we consider adding a blank card to extend the set if additional augmentations emerge in workshops. After development, we used the design cards with designers and developers to evaluate whether they can be used in real design processes and to learn more about the ideas and concepts they may lead to.
4 DEVELOPING AUGMENTATION-ORIENTED CONCEPTS FOR VR APPLICATIONS
We used the card set in brief (approx. 4 hours) structured ideation workshops. Since the card set was developed as part of a fully funded accompanying research project (i.e., supplementary research that is conducted alongside applied research projects to generate methodological insights or to further transfer between individual endeavors), we had the unique opportunity to be in close exchange with several ongoing interdisciplinary research projects, each consisting of research, industry, and application partners. Each research project explored the possibilities of VR in a specific domain and developed an application. A general invitation to ideation workshops using the augmentation-oriented design approach was accepted by five project groups. We then conducted five separate workshops, each with one research group (see table 4), each following a similar structure. The research projects selected the workshop participants. As all projects had just started working on their VR applications, the workshop format aimed at the early stages of conceptual design. It introduced the augmentation-oriented approach and used the cards to contextualize augmentation and to generate initial ideas and refine them. Note, however, that the use of the cards is not limited to this particular workshop format.
4.1 Workshop Format
We divided the workshops into five parts. First, in (1) scenario and insights, one or more participants from the research project presented their project in up to 30 minutes. Typically, participants presented the goals and scenario of the project, initial stakeholder and user insights, and ideas of how VR could enrich the selected scenario. We subsequently asked participants to work in pairs for 15 minutes to add their own further ideas and experiences. Participants were prompted to focus on positive aspects to avoid an overwhelming focus on problem-solving (see [17]). During a brief closing discussion phase, we visually summarized the goals of the project, the target audience, and the intended technological solution to ensure that everyone had a shared understanding.
Second, in (2) introduction to augmentation-oriented design, we introduced the notion of augmentation and “impossible experiences” as a general design approach. In a 30-minute presentation, we explained the concept of going beyond realism and showed example applications. Ultimately, we introduced our categories of augmentations and the design cards. We then handed out printed card sets and participants had time to familiarize with the cards.
Third, in (3) speed-ideation, we asked participants to pair up and to quickly generate as many ideas for augmenting VR applications for their scenario as possible. We instructed the participants to come up with ideas by combining the cards with their insights from the first step. We prepared simple concept sheets with space for a title, a sketch, and a brief description, to support the documentation of concepts. Ideas at this stage did not have to describe a complete application but could just outline a specific functionality or interaction. Pairs had ten minutes to develop ideas. After a quick round of presentations, we reshuffled the teams for a second round of ideation.
Fourth, in (4) enactments, all participants selected ideas that were seen as promising. The groups usually chose and combined several ideas into one that appeared in line with the overall goals of their project. We then formed one or two larger groups (up to five participants, depending on the overall group size) to develop the ideas further through enactments. Here, participants would take on different roles (e.g., of different users, sometimes of the system, to enact the experience) and act out the future use of their envisioned VR application in improvised role-play. To support the role-play and help participants to set the scene for their enactments, we provided a variety of placeholder props (see figure 3) [55]. Enactments are an established prototyping method, especially in early stages of VR development [30:422] and are a suitable way for experts and laypeople to work together [55]. We included enactments because internal pilot tests showed that concepts for augmenting VR applications remained relatively vague without a further step to “think them through”. Enactments force participants not only to talk about their concepts in the abstract, but also to describe them in detail and to inevitably make design decisions. The placeholder objects (see, e.g., [18]) stimulate to act out novel interactions and to immediately shape and “feel” them, as a form of bodystorming (see [14,15]). We took part in the enactments as moderators to help with inquiries about the method, but also asked questions to trigger reflection. After the enactments, which lasted about 30 minutes, the groups presented the resulting concept. In this presentation, we asked the different group members to reflect on their experiences from the enactments from the point of view they had taken on during the role-play.
Finally, in (5) back to reality, the participants jointly reflected on the outcome of the workshop and its impact on the further development of their VR applications. Participants now discussed ease of implementation, that is, which aspects of the concepts would transfer into their development process, and which aspects would be more challenging.
Figure 3: Enactment with props to support the imagination.
4.2 Participants and Analysis
The thirty-five participants had various backgrounds. In some groups, we worked with designers and developers only, while others further included stakeholders or future users (see table 4 for further details). The number of participants in the workshops varied between four and ten. Due to covid-19 restrictions, the first two workshops were performed remotely via online conferencing and whiteboard tools, such as Zoom and Miro. In this case, the workshops had to be slightly adapted. For example, in the enactments, participants would use the online whiteboard to draw the virtual environment they imagined and describe what they would do step-by-step, rather than physically enacting interactions. However, the concept and phases of the workshop remained identical, and the card set was provided as well. We performed the three remaining workshops in person at different sites.
| Group | Application domain / development goal | Participants (Gender) | Participant backgrounds | Online/ in person |
|---|---|---|---|---|
| 1 | Multi-user VR application to experience stage performances such as concerts or plays | 7 (2 female, 5 male) | Computer science, UX design, UX research, music artist | Online |
| 2 | Multi-user VR game to visualize and learn about the impacts of everyday decisions on climate change | 4 (2 female, 2 male) | Product design, game design, programming | Online |
| 3 | Cross-generational multi-user VR application for (grand-) children to spend meaningful time together with their (grand-) parents over a distance | 10 (5 female, 5 male) | Interaction design, computer science, healthcare technology development, elderly care | In Person |
| 4 | Application in VR for users with autism to practice stressful everyday situations | 6 (4 female, 2 male) | Computer science, learning systems | In Person |
| 5 | VR application including a haptic device to perform physiotherapeutic exercises by oneself at home | 7 (4 female, 3 male) | Computer science, healthcare technology development, interaction design, physiotherapy | In Person |
Table 4: Workshop groups and specifications.
We documented the outcomes of each step and recorded video and audio of all sessions. To gather feedback about the design cards, the workshop and approach of augmentation-oriented design itself, we added feedback sessions at the end of the workshops. We prepared a set of questions (e.g., Did the approach result in new/different ideas or insights?) and asked all participants to share their thoughts. In addition, in the weeks following the workshops, we conducted follow-up interviews with participants who had agreed to take part. We conducted one group interview (N=4, 2 female, 2 male, 0 other; participants stated their gender as part of the interviews) and six single interviews (3 female, 3 male, 0 other) with participants from three of the five workshops. To decrease bias, half of the single interviews were carried out by another researcher (3rd author), who was not directly involved in the creation of the cards. The semi-structured interviews lasted up to 60 minutes and were conducted through video calls. As the interviews took place in the weeks after the workshops, we showed a brief overview and summary of what had happened during the workshop along with some photos from the sessions to refresh the memory of the participant. The first part of the interview was about the workshop in general and the perceived impact on the development (e.g., Please describe your experience of the workshop. Do you think the workshop led to new insights?). In the second part, we asked participants to discuss the concepts and outcomes that emerged from the workshop (e.g., Were the ideas new/innovative/different from previous ideas in the development? Please describe the feasibility of the concepts.). In a third part, we asked specifically about the design cards (e.g., Were the cards easy to understand and use? How did you use the cards during the ideation?). Finally, we asked concluding questions about the next steps in their development process. After obtaining permission, we audio-recorded and transcribed the interviews for analysis. To analyze, we conducted a content-analysis as described by Mayring [45].
In the following, we present the results of the workshops: first, a reflection from the first and second authors as organizers; second, the results of the analysis of the feedback interviews; third, the concept ideas that emerged during the workshops and a secondary analysis of the concepts, using interaction criticism [6].
4.3 Reflection from the Perspective of the Organizers
The first part of the workshop, that is, the gathering of insights about users and stakeholders, depended largely on how much information had been already collected. Involving stakeholders in the workshops was enormously enriching, as insights were more direct and could be validated immediately. In these workshops, the first part took the form of an interview in which the participants asked the stakeholders about their experiences (e.g., what is important for an artist when interacting with an audience).
For ideation, the card set seemed inspiring and well-liked. At the beginning, the pairs sometimes took a little longer to develop the first idea, as the participants had to get used to the design cards. They grasped the general idea of each card quickly, but reading the full content took some time. In the future, more time could be allocated for familiarizing with the cards prior to ideation. Alternatively, participants could initially only receive a selection of cards.
To do two ideation rounds was very helpful, since the participants were able to respond to and build on ideas from the previous round. In all workshops, participants created a rather large number of ideas quite effortlessly and seemed to have no difficulties with applying the approach we presented – regardless of their background. Overall, participants developed 66 concept ideas during the speed-ideation across the five workshops (group 1: 11, group 2: 7, group 3: 20, group 4: 11, group 5: 17). The variation between the groups can be explained to a certain extend by the number of participants in each workshop. However, we also found differences in the scope of ideas, despite identical instructions: Some produced many small concepts (i.e., separate functionalities involving augmentations), and some produced fewer but more complex concepts (i.e., an entire application). As the participants were asked on the provided concept sheets (see figure 4 for an example) to indicate which cards of the card set had inspired them to create the respective concept, we could see that the entire card set was used during the workshops. The most frequently used cards (ten or more times) were “Take on other roles”, “Create and transform worlds”, “Perceive more and better”, and “Manipulate the passage of time”.
Figure 4: Concept sheets filled out in German language from different workshops. Translations (from left to right): Sheet 1: Insight: Social event with a crowd; Superpower: Focused and precise perception; Title: Switch off sensory overload; Description: eliminate individual sources of stimuli; focus on relevant stimuli; relief, stress reduction. Sheet 2: Insight: In the garden; Superpower: Perception; Title: Hearing aid of the future; Description: Listen to the flower grow. Sheet 3: Insight: Feedback; Superpower: perceive more and better; Title: Looking inside to see muscles and bones; Description: visualizing changes in the body; visualizing relationships; understanding why the exercise makes sense.
In our opinion, most of the concepts represented interesting and innovative starting points for further design. As expected, the short time frame of the speed-ideation gave the participants only little time to work on each idea. Therefore, most concepts consisted of a brief description of an augmenting ability and its use. For example, the concept “Character-split” (group 2) described how a user can split into two parts when making a potentially momentous choice and experience the consequences of both possible choices simultaneously. Most concept ideas were quite playful and successfully linked the chosen augmentation opportunities provided by the cards to the specific scenario and goal of the application. One example for augmented perception was the concept “I know how you feel” which allowed users to abstractly see the emotions and stress-level of their (elderly) relatives as part of the virtual environment during a shared activity in VR (group 3). Some concepts remained undefined and are best described as design opportunities rather than concepts. For example, in “the concert as an audiovisual journey” (group 1), artists could create unique worlds and send the audience on extraordinary journeys beyond current stage shows. These concepts define an application domain but without further specifying concrete augmentation-oriented designs. However, while quite open, some of these ideas found their way into the following enactments and became more defined. A small number of concepts stayed close to the description on the cards and simply transferred the augmentation opportunity to the application scenario without much further adaption. Most of these concepts were from the first round of ideation and can be seen as attempts familiarize oneself with the card set. Finally, a few concepts did not make use of any card, but still followed the overall approach of augmentation-oriented design. For example, a participant (group 1) described “Biofeedback as a new audience response”, where sensors measure bio signals (e.g., heart rate, eye movement), which are converted into feedback for artists or used for automatic changes of environment, such as the lighting. Some of these ideas could be potential starting points for a future expansion of the card set.
For the enactments, the groups chose and combined concepts to enact. The selection was not always straightforward as participants found many ideas promising. The groups applied different ways to choose. Often, preference was given to the concepts that were most conceivable and best suited the ideas that already existed before the workshop. In some cases, perceived technological feasibility was strongly considered, although we advised against this. All in all, selection seemed to tend towards the more conventional. We therefore believe that the concepts not selected can still be valuable for the designers, if the goal is to create especially innovative interactions.
Some groups had difficulties to begin the enactments. We had to actively facilitate to make sure the groups started to play out their concepts instead of only discussing them. Once started, however, the enacting helped greatly to expand and clarify the concepts. The physical aspect of playing led participants to make concrete decisions about interactions because they were confronted with how such an interaction might feel when carried out. For example, the concept “Training with a doppelgänger” (see table 5 for a brief description) began with the idea of obtaining additional visual feedback on the accuracy of the execution of a physiotherapy exercise on one's own body. Initially, the participants imagined the feedback visualization to appear directly while performing the exercise, perhaps in a virtual mirror. However, while enacting the interaction with such a mirror, the participants realized that it was very similar to smart mirrors already in development and wanted to make better use of VR. They returned to the card set, selected the "being several things at once" card and changed the visual feedback mechanism from a mirror to a virtual doppelgänger. In the subsequently adjusted role-play, now a virtual doppelgänger performed physiotherapy exercises previously recorded by the user, while the user could walk around the doppelgänger and view feedback visualizations showing how to improve the performance of the exercise. In the role-play, participants found this version of the concept more interesting, augmenting, and possibly even more user-friendly.
As intended, the placeholder props supported the participants in defining the elements of the virtual environment. The participants first discussed the scene in which they wanted to enact and then selected props as stand-ins for virtual elements, such as tools, interface elements, or objects to demarcate virtual rooms. Especially in larger groups, the props acted as anchors on which the groups agreed and around which the role-play was structured. Note that in the workshops that had to be conducted online, the enactments were not as embodied and unfolded rather by participants narrating the actions they would be doing. Over the course of the enactments, more and more design decisions were made by playing and replaying situations in different ways. Ultimately, through role-play, the groups settled on a concept as the outcome of this session (see table 5 and the appendix for more detailed descriptions).
In summary, the workshops benefited from the clear structure and the pre-prepared materials. Regardless of their background, participants took part very actively, participated in all steps, and made extensive use of the design cards. A wide range of concepts emerged.
4.4 Feedback from Participants
In the feedback sessions and interviews, all participants stated that they appreciated the workshop for how the different phases built upon another and its general pace. Especially working together with participants from different backgrounds was seen as very successful: “That [workshop process] was very well conceived, well thought out. […] You did a good job of getting us started, so that we could all get on the same page, mix up, and talk to each other” (P1). The workshop provided a focused session to work together with all partners from each research project and to create a shared understanding of their project and the possibilities.
All participants appreciated the design cards. In most cases, the brief and conclusive wordings enabled them to quickly grasp the idea of an augmentation opportunity and develop ideas with it. In combination with the insights about the scenario, participants ideated quickly: “I mean like it's always super easy, just taking this insight and then reading through it and thinking. […] it was cool how almost from any card and any insight, you could come up with something and so it really helped with it [idea generation]” (P3). Some participants shared that they had started to use the cards beyond the workshop: “I have the card sets with me. They're actually here, […] and I plan on giving them to other colleagues for, well, idea generation processes that they do, because we all were really satisfied with how well it worked in getting new concepts. So, I think not just for this project but also for other projects that are VR related, they can be super useful” (P3). For some participants, the time available seemed too short to comprehensively explore all augmentations offered by the cards: “I needed more time to explore the superpowers themselves and there was like five minutes of duration […]. I really like the superpowers, but given the workshop, it was really hard to explore every card. I don't know how many cards it was, but I think I explored only like four of them” (P4).
The enactments were seen as a successful way to further define the smaller and sometimes vague ideas from the previous ideation. Specifically, participants enjoyed the change of pace and joint active work on the ideas. It also made the ideas more memorable. As one participant said: “We actually got into a ‘doing’ phase through these role-playing games. […] I thought that was pretty good and that's also what many people are still saying about it now. [...] And I think a little bit of this body experience, actually doing it – that helps you to remember it even more. […] The fact that we played there is still pretty present in my mind” (P2). However, as several participants pointed out, role-play also required to take on a role, improvise, and play, which not everyone is comfortable with. Therefore, many participants had to overcome initial inhibitions. However, all participants felt that the enactments were very helpful, and fit the early stage of development: “Because the project is quite at the starting point, we didn't find the time to actually play or simulate kind of a virtual environment as done in this workshop. […] So, I think this was very good, the enactment phase, because we didn't realize so many tiny things which we need to care about, which we didn't talk [about] so far” (P4).
The participants saw the detailed concepts as the main outcome of the workshop. Most participants rated the ideas as new and innovative for the project, and as something they would not have thought of without the design cards. In some respects, the interviews revealed a discrepancy between participants from different backgrounds. Some participants wished for more simple and straightforward concepts, as they were afraid that too many augmentations would make a future application more interesting but also harder to learn and use: “They [VR concepts] should be easy-to-use, definitely as a requirement, in so that we should limit ourselves to, I don't know one or two superpower features […] in the application” (P4). In contrast, another participant (P6) pointed out that they were a bit disappointed that their group did not choose “crazier” concepts. They saw the workshop as a safe space, in which they could and should explore ideas that could stray further from the straightforward ideas they would otherwise pursue. In some cases, participants saw an emerging concept already as quite complete and a very promising starting point for development.
To develop ideas beyond transferring real-life situations into the virtual world, the workshop and the design cards were deemed very helpful and useful for the projects. One participant summarized: “It was important for us to bridge the gap between VR and reality. […] for us this has expanded the process and ideation a bit more, that you look further and think about how much more is really possible in VR” (P5). Since all projects wanted to tackle real-world challenges, it was hard for participants to not focus on realism at times. Some participants had reservations if learnings from a virtual world with different rules would transfer into the real world. Therefore, a basis in reality was very important for most participants, even though the workshop was framed as an exploratory ideation exercise: “So, I think that's why it's quite important to move slowly [through the process] and try to, even if you're using VR, it's quite important to kind of be realistic” (P4).
Overall, the approach of developing augmentation-oriented designs for VR applications was completely new to most participants. Nevertheless, the design cards and the overall workshop process made it easy to generate ideas they could realistically pursue further. Additionally, the developers we interviewed said most of the ideas they worked on seemed technologically feasible, and in some cases even easier to implement than flawless realism. In summary, following our approach was seen as enriching: “So the results are very interesting and we're continuing to work with them because they've given us a good push into this area” (P5).
4.5 Secondary Analysis of Conceptual Designs
Table 5 provides an overview of the final concepts created. Using interaction criticism [6], we further analyzed these concepts. Interaction criticism suggests four different perspectives: the designer (i.e., What are the intentions? How are they made explicit?), the interface (i.e., analyze the visual form and language; What are influences on the interaction design?), the user (i.e., Who is the intended audience? How are they addressed? Analysis of rhetoric and material), and the social context (i.e., What is the ideal behind the product? Which future impact does it have? Which social questions are addressed?). For our purposes, we focused on the role of augmentations in all four perspectives. For example, instead of analyzing the general intentions behind a concept, we rather focused on how participants utilized virtual augmentations in their concept to advance their intentions. This is in line with Bardzell's suggestion that all four perspectives do not need to be considered equally, since interaction criticism can be adjusted to a specific purpose [6].
| Group | Concept title | Shorter description |
|---|---|---|
| 1 | Audiovisual Journey | In a VR concert, artists take their audiences on interactive and immersive audiovisual journeys instead of a performance on a traditional stage and seats setup. The concert environment is changeable and influenced by the song being performed, creating unique experiences akin to music videos. Artists can transport the audience through different worlds and respond to their reactions. For the audience, the concert is a social experience, starting in a gathering room where attendees can interact with each other. After the show, the audience jointly reflects on their experience and revisits moments they want to remember. |
| 1 | Concert in Space | A VR concert in space lets the audience experience music performances in a unique environment where they float and navigate using magnetic forces. The interaction with other attendees is influenced by magnetic attraction and communication through physical contact with haptic feedback. Groups of attendees can make sure they stay around each other and form a magnetically connected network. The artists can utilize these elements to create new sensory experiences in line with their music and guide the audience to different points in space. |
| 2 | Garden Party Planning | In a VR/desktop application, players plan and prepare a garden party and experience the impact of their decisions on the environment through amplified changes in the weather and climate. One player uses a VR headset and haptic device while others join from desktop computers. Negative decisions can lead to adverse weather events, which the VR player can manage using magical tools and powers. To deal with escalating climate repercussions, the VR player can duplicate themself. Desktop players can assist by controlling instances of the VR player to complete tasks simultaneously. |
| 3 | Memory travelling | A grandparent and grandchild travel through time and space to experience a memory in a VR application. With a remote, they can travel between memories, like in a photo album. They can control the length and focus of their trip to relive the memory and use a haptic glove for a more immersive experience, e.g., to feel each other's presence. The virtual environment fluctuates based on emotions to enhance the feeling of the memory. Presumably more capable with technological systems, the grandchild is largely in charge of all control elements. |
| 3 | Generational Magic Garden | A grandparent and grandchild have a virtual gardening experience in a VR garden, where they can plant and grow anything they want. The grandparent uses their real-life gardening experiences, augmented by additional knowledge provided in the application to help the grandchild become a master gardener. The avatar of the grandparent has easy movements to enable them to perform actions they physically might not be able to perform otherwise. Further, they can control the weather using a magic wand. |
| 4 | Doctor's visit training | A VR application where users with autism can experience virtual doctor visits, including both positive and negative scenarios. This allows users to train and prepare for different experiences, reducing anxiety and nervousness when visiting the doctor's office. The user is in control and can navigate through the scenarios in different difficulties, and experience how interactions with staff and patients can vary. |
| 4 | Grocery shopping training | This VR application lets users with autism experience virtual shopping scenarios, from a smooth, efficient experience to one with obstacles, such as difficulty finding items or unforeseen events at the cash register. The user can manipulate the shopping environment and train for different situations to reduce anxiety when shopping in real life. They can also jump back in time to try out different strategies to react to a situation. |
| 5 | Movement Training with a Doppelgänger | A VR application to learn and perform exercises from physiotherapy at home. The user's virtual avatar is duplicated, to they see a copy of themselves in front of them mirroring their movements. This way, the user can see how they perform the exercises, and the application provides direct visual feedback to improve motion accuracy. If the user's form is incorrect, symbols indicate where improvements are needed, and anatomical views show why the posture should be corrected. The user can also walk around the avatar, view movements from all angles, and control the speed to focus on specific aspects. |
| 5 | Virtual Exercise Garden | A playful VR application that encourages patients to perform movements in a virtual garden through three tasks: walking through a thorn bush, balancing weights, and picking berries. The tasks require specific movements, and their completion and results are monitored and collected. The virtual environment can be altered to fit the needs of the test. Gamification elements motivate continued use. |
Table 5: Short descriptions of the final concepts from the workshops.
4.5.1 Patterns of applying augmentations.
In several concepts, the participants used the design cards to develop ideas about what the virtual space generally allowed instead of focusing on detailed interactions. Here, participants either referred to augmentations from the card set concerning the virtual world (i.e., augmented worlds, adjustable worlds) or based their concept on the general idea of “unreality” (e.g., there is no need for a concert hall with seats and a stage, the virtual location of a concert performance can be in a constant state of flux and change completely with every song). Consequently, interaction-specific augmentations were less in focus in these concepts. As a caveat, these concepts remained rather abstract and theoretical. Consistently, in the enactments, participants talked more about the rules of the virtual world in general and acted out fewer concrete interactions. Here, the ideation and enactments did not lead to fully defined concepts for applications but were rather a step to explore the opportunities of VR in a given scenario.
For other concepts, participants focused more strongly on augmenting specific interactions. In these cases, the card set was applied more directly. We identified two basic patterns of how and for what purpose the augmentations were implemented. On the one hand, participants applied augmentations for single interactions with the purpose of making these interactions more stimulating and interesting. Here, augmenting abilities were introduced to make the virtual world more fun and motivating. Key intentions are having fun, being able to do something that is usually impossible, and being stimulated by the novelty of these interactions. An example from the workshops is the “garden party planning” concept. While the overall goal of the application was to produce insights about how everyday decisions have an impact on climate change, the augmenting abilities are not directly applied to this. Rather, users receive new abilities to fight off weather effects in minigames. Comparably, concepts following this pattern tend to remain superficial and not all augmentations are in line with the overall goals of the application (e.g., to learn about the impacts of everyday choices on climate change).
On the other hand, augmentations were purposefully applied to support the overall goal of the application. In these cases, participants selected multiple augmentations and shaped them specifically to generate value in their scenario. Here, the augmentations are less isolated and fragmented, as they play together towards a common goal. As an example, one group enacted different augmentations to encourage physical learning and the correct execution of movements in physiotherapy. In this concept, the augmentations created valuable extended interactions, which have advantages over real-world physical activity learning. For example, users would be able to see another instance of themselves perform the movements. For further insight, various visual augmentations would show where their movement was incorrect. Additionally, anatomical views would enable interested participants to see a schematic simulation of their muscles and tendons to develop an understanding why they have to perform a movement in a certain way. This shows that the design approach we proposed was successful in developing an idea of how VR can have real value in addition to what is already taking place in the real world. Interestingly, in the feedback session, participants from this group pointed out that this was already a very “complete” concept which they would have in mind for their next development steps: “I think giving feedback on movement like this is good and important for our application and I think it can be implemented well. [...] In any case, I think that is something that is possible and makes sense” (P18).
While some of the concepts appear more fragmented and were not finished or easy to implement yet, they still are an important contribution. The enactments forced the actors to make preliminary decisions and assumptions, and even make concrete design decisions immediately [55]. That is, to freely explore many possibilities without the necessity to commit to the ideas yet and to learn from this exploration. Often, important learnings result from ideas that will never be implemented or are highly unrealistic. The results therefore are oftentimes not complete concepts ready for implementation, but rather inform decision-making in the development process. Therefore, while the emerging concepts themselves might be less promising in these cases, the learnings can be valuable results in their own right.
In terms of applying augmentations in VR applications, closely intertwining augmentations with the overall goals of an application seems the more favorable approach. Often, a vital question in VR development is how an application can generate persistent value beyond initial technological fascination and novelty. Augmentation-oriented design and the design cards as a guiding tool are a promising path to this. Well-implemented, virtual augmentations have the potential to provide experiences that are almost uniquely possible in VR and can still be translated into real-world learning.
4.5.2 Relationship between realism and augmentations.
A different perspective on the concepts is the question of in which aspects they follow realism and in which aspects they are augmenting. A notion shared by the participants was that realistic environments and interactions are necessary when transfer of knowledge from the application to the real world is important, or to create a sense of familiarity and security. Participants assumed that potential users would get acquainted more easily with a virtual world if it closely resembles what they already know. This concerned the way the world looks (e.g., the representation of a living room), as well as interactions and interaction elements (e.g., a TV remote control). One group suggested that an elderly user can first familiarize with the virtual environment and then only gradually receives additional augmentations. To enable a transfer of knowledge between what happens in the VR application and in the real world, most groups chose to anchor the virtual experience in realism, e.g., through familiar environments. Then, either to add stimulating or fun elements, or to enable new insights and learnings, participants introduced augmenting interactions. This approach may be a consequence of how we structured the workshop, that is to first gather real world insights and then introduce fitting augmentations.
5 DISCUSSION
Especially in the design of serious VR applications, realism plays a central role. We have argued that VR offers the possibility to use and experience unrealistic, augmentation-oriented interactions. Flying, looking through walls, or stopping time are just a few examples. While mimicking reality seems straightforward, designing meaningful unreality is more challenging. It requires methodological support, which is currently lacking. To this end, we developed a set of design cards and incorporated them into a workshop format. We then conducted workshops with design teams in early phases of development to test the proposed method in practice. All in all, participants experienced the workshops and especially the cards as helpful for generating novel “unreal” ideas. They found the approach inspiring and leading to innovative and promising concepts. Only a few participants criticized that the amount of content on the design cards may be too much to absorb in the short time available in the workshop. However, we see this more as a flaw in the structure and timing of the workshop than in the cards themselves. In general, we want the cards to be self-explanatory to be helpful in many different design processes. In this case, optimizing for understandability seems more important than optimizing for reading speed.
Most evaluations of design cards are conducted by their makers, which opens the possibility of bias [65]. Consequently, we took this into account in the feedback interviews by involving another researcher, who was not part of creating the cards and organizing and moderating the workshops, to carry out half of the feedback interviews. In our study, the results did not differ significantly between interviewers. While additional independent evaluations of the cards would further clarify their value, the present exploratory work already shows that applying an augmentation-oriented design approach and the card set to support this can yield promising results. Additionally, in the time since the workshops, we heard back from several participants who asked for permission to use the card set in their projects, but also for, for example, teaching in student projects.
While we already had the opportunity to test the new design tool and workshop format with many participants who were in the early stage of developing VR applications, the method would benefit from being evaluated with more diverse teams and participants. In the present case, all participants were from a very similar cultural background and of similar age. Especially the underlying concept of virtual augmentation through technology might elicit very different responses in other cultures and ages. Additionally, the cards could be tested with an even stronger focus on participatory design, including more stakeholders who are usually not involved in design processes to examine whether the design cards can be a facilitator in such co-design processes specifically. In our workshops, the cards enabled participants unfamiliar with VR to take part in the design of a future VR application. This makes them promising for making such design processes accessible to a wider audience. In the future, the card set should be used and evaluated with more audiences, including experts and non-experts alike, and at different stages of the design process.
More augmentations could be added to expand the scope of the design cards. Moreover, it might be beneficial to equip participants in ideation workshops with a deeper understanding of what virtual augmentations feel like. To this end, we have developed an application, in which users can try out examples of augmenting interactions mentioned on the design cards.
An important further question is whether the workshop had a lasting impact on the mindset of the participants and the teams, that is, whether augmentation-oriented design became an option to them. In this respect, it will be interesting to analyze whether augmentation-oriented interaction found its way into final interactive prototypes. Naturally, as all groups participated with the motivation of trying out a novel design method and potentially using the results of the workshop for their own research work, we could not carry out a workshop with a control group (e.g., focusing on reality-oriented designs). However, as this is exploratory work and examines the approach and outcomes of augmentation-oriented design in depth instead of comparing it to reality-oriented design, this is no limitation.
We analyzed the concepts created in the workshop with the help of interaction criticism [6]. Overall, emerging concepts clearly differed from what we would expect from a reality-oriented approach. In some cases, participants developed their concepts relatively superficially, focusing on the general possibilities of the virtual space, or simply using the augmentations for novel and fun interactions independent of the application's overall purpose. Others used augmenting interactions in a way to support the overall purpose of the application. In future workshops, the latter approach could be encouraged more clearly. Starting from the first ideation phase, the moderators should stress this link to make sure that the augmentations from the design cards are always applied with the overall purpose in mind. Further, it might be helpful to not only make use of the cards during the ideation, but also to use them during the enactments. Here, they could be used to trigger deeper reflection as enactments unfold. Moderators can become more active and ask participants, if the interactions they just played feel enriching in terms of the overall intention of the future application. If not, the group can take a step back and replay the interaction in a way that is more in line with their goals.
In addition to the concepts that were ultimately selected for enacting, many other promising concepts emerged during the ideation phase. Often these were not selected for further detailing because to the groups they seemed less imaginable, implementable, and maybe too challenging for future users. In other words, participants tended to be more conservative when moving from ideation to enactment. When looking for more innovative and challenging ideas, participants may need to be encouraged to select concepts from ideation that are less straightforwardly linked to their design problem at hand, especially in terms of feasibility. Nevertheless, the cards seem able to stimulate both.
Another topic for discussion is the role of realism in the concepts that emerged from the workshops. As pointed out before, realism is one of the main goals and drivers of VR research (e.g., [27,29,54]). However, several authors have questioned the sole focus on realism (e.g., [11,80]) and highlighted the opportunities of being magical or unrealistic in some aspects of a design [37,67]. Looking at the emerging concepts, it is helpful to be precise and distinguish between different types of realism. Rogers and colleagues [63] offer a hierarchical taxonomy of realism dimension and how these dimensions relate to one another. Viewed through this lens, the concepts generally focus on only one or two dimensions of realism. For example, the concept “Movement Training with a Doppelgänger” is realistic on most dimensions, only the ability to duplicate oneself to analyze the movement accuracy would be a significant change in either avatar realism or embodiment realism. Simply put, the participants used the design cards as inspiration for making selected aspects of their future application unreal, in order to add value for the user. This is very much in line with Shneiderman's recommendation for enhanced 3D-design features from 20 years ago, namely to “[…] facilitate user tasks rather than mimic reality” [76:12]. While doing so, participants were clearly wary of shifting their focus away from realism in too many aspects simultaneously. All in all, virtual unreality in an augmenting sense seems promising to generate innovative concepts or to make an application more engaging, unique, and fun. However, each decision to do so has to be carefully weighed up against the benefits of realism. Further, the integrity of the entire experience has to remain intact, and cannot become too complex, demanding, or unfamiliar.
Altogether, several open questions remain and provide opportunities for future work. While the overall approach is promising (e.g., [1,67,80]), the potentially emerging concepts and the role and extent of realism in serious VR applications should be evaluated further. While participants expressed that the approach led to innovative ideas, they also mentioned potential drawbacks of augmentation-oriented designs. While there was agreement that augmentation-oriented designs can add value, there was also uncertainty if they could be harder to learn and use, overwhelming for inexperienced users, and whether learnings made with augmented abilities would fully transfer into the real world. Some of these concerns are echoed in literature (e.g., [3,20]). Therefore, additional studies are needed to compare augmentation-oriented designs with realism-oriented designs, not only in terms of task performance but also in terms of the experiential consequences that emerge.
From our perspective, there is great potential in moving beyond realism in VR design by offering users augmenting capabilities – a “positive unreality”. While realism is technologically difficult to achieve, augmentation-oriented design can create lasting value in a variety of ways by allowing users to experience abilities and environments that are impossible in the real world. It also plays to the strengths of VR over other media by exploiting its potential for immersion and presence. However, further empirical work is needed to fully understand the benefits and challenges of augmentation-oriented design. Promisingly, an experimental vignette study [53] compared augmentation-oriented designs with reality-oriented designs and found that the resulting augmentation experience is positively related to key experiential factors, such as positive affect or need fulfillment. Further research is needed to examine the experiential consequences of augmentation experiences and how well insights gained through supernatural abilities in VR translate to the real world. Such work can also provide new instruments (e.g., questionnaires) to reliably measure augmentation experience and thus make it a verifiable design goal.
Acknowledgements
This work was funded by the
Appendix
Longer descriptions of the enacted concepts from the workshops
| Group | Concept title | Concept description |
|---|---|---|
| 1 | Audiovisual Journey | Concerts in virtual reality become a completely different experience than real-world concerts. Instead of a stage or seats, artists take their audiences on 'audiovisual journeys'. However, the concert remains a social experience and one can experience it together in a friend group. To accommodate this, the event starts in gathering room, where everyone comes together, slowly get used to the VR and can talk about their anticipation. To start, the artists bring their audience into a new environment. This environment is changeable depending on the song. It can be likened to music videos, but more interactive and immersive. The artists can transport the audience through these different worlds, tell stories, but can also influence the speed and progression based on their audience's reactions. A show might only last much shorter than a regular concert – a few songs in different settings might be enough. A setting can have a unique style, fitting with the song – think of music videos in which the protagonist moves through a stylized and scribbled world – only now the audience is in this role. When the show is over, the audience returns into the 'lobby' in which they can exchange thoughts about the experience. To vividly remember, they can go back in time and take their friends back to show them certain aspects. |
| 1 | Concert in Space | In virtual reality, a concert can take place anywhere imaginable – a fascinating and stimulating place to jointly experience a music performance could be in zero gravity: in space. Here, each member of an audience floats in space and can only navigate by magnetically pulling oneself to points of interest or be magnetically pulled by others. This influences especially the interaction with other members of the audience. One can loosely float together in magnetically connected groups, or even communicate with others by bumping into them. In this case, a haptic element reinforces the feeling of being bumped into by someone else, but also underline the sensation of the music (such as feeling the beat at one's body). The artists can also utilize these elements to create new sensations for their audience, in line with their music. They can also pull the audience to move to different places, while distinct centers of gravity between members of the audience stay intact. |
| 2 | Garden Party Planning | In a playful asynchronous VR/desktop application, players jointly organize and prepare a garden party. In the process, the environmental effect of their choices and actions in terms of climate change is extremely amplified, making changes to the weather and climate almost immediately visible. One player participates using a VR-HMD and a haptic device, while the other players join remotely from desktop computers. As a party is planned, many decisions negatively impact the climate, e.g., the choice of food. The player in the garden is then confronted with bad weather and horrible conditions, such as rising water levels or extreme weather events such as storms. To manage situations, the VR player can utilize powerful magical tools and powers, e.g., they can use a cardboard to fan back against the storm to make it milder. Sometimes the desktop users can also help along: the VR player can split themselves into multiple instances, to complete multiple tasks at once. In this case, the desktop players can help control the other instances of the VR player who see their other selves working for them. |
| 3 | Memory travelling | A grandmother is with her grandson on a vehicle that travels through time/space, and they virtually travel to a location and time that goes through a memory the grandmother once had. This concept allows someone, specifically the grandmother, to be able to share a story and memory with her grandchild that wouldn't otherwise be able to be done outside of simply explaining. By means of a remote control, the grandmother and grandchild can control how long and where they are during this trip to be able to fast-forward through memories or parts of their story, they might not want to focus on but can relive to an extent. Through pictures and memories stored within the HMD and with the means of the superpowers, they're able to travel to any story or place they want to think about. Furthermore, they wear a haptic glove that allows them to feel somewhat specific tactile sensations, like the feeling of holding hands with the grandmother's deceased husband. Based on emotions, the surroundings within the virtual environment also fluctuate, like when the grandmother were to be depressed by a certain aspect of the memory, the clouds become dark or the light within the environment dims. |
| 3 | Generational Magic Garden | A grandmother is with her granddaughter in a virtual garden where they both are planting all kinds of fruits and vegetables and having a garden experience. While the granddaughter is learning how to garden, the grandmother can help firsthand based on her previous experiences with gardening in real life. As such, the grandmother can go back in her memories and assist her granddaughter in becoming a master gardener while allowing her to ask questions and make mistakes but to immediately rectify them and without any consequences to the virtual garden. Within this environment, they're able to plant anything they want, they only have to decide what to plant and it's able to happen. Additionally, since the grandmother is older, their virtual avatars that they interact through are able to make movements much easier than what would be necessary to do in reality. For example, the grandmother would only have to bend over slightly in reality to be able to touch the floor in the virtual garden. The materials within the garden also have haptic sensations that feel as though they're real, like the dirt and leaves. Also, they're able to control the weather and physics using a magic wand, wherein if the granddaughter were to plant something the grandmother is allergic to, she can just make it go away. |
| 4 | Doctor's visit training | For this application, the user can experience a virtual doctors visit. In the first scenario, the user started off with a pleasant and seamless experience at the doctors where they were treated with the utmost respect. In the following scenario, they were treated as the opposite, with rude staff and personnel interactions throughout the experience. The user purposefully put themself into this type of scenario to be able to see and experience what something like that would be like, as they've never had one as such. This application allows its users to virtually run through various loops and extremes within different hospitals and doctors to prepare and understand how things might happen within those physical situations; being able to train for different scenarios allows for less anxiety and nervousness when visiting a new doctor's office. |
| 4 | Grocery shopping training | For this application, the user can have multiple virtual shopping experiences. In the first scenario, the user plays out a potential situation where they seem to know where everything is, they collect it, pay, and leave without a problem. In the following scenario, this time, the user is unsure of where the products are and is able to interact with an employee there and has a bit of a clumsy moment at the register afterwards. This application is meant to be able to see how things could play out in many different situations at the store so that one might be able to train for many different variables. In all these variants, the shopping environment can be manipulated and controlled to add or take out a multitude of different factors, like shopping carts, people, and even whole aisles of the store. |
| 5 | Movement Training with a Doppelgänger | A user stands in the virtual space of the application opposite of their virtual avatar. Within this space, the user is meant to copy the exact predefined movements of the avatar while their motions and movements are relayed into the app via direct feedback. The avatar can move in a way that is seen as the precise motion, so when the user isn't able to copy the movement as closely as humanly possible, for example squatting, the avatar's silhouette changes colors and certain symbols become visible, for example if the user's knees are going too far out, arrows will appear to indicate that your knees should be closer together. If the user so wishes, they can have the avatar continue the motion while they walk around the avatar and see how the movement is done from all angles and even control the space and time surrounding the avatar to control the desired aspects of the motion. |
| 5 | Virtual Exercise Garden | In a virtual environment, a young patient stands in an open world that has the potential to be changed and shifted as seen fit. To test desired movements of the person, three tasks are designed for them in a made-up garden-like environment: to walk through and avoid being pricked by a thorn bush, a weight balancing movement, and picking berries. While trying to accomplish these tasks, they were set up in the virtual space in such a way that odd but necessary movements are needed to get through. These movements aren't meant to be random, but instead are to test the subject's ability to complete the movement. For example, in the berry picking task the user can only pick them when they bend their knees in the desired way. In each of these tasks, the process and their results are being collected and monitored. |
Table 6: Long versions of the concept descriptions from the enactments.
The complete design card set
Find all design cards here or in the supplemental materials.
Footnotes
Computer program for the analysis of qualitative and mixed methods data; https://www.maxqda.com/products/maxqda-standard
1 Computer program for the analysis of qualitative and mixed methods data; https://www.maxqda.com/products/maxqda-standard
Footnote
Supplemental Material
Available for Download
This material contains a PDF of the design card set outlined in the paper.
- Parastoo Abtahi. 2021. From Illusions to Beyond-Real Interactions in Virtual Reality. In The Adjunct Publication of the 34th Annual ACM Symposium on User Interface Software and Technology, October 10, 2021. ACM, New York, NY, USA, 153–157. https://doi.org/10.1145/3474349.3477586Google Scholar
Digital Library
- Merwan Achibet, Adrien Girard, Anthony Talvas, Maud Marchal, and Anatole Lecuyer. 2015. Elastic-Arm: Human-scale passive haptic feedback for augmenting interaction and perception in virtual environments. In 2015 IEEE Virtual Reality (VR), March 25, 2015. IEEE, 63–68. https://doi.org/10.1109/VR.2015.7223325Google ScholarCross Ref
- Anastassia Andreasen, Michele Geronazzo, Niels Christian Nilsson, Jelizaveta Zovnercuka, Kristian Konovalov, and Stefania Serafin. 2019. Auditory Feedback for Navigation with Echoes in Virtual Environments: Training Procedure and Orientation Strategies. IEEE Transactions on Visualization and Computer Graphics 25, 5 (May 2019), 1876–1886. https://doi.org/10.1109/TVCG.2019.2898787Google ScholarCross Ref
- Anastassia Andreasen, Jelizaveta Zovnercuka, Kristians Konovalovs, Michele Geronazzo, Razvan Paisa, and Stefania Serafin. 2018. Navigate as a Bat. Real-Time Echolocation System in Virtual Reality. Proceedings of the 15th International Sound & Music Computing Conference (SMC2018) (2018), 192–199. https://doi.org/https://doi.org/10.5281/zenodo.1422555Google ScholarCross Ref
- Domna Banakou and Mel Slater. 2014. Body ownership causes illusory self-attribution of speaking and influences subsequent real speaking. Proceedings of the National Academy of Sciences 111, 49 (December 2014), 17678–17683. https://doi.org/10.1073/pnas.1414936111Google ScholarCross Ref
- Jeffrey Bardzell. 2011. Interaction criticism: An introduction to the practice. Interacting with Computers 23, 6 (November 2011), 604–621. https://doi.org/10.1016/j.intcom.2011.07.001Google ScholarDigital Library
- Oloff C. Biermann, Daniel Ajisafe, and Dongwook Yoon. 2022. Interaction Design for VR Applications: Understanding Needs for University Curricula; Interaction Design for VR Applications: Understanding Needs for University Curricula. CHI EA ’22: Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (April 2022), 1–7. https://doi.org/10.1145/3491101Google ScholarDigital Library
- Ann Blandford, Dominic Furniss, and Stephann Makri. 2016. Qualitative HCI Research: Going Behind the Scenes. Synthesis Lectures on Human-Centered Informatics 9, 1 (April 2016), 1–115. https://doi.org/10.2200/S00706ED1V01Y201602HCI034Google ScholarCross Ref
- Tobias Blum, Valerie Kleeberger, Christoph Bichlmeier, and Nassir Navab. 2012. mirracle: An augmented reality magic mirror system for anatomy education. In 2012 IEEE Virtual Reality (VR), March 2012. IEEE, 115–116. https://doi.org/10.1109/VR.2012.6180909Google ScholarDigital Library
- Jennifer Boger, Mahzar Eisapour, Shi Cao, and Laura Domenicucci. 2018. Participatory design of a virtual reality exercise for people with mild cognitive impairment. In Conference on Human Factors in Computing Systems - Proceedings, April 20, 2018. Association for Computing Machinery. https://doi.org/10.1145/3170427.3174362Google ScholarDigital Library
- Doug A. Bowman, Ryan P. McMahan, and Eric D. Ragan. 2012. Questioning naturalism in 3D user interfaces. Communications of the ACM 55, 9 (September 2012), 78–88. https://doi.org/10.1145/2330667.2330687Google ScholarDigital Library
- Evren Bozgeyikli, Andrew Raij, Srinivas Katkoori, and Rajiv Dubey. 2016. Point & Teleport Locomotion Technique for Virtual Reality. In Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play, October 15, 2016. ACM, New York, NY, USA, 205–216. https://doi.org/10.1145/2967934.2968105Google ScholarDigital Library
- Tim Brown. 2008. Design Thinking. Harvard business review, 84.Google Scholar
- Marion Buchenau and Jane Fulton Suri. 2000. Experience prototyping. In Proceedings of the 3rd conference on Designing interactive systems: processes, practices, methods, and techniques, August 2000. ACM, New York, NY, USA, 424–433. https://doi.org/10.1145/347642.347802Google ScholarDigital Library
- Colin Burns, Eric Dishman, William Verplank, and Bud Lassiter. 1994. Actors, hairdos & videotape - informance design. In Conference companion on Human factors in computing systems - CHI ’94, 1994. ACM Press, New York, New York, USA, 119–120. https://doi.org/10.1145/259963.260102Google ScholarDigital Library
- Huiwen Chang and Michael F. Cohen. 2017. Panning and Zooming High-Resolution Panoramas in Virtual Reality Devices. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, October 20, 2017. ACM, New York, NY, USA, 279–288. https://doi.org/10.1145/3126594.3126617Google ScholarDigital Library
- Pieter M. A. Desmet and Marc Hassenzahl. 2012. Towards happiness: Possibility-driven design. In Human-Computer Interaction: The Agency Perspective, Marielba Zacarias and José Valente de Oliveira (eds.). Springer, Heidelberg, Berlin, 3–27.Google Scholar
- Judith Dörrenbächer and Marc Hassenzahl. 2019. Changing perspective: A co-design approach to explore future possibilities of divergent hearing. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, May 02, 2019. ACM, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300259Google ScholarDigital Library
- Douglas C. Engelbart. 1962. Augmenting human intellect: A conceptual framework. Menlo Park, CA.Google Scholar
- Kevin Fan, Jochen Huber, Suranga Nanayakkara, and Masahiko Inami. 2014. SpiderVision. In Proceedings of the 5th Augmented Human International Conference, March 07, 2014. ACM, New York, NY, USA, 1–8. https://doi.org/10.1145/2582051.2582100Google ScholarDigital Library
- Eivind Flobak, Jo D. Wake, Joakim Vindenes, Smiti Kahlon, Tine Nordgreen, and Frode Guribye. 2019. Participatory design of VR scenarios for exposure therapy. In Conference on Human Factors in Computing Systems - Proceedings, May 02, 2019. Association for Computing Machinery. https://doi.org/10.1145/3290605.3300799Google ScholarDigital Library
- Joseph L. Gabbard, Deborah Hix, and J. Edward Swan. 1999. User-centered design and evaluation of virtual environments. IEEE Computer Graphics and Applications 19, 6 (November 1999), 51–59. https://doi.org/10.1109/38.799740Google ScholarDigital Library
- Nirit Gavish, Teresa Gutiérrez, Sabine Webel, Jorge Rodríguez, Matteo Peveri, Uli Bockholt, and Franco Tecchia. 2015. Evaluating virtual reality and augmented reality training for industrial maintenance and assembly tasks. Interactive Learning Environments 23, 6 (November 2015), 778–798. https://doi.org/10.1080/10494820.2013.815221Google ScholarCross Ref
- Michele Geronazzo, Luis S. Vieira, Niels Christian Nilsson, Jesper Udesen, and Stefania Serafin. 2020. Superhuman Hearing - Virtual Prototyping of Artificial Hearing: a Case Study on Interactions and Acoustic Beamforming. IEEE Transactions on Visualization and Computer Graphics 26, 5 (May 2020), 1912–1922. https://doi.org/10.1109/TVCG.2020.2973059Google ScholarCross Ref
- Jonathan Harth, Alexandra Hofmann, Mike Karst, David Kempf, Annelie Ostertag, Isabell Przemus, and Bernhard Schaefermeyer. 2018. Different Types of Users, Different Types of Immersion: A User Study of Interaction Design and Immersion in Consumer Virtual Reality. IEEE Consumer Electronics Magazine 7, 4 (July 2018), 36–43. https://doi.org/10.1109/MCE.2018.2816218Google ScholarCross Ref
- Eva Hornecker. 2010. Creative idea exploration within the structure of a guiding framework. In Proceedings of the fourth international conference on Tangible, embedded, and embodied interaction, January 24, 2010. ACM, New York, NY, USA, 101–108. https://doi.org/10.1145/1709886.1709905Google ScholarDigital Library
- Jonatan Hvass, Oliver Larsen, Kasper Vendelbo, Niels Nilsson, Rolf Nordahl, and Stefania Serafin. 2017. Visual realism and presence in a virtual reality game. In 2017 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON), June 2017. IEEE, 1–4. https://doi.org/10.1109/3DTV.2017.8280421Google ScholarCross Ref
- Victoria Interrante, Brian Ries, and Lee Anderson. 2007. Seven League Boots: A New Metaphor for Augmented Locomotion through Moderately Large Scale Immersive Virtual Environments. In 2007 IEEE Symposium on 3D User Interfaces, 2007. IEEE. https://doi.org/10.1109/3DUI.2007.340791Google ScholarCross Ref
- Robert J.K. Jacob, Audrey Girouard, Leanne M. Hirshfield, Michael S. Horn, Orit Shaer, Erin Treacy Solovey, and Jamie Zigelbaum. 2008. Reality-Based Interaction: A Framework for Post-WIMP Interfaces. In Proceeding of the twenty-sixth annual CHI conference on Human factors in computing systems - CHI ’08, January 01, 2008. ACM Press, New York, New York, USA, 201. https://doi.org/10.1145/1357054.1357089Google ScholarDigital Library
- Jason Jerald. 2015. The VR Book: Human-Centered Design for Virtual Reality. Association for Computing Machinery. https://doi.org/10.1145/2792790Google ScholarDigital Library
- Wee Sim Khor, Benjamin Baker, Kavit Amin, Adrian Chan, Ketan Patel, and Jason Wong. 2016. Augmented and virtual reality in surgery—the digital surgical environment: applications, limitations and legal pitfalls. Annals of Translational Medicine 4, 23 (December 2016), 454–454. https://doi.org/10.21037/atm.2016.12.23Google ScholarCross Ref
- Konstantina Kilteni, Jean-Marie Normand, Maria v. Sanchez-Vives, and Mel Slater. 2012. Extending Body Space in Immersive Virtual Reality: A Very Long Arm Illusion. PLoS ONE 7, 7 (July 2012), e40867. https://doi.org/10.1371/journal.pone.0040867Google ScholarCross Ref
- Pascal Knierim, Thomas Kosch, Gabrielle LaBorwit, and Albrecht Schmidt. 2020. Altering the Speed of Reality? In Proceedings of the Augmented Humans International Conference, March 16, 2020. ACM, New York, NY, USA, 1–5. https://doi.org/10.1145/3384657.3384659Google ScholarDigital Library
- Olli Koskinen, Ismo Rakkolainen, and Roope Raisamo. 2018. Gigapixel virtual reality employing live superzoom cameras. In Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology, November 28, 2018. ACM, New York, NY, USA, 1–2. https://doi.org/10.1145/3281505.3281586Google ScholarDigital Library
- Veronika Krauß, Alexander Boden, Leif Oppermann, and René Reiners. 2021. Current Practices, Challenges, and Design Implications for Collaborative AR/VR Application Development. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, May 06, 2021. ACM, New York, NY, USA, 1–15. https://doi.org/10.1145/3411764.3445335Google ScholarDigital Library
- Andrey Krekhov, Sebastian Cmentowski, Katharina Emmerich, and Jens Krüger. 2019. Beyond Human. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play, October 17, 2019. ACM, New York, NY, USA, 439–451. https://doi.org/10.1145/3311350.3347172Google ScholarDigital Library
- Alexander Kulik. 2009. Building on Realism and Magic for Designing 3D Interaction Techniques. IEEE Computer Graphics and Applications 29, 6 (November 2009), 22–33. https://doi.org/10.1109/MCG.2009.115Google ScholarCross Ref
- Joanna Kwiatkowska, Agnieszka Szóstek, and David Lamas. 2014. (Un)structured sources of inspiration. In Proceedings of the 13th Participatory Design Conference on Research Papers - PDC ’14, September 15, 2014. ACM Press, New York, New York, USA, 31–39. https://doi.org/10.1145/2661435.2661442Google ScholarDigital Library
- Jie Li, Guo Chen, Huib de Ridder, and Pablo Cesar. 2020. Designing a social VR clinic for medical consultations. In Conference on Human Factors in Computing Systems - Proceedings, April 25, 2020. Association for Computing Machinery. https://doi.org/10.1145/3334480.3382836Google ScholarDigital Library
- Yang Li, Jin Huang, Feng Tian, Hong-An Wang, and Guo-Zhong Dai. 2019. Gesture interaction in virtual reality. Virtual Reality & Intelligent Hardware 1, 1 (February 2019), 84–112. https://doi.org/10.3724/SP.J.2096-5796.2018.0006Google ScholarCross Ref
- Feng Liang, Stevanus Kevin, Holger Baldauf, Kai Kunze, and Yun Suen Pai. 2020. OmniView. In Proceedings of the Augmented Humans International Conference, March 16, 2020. ACM, New York, NY, USA, 1–10. https://doi.org/10.1145/3384657.3384796Google ScholarDigital Library
- Lara Maister, Mel Slater, Maria v. Sanchez-Vives, and Manos Tsakiris. 2015. Changing bodies changes minds: owning another body affects social cognition. Trends in Cognitive Sciences 19, 1 (January 2015), 6–12. https://doi.org/10.1016/j.tics.2014.11.001Google ScholarCross Ref
- Fabrizia Mantovani, Gianluca Castelnuovo, Andrea Gaggioli, and Giuseppe Riva. 2003. Virtual Reality Training for Health-Care Professionals. CyberPsychology & Behavior 6, 4 (August 2003), 389–395. https://doi.org/10.1089/109493103322278772Google ScholarCross Ref
- Jessica L. Maples-Keller, Carly Yasinski, Nicole Manjin, and Barbara Olasov Rothbaum. 2017. Virtual Reality-Enhanced Extinction of Phobias and Post-Traumatic Stress. Neurotherapeutics 14, 3 (July 2017), 554–563. https://doi.org/10.1007/s13311-017-0534-yGoogle ScholarCross Ref
- Philipp Mayring. 2022. Qualitative Content Analysis. Sage, London.Google Scholar
- Ryan P. McMahan, Chengyuan Lai, and Swaroop K. Pal. 2016. Interaction fidelity: The uncanny valley of virtual reality interactions. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2016. Springer Verlag, 59–70. https://doi.org/10.1007/978-3-319-39907-2_6Google ScholarCross Ref
- Christoph Meinel, Larry Leifer, and Hasso Plattner (Eds.). 2011. Design Thinking. Springer Berlin Heidelberg, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-13757-0Google ScholarCross Ref
- Florian Mueller, Martin R. Gibbs, Frank Vetere, and Darren Edge. 2014. Supporting the creative game design process with exertion cards. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, April 26, 2014. ACM, New York, NY, USA, 2211–2220. https://doi.org/10.1145/2556288.2557272Google ScholarDigital Library
- Miriam Mulders and Raphael Zender. 2021. An Academic Conference in Virtual Reality?-Evaluation of a SocialVR Conference. In 2021 7th International Conference of the Immersive Learning Research Network (iLRN), May 17, 2021. IEEE, 1–6. https://doi.org/10.23919/iLRN52045.2021.9459319Google ScholarCross Ref
- Takaki Murakami, Tanner Person, Charith Lasantha Fernando, and Kouta Minamizawa. 2017. Altered touch. In ACM SIGGRAPH 2017 Posters, July 30, 2017. ACM, New York, NY, USA, 1–2. https://doi.org/10.1145/3102163.3102225Google ScholarDigital Library
- Mahdi Nabioyuni and Doug A. Bowman. 2015. An Evaluation of the Effects of Hyper-Natural Components of Interaction Fidelity on Locomotion Performance in Virtual Reality. In International Conference on Artificial Reality and Telexistence and Eurographics Symposium on Virtual Environments, ICAT-EGVE 2015, 2015. Eurographics Association, 167–174. https://doi.org/10.2312/egve.20151325Google ScholarCross Ref
- Camille Nadal, Shane McCully, Kevin Doherty, Corina Sas, and Gavin Doherty. 2022. The TAC Toolkit: Supporting Design for User Acceptance of Health Technologies from a Macro-Temporal Perspective. In CHI Conference on Human Factors in Computing Systems, April 29, 2022. ACM, New York, NY, USA, 1–18. https://doi.org/10.1145/3491102.3502039Google ScholarDigital Library
- Robin Neuhaus, Ronda Ringfort-Felner, Shadan Sadeghian, and Marc Hassenzahl. 2024. To mimic reality or to go beyond? “Superpowers” in virtual reality, the experience of augmentation and its consequences. International Journal of Human-Computer Studies 181, (January 2024), 103165. https://doi.org/10.1016/j.ijhcs.2023.103165Google ScholarDigital Library
- Mark Newman, Birgitta C. M. Gatersleben, Kayleigh J. Wyles, and Eleanor Ratcliffe. 2022. The use of virtual reality in environment experiences and the importance of realism. Journal of Environmental Psychology 79, (February 2022), 101733. https://doi.org/10.1016/j.jenvp.2021.101733Google ScholarCross Ref
- William Odom, John Zimmerman, Scott Davidoff, Jodi Forlizzi, Anind K. Dey, and Min Kyung Lee. 2012. A fieldwork of the future with user enactments. In Proceedings of the Designing Interactive Systems Conference on - DIS ’12, 2012. ACM Press, New York, New York, USA, 338. https://doi.org/10.1145/2317956.2318008Google ScholarDigital Library
- Ekaterina Olshannikova, Aleksandr Ometov, Yevgeni Koucheryavy, and Thomas Olsson. 2015. Visualizing Big Data with augmented and virtual reality: challenges and research agenda. Journal of Big Data 2, 1 (December 2015), 22. https://doi.org/10.1186/s40537-015-0031-2Google ScholarCross Ref
- Randy Pausch, Jon Snoddy, Robert Taylor, Scott Watson, and Eric Haseltine. 1996. Disney's Aladdin. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques - SIGGRAPH ’96, 1996. ACM Press, New York, New York, USA, 193–203. https://doi.org/10.1145/237170.237257Google ScholarDigital Library
- Tabitha C. Peck, Sofia Seinfeld, Salvatore M. Aglioti, and Mel Slater. 2013. Putting yourself in the skin of a black avatar reduces implicit racial bias. Consciousness and Cognition 22, 3 (September 2013), 779–787. https://doi.org/10.1016/j.concog.2013.04.016Google ScholarCross Ref
- Michael Pfandler, Marc Lazarovici, Philipp Stefan, Patrick Wucherer, and Matthias Weigl. 2017. Virtual reality-based simulators for spine surgery: a systematic review. The Spine Journal 17, 9 (September 2017), 1352–1363. https://doi.org/10.1016/j.spinee.2017.05.016Google ScholarCross Ref
- Jaziar Radianti, Tim A. Majchrzak, Jennifer Fromm, and Isabell Wohlgenannt. 2020. A systematic review of immersive virtual reality applications for higher education: Design elements, lessons learned, and research agenda. Computers and Education 147, (April 2020). https://doi.org/10.1016/j.compedu.2019.103778Google ScholarDigital Library
- Roope Raisamo, Ismo Rakkolainen, Päivi Majaranta, Katri Salminen, Jussi Rantala, and Ahmed Farooq. 2019. Human augmentation: Past, present and future. International Journal of Human-Computer Studies 131, May (November 2019), 131–143. https://doi.org/10.1016/j.ijhcs.2019.05.008Google ScholarDigital Library
- Ismo Rakkolainen, Roope Raisamo, Matthew Turk, Tobias Hollerer, and Karri Palovuori. 2017. Extreme field-of-view for head-mounted displays. In 2017 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON), June 2017. IEEE, 1–4. https://doi.org/10.1109/3DTV.2017.8280417Google ScholarCross Ref
- Katja Rogers, Sukran Karaosmanoglu, Maximilian Altmeyer, Ally Suarez, and Lennart E. Nacke. 2022. Much Realistic, Such Wow! A Systematic Literature Review of Realism in Digital Games. In CHI Conference on Human Factors in Computing Systems, April 29, 2022. ACM, New York, NY, USA, 1–21. https://doi.org/10.1145/3491102.3501875Google ScholarDigital Library
- Robin S. Rosenberg, Shawnee L. Baughman, and Jeremy N. Bailenson. 2013. Virtual Superheroes: Using Superpowers in Virtual Reality to Encourage Prosocial Behavior. PLoS ONE 8, 1 (January 2013), e55003. https://doi.org/10.1371/journal.pone.0055003Google ScholarCross Ref
- Robin Roy and James P. Warren. 2019. Card-based design tools: a review and analysis of 155 card decks for designers and designing. Design Studies 63, (July 2019), 125–154. https://doi.org/10.1016/j.destud.2019.04.002Google ScholarCross Ref
- Jose Luis Rubio-Tamayo, Manuel Gertrudix Barrio, and Francisco García García. 2017. Immersive environments and virtual reality: Systematic review and advances in communication, interaction and simulation. Multimodal Technologies and Interaction 1. https://doi.org/10.3390/mti1040021Google ScholarCross Ref
- Shadan Sadeghian and Marc Hassenzahl. 2021. From Limitations to “Superpowers”: A Design Approach to Better Focus on the Possibilities of Virtual Reality to Augment Human Capabilities. In Designing Interactive Systems Conference 2021, June 28, 2021. ACM, New York, NY, USA, 180–189. https://doi.org/10.1145/3461778.3462111Google ScholarDigital Library
- Maria v. Sanchez-Vives and Mel Slater. 2005. From presence to consciousness through virtual reality. Nature Reviews Neuroscience 6, 4 (April 2005), 332–339. https://doi.org/10.1038/nrn1651Google ScholarCross Ref
- Antti Sand, Ismo Rakkolainen, Poika Isokoski, Jari Kangas, Roope Raisamo, and Karri Palovuori. 2015. Head-mounted display with mid-air tactile feedback. In Proceedings of the 21st ACM Symposium on Virtual Reality Software and Technology, November 13, 2015. ACM, New York, NY, USA, 51–58. https://doi.org/10.1145/2821592.2821593Google ScholarDigital Library
- Tomoya Sasaki, Kao-Hua Liu, Taiki Hasegawa, Atsushi Hiyama, and Masahiko Inami. 2019. Virtual Super-Leaping. In Proceedings of the 10th Augmented Human International Conference 2019, March 11, 2019. ACM, New York, NY, USA, 1–8. https://doi.org/10.1145/3311823.3311861Google ScholarDigital Library
- Jonas Schild, Sebastian Misztal, Beniamin Roth, Leonard Flock, Thomas Luiz, Dieter Lerner, Markus Herkersdorf, Konstantin Weaner, Markus Neuberaer, Andreas Franke, Claus Kemp, Johannes Pranqhofer, Sven Seele, Helmut Buhler, and Rainer Herpers. 2018. Applying Multi-User Virtual Reality to Collaborative Medical Training. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), March 24, 2018. IEEE, 775–776. https://doi.org/10.1109/VR.2018.8446160Google ScholarCross Ref
- Jonas Schjerlund, Kasper Hornbæk, and Joanna Bergström. 2021. Ninja Hands: Using Many Hands to Improve Target Selection in VR. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, May 06, 2021. ACM, New York, NY, USA, 1–14. https://doi.org/10.1145/3411764.3445759Google ScholarDigital Library
- Albrecht Schmidt. 2017. Augmenting Human Intellect and Amplifying Perception and Cognition. IEEE Pervasive Computing 16, 1 (January 2017), 6–10. https://doi.org/10.1109/MPRV.2017.8Google ScholarDigital Library
- Danny Schott, Patrick Saalfeld, Gerd Schmidt, Fabian Joeres, Christian Boedecker, Florentine Huettl, Hauke Lang, Tobias Huber, Bernhard Preim, and Christian Hansen. 2021. A VR/AR Environment for Multi-User Liver Anatomy Education. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR), March 01, 2021. IEEE, 296–305. https://doi.org/10.1109/VR50410.2021.00052Google ScholarCross Ref
- Jun'ichiro Seyama and Ruth S. Nagayama. 2007. The Uncanny Valley: Effect of Realism on the Impression of Artificial Human Faces. Presence: Teleoperators and Virtual Environments 16, 4 (August 2007), 337–351. https://doi.org/10.1162/pres.16.4.337Google ScholarDigital Library
- Ben Shneiderman. 2003. Why not make interfaces better than 3D reality? IEEE Computer Graphics and Applications 23, 6 (November 2003), 12–15. https://doi.org/10.1109/MCG.2003.1242376Google ScholarDigital Library
- Mel Slater. 2009. Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philosophical Transactions of the Royal Society B: Biological Sciences 364, 1535 (December 2009), 3549–3557. https://doi.org/10.1098/rstb.2009.0138Google ScholarCross Ref
- Yongbin Sun, Alexandre Armengol-Urpi, Sai Nithin Reddy Kantareddy, Joshua Siegel, and Sanjay Sarma. 2019. MagicHand: Interact with IoT Devices in Augmented Reality Environment. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), March 01, 2019. IEEE, 1738–1743. https://doi.org/10.1109/VR.2019.8798053Google ScholarCross Ref
- Alistair G. Sutcliffe, Charalambos Poullis, Andreas Gregoriades, Ireni Katsouri, Aimilia Tzanavari, and Kyriakos Herakleous. 2019. Reflecting on the Design Process for Virtual Reality Applications. International Journal of Human–Computer Interaction 35, 2 (January 2019), 168–179. https://doi.org/10.1080/10447318.2018.1443898Google ScholarCross Ref
- Carlos Velasco, Francisco Barbosa Escobar, Olivia Petit, and Qian Janice Wang. 2021. Impossible (Food) Experiences in Extended Reality. Frontiers in Computer Science 3, (August 2021). https://doi.org/10.3389/fcomp.2021.716846Google ScholarCross Ref
- Catalin Voss, Peter Washington, Nick Haber, Aaron Kline, Jena Daniels, Azar Fazel, Titas De, Beth McCarthy, Carl Feinstein, Terry Winograd, and Dennis Wall. 2016. Superpower glass. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct, September 12, 2016. ACM, New York, NY, USA, 1218–1226. https://doi.org/10.1145/2968219.2968310Google ScholarDigital Library
- Stefan Weber, David Weibel, and Fred W. Mast. 2021. How to Get There When You Are There Already? Defining Presence in Virtual Reality and the Importance of Perceived Realism. Frontiers in Psychology 12, (May 2021). https://doi.org/10.3389/fpsyg.2021.628298Google ScholarCross Ref
- Richard Wetzel, Tom Rodden, and Steve Benford. 2016. Developing Ideation Cards for Mixed Reality Game Design. In First International Joint Conference of DiGRA and FDG, August 2016. Abertay University, Dundee, Scotland.Google Scholar
- Christiane Wölfel and Timothy Merritt. 2013. Method Card Design Dimensions: A Survey of Card-Based Design Tools. In LNCS. 479–486. https://doi.org/10.1007/978-3-642-40483-2_34Google ScholarCross Ref
Index Terms
- Virtual Unreality: Augmentation-Oriented Ideation Through Design Cards
Recommendations
To mimic reality or to go beyond? “Superpowers” in virtual reality, the experience of augmentation and its consequences
Highlights- Augmentation-oriented VR designs lead to higher augmentation experience.
- Augmentation experience relates positively to key experiential factors, e.g., affect.
- Findings support augmenting users as a design strategy for VR.
- ...
AbstractVirtual reality (VR) offers the outstanding possibility to virtually extend human capabilities beyond reality (e.g., telekinesis). Previous work (Sadeghian and Hassenzahl, 2021) has proposed such augmentation as a core design principle of VR by ...
REVEAL: REal and Virtual Environments Augmentation Lab @ Bath
CHI EA '24: Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing SystemsThe REal and Virtual Environments Augmentation Lab (REVEAL) at the University of Bath is an interdisciplinary research centre focusing on immersive technology. REVEAL investigates the fundamental principles, applications and interaction techniques of ...
Envisioning Haptic Design for Immersive Virtual Environments
DIS' 20 Companion: Companion Publication of the 2020 ACM Designing Interactive Systems ConferenceCurrent techniques for haptics in immersive virtual environments (IVEs) allow users to perceive materials while exploring virtual surfaces. However, these experiences are usually restricted to the properties defined during the design phase of the IVE. ...
Comments