Immersive In-Situ Prototyping: Influence of Real-World Context on Evaluating Future Pedestrian Interfaces in Virtual Reality

2.1 Immersive Real-World Video Applications

360-degree videos, offering the ability to capture reality in panoramic views, have become an increasingly popular technique for creating immersive experiences [19, 35, 46, 49]. Since 360-degree videos contain ample on-site information, they have been utilised in areas like tourism (e.g., cultural heritage visiting [3], destination promotion [47]), education (e.g., remote lecture [19], surgical training [49]) and journalism [22, 43]. Empirical studies in these fields have discovered that immersive 360-degree videos, i.e., viewing via HMDs, provide users with a high visual-audio realism [38, 46, 49, 50], sense of presence [19, 35, 43, 47], and situational awareness [46, 50]. Their applications in experiential media have been found to be engaging [3, 19, 38] and effective for storytelling [22, 43].

As 3D development platforms (e.g., Unity 3D², Unreal Engine³) currently provide vast libraries and ease of deployment to various devices, researchers have started to explore methods to augment solely 360-degree video-based environments with virtual content. Hoggenmueller and Tomitsch [16] proposed the concept of “hyperreal prototyping” for urban pervasive displays, referring to the potential of such techniques in creating VR simulations where the distinction between the virtual and the physical becomes blurred. Similarly, Lee et al. [21] proposed “augmented virtual reality” for comparing interior design plans, emphasising real-world videos could help enhance the realism of completely CG VR. Using 360-degree videos to simulate a presence in the real world, some studies have prototyped AR experiences [5, 33] or added UI elements for interaction purposes [2, 3, 18, 50]. Our research builds on the conceptual and empirical foundations in literature and contributes to prototyping pedestrian experiences with HMIs in urban traffic situations.

2.2 Simulate Human-Machine Interfaces in Traffic

VR simulators are increasingly recognised for their flexibility and safety for pedestrian research [9, 40], allowing for the creation of diverse traffic scenarios with reduced time, cost, and safety risks compared to physical setups [10, 27, 31]. They are also useful for developing mockups of speculative HMI concepts that are difficult to physically implement with existing technologies, and therefore, facilitating early user feedback and concept refinement [11, 25, 32, 41]. Studies also highlight the importance of contextual setups in VR; for example, the visual realism and social atmosphere of VR environments can influence experiential qualities like sense of presence, level of comfort, and feeling of naturalness [36, 37, 40]. Real-world videos provide authentic representations of reality and hence are often employed in traffic research to increase the ecological validity of simulations, such as monitor-based videos [1, 26], projector-based immersive “CAVE” [14], and 360-degree video-based VR [8]. The latter has gained increasing attention in recent years, demonstrating that immersive real-world videos are effective in conveying contextual information with high visual fidelity, spatial presence, and engagement [13, 17, 48].

A few studies related to traffic HMIs have started to apply rendered overlays onto real-world video-based VR, including driving simulators (interior UIs [15, 48] and other on-road cars [48]) and AV–pedestrian HMIs (preprocessed static interfaces [42] and synthesised sounds [13]). However, so far, there has been no empirical evaluation of dynamic visual overlays of pedestrian interfaces that are integrated into immersive real-world videos. Furthermore, it is unclear how realistic environments can impact pedestrian evaluations of futuristic HMI proposals.

3.1 Case Study Context

Investigating HMIs that can support pedestrian safety has gained considerable research attention in the last decade due to the rising of autonomous systems in the urban mobility infrastructure [10, 31]. This is motivated by the critical role of HMIs in conveying vehicle intention and facilitating pedestrian interactions [4, 12]. Considering future traffic as an intelligent, interconnected network, the Drone–Pedestrian study investigated how drone-based interfaces could guide pedestrians through dangerous road situations, utilising the birds-eye view advantage of drones for traffic monitoring [16, 20]. Building upon this body of work, the AV–Pedestrian study explored design options of AV–pedestrian communication interfaces in highly urbanised areas [27, 40]. The speculative nature of the HMIs involved in both case studies introduced difficulties in evaluating them in the real world. Therefore, the immersive in-situ prototypes provided opportunities to collect early user feedback with the inclusion of real-world contexts.

Figure 2:

View Figure

Figure 3:

View Figure

3.2 Prototype Development

We summarised the development procedures based on the two case studies and present an overview of the process in Figure 2. Figure 3 reports details of the prototype setups. To record the 360-degree videos, we selected filming locations in a neighbourhood close to the city centre and used Insta360 Pro 2 as our video and sound recording device. The Drone–Pedestrian study contained two scenarios requiring participants to cross back and forth respectively on a busy public road next to a popular park. The AV–Pedestrian study involved participants walking down a pedestrianised corridor connecting a main road to a university campus. We obtained the 3D quadcopter model from Unity Asset Store and created the passenger transport pod using Autodesk 3ds Max (as a replica of one of our university’s real-world AVs). For the HMIs, a variety of interface modalities were encompassed by the two studies, ranging from displays to projections to tile changes (see Figure 3). Besides the drone displays and drone projections, which were modelled in Blender, the rest of the interfaces were developed using Unity 3D libraries.

We developed the 3D scenes integrating the 360-degree videos and the virtual 3D objects using Unity 3D (see Figure 4 for final scenes). To simulate a 360-degree video as background environment, we applied the video as a render texture to a panoramic skybox material. The viewers would be able to see the environment from the perspective of the 360-degree camera that filmed the video. Based on the design scenarios, we mapped the 3D objects into the 3D skybox environment with geometric adjustments, including scaling, position and rotation, to overlay the objects in relation to the spatial layout in the video. Further, we animated the 3D objects, e.g., changing their movements or appearances, using coding scripts and Unity built-in animators. In this process, we repeatedly adjusted parameters of the animations and of the geometric or visual properties of the objects to achieve good synchronisations with the videos. Finally, the prototypes in both studies were deployed to Oculus Quest 2 for user evaluations.

Figure 4:

View Figure

3.3 User Evaluation

4.1 Sense of Presence

Results of the ITC-SOPI questionnaire (see Figure 5) show similar high ratings for the perceived naturalness or the ecological validity of the VR environments for both studies. Similarly, engagement ratings are generally high across the two studies. While spatial presence receives above middle ratings for both studies, the rating for the AV–Pedestrian study is lower than that for the Drone–Pedestrian study. Negative effects are low for both studies.

Figure 5:

View Figure

4.2 Influence of Real-World Contexts

The qualitative analysis from both studies revealed three common patterns in how participants reacted to the prototypes.

5.1 Implications for Using Immersive In-Situ Prototyping

Our results suggest that immersive VR environments created from real-world recordings can enhance the realism of scenes through their high naturalness and familiarity. Based on prior studies using 360-degree video-based VR, immersive environments recorded from reality might have inherent advantages in ecological validity compared to those synthesised by computers, even in high visual fidelity [17]. Creating interaction scenarios in good naturalness can be important for evaluating traffic HMIs, as it can reduce potential uncanny valley effects of avatars or distractions from the novelty of virtual simulations [37, 40]. Furthermore, since the 360-degree video method supports conveying narratives with high plausibility [22, 43], it can be used to set up various environmental or social aspects often considered in testing pedestrian interfaces, such as the influence of other pedestrians [9, 27].

Our results further indicate that the familiarity of settings facilitated participants to immerse themselves as pedestrians and, furthermore, explicitly recognised the scenes as part of their daily lives and actively related to their everyday behaviours when assessing the futuristic interfaces. Such approaches open up opportunities for inquiring user requirements using representations of local contexts when conducting actual field studies is not feasible. This could serve as a useful phase in iterative design processes, for example, supporting the testing of early proposals with socio-cultural considerations, such as local traffic norms. With continued investigations, the immersive in-situ prototyping technique holds promise to assist the broader user-centred design research, as prior work using similar prototyping methods suggests its potential for user-engaged design [21], co-creation [50], and interface learning [35, 50].

5.2 Technical Considerations

In examining the efficacy of integrating 360-degree videos and virtual overlays, we observed that most participants did not emphasise or distinctly mention the technical aspect of overlaying effects in our VR simulations. Their qualitative feedback around VR experiences primarily highlighted the feeling of realism in the scenes in general. We suspect two reasons behind this observation: (1) participants were mostly captivated by the novelty of those interfaces; and (2) they focused on the assigned tasks, which diverted their attention from analysing the technical aspects of the simulations. Despite this, we found indications of good integration of the two “materials”: (1) most participants appeared to naturally engage with the scenes and express their views on the interfaces without any hindrance; (2) there were a few comments on the relationship between the rendered AV (including its interfaces) and the other real-world pedestrians in the videos, suggesting the perception that the virtual and the real elements were in the same world. Nevertheless, we report below technical challenges in blending the two materials, along with participant feedback about the potential unnaturalness associated with these challenges.

A notable challenge was the spatial mapping of the two materials. For example, we found that there could be slight discrepancies between the scale of objects at varying distances from the viewer (i.e., the camera’s position). This phenomenon was primarily due to the 360-degree videos being projected onto a spherical surface, i.e., the panoramic skybox. Therefore, a practical solution that we employed was utilising the player’s egocentric perspective, as seen in Unity’s game mode, for closely tracking the overlaying effects during the video playback. This approach enabled us to identify and and adjust (e.g., through modifying scripts or animators) any misalignment at specific video moments. Nevertheless, three participants noted the feeling of the AV not being firmly attached to the ground, while this sensation was not observed in the Drone study, possibly because the drone was operating in the air. This could also explain why the spatial presence in the AV study was lower than that in the drone study (Figure 5). Other reasons might include the lower visibility during the night time, the different area sizes covered by the interfaces, among other factors.

Furthermore, to enhance the natural appearance of overlays, it is essential to accurately simulate natural phenomena that can support the intended interpretation of the modality. For example, a successful implementation in our work was the reflection of light on the AV’s wheels from the paving tile lighting, which effectively observed by participants as lights emanating from the ground. Nonetheless, one participant mentioned the drone projections might be too bright to be considered fully realistic, despite the increased transparency of those projections being implemented.

5.3 Limitations and Future Work

Since immersive 360-degree videos are not inherently interactive beyond the viewer’s ability to turn head and look in all directions [50], interactions initiated by pedestrians in our studies were represented through key-pressing on controllers (Drone) and through the think-aloud protocol (AV). While the two methods still allowed us to collect data essential to indicate participants’ decision-making, some participants expressed the desire to physically move around the environments. We currently experiment with (1) filming multiple videos at various positions to support users to move through the space and (2) enriching controller- or gesture-based commands for interaction with the dynamic overlays [2]. Future research could explore methods with more sophisticated technical setups, such as live streaming 360-degree videos [34] with dynamic insertion of real-time objects [39].