Joon Hyub Lee
ABSTRACT
In this study, we explore a new way to complementarily utilize the immersive visual output of VR and the physical haptic input of a smartphone. In particular, we focus on interacting with distant virtual objects using a smartphone in a through-plane manner and present a novel selection technique that overcomes the binocular parallax that occurs in such an arrangement. In our proposed technique, when a user in the stereoscopic viewing mode needs to perform a distant selection, the user brings the fingertip near the screen of the mobile device, triggering a smoothly animated transition to the monoscopic touching mode. Using a novel proof-of-concept implementation that utilizes a transparent acrylic panel, we conducted a user study and found that the proposed technique is significantly quicker, more precise, more direct, and more intuitive compared to the ray casting baseline. Subsequently, we created VR applications that explore the rich and interesting use cases of the proposed technique.
Supplemental Material
Available for Download
zip
3606809.zip (66.2 MB)
Supplemental File
- Ferran Argelaguet and Carlos Andujar. 2009. Visual feedback techniques for virtual pointing on stereoscopic displays. In Proc. VRST ’09. 163–170. https://doi.org/10.1145/1643928.1643966Google Scholar
Digital Library
- Rahul Arora, Rubaiat Habib Kazi, Tovi Grossman, George Fitzmaurice, and Karan Singh. 2018. SymbiosisSketch: combining 2D & 3D sketching for designing detailed 3D objects in situ. In Proc. CHI ’18. Article 185, 15 pages. https://doi.org/10.1145/3173574.3173759Google ScholarDigital Library
- Domagoj Baričević, Cha Lee, Matthew Turk, Tobias Höllerer, and Doug A. Bowman. 2012. A hand-held AR magic lens with user-perspective rendering. In Proc. ISMAR ’12. 197–206. https://doi.org/10.1109/ISMAR.2012.6402557Google ScholarDigital Library
- Patrick Baudisch, Nathaniel Good, and Paul Stewart. 2001. Focus plus context screens: combining display technology with visualization techniques. In Proc. UIST ’01. 31–40. https://doi.org/10.1145/502348.502354Google ScholarDigital Library
- Verena Biener, Travis Gesslein, Daniel Schneider, Felix Kawala, Alexander Otte, Per Ola Kristensson, Michel Pahud, Eyal Ofek, Cuauhtli Campos, Matjaž Kljun, Klen Čopič Pucihar, and Jens Grubert. 2022. PoVRPoint: authoring presentations in mobile virtual reality. TVCG 28, 5 (May 2022), 2069–2079. https://doi.org/10.1109/TVCG.2022.3150474Google ScholarCross Ref
- Verena Biener, Daniel Schneider, Travis Gesslein, Alexander Otte, Bastian Kuth, Per Ola Kristensson, Eyal Ofek, Michel Pahud, and Jens Grubert. 2020. Breaking the screen: interaction across touchscreen boundaries in virtual reality for mobile knowledge workers. TVCG 26, 12 (Dec 2020), 3490–3502. https://doi.org/10.1109/TVCG.2020.3023567Google ScholarCross Ref
- Xiang Cao, Jacky Jie Li, and Ravin Balakrishnan. 2008. Peephole pointing: modeling acquisition of dynamically revealed targets. In Proc. CHI ’08. 1699–1708. https://doi.org/10.1145/1357054.1357320Google ScholarDigital Library
- Rajkumar Darbar, Arnaud Prouzeau, Joan Odicio-Vilchez, Thibault Lainé, and Martin Hachet. 2021. Exploring smartphone-enabled text selection in AR-HMD. In Proc. GI ’21. 117–126. https://doi.org/10.20380/GI2021.14Google ScholarCross Ref
- Tobias Drey, Jan Gugenheimer, Julian Karlbauer, Maximilian Milo, and Enrico Rukzio. 2020. VRSketchIn: exploring the design space of pen and tablet interaction for 3D sketching in virtual reality. In Proc. CHI ’20. Article 501, 14 pages. https://doi.org/10.1145/3313831.3376628Google ScholarDigital Library
- Danilo Gasques, Janet G. Johnson, Tommy Sharkey, and Nadir Weibel. 2019. What you sketch is what you get: quick and easy augmented reality prototyping with PintAR. In CHI ’19 Extended Abstracts. Article LBW1416, 6 pages. https://doi.org/10.1145/3290607.3312847Google ScholarDigital Library
- Travis Gesslein, Verena Biener, Philipp Gagel, Daniel Schneider, Per Ola Kristensson, Eyal Ofek, Michel Pahud, and Jens Grubert. 2020. Pen-based interaction with spreadsheets in mobile virtual reality. In Proc. ISMAR ’20. 361–373. https://doi.org/10.1109/ISMAR50242.2020.00063Google ScholarCross Ref
- Tovi Grossman and Ravin Balakrishnan. 2004. Pointing at trivariate targets in 3D environments. In Proc. CHI ’04. 447–454. https://doi.org/10.1145/985692.985749Google ScholarDigital Library
- Jens Grubert, Matthias Heinisch, Aaron Quigley, and Dieter Schmalstieg. 2015. MultiFi: multi fidelity interaction with displays on and around the body. In Proc. CHI ’15. 3933–3942. https://doi.org/10.1145/2702123.2702331Google ScholarDigital Library
- Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (task load index): results of empirical and theoretical research. Adv. Psychol., Vol. 52. 139–183. https://doi.org/10.1016/S0166-4115(08)62386-9Google ScholarCross Ref
- Sebastian Hubenschmid, Johannes Zagermann, Simon Butscher, and Harald Reiterer. 2021. STREAM: exploring the combination of spatially-aware tablets with augmented reality head-mounted displays for immersive analytics. In Proc. CHI ’21. Article 469, 14 pages. https://doi.org/10.1145/3411764.3445298Google ScholarDigital Library
- Ernst Kruijff, J. Edward Swan, and Steven Feiner. 2010. Perceptual issues in augmented reality revisited. In Proc. ISMAR ’10. 3–12. https://doi.org/10.1109/ISMAR.2010.5643530Google ScholarCross Ref
- Ricardo Langner, Marc Satkowski, Wolfgang Büschel, and Raimund Dachselt. 2021. MARVIS: combining mobile devices and augmented reality for visual data analysis. In Proc. CHI ’21. Article 468, 17 pages. https://doi.org/10.1145/3411764.3445593Google ScholarDigital Library
- Khanh-Duy Le, Tanh Quang Tran, Karol Chlasta, Krzysztof Krejtz, Morten Fjeld, and Andreas Kunz. 2021. VXSlate: exploring combination of head movements and mobile touch for large virtual display interaction. In Proc. DIS ’21. 283–297. https://doi.org/10.1145/3461778.3462076Google ScholarDigital Library
- Joon Hyub Lee and Seok-Hyung Bae. 2013. Binocular cursor: enabling selection on transparent displays troubled by binocular parallax. In Proc. CHI ’13. 3169–3172. https://doi.org/10.1145/2470654.2466433Google ScholarDigital Library
- Joon Hyub Lee, Seok-Hyung Bae, Jinyung Jung, and Hayan Choi. 2012. Transparent display interaction without binocular parallax. In UIST ’12 Adjunct. 97–98. https://doi.org/10.1145/2380296.2380340Google ScholarDigital Library
- Joon Hyub Lee, Donghyeok Ma, Haena Cho, and Seok-Hyung Bae. 2021. Post-Post-It: a spatial ideation system in VR for overcoming limitations of physical Post-it notes. In CHI ’21 Extended Abstracts. Article 300, 7 pages. https://doi.org/10.1145/3411763.3451786Google ScholarDigital Library
- Weizhou Luo, Eva Goebel, Patrick Reipschläger, Mats Ole Ellenberg, and Raimund Dachselt. 2021. Exploring and slicing volumetric medical data in augmented reality using a spatially-aware mobile device. In ISMAR ’21 Adjunct. 334–339. https://doi.org/10.1109/ISMAR-Adjunct54149.2021.00076Google ScholarCross Ref
- Akhmajon Makhsadov, Donald Degraen, André Zenner, Felix Kosmalla, Kamila Mushkina, and Antonio Krüger. 2022. VRySmart: a framework for embedding smart devices in virtual reality. In CHI ’22 Extended Abstracts. Article 358, 8 pages. https://doi.org/10.1145/3491101.3519717Google ScholarDigital Library
- Fabrice Matulic, Aditya Ganeshan, Hiroshi Fujiwara, and Daniel Vogel. 2021. Phonetroller: visual representations of fingers for precise touch input with mobile phones in VR. In Proc. CHI ’21. Article 129, 13 pages. https://doi.org/10.1145/3411764.3445583Google ScholarDigital Library
- Manuel Meier, Paul Streli, Andreas Fender, and Christian Holz. 2021. TapID: rapid touch interaction in virtual reality using wearable sensing. In Proc. VR ’21. 519–528. https://doi.org/10.1109/VR50410.2021.00076Google ScholarCross Ref
- Meta. 2020. Infinite Office. Retrieved Sep 1, 2022 from https://youtu.be/5_bVkbG1ZCoGoogle Scholar
- Mark R. Mine. 1995. Virtual environment interaction techniques. Technical Report TR95-018. UNC Chapel Hill CS Dept.1–18 pages.Google Scholar
- Roberto A. Montano-Murillo, Cuong Nguyen, Rubaiat Habib Kazi, Sriram Subramanian, Stephen DiVerdi, and Diego Martinez-Plasencia. 2020. Slicing-volume: hybrid 3D/2D multi-target selection technique for dense virtual environments. In Proc. VR ’20. 53–62. https://doi.org/10.1109/VR46266.2020.00023Google ScholarCross Ref
- Erwan Normand and Michael J. McGuffin. 2018. Enlarging a smartphone with AR to create a handheld VESAD (virtually extended screen-aligned display). In Proc. ISMAR ’18. 123–133. https://doi.org/10.1109/ISMAR.2018.00043Google ScholarCross Ref
- Jeffrey S. Pierce, Andrew S. Forsberg, Matthew J. Conway, Seung Hong, Robert C. Zeleznik, and Mark R. Mine. 1997. Image plane interaction techniques in 3D immersive environments. In Proc. I3D ’97. 39–ff.https://doi.org/10.1145/253284.253303Google ScholarDigital Library
- Carolin Reichherzer, Jack Fraser, Damien Constantine Rompapas, and Mark Billinghurst. 2021. SecondSight: a framework for cross-device augmented reality interfaces. In CHI ’21 Extended Abstracts. Article 234, 6 pages. https://doi.org/10.1145/3411763.3451839Google ScholarDigital Library
- Houssem Saidi, Emmanuel Dubois, and Marcos Serrano. 2021. HoloBar: rapid command execution for head-worn AR exploiting around the field-of-view interaction. In Proc. CHI ’21. Article 745, 17 pages. https://doi.org/10.1145/3411764.3445255Google ScholarDigital Library
- Dieter Schmalstieg, L. Miguel Encarnação, and Zsolt Szalavári. 1999. Using transparent props for interaction with the virtual table. In Proc. I3D ’99. 147–153. https://doi.org/10.1145/300523.300542Google ScholarDigital Library
- Martin Spindler, Wolfgang Büschel, and Raimund Dachselt. 2012. Use your head: tangible windows for 3D information spaces in a tabletop environment. In Proc. ITS ’12. 245–254. https://doi.org/10.1145/2396636.2396674Google ScholarDigital Library
- Hemant Bhaskar Surale, Aakar Gupta, Mark Hancock, and Daniel Vogel. 2019. TabletInVR: exploring the design space for using a multi-touch tablet in virtual reality. In Proc. CHI ’19. Article 13, 13 pages. https://doi.org/10.1145/3290605.3300243Google ScholarDigital Library
- Arda Ege Unlu and Robert Xiao. 2021. PAIR: phone as an augmented immersive reality controller. In Proc. VRST ’21. Article 27, 6 pages. https://doi.org/10.1145/3489849.3489878Google ScholarDigital Library
- Klen Čopič Pucihar, Paul Coulton, and Jason Alexander. 2014. The use of surrounding visual context in handheld AR: device vs. user perspective rendering. In Proc. CHI ’14. 197–206. https://doi.org/10.1145/2556288.2557125Google ScholarDigital Library
- Shengzhi Wu, Daragh Byrne, and Molly Wright Steenson. 2020. “Megereality”: leveraging physical affordances for multi-device gestural interaction in augmented reality. In CHI ’20 Extended Abstracts. Article INT008, 4 pages. https://doi.org/10.1145/3334480.3383170Google ScholarDigital Library
- Fengyuan Zhu and Tovi Grossman. 2020. BISHARE: exploring bidirectional interactions between smartphones and head-mounted augmented reality. In Proc. CHI ’20. Article 106, 14 pages. https://doi.org/10.1145/3313831.3376233Google ScholarDigital Library
Index Terms
- Stereoscopic Viewing and Monoscopic Touching: Selecting Distant Objects in VR Through a Mobile Device
Recommendations
Effective stereoscopic rendering for mobile VR
SA '16: SIGGRAPH ASIA 2016 PostersLately, the VR market is expanding rapidly with the advent of mobile phone based VR such as Google Cardboard, Samsung Gear VR, LG 360VR starting with VR(Virtual Reality) HMD(Head Mount Device) PC based VR using Stereoscopic rendering such as Oculus Rift,...
Physically touching and tasting virtual objects enhances the realism of virtual experiences
Experiment 1 explored the impact of physically touching a virtual object on how realistic the virtual environment (VE) seemed to the user. Subjects in a `no touch' group picked up a 3D virtual image of a kitchen plate in a VE, using a traditional 3D ...
VR Grabbers: Ungrounded Haptic Retargeting for Precision Grabbing Tools
UIST '18: Proceedings of the 31st Annual ACM Symposium on User Interface Software and TechnologyHaptic feedback in VR is important for realistic simulation in virtual reality. However, recreating the haptic experience for hand tools in VR traditionally requires hardware with precise actuators, adding complexity to the system. We propose Ungrounded ...
Comments