ABSTRACT
To understand how well a proposed augmented reality (AR) solution works, existing papers often conducted tailored and isolated evaluations for specific AR tasks, e.g., depth or lighting estimation, and compared them to easy-to-setup baselines, either using datasets or resorting to time-consuming data capturing. Conceptually simple, it can be extremely difficult to evaluate an AR system fairly and in scale to understand its real-world performance. The difficulties arise for three key reasons: lack of control of the physical environment, the time-consuming data capturing, and the difficulties to reproduce baseline results.
This paper presents our design of an AR experimentation platform, ExpAR, aiming to provide scalable and controllable AR experimentation. ExpAR is envisioned to operate as a standalone deployment or a federated platform; in the latter case, AR researchers can contribute physical resources, including scene setup and capturing devices, and allow others to time share these resources. Our design centers around the generic sensing-understanding-rendering pipeline and is driven by the evaluation limitations observed in recent AR systems papers. We demonstrate the feasibility of this vision with a preliminary prototype and our preliminary evaluations suggest the importance of further investigating different device capabilities to stream in 30 FPS.
- Kittipat Apicharttrisorn, Jiasi Chen, Vyas Sekar, Anthony Rowe, and Srikanth V. Krishnamurthy. 2023. Breaking Edge Shackles: Infrastructure-Free Collaborative Mobile Augmented Reality (SenSys '22). Association for Computing Machinery, New York, NY, USA, 1--15. Google ScholarDigital Library
- Ali J Ben Ali, Marziye Kouroshli, Sofiya Semenova, Zakieh Sadat Hashemifar, Steven Y Ko, and Karthik Dantu. 2022. Edge-SLAM: Edge-Assisted Visual Simultaneous Localization and Mapping. ACM Trans. Embed. Comput. Syst. 22, 1 (Oct. 2022), 1--31.Google ScholarDigital Library
- Ying Chen, Hazer Inaltekin, and Maria Gorlatova. 2023. AdaptSLAM: Edge-Assisted Adaptive SLAM with Resource Constraints via Uncertainty Minimization. In Proc. IEEE INFOCOM.Google ScholarCross Ref
- Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarte, Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces, Shahram Izadi, Adarsh Kowdle, Konstantine Tsotsos, and David Kim. 2020. DepthLab: Real-time 3D Interaction with Depth Maps for Mobile Augmented Reality. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST '20).Google ScholarDigital Library
- Dmitry Duplyakin, Robert Ricci, Aleksander Maricq, Gary Wong, Jonathon Duerig, Eric Eide, Leigh Stoller, Mike Hibler, David Johnson, Kirk Webb, Aditya Akella, Kuangching Wang, Glenn Ricart, Larry Landweber, Chip Elliott, Michael Zink, Emmanuel Cecchet, Snigdhaswin Kar, and Prabodh Mishra. 2019. The Design and Operation of CloudLab. In Proceedings of the USENIX Annual Technical Conference (ATC). 1--14.Google Scholar
- The Green Planet AR Experience. 2022. https://www.factory42.uk/greenplanetexperience.Google Scholar
- Hugging Face. 2023. https://huggingface.co/.Google Scholar
- AWS Device Farm. 2023. https://aws.amazon.com/device-farm/.Google Scholar
- Immersive Stream for XR (Preview). 2023. https://xr.withgoogle.com/.Google Scholar
- Yongjie Guan, Xueyu Hou, Nan Wu, Bo Han, and Tao Han. 2022. DeepMix: mobility-aware, lightweight, and hybrid 3D object detection for headsets. In Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services (MobiSys '22). 28--41.Google ScholarDigital Library
- Bo Han, Parth Pathak, Songqing Chen, and Lap-Fai Craig Yu. 2022. CoMIC: A Collaborative Mobile Immersive Computing Infrastructure for Conducting Multi-user XR Research. IEEE Netw. (2022), 1--9.Google Scholar
- Muhammad Huzaifa, Rishi Desai, Samuel Grayson, Xutao Jiang, Ying Jing, Jae Lee, Fang Lu, Yihan Pang, Joseph Ravichandran, Finn Sinclair, Boyuan Tian, Hengzhi Yuan, Jeffrey Zhang, and Sarita V Adve. 2021. ILLIXR: Enabling End-to-End Extended Reality Research. In 2021 IEEE International Symposium on Workload Characterization (IISWC). 24--38.Google Scholar
- Hyoukjun Kwon, Krishnakumar Nair, Jamin Seo, Jason Yik, Debabrata Mohapatra, Dongyuan Zhan, Jinook Song, Peter Capak, Peizhao Zhang, Peter Vajda, Colby Banbury, Mark Mazumder, Liangzhen Lai, Ashish Sirasao, Tushar Krishna, Harshit Khaitan, Vikas Chandra, and Vijay Janapa Reddi. 2023. XRBench: An extended reality (XR) machine learning benchmark suite for the metaverse. In Proceedings of Machine Learning and Systems, Vol. 5.Google Scholar
- Larry L. Peterson and Prof. David Culler. 2002. PlanetLab. https://planetlab.cs.princeton.edu/.Google Scholar
- Liangkai Liu, Ren Zhong, Aaron Willcock, Nathan Fisher, and Weisong Shi. 2023. An Open Approach to Energy-Efficient Autonomous Mobile Robots citation. In 2023 International Conference on Robotics and Automation (ICRA). IEEE Press, 7 pages.Google Scholar
- Z Liu, G Lan, J Stojkovic, Y Zhang, C Joe-Wong, and M Gorlatova. 2020. CollabAR: Edge-assisted Collaborative Image Recognition for Mobile Augmented Reality. In 2020 19th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN). 301--312.Google Scholar
- Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, Silvio Savarese, and Li Fei-Fei. 2018. ROBOTURK: A Crowd-sourcing Platform for Robotic Skill Learning through Imitation. In Conference on Robot Learning.Google Scholar
- Adithyavairavan Murali, Tao Chen, Kalyan Vasudev Alwala, Dhiraj Gandhi, Lerrel Pinto, Saurabh Gupta, and Abhinav Gupta. 2019. Py-Robot: An Open-source Robotics Framework for Research and Benchmarking. arXiv preprint arXiv:1906.08236 (2019).Google Scholar
- Oliver Kroemer. 2023. LoCoBot: An Open Source Low Cost Robot. http://www.locobot.org/.Google Scholar
- Siddhant Prakash, Alireza Bahremand, Linda D Nguyen, and Robert LiKamWa. 2019. GLEAM: An Illumination Estimation Framework for Real-time Photorealistic Augmented Reality on Mobile Devices. In Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services (Seoul, Republic of Korea) (MobiSys '19). Association for Computing Machinery, New York, NY, USA.Google ScholarDigital Library
- SunFounder. 2023. Raspberry Pi Ai Car Kit - PiCar-X. https://www.sunfounder.com/products/picar-x. Accessed: 2023-6-15.Google Scholar
- Albert Tung, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, Yuke Zhu, Li Fei-Fei, and Silvio Savarese. 2021. Learning Multi-Arm Manipulation Through Collaborative Teleoperation. In 2021 IEEE International Conference on Robotics and Automation (ICRA). 9212--9219.Google Scholar
- Jamie Watson, Mohamed Sayed, Zawar Qureshi, Gabriel J. Brostow, Sara Vicente, Oisin Mac Aodha, and Michael Firman. 2023. Virtual Occlusions Through Implicit Depth. In CVPR.Google Scholar
- Jingao Xu, Guoxuan Chi, Zheng Yang, Danyang Li, Qian Zhang, Qiang Ma, and Xin Miao. 2021. FollowUpAR: enabling follow-up effects in mobile AR applications. In Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys '21). Association for Computing Machinery, 1--13.Google ScholarDigital Library
- Juheon Yi and Youngki Lee. 2020. Heimdall: mobile GPU coordination platform for augmented reality applications. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking (London, United Kingdom) (MobiCom '20, Article 35). Association for Computing Machinery, New York, NY, USA, 1--14.Google ScholarDigital Library
- Jinrui Zhang, Huan Yang, Ju Ren, Deyu Zhang, Bangwen He, Ting Cao, Yuanchun Li, Yaoxue Zhang, and Yunxin Liu. 2022. MobiDepth: real-time depth estimation using on-device dual cameras. In Proceedings of the 28th Annual International Conference on Mobile Computing And Networking (MobiCom '22).Google ScholarDigital Library
- Wenxiao Zhang, Bo Han, and Pan Hui. 2022. SEAR: Scaling Experiences in Multi-user Augmented Reality. IEEE Trans. Vis. Comput. Graph. 28, 5 (May 2022), 1982--1992.Google ScholarCross Ref
- Yunfan Zhang, Tim Scargill, Ashutosh Vaishnav, Gopika Premsankar, Mario Di Francesco, and Maria Gorlatova. 2022. InDepth: Real-time Depth Inpainting for Mobile Augmented Reality. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 6, 1 (March 2022), 1--25.Google ScholarDigital Library
- Yiqin Zhao, Sean Fanello, and Tian Guo. 2023. Multi-Camera Lighting Estimation for Photorealistic Front-Facing Mobile Augmented Reality. In Proceedings of the 24th International Workshop on Mobile Computing Systems and Applications (Newport Beach, California) (HotMobile '23). Association for Computing Machinery, New York, NY, USA, 68--73. Google ScholarDigital Library
- Yiqin Zhao and Tian Guo. 2020. PointAR: Efficient Lighting Estimation for Mobile Augmented Reality. In The European Conference on Computer Vision (ECCV '20). 678--693.Google Scholar
- Yiqin Zhao and Tian Guo. 2021. Xihe: a 3D vision-based lighting estimation framework for mobile augmented reality. In Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys '21). 28--40.Google ScholarDigital Library
- Yiqin Zhao, Chongyang Ma, Haibin Huang, and Tian Guo. 2022. LITAR: Visually Coherent Lighting for Mobile Augmented Reality. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 6, 3 (Sept. 2022), 1--29.Google ScholarDigital Library
Index Terms
- Toward Scalable and Controllable AR Experimentation
Recommendations
Exhibition approach using an AR and VR pillar
SA '17: SIGGRAPH Asia 2017 Mobile Graphics & Interactive ApplicationsThis demonstration presents a development of an Augmented Reality (AR) and Virtual Reality (AR) pillar, a novel approach for showing AR and VR content in a public setting. A pillar in a public exhibition venue was converted to a four-sided AR and VR ...
Requirements for Virtualization of AR Displays within VR Environments
Proceedings, Part I, of the 6th International Conference on Virtual, Augmented and Mixed Reality. Designing and Developing Virtual and Augmented Environments - Volume 8525Everybody has been talking about new emerging products in augmented reality AR and their potential to enhance our daily life and work. The AR technology has been around for quite a while and various use cases have been thought and tested. Clearly, the ...
Membrane AR: varifocal, wide field of view augmented reality display from deformable membranes
SIGGRAPH '17: ACM SIGGRAPH 2017 Emerging TechnologiesAccommodative depth cues, a wide field of view, and ever-higher resolutions present major design challenges for near-eye displays. Optimizing a design to overcome one of them typically leads to a trade-off in the others. We tackle this problem by ...
Comments