Abstract
We develop a method for predicting virtual object occlusions during hand-object interaction in mixed reality. We reconstruct a 3D hand model from a monocular RGB camera frame and use it to predict occlusions. The quality of these occlusions, evaluated using dice coefficient, is at the same level as reported for other depth sensor and hand model based methods. Our model runs in real-time on off-the-shelf smartphone and gives a plausible experience of grabbing, rotating and translating virtual objects, which we confirmed through a dedicated user study. Volume penetration is the main reason for occlusion errors. However, the suitability of the object to the user scenario and its appealing look can compensate for the occlusion errors, and make a positive mixed reality experience.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Battisti, C., Messelodi, S., Poiesi, F.: Seamless bare-hand interaction in mixed reality. In: 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), pp. 198–203 (2018). https://doi.org/10.1109/ISMAR-Adjunct.2018.00066
Chantara, W., Mun, J.H., Shin, D.W., Ho, Y.: Object tracking using adaptive template matching. IEIE Trans. Smart Process. Comput. 4, 1–9 (2015)
Feng, Q., Shum, H.P.H., Morishima, S.: Resolving hand-object occlusion for mixed reality with joint deep learning and model optimization. Comput. Anim. Virtual Worlds 31(4–5) (2020). https://doi.org/10.1002/cav.1956
Ge, L., et al.: 3D hand shape and pose estimation from a single RGB image. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10825–10834 (2019). https://doi.org/10.1109/CVPR.2019.01109
Lee, J., Kunii, T.L.: Model-based analysis of hand posture. IEEE Comput. Graph. Appl. 15(5), 77–86 (1995). https://doi.org/10.1109/38.403831
Liu, W., et al.: SSD: single shot MultiBox detector. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 21–37. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_2
Tang, X., Hu, X., Fu, C.W., Cohen-Or, D.: GrabAR: occlusion-aware grabbing virtual objects in AR. In: Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, UIST 2020, pp. 697–708. Association for Computing Machinery, New York (2020). https://doi.org/10.1145/3379337.3415835
Walton, D.R., Steed, A.: Accurate real-time occlusion for mixed reality. In: Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology, VRST 2017. Association for Computing Machinery, New York (2017). https://doi.org/10.1145/3139131.3139153
Zhang, X., Li, Q., Zhang, W., Zheng, W.: End-to-end hand mesh recovery from a monocular RGB image. CoRR abs/1902.09305 (2019). http://arxiv.org/abs/1902.09305
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Olshevsky, V., Bondarets, I., Trunov, O., Shcherbina, A. (2021). Realistic Occlusion of Virtual Objects Using Three-Dimensional Hand Model. In: Stephanidis, C., Antona, M., Ntoa, S. (eds) HCI International 2021 - Posters. HCII 2021. Communications in Computer and Information Science, vol 1420. Springer, Cham. https://doi.org/10.1007/978-3-030-78642-7_40
Download citation
DOI: https://doi.org/10.1007/978-3-030-78642-7_40
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-78641-0
Online ISBN: 978-3-030-78642-7
eBook Packages: Computer ScienceComputer Science (R0)