Jascha Achenbach
Marc Erich Latoschik
Mario Botsch
ABSTRACT
In this paper we present a complete pipeline to create ready-to-animate virtual humans by fitting a template character to a point set obtained by scanning a real person using multi-view stereo reconstruction. Our virtual humans are built upon a holistic character model and feature a detailed skeleton, fingers, eyes, teeth, and a rich set of facial blendshapes. Furthermore, due to the careful selection of techniques and technology, our reconstructed humans are quite realistic in terms of both geometry and texture. Since we represent our models as single-layer triangle meshes and animate them through standard skeleton-based skinning and facial blendshapes, our characters can be used in standard VR engines out of the box. By optimizing for computation time and minimizing manual intervention, our reconstruction pipeline is capable of processing whole characters in less than ten minutes.
Supplemental Material
- Jascha Achenbach, Eduard Zell, and Mario Botsch. 2015. Accurate Face Reconstruction through Anisotropic Fitting and Eye Correction. In Proc. of Vision, Modeling & Visualization. 1--8.Google Scholar
- Oleg Alexander, Mike Rogers, William Lambeth, Matt Chiang, and Paul Debevec. 2009. The Digital Emily Project: Photoreal Facial Modeling and Animation. In SIGGRAPH 2009 Courses. ACM, 1--15. Google ScholarDigital Library
- Brett Allen, Brian Curless, and Zoran Popović. 2003. The Space of Human Body Shapes: Reconstruction and Parameterization from Range Scans. ACM Transactions on Graphics 22, 3 (2003), 587--594. Google ScholarDigital Library
- Brett Allen, Brian Curless, Zoran Popović, and Aaron Hertzmann. 2006. Learning a Correlated Model of Identity and Pose-Dependent Body Shape Variation for Real-Time Synthesis. In Proc. of Eurographics Symposium on Computer Animation. 147--156.Google Scholar
- Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. 2005. SCAPE: Shape Completion and Animation of People. ACM Transactions on Graphics 24, 3 (2005), 408--416. Google ScholarDigital Library
- Akshay Asthana, Stefanos Zafeiriou, Shiyang Cheng, and Maja Pantic. 2013. Robust discriminative response map fitting with constrained local models. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition. 3444--3451. Google ScholarDigital Library
- Autodesk. 2014. Character Generator. https://charactergenerator.autodesk.com/. (2014).Google Scholar
- Domna Banakou and Mel Slater. 2014. Body ownership causes illusory self-attribution of speaking and influences subsequent real speaking. Proceedings of the National Academy of Sciences 111, 49 (2014), 17678--17683. Google ScholarCross Ref
- Ilya Baran and Jovan Popović. 2007. Automatic Rigging and Animation of 3D Characters. ACM Transactions on Graphics 26, 3, Article 72 (2007), 8 pages.Google ScholarDigital Library
- Thabo Beeler, Bernd Bickel, Paul Beardsley, Bob Sumner, and Markus Gross. 2010. High-quality Single-shot Capture of Facial Geometry. ACM Transactions on Graphics 29, 4 (2010), 1--9. Google ScholarDigital Library
- Volker Blanz and Thomas Vetter. 1999. A morphable model for the synthesis of 3D faces. In Proc. of SIGGRAPH. 187--194. Google ScholarDigital Library
- Federica Bogo, Michael J Black, Matthew Loper, and Javier Romero. 2015. Detailed full-body reconstructions of moving people from monocular RGB-D sequences. In Proc. of IEEE International Conference on Computer Vision. 2300--2308. Google ScholarDigital Library
- Federica Bogo, Javier Romero, Matthew Loper, and Michael J. Black. 2014. FAUST: Dataset and Evaluation for 3D Mesh Registration. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition. 3794--3801. Google ScholarDigital Library
- Sofien Bouaziz, Andrea Tagliasacchi, and Mark Pauly. 2014. Dynamic 2D/3D Registration. In Eurographics Tutorials.Google Scholar
- Sofien Bouaziz, Yangang Wang, and Mark Pauly. 2013. Online Modeling for Realtime Facial Animation. ACM Transactions on Graphics 32, 4, Article 40 (2013), 10 pages.Google ScholarDigital Library
- Samuel R Buss. 2004. Introduction to inverse kinematics with jacobian transpose, pseudoinverse and damped least squares methods. IEEE Journal of Robotics and Automation 17 (2004), 1--19.Google Scholar
- Chen Cao, Qiming Hou, and Kun Zhou. 2014a. Displaced Dynamic Expression Regression for Real-time Facial Tracking and Animation. ACM Transactions on Graphics 33, 4 (2014), 1--10.Google ScholarDigital Library
- Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou. 2014b. FaceWare-house: A 3D facial expression database for visual computing. IEEE Transactions on Visualization and Computer Graphics 20, 3 (2014), 413--425. Google ScholarDigital Library
- P. Ekman and W. Friesen. 1978. Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press.Google Scholar
- Andrew Feng, Dan Casas, and Ari Shapiro. 2015. Avatar Reshaping and Automatic Rigging Using a Deformable Model. In Proc. of ACM Motion in Games. 57--64. Google ScholarDigital Library
- Andrew Feng, Evan Suma Rosenberg, and Ari Shapiro. 2017. Just-in-time, viable, 3-D avatars from scans. Computer Animation and Virtual Worlds 28 (2017), 3--4. Google ScholarCross Ref
- Andrew Feng, Ari Shapiro, Wang Ruizhe, Mark Bolas, Gerard Medioni, and Evan Suma. 2014. Rapid Avatar Capture and Simulation Using Commodity Depth Sensors. In SIGGRAPH 2014 Talks. ACM. Google ScholarDigital Library
- Pablo Garrido, Michael Zollhöfer, Dan Casas, Levi Valgaerts, Kiran Varanasi, Patrick Pérez, and Christian Theobalt. 2016. Reconstruction of Personalized 3D Face Rigs from Monocular Video. ACM Transactions on Graphics 35, 3, Article 28 (2016), 15 pages.Google Scholar
- Abhijeet Ghosh, Graham Fyffe, Borom Tunwattanapong, Jay Busch, Xueming Yu, and Paul Debevec. 2011. Multiview Face Capture Using Polarized Spherical Gradient Illumination. ACM Transactions on Graphics 30, 6, Article 129 (2011), 10 pages.Google ScholarDigital Library
- Mar González-Franco, Daniel Perez-Marcos, Bernhard Spanlang, and Mel Slater. 2010. The contribution of real-time mirror reflections of motor actions on virtual body ownership in an immersive virtual environment. In Proc. of IEEE Virtual Reality Conference. 111--114. Google ScholarDigital Library
- P. Guan, A. Weiss, A. Balan, and M. J. Black. 2009. Estimating human shape and pose from a single image. In Proc. of International Conference on Computer Vision. 1381--1388.Google Scholar
- Nils Hasler, Carsten Stoll, Martin Sunkel, Bodo Rosenhahn, and H-P Seidel. 2009. A statistical model of human pose and body shape. Computer Graphics Forum 28, 2 (2009), 337--346. Google ScholarCross Ref
- David A. Hirshberg, Matthew Loper, Eric Rachlin, and Michael J. Black. 2012. Coregistration: Simultaneous Alignment and Modeling of Articulated 3D Shape. In Proc. of European Conference on Computer Vision. 242--255. Google ScholarDigital Library
- Berthold K. P. Horn. 1987. Closed-form solution of absolute orientation using unit quaternions. Journal of the Optical Society of America A 4, 4 (1987), 629--642. Google ScholarCross Ref
- Pei-Lun Hsieh, Chongyang Ma, Jihun Yu, and Hao Li. 2015. Unconstrained Realtime Facial Performance Capture. In Proc. of Computer Vision and Pattern Recognition. 1675--1683. Google ScholarCross Ref
- Alexandru Eugen Ichim, Sofien Bouaziz, and Mark Pauly. 2015. Dynamic 3D Avatar Creation from Hand-held Video Input. ACM Transactions on Graphics 34, 4, Article 45 (2015), 14 pages.Google ScholarDigital Library
- Tao Ju, Scott Schaefer, and Joe Warren. 2005. Mean Value Coordinates for Closed Triangular Meshes. ACM Transactions on Graphics 24, 3 (2005), 561--566. Google ScholarDigital Library
- Marc Latoschik, Daniel Roth, Dominik Gall, Jascha Achenbach, Thomas Waltemate, and Mario Botsch. 2017. The Effect of Avatar Realism in Immersive Social Virtual Realities. In Proc. of ACM Symposium on Virtual Reality Software and Technology. to appear.Google ScholarDigital Library
- Marc Erich Latoschik, Jean-Luc Lugrin, and Daniel Roth. 2016. FakeMi: a fake mirror system for avatar embodiment studies. In Proc. of ACM Virtual Reality Software and Technology. 73--76. Google ScholarDigital Library
- J. P. Lewis, Ken Anjyo, Taehyun Rhee, Mengjie Zhang, Fred Pighin, and Zhigang Deng. 2014. Practice and Theory of Blendshape Facial Models. In Eurographics 2014 - State of the Art Reports.Google Scholar
- Hao Li, Etienne Vouga, Anton Gudym, Linjie Luo, Jonathan T. Barron, and Gleb Gusev. 2013. 3D Self-portraits. ACM Transactions on Graphics 32, 6, Article 187 (2013), 9 pages.Google ScholarDigital Library
- Hao Li, Thibaut Weise, and Mark Pauly. 2010. Example-based Facial Rigging. ACM Transactions on Graphics 29, 4, Article 32 (2010), 6 pages.Google ScholarDigital Library
- Shu Liang, Ira Kemelmacher-Shlizerman, and Linda G. Shapiro. 2014. 3D Face Hallucination from a Single Depth Frame. In Proc. of International Conference on 3D Vision. 31--38.Google Scholar
- Matthew Loper, Naureen Mahmood, and Michael J. Black. 2014. MoSh: Motion and Shape Capture from Sparse Markers. ACM Transactions on Graphics 33, 6, Article 220 (2014), 13 pages.Google ScholarDigital Library
- Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. 2015. SMPL: A Skinned Multi-person Linear Model. ACM Transactions on Graphics 34, 6, Article 248 (2015), 16 pages.Google ScholarDigital Library
- Jean-Luc Lugrin, Johanna Latt, and Marc Erich Latoschik. 2015. Anthropomorphism and illusion of virtual body ownership. In Proc. of the 25th International Conference on Artificial Reality and Telexistence and 20th Eurographics Symposium on Virtual Environments. 1--8.Google ScholarDigital Library
- C. Malleson, M. Kosek, M. Klaudiny, I. Huerta, J. C. Bazin, A. Sorkine-Hornung, M. Mine, and K. Mitchell. 2017. Rapid one-shot acquisition of dynamic VR avatars. In Proc. of IEEE Virtual Reality Conference. 131--140. Google ScholarCross Ref
- Tabitha C Peck, Sofia Seinfeld, Salvatore M Aglioti, and Mel Slater. 2013. Putting yourself in the skin of a black avatar reduces implicit racial bias. Consciousness and cognition 22, 3 (2013), 779--787.Google Scholar
- Patrick Pérez, Michel Gangnet, and Andrew Blake. 2003. Poisson Image Editing. ACM Transactions on Graphics 22, 3 (2003), 313--318. Google ScholarDigital Library
- Leonid Pishchulin, Stefanie Wuhrer, Thomas Helten, Christian Theobalt, and Bernt Schiele. 2017. Building Statistical Shape Spaces for 3D Human Modeling. Pattern Recognition (2017), 276--286. Google ScholarDigital Library
- Gerard Pons-Moll, Sergi Pujades, Sonny Hu, and Michael Black. 2017. ClothCap: Seamless 4D Clothing Capture and Retargeting. ACM Transactions on Graphics 36, 4, Article 73 (2017), 15 pages.Google ScholarDigital Library
- Daniel Roth, Kristoffer Waldow, Felix Stetter, Gary Bente, Marc Erich Latoschik, and Arnulph Fuhrmann. 2016. SIAMC: a socially immersive avatar mediated communication platform. In Proc. of ACM Virtual Reality Software and Technology. 357--358. Google ScholarDigital Library
- Fuhao Shi, Hsiang-Tao Wu, Xin Tong, and Jinxiang Chai. 2014. Automatic Acquisition of High-fidelity Facial Performances Using Monocular Videos. ACM Transactions on Graphics 33, 6, Article 222 (2014), 13 pages.Google ScholarDigital Library
- Leonid Sigal, Alexandru O. Balan, and Michael J. Black. 2007. Combined discriminative and generative articulated pose and non-rigid shape estimation. In Proc. of International Conference on Neural Information Processing Systems. 1337--1344.Google Scholar
- Mel Slater, Bernhard Spanlang, Maria V Sanchez-Vives, and Olaf Blanke. 2010. First person experience of body transfer in virtual reality. PloS one 5, 5 (2010).Google Scholar
- Matthias Straka, Stefan Hauswiesner, Matthias Ruther, and Horst Bischof. 2012. Rapid Skin: Estimating the 3D Human Pose and Shape in Real-Time. In Proc. of International Conference on 3D Imaging, Modeling, Processing, Visualization Transmission. 41--48.Google ScholarDigital Library
- Jürgen Sturm, Erik Bylow, Fredrik Kahl, and Daniel Cremers. 2013. CopyMe3D: Scanning and Printing Persons in 3D. In Proc. of German Conference on Pattern Recognition. 405--414. Google ScholarCross Ref
- Robert W. Sumner and Jovan Popović. 2004. Deformation Transfer for Triangle Meshes. ACM Transactions on Graphics 23, 3 (2004), 399--405. Google ScholarDigital Library
- Justus Thies, Michael Zollhöfer, Matthias Nießner, Levi Valgaerts, Marc Stamminger, and Christian Theobalt. 2015. Real-time Expression Transfer for Facial Reenactment. ACM Transactions on Graphics 34, 6, Article 183 (2015), 14 pages.Google ScholarDigital Library
- J. Thies, M. Zollhöfer, M. Stamminger, C. Theobalt, and M. Nießner. 2016. Face2Face: Real-time Face Capture and Reenactment of RGB Videos. In Proc. of IEEE Computer Vision and Pattern Recognition. 2387--2395. Google ScholarDigital Library
- Jing Tong, Jin Zhou, Ligang Liu, Zhigeng Pan, and Hao Yan. 2012. Scanning 3D Full Human Bodies Using Kinects. IEEE Transactions on Visualization and Computer Graphics 18, 4 (2012), 643--650. Google ScholarDigital Library
- Aggeliki Tsoli, Naureen Mahmood, and Michael J. Black. 2014. Breathing Life into Shape: Capturing, Modeling and Animating 3D Human Breathing. ACM Transactions on Graphics 33, 4, Article 52 (2014), 11 pages.Google ScholarDigital Library
- Thibaut Weise, Sofien Bouaziz, Hao Li, and Mark Pauly. 2011. Realtime Performance-based Facial Animation. ACM Transactions on Graphics 30, 4, Article 77 (2011), 10 pages.Google ScholarDigital Library
- Alexander Weiss, David Hirshberg, and Michael J. Black. 2011. Home 3D Body Scans from Noisy Image and Range Data. In Proc. of IEEE International Conference on Computer Vision. 1951--1958. Google ScholarDigital Library
- Chenglei Wu, Derek Bradley, Markus Gross, and Thabo Beeler. 2016. An Anatomically-Constrained Local Deformation Model for Monocular Face Capture. ACM Transactions on Graphics 35, 4, Article 115 (2016), 12 pages.Google ScholarDigital Library
- Stefanie Wuhrer, Leonid Pishchulin, Alan Brunton, Chang Shu, and Jochen Lang. 2014. Estimation of Human Body Shape and Posture Under Clothing. Computer Vision and Image Understanding 127 (2014), 31--42. Google ScholarDigital Library
Index Terms
- Fast generation of realistic virtual humans
Recommendations
Realistic Virtual Humans from Smartphone Videos
VRST '20: Proceedings of the 26th ACM Symposium on Virtual Reality Software and TechnologyThis paper introduces an automated 3D-reconstruction method for generating high-quality virtual humans from monocular smartphone cameras. The input of our approach are two video clips, one capturing the whole body and the other providing detailed close-...
Virtual Human Representation and Communication in VLNet
The realism in participant representation in networked virtual environments involves two elements: believable appearance and realistic movements. Using virtual human figures for participant representation fulfills these functionalities with realism, as ...
Investigating Social Distances between Humans, Virtual Humans and Virtual Robots in Mixed Reality
AAMAS '18: Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent SystemsMixed reality environments offer new potentials for the design of compelling social interaction experiences with virtual characters. In this paper, we summarise initial experiments we are conducting in which we measure comfortable social distances ...
Comments