Thomas Whelan
Michael Goesele
Richard Szeliski
Skip Abstract Section
Skip Supplemental Material SectionAbstract
Planar reflective surfaces such as glass and mirrors are notoriously hard to reconstruct for most current 3D scanning techniques. When treated naïvely, they introduce duplicate scene structures, effectively destroying the reconstruction altogether. Our key insight is that an easy to identify structure attached to the scanner---in our case an AprilTag---can yield reliable information about the existence and the geometry of glass and mirror surfaces in a scene. We introduce a fully automatic pipeline that allows us to reconstruct the geometry and extent of planar glass and mirror surfaces while being able to distinguish between the two. Furthermore, our system can automatically segment observations of multiple reflective surfaces in a scene based on their estimated planes and locations. In the proposed setup, minimal additional hardware is needed to create high-quality results. We demonstrate this using reconstructions of several scenes with a variety of real mirrors and glass.
Supplemental Material
- Sameer Agarwal, Keir Mierle, and Others. 2018. Ceres Solver, http://ceres-solver.org. (2018).Google Scholar
- N. Arvanitopoulos, R. Achanta, and S. Süsstrunk. 2017. Single Image Reflection Suppression. In CVPR 2017. 1752--1760.Google Scholar
- J. Balzer, D. Acevedo-Feliz, S. Soatto, S. Höfer, M. Hadwiger, and J. Beyerer. 2014. Cavlectometry: Towards Holistic Reconstruction of Large Mirror Objects. In 2nd International Conference on 3D Vision (3DV). 448--455. Google ScholarDigital Library
- J. Balzer, S. Höfer, and J. Beyerer. 2011. Multiview specular stereo reconstruction of large mirror surfaces. In CVPR 2011. 2537--2544. Google ScholarDigital Library
- L. G. Brown. 1992. A Survey of Image Registration Techniques. Computing Surveys 24, 4 (December 1992), 325--376. Google ScholarDigital Library
- Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. 2017. Matterport3D: Learning from RGB-D Data in Indoor Environments. In 5th International Conference on 3D Vision (3DV).Google ScholarCross Ref
- Tongbo Chen, Michael Goesele, and Hans-Peter Seidel. 2006. Mesostructure from Specularity. In CVPR 2006, Vol. 2. 1825--1832. Google ScholarDigital Library
- Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. 2017. ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE.Google ScholarCross Ref
- A. DelPozo and S. Savarese. 2007. Detecting Specular Surfaces on Natural Images. In CVPR 2007.Google Scholar
- Yuanyuan Ding and Jingyi Yu. 2008. Recovering shape characteristics on near-flat specular surfaces. In CVPR 2008.Google Scholar
- J. Engel, V. Koltun, and D. Cremers. 2018. Direct Sparse Odometry. PAMI 40, 3 (2018), 611--625.Google ScholarCross Ref
- A. Fasano, M. Callieri, P. Cignoni, and R. Scopigno. 2003. Exploiting mirrors for laser stripe 3D scanning. In 3DIM 2003. 243--250.Google Scholar
- Paul Foster, Zhenghong Sun, Jong Jin Park, and Benjamin Kuipers. 2013. VisAGGE: Visible angle grid for glass environments. In CVPR 2013. 2213--2220.Google ScholarCross Ref
- C. Godard, P. Hedman, W. Li, and G. J. Brostow. 2015. Multi-view Reconstruction of Highly Specular Surfaces in Uncontrolled Environments. In 3rd International Conference on 3D Vision (3DV). 19--27. Google ScholarDigital Library
- Ivo Ihrke, Kiriakos N. Kutulakos, Hendrik P. A. Lensch, Marcus Magnor, and Wolfgang Heidrich. 2010. Transparent and Specular Object Reconstruction. Computer Graphics Forum 29, 8 (2010), 2400--2426.Google ScholarCross Ref
- B. Jacquet, C. Häne, K. Köser, and M. Pollefeys. 2013. Real-World Normal Map Capture for Nearly Flat Reflective Surfaces. In CVPR 2013. 713--720. Google ScholarDigital Library
- Jun Jiang, Renato Miyagusuku, Atsushi Yamashita, and Hajime Asama. 2017. Glass Confidence Maps Building Based on Neural Networks Using Laser Range-Finders for Mobile Robots. In IEEE/SICE International Symposium on System Integration.Google Scholar
- O. Kähler, V. Adrian Prisacariu, C. Yuheng Ren, X. Sun, P. Torr, and D. Murray. 2015. Very High Frame Rate Volumetric Integration of Depth Images on Mobile Devices. TVCG 21, 11 (Nov 2015), 1241--1250. Google ScholarDigital Library
- J. Kannala and S. S. Brandt. 2006. A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses. PAMI 28, 8 (Aug 2006), 1335--1340. Google ScholarDigital Library
- P.-F. Käshammer and A. Nüchter. 2015. Mirror identification and correction of 3D point clouds. The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences 40, 5 (2015), 109.Google Scholar
- U Klank, D. Carton, and M. Beetz. 2011. Transparent object detection and reconstruction on a mobile platform. In ICRA 2011. 5971--5978.Google Scholar
- Rainer Koch, Stefan May, Patrick Murmann, and Andreas Nüchter. 2017b. Identification of Transparent and Specular Reflective Material in Laser Scans to Discriminate Affected Measurements for Faultless Robotic SLAM. Journal of Robotics and Autonomous Systems (JRAS) 87 (2017), 296--312.Google ScholarCross Ref
- R. Koch, S. May, and A. Nüchter. 2017a. Effective distinction of transparent and specular reflective objects in point clouds of a multi-echo laser scanner. In ICAR 2017. 566--571.Google Scholar
- Brian Kulis and Michael I. Jordan. 2011. Revisiting k-means: New algorithms via Bayesian nonparametrics. arXiv preprint arXiv.1111.0352 (2011). Google ScholarDigital Library
- J. P. Lewis. 1995. Fast Normalized Cross-Correlation. In Vision Interface '95. Canadian Image Processing and Pattern Recognition Society.Google Scholar
- M. Liu, R. Hartley, and M. Salzmann. 2015. Mirror Surface Reconstruction from a Single Image. PAMI 37, 4 (April 2015), 760--773.Google ScholarDigital Library
- William E. Lorensen and Harvey E. Cline. 1987. Marching cubes: A high resolution 3D surface construction algorithm. In SIGGRAPH. ACM, 163--169. Google ScholarDigital Library
- D. Miyazaki, M. Kagesawa, and K. Ikeuchi. 2004. Transparent surface modeling from a pair of polarization images. PAMI 26, 1 (2004), 73--82. Google ScholarDigital Library
- R. Mur-Artal, J. M. M. Montiel, and J. D. Tardós. 2015. ORB-SLAM: A Versatile and Accurate Monocular SLAM System. IEEE Transactions on Robotics 31, 5 (Oct 2015), 1147--1163.Google ScholarDigital Library
- R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohi, J. Shotton, S. Hodges, and A. Fitzgibbon. 2011. KinectFusion: Real-time dense surface mapping and tracking. In ISMAR 2011. 127--136. Google ScholarDigital Library
- Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Marc Stamminger. 2013. Real-time 3D Reconstruction at Scale Using Voxel Hashing. ACM Trans. Graph. 32, 6, Article 169 (Nov. 2013), 11 pages. Google ScholarDigital Library
- Edwin Olson. 2011. AprilTag: A robust and flexible visual fiducial system. In ICRA 2011. 3400--3407.Google ScholarCross Ref
- Rui Rodrigues, João P. Barreto, and Urbano Nunes. 2010. Camera Pose Estimation Using Images of Planar Mirror Reflections. In ECCV 2010. 382--395. Google ScholarDigital Library
- YiChang Shih, D. Krishnan, F. Durand, and W. T. Freeman. 2015. Reflection removal using ghosting cues. In CVPR 2015. 3193--3201.Google Scholar
- Sudipta N. Sinha, Johannes Kopf, Michael Goesele, Daniel Scharstein, and Richard Szeliski. 2012. Image-based Rendering for Scenes with Reflections. ACM Trans. Graph. 31, 4, Article 100 (July 2012), 10 pages. Google ScholarDigital Library
- Julian Straub, Trevor Campbell, Jonathan P How, and John W Fisher. 2015. Small-variance nonparametric clustering on the hypersphere. In CVPR 2015. 334--342.Google ScholarCross Ref
- Marco Tarini, Hendrik P.A. Lensch, Michael Goesele, and Hans-Peter Seidel. 2005. 3D acquisition of mirroring objects using striped patterns. Graphical Models 67, 4 (2005), 233 -- 259. Google ScholarDigital Library
- Markus Unger, Thomas Pock, and Horst Bischof. 2008a. Interactive globally optimal image segmentation. Technical Report ICG-TR-08/02. Graz University of Technology.Google Scholar
- Markus Unger, Thomas Pock, Werner Trobin, Daniel Cremers, and Horst Bischof. 2008b. TVSeg - Interactive Total Variation Based Image Segmentation. In BMVC 2008.Google Scholar
- J. Wang and E. Olson. 2016. AprilTag 2: Efficient and robust fiducial detection. In IROS 2016. 4193--4198.Google Scholar
- Qiaosong Wang, Haiting Lin, Yi Ma, Sing Bing Kang, and Jingyi Yu. 2015. Automatic Layer Separation using Light Field Imaging. CoRR abs/1506.04721 (2015). arXiv:1506.04721 http://arxiv.org/abs/1506.04721Google Scholar
- Sven Wanner and Bastian Goldluecke. 2013. Reconstructing Reflective and Transparent Surfaces from Epipolar Plane Images. In GCPR 2013.Google ScholarCross Ref
- Tianfan Xue, Michael Rubinstein, Ce Liu, and William T. Freeman. 2015. A computational approach for obstruction-free photography. ACM Trans. Graph. 34, 4 (2015), 79:1--79:11. Google ScholarDigital Library
- Shao-Wen Yang and Chieh-Chih Wang. 2008. Dealing with laser scanner failure: Mirrors and windows. In ICRA 2008. 3009--3015.Google Scholar
- S. W Yang and C. C. Wang. 2011. On Solving Mirror Reflection in LIDAR Sensing. IEEE/ASME Transactions on Mechatronics 16, 2 (April 2011), 255--265.Google ScholarCross Ref
- Y. Zhang, M. Ye, D. Manocha, and R. Yang. 2017. 3D Reconstruction in the Presence of Glass and Mirrors by Acoustic and Visual Fusion. PAMI (2017).Google Scholar
- Zhengyou Zhang. 2000. A Flexible New Technique for Camera Calibration. PAMI 22, 11 (Nov. 2000), 1330--1334. Google ScholarDigital Library
Recommendations
Scalable image-based indoor scene rendering with reflections
This paper proposes a novel scalable image-based rendering (IBR) pipeline for indoor scenes with reflections. We make substantial progress towards three sub-problems in IBR, namely, depth and reflection reconstruction, view selection for temporally ...
Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography
A new paradigm, Random Sample Consensus (RANSAC), for fitting a model to experimental data is introduced. RANSAC is capable of interpreting/smoothing data containing a significant percentage of gross errors, and is thus ideally suited for applications ...
Full 3D reconstruction of transparent objects
Numerous techniques have been proposed for reconstructing 3D models for opaque objects in past decades. However, none of them can be directly applied to transparent objects. This paper presents a fully automatic approach for reconstructing complete 3D ...
Comments