Skip to main content
SpringerLink
Log in

Real-time indoor scene reconstruction with Manhattan assumption

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper presents a novel end-to-end system for real-time indoor scene reconstruction, which outperforms traditional image feature point-based method and dense geometry correspondence-based method in handling indoor scenes with less texture and geometry features. In our method, we fully explore the Manhattan assumption, i.e. scenes are majorly consisted with planar surfaces with orthogonal normal directions. Given an input depth frame, we first extract dominant axes coordinates via principle component analysis which involves the orthogonal prior and reduce the influence of noise. Then we calculate the coordinates of dominant planes (such as walls, floor and ceiling) in the coordinates using mean shift. Finally, we compute the camera orientation and reconstruct the scene by proposing a fast scheme based on matching the dominant axes and planes to the previous frame. We have tested our approach on several datasets and demonstrated that it outperforms some well known existing methods in these experiments. The performance of our method is also able to meet the requirement of real-time with an unoptimized CPU implementation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Arun KS, Huang TS, Blostein SD (1987) Least-squares fitting of two 3-d point sets. IEEE Trans Pattern Anal Mach Intell 9(5):698–700

    Article  Google Scholar 

  2. Besl PJ, Mckay ND (1992) A method for registration of 3-D shapes. IEEE Trans Pattern Anal Mach Intell 14(2):239–256

    Article  Google Scholar 

  3. Calonder M, Lepetit V, Strecha C, Fua P (2010) Brief: Binary robust independent elementary features. In: European conference on computer vision, pp 778–792

  4. Chen HH (1991) Pose determination from line-to-plane correspondences: existence condition and closed-form solutions. IEEE Trans Pattern Anal Mach Intell 13(6):530–541

    Article  Google Scholar 

  5. Comaniciu D, Meer P (2002) Mean shift: a robust approach toward feature space analysis. IEEE Trans Pattern Anal Mach Intell 24(5):603–619

    Article  Google Scholar 

  6. Dai A, Nießner M, Zollöfer M, Izadi S, Theobalt C (2017) BundleFusion: real-time globally consistent 3D reconstruction using on-the-fly surface re-integration. ACM Trans Graph 2017 (TOG) 36(3). https://doi.org/10.1145/3054739

  7. Eric W, Grimson L, Lozano-Perez T (1987) Model-based recognition and localization from sparse range or tactile data. Morgan Kaufmann Publishers Inc., Burlington

    Book  Google Scholar 

  8. Furukawa Y, Curless B, Seitz SM, Szeliski R (2009) Manhattan-world stereo. In: IEEE conference on computer vision and pattern recognition 2009. CVPR 2009, pp 1422–1429

  9. Henry Peter, Krainin Michael, Herbst Evan, Ren Xiaofeng, Fox D (2014) RGB-D mapping: using depth cameras for dense 3D modeling of indoor environments. Springer, Berlin

    Google Scholar 

  10. Jun HE, Hao D, Xie YQ, Liu BS (2006) Fast improved delaunay triangulation algorithm. Journal of System Simulation 18(11):3055–3057

    Google Scholar 

  11. Kahler O, Prisacariu VA, Ren CY, Sun X, Torr PHS, Murray DW (2015) Very high frame rate volumetric integration of depth images on mobile device. IEEE Trans Vis Comput Graph (Proceedings International Symposium on Mixed and Augmented Reality) 21(11):1241–1250

    Article  Google Scholar 

  12. Lee TK, Lim S, Lee S, An S (2012) Indoor mapping using planes extracted from noisy rgb-d sensors. In: Ieee/rsj international conference on intelligent robots and systems, pp 1727–1733

  13. Lepetit V, Fua P (2006) Keypoint recognition using randomized trees. IEEE Trans Pattern Anal Mach Intell 28(9):1465–79

    Article  Google Scholar 

  14. Newcombe RA, Izadi S, Hilliges O, Molyneaux D, Kim D, Davison AJ, Kohli P, Shotton J, Hodges S, Fitzgibbon A (2011) Kinectfusion: real-time dense surface mapping and tracking. In: IEEE ISMAR. IEEE, Piscataway

  15. Nistér D, Stewénius H (2007) A minimal solution to the generalised 3-point pose problem. J Math Imaging Vision 27(1):67–79

    Article  MathSciNet  Google Scholar 

  16. Prisacariu VA, Kahler O, Cheng MM, Ren CY, Valentin J, Torr PHS, Reid ID, Murray DW (2014) A framework for the volumetric integration of depth images. arXiv:1410.0925

  17. Ramalingam S, Taguchi Y (2013) A theory of minimal 3d point to 3d plane registration and its generalization. Int J Comput Vis 102(1):73–90

    Article  MathSciNet  Google Scholar 

  18. Rosten E, Porter R, Drummond T (2010) Faster and better: a machine learning approach to corner detection. IEEE Trans Pattern Anal Mach Intell 32(1):105–119

    Article  Google Scholar 

  19. Rublee E, Rabaud V, Konolige K, Bradski G (2011) Orb: An efficient alternative to sift or surf. In: IEEE international conference on computer vision, pp 2564–2571

  20. Shotton J, Glocker B, Zach C, Izadi S, Criminisi A, Fitzgibbon A (2013) Scene coordinate regression forests for camera relocalization in rgb-d images. In: IEEE conference on computer vision and pattern recognition, pp 2930–2937

  21. Steinbrücker F, Kerl C, Cremers D (2013) Large-scale multi-resolution surface reconstruction from rgb-d sequences. In: IEEE international conference on computer vision, pp 3264–3271

  22. Taguchi Y, Jian YD, Ramalingam S, Feng C (2013) Point-plane slam for hand-held 3d sensors. In: IEEE international conference on robotics and automation, pp 5182–5189

  23. Trevor AJ, Rogers J, Christensen H (2012) Planar surface slam with 3d and 2d sensors. In: IEEE international conference on robotics and automation, pp 3041–3048

  24. Umeyama S (1991) Least-squares estimation of transformation parameters between two point patterns. IEEE Trans Pattern Anal Mach Intell 13(4):376–380

    Article  Google Scholar 

  25. Whelan T, Johannsson H, Kaess M, Leonard JJ (2013) Robust real-time visual odometry for dense rgb-d mapping. In: IEEE international conference on robotics and automation, pp 5724–5731

  26. Yan C, Zhang Y, Dai F, Xi W (2014) Parallel deblocking filter for hevc on many-core processor. Electron Lett 50(5):367–368

    Article  Google Scholar 

  27. Yan C, Zhang Y, Jizheng X, Dai F (2014) Efficient parallel framework for hevc motion estimation on many-core processors. IEEE Trans Circuits Syst Video Technol 24(12):2077–2089

    Article  Google Scholar 

  28. Yan C, Zhang Y, Xu J, Dai F, Li L, Dai Q, Wu F (2014) A highly parallel framework for hevc coding unit partitioning tree decision on many-core processors. IEEE Signal Process Lett 21(5):573–576

    Article  Google Scholar 

  29. Yan C, Xie H, Liu X, Yin J, Zhang Y, Dai Q (2017) Effective uyghur language text detection in complex? background images for traffic prompt identification. https://doi.org/10.1109/TITS.2017.2749977

  30. Yan C, Xie H, Yang D, Yin J, Zhang Y, Dai Q (2017) Supervised hash coding with deep neural network for environment perception of intelligent vehicles. IEEE Trans Intell Transp Syst

  31. Zhang Z, Faugeras OD (1991) Determining motion from 3d line segment matches: a comparative study. Image Vis Comput 9(1):10–19

    Article  Google Scholar 

Download references

Acknowledgments

This work is supported by National Nature Science Foundation of China (61671196, 61327902, 61671268, 61727808), Zhejiang Province Nature Science Foundation of China LR17F030006.

Author information

Authors and Affiliations

  1. Institute of Information and Control, Hangzhou Dianzi University, Hangzhou, China

    Zunjie Zhu, Chenggang Yan, Ning Li & Bingjian Gong

  2. School of Software, Tsinghua University, Beijing, China

    Feng Xu

  3. Institute of Computing Technology, Chinese Academy of Sciences(CAS), Beijing, China

    Yongdong Zhang

  4. Department of Automation, Tsinghua University, Beijing, China

    Qionghai Dai

Authors
  1. Zunjie Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Feng Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chenggang Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ning Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Bingjian Gong
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yongdong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Qionghai Dai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Feng Xu or Chenggang Yan.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhu, Z., Xu, F., Yan, C. et al. Real-time indoor scene reconstruction with Manhattan assumption. Multimed Tools Appl 78, 713–726 (2019). https://doi.org/10.1007/s11042-017-5519-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-5519-7

Keywords

Search

Navigation