Skip to main content
SpringerLink
Log in

Robust and Optimal Registration of Image Sets and Structured Scenes via Sum-of-Squares Polynomials

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

This paper addresses the problem of registering a known structured 3D scene, typically a 3D scan, and its metric Structure-from-Motion (SfM) counterpart. The proposed registration method relies on a prior plane segmentation of the 3D scan. Alignment is carried out by solving either the point-to-plane assignment problem, should the SfM reconstruction be sparse, or the plane-to-plane one in case of dense SfM. A Polynomial Sum-of-Squares optimization theory framework is employed for identifying point-to-plane and plane-to-plane mismatches, i.e. outliers, with certainty. An inlier set maximization approach within a Branch-and-Bound search scheme is adopted to iteratively build potential inlier sets and converge to the solution satisfied by the largest number of assignments. Plane visibility conditions and vague camera locations may be incorporated for better efficiency without sacrificing optimality. The registration problem is solved in two cases: (i) putative correspondences (with possibly overwhelmingly many outliers) are provided as input and (ii) no initial correspondences are available. Our approach yields outstanding results in terms of robustness and optimality.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

Notes

  1. Our source code can be found at: www.dropbox.com/sh/vvkeaf5fcaxwsyr/AACqTUJE3FTXeiPnFkCbOSSXa?dl=0.

References

  • Bartoli, A., & Castellani, U. (2012). 3D shape registration. In 3D imaging, analysis, and applications, Springer (pp. 221–264).

  • Bazin, J., Li, H., Kweon, I. S., Demonceaux, C., Vasseur, P., & Ikeuchi, K. (2013). A branch-and-bound approach to correspondence and grouping problems. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 35, 1565–1576.

    Article  Google Scholar 

  • Borrmann, D., Elseberg, J., Lingemann, K., & Nüchter, A. (2011). The 3D hough transform for plane detection in point clouds: A review and a new accumulator design. 3D. Research, 32(1–32), 13.

    Google Scholar 

  • Boyd, S., & Vandenberghe, L. (2004). Convex optimization. New York, NY: Cambridge University Press.

    Book  MATH  Google Scholar 

  • Breuel, T. M. (2003). Implementation techniques for geometric branch-and-bound matching methods. Computer Vision and Image Understanding, 90(3), 258–294.

    Article  MATH  Google Scholar 

  • Chandraker, M., Agarwal, S., Kahl, F., Nister, D., & Kriegman, D. (2007). Autocalibration via rank-constrained estimation of the absolute quadric. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 1–8).

  • Chesi, G., Garulli, A., Vicino, A., & Cipolla, R. (2002). Estimating the fundamental matrix via constrained least-squares: A convex approach. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 24, 397–401.

    Article  Google Scholar 

  • Choi, M. D., Lam, T. Y., & Reznick, B. (1995). Sums of squares of real polynomials. Proceedings of Symposia in Pure Mathematics, 2(58), 103–126.

    MathSciNet  MATH  Google Scholar 

  • Christy, S., & Horaud, R. (1999). Iterative pose computation from line correspondences. Computer Vision and Image Understanding (CVIU), 73, 137–144.

    Article  MATH  Google Scholar 

  • Corsini, M., Dellepiane, M., Ganovelli, F., Gherardi, R., Fusiello, A., & Scopigno, R. (2013). Fully automatic registration of image sets on approximate geometry. International Journal of Computer Vision (IJCV), 102, 91–111.

    Article  Google Scholar 

  • Du, S., Zheng, N., Ying, S., You, Q., & Wu, Y. (2007). An extension of the ICP algorithm considering scale factor. In IEEE international conference on image processing (ICIP) (pp. V–193).

  • Enqvist, O., Josephson, K., & Kahl, F. (2009). Optimal correspondences from pairwise constraints. In 2009 IEEE 12th international conference on computer vision (pp. 1295–1302). IEEE.

  • Enqvist, O., & Kahl, F. (2008). Robust optimal pose estimation. Computer Vision-ECCV, 2008, 141–153.

    Google Scholar 

  • Ferraz, L., Binefa, X., & Moreno-Noguer, F. (2014). Very fast solution to the pnp problem with algebraic outlier rejection. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 501–508).

  • Finsler, P. (1936/37). Uber das vorkommen definiter und semidefiniter formen in scharen quadratischer formen. Commentarii Mathematici Helvetici, 9, 188–192.

  • Fischler, M. A., & Bolles, R. C. (1981). Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communication of the ACM, 24, 381–395.

    Article  MathSciNet  Google Scholar 

  • Fitzgibbon, A. W. (2001). Robust registration of 2D and 3D point sets. In The British machine vision conference (BMVC) (pp. 662–670).

  • Fraundorfer, F., & Scaramuzza, D. (2012). Visual odometry: Part II: Matching, robustness, optimization, and applications. IEEE Robotics & Automation Magazine, 19(2), 78–90.

    Article  Google Scholar 

  • Habed, A., Al Ismaeil, K., & Fofi, D. (2012). A new set of quartic trivariate polynomial equations for stratified camera self-calibration under zero-skew and constant parameters assumptions. In European conference on computer vision (ECCV) (pp. 710–723).

  • Hartley, R. I., & Zisserman, A. (2004). Multiple view geometry in computer vision (2nd ed.). Cambridge: Cambridge University Press.

    Book  MATH  Google Scholar 

  • Hilbert, D. (1888). Uber die darstellung definiter formen als summe von formen quadraten. Mathematische Annalen, 32, 342–350.

    Article  MathSciNet  MATH  Google Scholar 

  • Hough Transform Plane Detector. (2015). Howpublished. https://github.com/daviddoria/vtkhoughplanes/

  • Jensen, R., Dahl, A., Vogiatzis, G., Tola, E., & Aanæs, H. (2014). Large scale multi-view stereopsis evaluation. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 406–413).

  • Jurie, F. (1999). Solution of the simultaneous pose and correspondence problem using gaussian error model. Computer Vision and Image Understanding, 73(3), 357–373.

    Article  MATH  Google Scholar 

  • Kahl, F., & Henrion, D. (2007). Globally optimal estimates for geometric reconstruction problems. International Journal of Computer Vision, 74(1), 3–15.

    Article  Google Scholar 

  • Lasserre, J. B. (2000). Global optimization with polynomials and the problem of moments. SIAM Journal on Optimization, 11, 796–817.

    Article  MathSciNet  MATH  Google Scholar 

  • Li, H. (2009). Consensus set maximization with guaranteed global optimality for robust geometry estimation. In IEEE international conference on computer vision (ICCV) (pp. 1074–1080).

  • Liu, L., & Stamos, I. (2005). Automatic 3D to 2D registration for the photorealistic rendering of urban scenes. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 137–143).

  • Mastin, A., Kepner, J., & Fisher III, J. W. (2009). Automatic registration of LIDAR and optical images of urban scenes. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 2639–2646).

  • Moulon, P., Monasse, P., & Marlet, R. (2013). Adaptive structure from motion with a contrario model estimation. In Asian conference on computer vision (ACCV) (pp. 257–270).

  • Olsson, C., Kahl, F., & Oskarsson, M. (2006). The registration problem revisited: Optimal solutions from points, lines and planes. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 1206–1213).

  • Parrilo, P. A. (2000). Structured semidefinite programs and semialgebraic geometry methods in robustness and optimization. Technical report, California Institute of Technology

  • Paudel, D. P., Demonceaux, C., Habed, A., & Vasseur, P. (2014). Localization of 2D cameras in a known environment using direct 2D–3D registration. In International conference on pattern recognition (ICPR) (pp. 1–6).

  • Plotz, T., & Roth, S. (2015). Registering images to untextured geometry using average shading gradients. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 2030–2038).

  • Powers, V., & Wörmann, T. (1998). An algorithm for sums of squares of real polynomials. Journal of Pure and Applied Algebra, 127(1), 99–104.

    Article  MathSciNet  MATH  Google Scholar 

  • Putinar, M. (1993). Positive polynomials on compact semi-algebraic sets. Indiana University Mathematics Journal, 42, 969–984.

    Article  MathSciNet  MATH  Google Scholar 

  • Ramalingam, S., & Taguchi, Y. (2013). A theory of minimal 3D point to 3D plane registration and its generalization. International Journal of Computer Vision, 102(1–3), 73–90.

    Article  MathSciNet  MATH  Google Scholar 

  • Rusinkiewicz, S., & Levoy, M. (2001). Efficient variants of the ICP algorithm. In 3-D digital imaging and modeling (3DIM) (pp. 145–152).

  • Schindler, G., Krishnamurthy, P., Lublinerman, R., Liu, Y., & Dellaert, F. (2008). Detecting and matching repeated patterns for automatic geo-tagging in urban environments. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 1–7).

  • Schweighofer, G., & Pinz, A. (2008). Globally optimal o(n) solution to the pnp problem for general camera models. In The British machine vision conference (BMVC) (pp. 1–10).

  • Segal, A. V., Haehnel, D., & Thrun, S. (2009). Generalized-ICP. In Robotics: Science and systems (RSS).

  • Strecha, C., von Hansen, W., Van Gool, L., Fua, P., & Thoennessen, U. (2008). On benchmarking camera calibration and multi-view stereo for high resolution imagery. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 1–8).

  • Tamaazousti, M., Gay-Bellile, V., Collette, S. N., Bourgeois, S., & Dhome, M. (2011). Nonlinear refinement of structure from motion reconstruction by taking advantage of a partial knowledge of the environment. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 3073–3080).

  • Taneja, A., Ballan, L., & Pollefeys, M. (2013). City-scale change detection in cadastral 3D models using images. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 113–120).

  • Verschelde, J. (1999). Algorithm 795: Phcpack: A general-purpose solver for polynomial systems by homotopy continuation. ACM Transactions on Mathematical Software (TOMS), 25(2), 251–276.

    Article  MATH  Google Scholar 

  • Viola, P., & Wells, M, I. I. I. (1997). Alignment by maximization of mutual information. International Journal of Computer Vision (IJCV), 24, 137–154.

    Article  Google Scholar 

  • Wagner, S. (2009). Archimedean quadratic modules: A decision problem for real multivariate polynomials. Ph.D. thesis, Universität Konstanz.

  • Yang, J., Li, H., & Jia, Y. (2013). Go-icp: Solving 3D registration efficiently and globally optimally. In IEEE international conference on computer vision (ICCV) (pp. 1457–1464).

  • Yang, J., Li, H., & Jia, Y. (2014). Optimal essential matrix estimation via inlier-set maximization. In European conference on computer vision (ECCV) (pp. 111–126).

  • Zhang, X., Agam, G., & Chen, X. (2014). Alignment of 3d building models with satellite images using extended chamfer matching. In IEEE conference on computer vision and pattern recognition workshops (CVPRW) (pp. 746–753).

Download references

Acknowledgements

This research has been funded by the International Project NRF-ANR DrAACaR: ANR-11-ISO3-0003, the Regional Council of Bourgogne and European Regional Development Fund.

Author information

Authors and Affiliations

  1. Computer Vision Lab, ETH Zurich, Zurich, Switzerland

    Danda Pani Paudel

  2. ICube Laboratory, CNRS, University of Strasbourg, Strasbourg, France

    Adlane Habed

  3. Le2i Laboratory, CNRS, University of Bourgogne Franche-Comté, Dijon, France

    Cédric Demonceaux

  4. Laboratoire d’Informatique, de Traitement de l’Information et des Systèmes, Normandie Univ, UNIROUEN, UNIHAVRE, INSA Rouen, LITIS, 76000, Rouen, France

    Pascal Vasseur

Authors
  1. Danda Pani Paudel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Adlane Habed
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cédric Demonceaux
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Pascal Vasseur
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Danda Pani Paudel.

Additional information

Communicated by Josef Sivic.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Paudel, D.P., Habed, A., Demonceaux, C. et al. Robust and Optimal Registration of Image Sets and Structured Scenes via Sum-of-Squares Polynomials. Int J Comput Vis 127, 415–436 (2019). https://doi.org/10.1007/s11263-018-1114-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-018-1114-2

Keywords

Search

Navigation