Skip to main content
SpringerLink
Log in

Real-time vision-based tracking and reconstruction

  • Special Issue
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Many of the recent real-time markerless camera tracking systems assume the existence of a complete 3D model of the target scene. Also the system developed in the MATRIS project assumes that a scene model is available. This can be a freeform surface model generated automatically from an image sequence using structure from motion techniques or a textured CAD model built manually using a commercial software. The offline model provides 3D anchors to the tracking. These are stable natural landmarks, which are not updated and thus prevent an accumulating error (drift) in the camera registration by giving an absolute reference. However, sometimes it is not feasible to model the entire target scene in advance, e.g. parts, which are not static, or one would like to employ existing CAD models, which are not complete. In order to allow camera movements beyond the parts of the environment modelled in advance it is desired to derive additional 3D information online. Therefore, a markerless camera tracking system for calibrated perspective cameras has been developed, which employs 3D information about the target scene and complements this knowledge online by reconstruction of 3D points. The proposed algorithm is robust and reduces drift, the most dominant problem of simultaneous localisation and mapping (SLAM), in real-time by a combination of the following crucial points: (1) stable tracking of longterm features on the 2D level; (2) use of robust methods like the well-known Random Sampling Consensus (RANSAC) for all 3D estimation processes; (3) consequent propagation of errors and uncertainties; (4) careful feature selection and map management; (5) incorporation of epipolar constraints into the pose estimation. Validation results on the operation of the system on synthetic and real data are presented.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

Notes

  1. Note that m i,t and m i,t−1 are already centred around the principle point.

  2. Note that identical results are obtained by normalising the 2D/2D correspondences (A −1 m i,t , A −1 m i,t−1) and using the essential matrix E = [T] x ·R instead of the fundamental matrix F in (35).

  3. The system is single-threaded.

References

  1. Baker, S., Matthews, I.: Equivalence and efficiency of image alignment algorithms. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2001)

  2. Bay, H., Tuytelaars, T., Van Gool, L.: Surf: Speeded up robust features. In: European Conference on Computer Vision (ECCV), Graz, Austria (2006)

  3. Bleser, G., Wuest, H., Stricker, D.: Online camera pose estimation in partially known and dynamic scenes. In: International Symposium on Mixed and Augmented Reality (ISMAR), pp 56–65, Santa Barabara, October (2006)

  4. Cornelis, K.: From uncalibrated video to augmented reality. PhD Thesis, Catholic University Leuven, Leuven (2004)

  5. Davison, A.J., Reid, I., Molton, N., Stasse, O.: Monoslam: Real-time single camera slam. In: IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) (2007)

  6. Eade, E., Drummond, T.: Scalable monocular slam. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol 1, pp 469–476 (2006)

  7. Foerstner, W.: Uncertainty and projective geometry. Technical report, Institut fr Potogrammetrie, Universitaet Bonn (2004)

  8. Gustafsson, F.: Adaptive Filtering and Change Detection. Wiley, New York (2000)

  9. Hartley, R., Sturm, P.: Triangulation. In: International Conference on Computer Analysis of Images and Patterns, pp 190–197, Prague (1995)

  10. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision, 1st edn. Cambridge University Press, Cambridge. ISBN: 0 521 62304 9 (2001)

  11. Hol, J., Slycke, P., Schoen, T., Gustafsson, F.: 2d–3d model correspondence for camera pose estimation using sensor fusion (2005)

  12. Huber, P.J.: Robust Statistics. Wiley, New York (1981)

  13. Jin, H., Favaro, P., Soatto, S.: Real-time feature tracking and outlier rejection with changes in illumination. In: IEEE International Conference on Computer Vision (ICCV), pp 684–689 (2001)

  14. Davison, A.J., Montiel, J.M.M.: A visual compass based on slam. In: International Conference on Robotics and Automation (ICRA), pp 1917–1922 (2006)

  15. Julier, S.J., Uhlmann, J.K.: Unscented filtering and nonlinear estimation. In: Proceedings IEEE, vol 92, pp 401–422. IEEE, March (2004)

  16. Koeser, K., Bartczak, B., Koch, R.: Markerless augmented reality with light source estimation for direct illumination. In: Conference on Visual Media Production (CVMP), London, December (2005)

  17. Lepetit, V., Fua, P.: Keypoint recognition using randomized trees. In: Transactions on Pattern Analysis and Machine Intelligence (PAMI), vol 28, pp 1465–1479 (2006)

  18. Montemerlo, M., Thrun, S.: Fastslam: A factored solution to the simultaneous localization and mapping problem. In: AAAI (2002)

  19. Montemerlo, M., Thrun, S.: Fastslam 2.0: An improved particle filtering algorithm for simultaneous localization and mapping that provably converges. In: International Joint Conferences on Artificial Intelligence (IJCAI) (2003)

  20. Montiel, J.M.M, Civera, J., Davison, A.J.: Unified inverse depth parametrization for monocular slam. In: Internaltional Conference of the Royal Statistical Society (RSS) (2006)

  21. Mouragnon, E., Dekeyser, F., Sayd, P., Lhuillier, M., Dhome M.: Real time localization and 3d reconstruction. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp 363–370 (2006)

  22. Nister, D.: Preemptive ransac for live structure and motion estimation. In: International Conference on Computer Vision (ICCV), November (2003)

  23. Press, W.H., Teukolsky, S.A., Vetterling, W.T., Flannery, B.P.: Numerical Recipes: The Art of Scientific Computing, 3rd edn. Cambridge University Press, Cambridge (2007)

  24. Pupilli, M., Calway, A.: Real-time camera tracking using a particle filter. In: British Machine Vision Conference (BMVC), pp 519–528. Department of Computer Science, University of Bristol. BMVA Press, September (2005)

  25. Reitmayr, G., Drummond, T.: Going out: Robust model-based tracking for outdoor augmented reality. In: International Symposium on Mixed and Augmented Reality (ISMAR) (2006)

  26. Rosten, E., Drummond, T.: Fusing points and lines for high performance tracking. In: IEEE International Conference on Computer Vision (ICCV), vol 2, pp 1508–1511, October (2005)

  27. Shi, J., Tomasi, C.: Good features to track. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp 593–600 (1994)

  28. Simon, G., Lepetit, V., Berger, M.-O.: Computer vision methods for registration: Mixing 3d knowledge and 2d correspondences for accurate image composition. In: International Workshop on Augmented Reality (1998)

  29. Subbarao, R., Meer, P., Genc, Y.: A balanced approach to 3d tracking from image streams. In: International Symposium on Mixed and Augmented Reality (ISMAR), pp 70–78 (2005)

  30. Thrun, S., Burgard, W., Fox, D.: Probabilistic Robotics. Massachusetts Institute of Technology (2005)

  31. Vacchetti, L., Lepetit, V., Fua, P.: Fusing online and offline information for stable real-time 3d tracking in real-time. In: Conference on Computer Vision and Pattern Recognition (CVPR), vol 26, pp 1391–1391 (2003)

  32. Wuest, H., Vial, F., Stricker, D.: Adaptive line tracking with multiple hypotheses for augmented reality. In: International Symposium on Mixed and Augmented Reality (ISMAR), pp 62–69 (2005)

  33. Zhang, Z., Faugeras, O.: 3D Dynamic Scene Analysis. Springer, Heidelberg (1992)

  34. Zinßer, T., Gräßl, Ch., Niemann, H.: Efficient feature tracking for long video sequences. In: Symposium der Deutschen Arbeitsgemeinschaft fr Mustererkennung (DAGM), pp 326–333 (2004)

Download references

Acknowledgments

This work has been performed within the EU research program MATRIS (IST-002013). The authors would like to thank the project partners within the consortium and the EU for the financial support. For more information, please visit the website http://www.ist-matris.org.

Author information

Authors and Affiliations

  1. Department of Virtual and Augmented Reality, Fraunhofer IGD, Darmstadt, Germany

    Gabriele Bleser, Mario Becker & Didier Stricker

  2. Interactive Graphics Systems Group (GRIS), Technische Universität Darmstadt (TUD), Darmstadt, Germany

    Gabriele Bleser

Authors
  1. Gabriele Bleser
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mario Becker
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Didier Stricker
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gabriele Bleser.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bleser, G., Becker, M. & Stricker, D. Real-time vision-based tracking and reconstruction. J Real-Time Image Proc 2, 161–175 (2007). https://doi.org/10.1007/s11554-007-0034-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-007-0034-0

Keywords

Search

Navigation