Skip to main content
SpringerLink
Log in

Camera–projector matching using unstructured video

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

This paper presents a novel approach for matching 2-D points between a video projector and a digital camera. Our method is motivated by camera–projector applications for which the projected image needs to be warped to prevent geometric distortion. Since the warping process often needs geometric information on the 3-D scene obtained from a triangulation, we propose a technique for matching points in the projector to points in the camera based on arbitrary video sequences. The novelty of our method lies in the fact that it does not require the use of pre-designed structured light patterns as is usually the case. The backbone of our application lies in a function that matches activity patterns instead of colors. This makes our method robust to pose, severe photometric and geometric distortions. It also does not require calibration of the color response curve of the camera–projector system. We present quantitative and qualitative results with synthetic and real-life examples, and compare the proposed method with the scale invariant feature transform (SIFT) method and with a state-of-the-art structured light technique. We show that our method performs almost as well as structured light methods and significantly outperforms SIFT when the contrast of the video captured by the camera is degraded.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Al-Subaihi I.A., Watson G.A.: Algebraic fitting of quadric surfaces to data. Commun. Appl. Anal. 9, 539–548 (2005)

    MathSciNet  MATH  Google Scholar 

  2. Ashdown, M., Sato, Y.: Steerable projector calibration. In: Proceedings of the IEEE International Work. Projector–Camera Systems (2005)

  3. Belhumeur P.N.: A Bayesian approach to binocular stereopsis. Int. J. Comput. Vis. 19(3), 237–260 (1996)

    Article  Google Scholar 

  4. Boulaassal, H., Landes, T., Grussenmeyer, P., Tarsha Kurdi, F.: Automatic segmentation of building facades using terrestrial laser data. In: ISPRS Workshop on Laser Scanning and SilviLaser, pp. 65–70 (2007)

  5. Cotting D., Ziegler R., Gross M., Fuchs H.: Adaptive instant displays: continuously calibrated projections using per-pixel light control. Comput. Graph. Forum 24(3), 705–714 (2005)

    Article  Google Scholar 

  6. Davis J., Nehab D., Ramamoorthi R., Rusinkiewicz S.: Spacetime stereo: a unifying framework for depth from triangulation. IEEE Trans. Pattern Anal. Machine Intell. 27(2), 296–302 (2005)

    Article  Google Scholar 

  7. Drouin, M.-A., Jodoin, P.-M., Premont, J.: Camera–projector matching using an unstructured video stream. In: Proceedings of the IEEE International Work. Projector–Camera Systems, pp. 1–8 (2010)

  8. Drouin, M.-A., Trudeau, M., Roy, S.: Fast multiple-baseline stereo with occlusion. In: Proceedings of the Interantional Conference 3-D Digital Imaging and Modeling, pp. 540–547 (2005)

  9. Egnal G., Wildes R.P.: Detecting binocular half-occlusions: empirical comparisons of five approaches. IEEE Trans. Pattern Anal. Machine Intell. 24(8), 1127–1133 (2002)

    Article  Google Scholar 

  10. Felzenszwalb P.F., Huttenlocheri D.P.: Efficient belief propagation for early vision. Int. J. Comput. Vis. 70(1), 41–54 (2006)

    Article  Google Scholar 

  11. Fiala, M.: Artag, a fiducial marker system using digital techniques. In: Proceedings of the IEEE Conference. Comp. Vis. Pattern. Recogn., pp. 590–596 (2005)

  12. Fusiello A., Trucco E., Verri A.: A compact algorithm for rectification of stereo pairs. Mach. Vis. Appl. 12(1), 16–22 (2000)

    Article  Google Scholar 

  13. Gluckman, J., Nayar, S.K.: Rectifying transformations that minimize resampling effects. In: Proceedings of the IEEE Conference. Comp. Vis. Pattern Recogn., pp. I:111–117 (2001)

  14. Hartley R., Sturm P.: Triangulation. Comput. Vis. Image Understand. 68(2), 146–157 (1997)

    Article  Google Scholar 

  15. Hartley R.I., Zisserman A.: Multiple View Geometry in Computer Vision 2nd edn. Cambridge University Press, Cambridge (2004)

    Book  MATH  Google Scholar 

  16. Heckbert P.: Color image quantization for frame buffer display. Comput. Graph. 16, 297–307 (1982)

    Article  Google Scholar 

  17. Hirschmuller, H.: Accurate and efficient stereo processing by semi-global matching and mutual information. In: Proceedings of the IEEE Conference. Comp. Vis. Pattern Recogn., pp. 807–814 (2005)

  18. Johnson, T., Fuchs, H.: Real-time projector tracking on complex geometry using ordinary imagery. In: Proceedings of the ACM/IEEE International Workshop on Projector Camera Systems, pp. 1–8 (2007)

  19. Johnson, T., Welch, G., Fuchs, H., Force, E.L., Towles, H.: A distributed cooperative framework for continuous multi-projector pose estimation. In: IEEE Virt. Reality Conf., pp. 35–42 (2009)

  20. Lowe D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)

    Article  Google Scholar 

  21. Mufti, F., Mahony, R., Heinzmann, J.: Spatio-temporal ransac for robust estimation of ground plane in video range images for automotive applications. In: IEEE ITS, pp. 12–15 (2008)

  22. Ohta Y., Kanade T.: Stereo by intra- and inter-scanline using dynamic programming. IEEE Trans. Pattern Anal. Machine Intell. 7(2), 139–154 (1985)

    Article  Google Scholar 

  23. Otsu N.: A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man Cybern. 9(1), 62–66 (1979)

    Article  MathSciNet  Google Scholar 

  24. Pratt V.: Direct least-squares fitting of algebraic surfaces. SIGGRAPH 21(4), 145–152 (1987)

    Article  MathSciNet  Google Scholar 

  25. Quirk, P., Johnson, T., Skarbez, R., Towles, H., Gyarfas, F., Fuchs, H.: Ransac-assisted display model reconstruction for projective display. In: IEEE Virt. Reality Conf., p. 318 (2006)

  26. Raskar, R., Beardsley, P.: A self-correcting projector. In: Proceedings of the IEEE Conference. Comp. Vis. Pattern Recogn., pp. II:504–508 (2001)

  27. Raskar, R., van Baar, J., Beardsley, P., Willwacher, T., Rao, S., Forlines, C.: iLamps: geometrically aware and self-configuring projectors. In: ACM Trans. Graph., p. 5, ACM (2003)

  28. Salvi J., Pages J., Batlle J.: Pattern codification strategies in structured light systems. Pattern Recogn. 37(4), 827–849 (2004)

    Article  MATH  Google Scholar 

  29. Scharstein D., Szeliski R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47(1–3), 7–42 (2002)

    Article  MATH  Google Scholar 

  30. Shimamura, J., Arai, H., Yasuno, T.: Multiple plane detection for flexible projection. In: Proceedings of the IEEE International Work. Projector–Camera Systems, pp. 1–2 (2006)

  31. Tardif, J., Trudeau, M., Roy, S.: Multi-projectors for arbitrary surfaces without explicit calibration nor reconstruction. In: Proceedings of the International Conference. 3-D Digital Imaging and Modeling, pp. 217–224 (2003)

  32. Yang, R., Welch, G.: Automatic projector display surface estimation using every-day imagery. In: Proceedings of the International Conference. Cent. Eur Comput. Graph. Vis. Comput. Vis. (2001)

  33. Yapo, T., Sheng, Y., Nasman, J., Dolce, A., Li, E., Cutler, B.: Dynamic projection surfaces for immersive visualization. In: Proceedings of the IEEE International Work. Projector–Camera Systems, pp. 1–8 (2010)

  34. Zhang, L., Curless, B., Seitz, S.: Rapid shape acquisition using color structured light and multi-pass dynamic programming. In: International Symposium. 3D Data Proc. Vis Transm., pp. 24–36 (2002)

  35. Zhang, L., Curless, B., Seitz, S.: Spacetime stereo shape recovery for dynamic scenes. In: Proceedings of the IEEE Conference. Comp. Vis. Pattern. Recogn., pp. 367–374 (2003)

  36. Zhang L., Nayar S.: Projection defocus analysis for scene capture and image display. ACM Trans. Graph. 25(3), 907–915 (2006)

    Article  Google Scholar 

  37. Zhou, J., Wang, L., Akbarzadeh, A., Yang, R.: Multi-projector display with continuous self-calibration. In: Proceedings of the ACM/IEEE International Workshop on Projector camera systems, pp. 1–7 (2008)

  38. Zollmann S., Langlotz T., Bimber O.: Passive-active geometric calibration for view-dependent projections onto arbitrary surfaces. J. Virt. Real. Broadcasting 4(6), 1–10 (2007)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. NRC Institute for Information Technology, 1200 Montreal Road, Building M-50, Ottawa, ON, K1A 0R6, Canada

    Marc-Antoine Drouin

  2. Université de Sherbrooke, 2500, boulevard de l’Université, Sherbrooke, QC, J1K 2R1, Canada

    Pierre-Marc Jodoin & Julien Prémont

Authors
  1. Marc-Antoine Drouin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pierre-Marc Jodoin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Julien Prémont
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pierre-Marc Jodoin.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Drouin, MA., Jodoin, PM. & Prémont, J. Camera–projector matching using unstructured video. Machine Vision and Applications 23, 887–902 (2012). https://doi.org/10.1007/s00138-011-0358-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-011-0358-4

Keywords

Search

Navigation