Skip to main content
SpringerLink
Log in

A Multigrid Platform for Real-Time Motion Computation with Discontinuity-Preserving Variational Methods

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Variational methods are among the most accurate techniques for estimating the optic flow. They yield dense flow fields and can be designed such that they preserve discontinuities, estimate large displacements correctly and perform well under noise and varying illumination. However, such adaptations render the minimisation of the underlying energy functional very expensive in terms of computational costs: Typically one or more large linear or nonlinear equation systems have to be solved in order to obtain the desired solution. Consequently, variational methods are considered to be too slow for real-time performance. In our paper we address this problem in two ways: (i) We present a numerical framework based on bidirectional multigrid methods for accelerating a broad class of variational optic flow methods with different constancy and smoothness assumptions. Thereby, our work focuses particularly on regularisation strategies that preserve discontinuities. (ii) We show by the examples of five classical and two recent variational techniques that real-time performance is possible in all cases—even for very complex optic flow models that offer high accuracy. Experiments show that frame rates up to 63 dense flow fields per second for image sequences of size 160 × 120 can be achieved on a standard PC. Compared to classical iterative methods this constitutes a speedup of two to four orders of magnitude.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Glazer, F. 1984. Multilevel relaxation in low-level computer vision. In A. Rosenfeld, editor, Multiresolution Image Processing and Analysis, pages 312–330. Springer, Berlin.

    Google Scholar 

  • Alvarez, L., Esclarín J., Lefébure M., and Sánchez J. 1999a. A PDE model for computing the optical flow. In Proc. XVI Congreso de Ecuaciones Diferenciales y Aplicaciones, pp. 1349–1356, Las Palmas de Gran Canaria, Spain, Sept.

  • Alvarez, L., Weickert, J., and Sánchez J. 1999b. A scale-space approach to nonlocal optical flow calculations. In M. Nielsen, P. Johansen, O.F. Olsen, and J. Weickert, (eds.), Scale-Space Theories in Computer Vision, vol. 1682 of Lecture Notes in Computer Science, pp. 235–246. Springer, Berlin.

    Google Scholar 

  • Alvarez, L., Weickert, J., and Sánchez J. 2000. Reliable estimation of dense optical flow fields with large displacements. International Journal of Computer Vision, 39(1):41–56.

    Article  MATH  Google Scholar 

  • Anandan, P. 1989. A computational framework and an algorithm for the measurement of visual motion. International Journal of Computer Vision, 2:283–310.

    Article  Google Scholar 

  • Barron, J.L., Fleet, D.J., and Beauchemin, S.S. 1994. Performance of optical flow techniques. International Journal of Computer Vision, 12(1):43–77.

    Article  Google Scholar 

  • Bergen, J.R., Anandan, P., Hanna, K.J., and Hingorani, R. 1992. Hierarchical model-based motion estimation. In F. Hodnett, (ed.), Proc. Sixth European Conference on Mathematics in Industry, pp. 237–252. Teubner, Stuttgart.

    Google Scholar 

  • Black, M.J. and Anandan, P. 1996. The robust estimation of multiple motions: parametric and piecewise smooth flow fields. Computer Vision and Image Understanding, 63(1):75–104.

    Article  Google Scholar 

  • Black, M.J. and Anandan, P. 1991. Robust dynamic motion estimation over time. In Proc. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 292–302, Maui, HI, IEEE Computer Society Press.

    Google Scholar 

  • Bornemann, F. and Deuflhard, P. 1996. The cascadic multigrid method for elliptic problems. Numerische Mathematik, 75:135–152.

    Article  MATH  MathSciNet  Google Scholar 

  • Borzi, A., Ito, K., and Kunisch, K. 2002. Optimal control formulation for determining optical flow. SIAM Journal on Scientific Computing, 24(3):818–847.

    Article  MATH  MathSciNet  Google Scholar 

  • Brandt, A. 1977. Multi-level adaptive solutions to boundary-value problems. Mathematics of Computation, 31(138):333–390.

    Article  MATH  MathSciNet  Google Scholar 

  • Briggs, W.L., Henson, V.E., and McCormick S.F. 2000. A Multigrid Tutorial. 2 edition, SIAM, Philadelphia.

    MATH  Google Scholar 

  • Brox, T., Bruhn, A., Papenberg, N., and Weickert, J. 2004. High accuracy optic flow estimation based on a theory for warping. In T. Pajdla and J. Matas, editors, Computer Vision — ECCV 2004, vol. 3024 of Lecture Notes in Computer Science, pages 25–36. Springer, Berlin.

    Google Scholar 

  • Bruhn, A. and Weickert, J. 2005. Towards ultimate motion estimation: Combining highest accuracy with real-time performance. In Proc. Tenth International Conference on Computer Vision, vol. 1, pages 749–755, Beijing, China, IEEE Computer Society Press.

    Google Scholar 

  • Bruhn, A., Weickert, J., Feddern, C., Kohlberger, T., and Schnörr C. 2003. Real-time optic flow computation with variational methods. In N. Petkov and M.A. Westberg, editors, Computer Analysis of Images and Patterns, volume 2756 of Lecture Notes in Computer Science, pp. 222–229. Springer, Berlin.

    Google Scholar 

  • Bruhn, A., Weickert, J., and Schnörr C. 2005c. Lucas/Kanade meets Horn/Schunck: Combining local and global optic flow methods. International Journal of Computer Vision, 61(3):211–231.

    Article  Google Scholar 

  • Bruhn, A., Weickert, J., Feddern, C., Kohlberger, T., and Schnörr C. 2005a. Variational optical flow computation in real-time. IEEE Transactions on Image Processing, 14(5):608–615.

    Article  Google Scholar 

  • Bruhn, A., Weickert, J., Kohlberger, T., and Schnörr C. 2005b. Discontinuity-preserving computation of variational optic flow in real-time. In R. Kimmel, N. Sochen, and J. Weickert, (eds.), Scale-Space and PDE Methods in Computer Vision, volume 3459 of Lecture Notes in Computer Science, pp. 279–290. Springer, Berlin.

    Google Scholar 

  • Chan, R.H., Chan, T.F., and Wan, W.L. 1997. Multigrid for differential-convolution problems arising from image processing. In G. Golub, S.H. Lui, F. Luk, and R. Plemmons, (eds.), Proc. Workshop on Scientific Computing, pp. 58–72, Hong Kong.

  • Chan, T.F. and Mulet, P. 1999. On the convergence of the lagged diffusivity fixed point method in total variation image restoration. SIAM Journal on Numerical Analysis, 36(2):354–367.

    Article  MathSciNet  Google Scholar 

  • Charbonnier, P., Blanc-Féraud L., Aubert, G., and Barlaud, M. 1994. Two deterministic half-quadratic regularization algorithms for computed imaging. In Proc. 1994 IEEE International Conference on Image Processing, vol. 2, pp. 168–172, Austin, TX, IEEE Computer Society Press.

    Google Scholar 

  • Cohen, I. 1993. Nonlinear variational method for optical flow computation. In Proc. Eighth Scandinavian Conference on Image Analysis, vol. 1, pp. 523–530, Tromsø, Norway.

    Google Scholar 

  • Deriche, R., Kornprobst, P., and Aubert, G. 1995. Optical-flow estimation while preserving its discontinuities: a variational approach. In Proc. Second Asian Conference on Computer Vision, vol. 2, pp. 290–295, Singapore.

  • El Kalmoun M. and Rüde U. 2003. A variational multigrid for computing the optical flow. In T. Ertl, B. Girod, G. Greiner, H. Niemann, H.-P. Seidel, E. Steinbach, and R. Westermann, (eds.), Vision, Modelling and Visualization, pp. 577–584. IOS Press.

  • Elsgolc, L.E. 1961. Calculus of Variations. Pergamon, Oxford.

    MATH  Google Scholar 

  • Enkelmann, W. 1987. Investigation of multigrid algorithms for the estimation of optical flow fields in image sequences. Computer Vision, Graphics and Image Processing, 43:150–177.

    Article  Google Scholar 

  • Farnebäck G. 2001. Very high accuracy velocity estimation using orientation tensors, parametric motion, and simultaneous segmentation of the motion field. In Proc. Eighth International Conference on Computer Vision, vol. 1, pp. 171–177, Vancouver, Canada, IEEE Computer Society Press.

    Google Scholar 

  • Frohn-Schnauf C., Henn, S., and Witsch, K. 2004. Nonlinear multigrid methods for total variation denoising. Computating and Visualization in Sciene, 7(3-4):199–206.

    Google Scholar 

  • Förstner W. and Gülch E. 1987. A fast operator for detection and precise location of distinct points, corners and centres of circular features. In Proc. ISPRS Intercommission Conference on Fast Processing of Photogrammetric Data, pp. 281–305, Interlaken, Switzerland.

    Google Scholar 

  • Ghosal, S. and Vaněk P. Č. 1996. Scalable algorithm for discontinuous optical flow estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(2):181–194.

    Article  Google Scholar 

  • Glazer, F. 1984. Multilevel relaxation in low-level computer vision. In A. Rosenfeld, (ed.), Multiresolution Image Processing and Analysis, pp. 312–330. Springer, Berlin.

    Google Scholar 

  • Hackbusch, W. 1985. Multigrid Methods and Applications. Springer, New York.

    Google Scholar 

  • Horn, B. and Schunck, B. 1981. Determining optical flow. Artificial Intelligence, 17:185–203.

    Article  Google Scholar 

  • Huber, P.J. 1981. Robust Statistics. Wiley, New York.

    MATH  Google Scholar 

  • Kimmel, R. and Yavneh, I. 2003. An algebraic multigrid approach for image analysis. SIAM Journal on Scientific Computing, 24(4):1218–1231.

    Article  MATH  MathSciNet  Google Scholar 

  • Kohlberger, T., Schnörr C., Bruhn, A., and Weickert, J. 2004. Parallel variational motion estimation by domain decomposition and cluster computing. In T. Pajdla and J. Matas, (eds.), Computer Vision — ECCV 2004, volume 3024 of Lecture Notes in Computer Science, pages 205–216. Springer, Berlin.

    Google Scholar 

  • Kohlberger, T., Schnörr C., Bruhn, A., and Weickert, J. 2005. Domain decomposition for variational optical flow computation. IEEE Transactions on Image Processing, to appear.

  • Kumar, A., Tannenbaum, A.R., and Balas, G.J. 1996. Optic flow: a curve evolution approach. IEEE Transactions on Image Processing, 5(4):598–610.

    Article  Google Scholar 

  • Luettgen, M.R., Karl, W.C., and Willsky, A.S. 1994. Efficient multiscale regularization with applications to the computation of optical flow. IEEE Transactions on Image Processing, 3(1):41–64.

    Article  Google Scholar 

  • Mémin E. and Pérez P. 1998a. A multigrid approach for hierarchical motion estimation. In Proc. 6th International Conference on Computer Vision, pp. 933–938, Bombay, India.

  • Mémin E. and Pérez P. 1998b. Dense estimation and object-based segmentation of the optical flow with robust techniques. IEEE Transactions on Image Processing, 7(5):703–719.

    Article  Google Scholar 

  • Nagel H.-H. and Enkelmann, W. 1986. An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence, 8:565–593.

    Google Scholar 

  • Ortega, J.M. and Rheinboldt, W.C. 2000. Iterative Solution of Nonlinear Equations in Several Variables, vol. 30 of Classics in Applied Mathematics. SIAM, Philadelphia.

    Google Scholar 

  • Papenberg, N., Bruhn, A., Brox, T., Didas, S., and Weickert, J. 2005. Highly accurate optic flow computation with theoretically justified warping. International Journal of Computer Vision, to appear.

  • Rudin, L.I., Osher, S., and Fatemi, E. 1992. Nonlinear total variation based noise removal algorithms. Physica D, 60:259–268.

    Article  MATH  Google Scholar 

  • Schnörr C. 1994. Segmentation of visual motion by minimizing convex non-quadratic functionals. In Proc. Twelfth International Conference on Pattern Recognition, vol. A, pp. 661–663, Jerusalem, Israel, IEEE Computer Society Press.

    Google Scholar 

  • Terzopoulos, D. 1986. Image analysis using multigrid relaxation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 8(2):129–139.

    Article  Google Scholar 

  • Tikhonov, A.N. and Arsenin, V.Y. 1977. Solutions of Ill—Posed Problems. Wiley, Washington, DC.

    MATH  Google Scholar 

  • Trottenberg, U., Oosterlee, C., and Schüller A. 2001. Multigrid. Academic Press, San Diego.

    MATH  Google Scholar 

  • Vogel, C.R. 1995. A multigrid method for total variation-based image denoising. Computation and Control IV, 20:323–331.

    MATH  Google Scholar 

  • Vogel, C.R. 2002. Computational Methods for Inverse Problems. SIAM, Philadelphia.

    MATH  Google Scholar 

  • Weickert, J. 1998. Anisotropic Diffusion in Image Processing. Teubner, Stuttgart.

    MATH  Google Scholar 

  • Weickert, J. and Schnörr C. 2001a. A theoretical framework for convex regularizers in PDE-based computation of image motion. International Journal of Computer Vision, 45(3):245–264.

    Article  MATH  Google Scholar 

  • Weickert, J. and Schnörr C. 2001b. Variational optic flow computation with a spatio-temporal smoothness constraint. Journal of Mathematical Imaging and Vision, 14(3):245–255.

    Article  MATH  Google Scholar 

  • Wesseling, P. 1992. An Introduction to Multigrid Methods. Wiley, Chichester.

    MATH  Google Scholar 

  • Young, D.M. 1971. Iterative Solution of Large Linear Systems. Academic Press, New York.

    MATH  Google Scholar 

  • Zini, G., Sarti, A., and Lamberti, C. 1997. Application of continuum theory and multi-grid methods to motion evaluation from 3D echocardiography. IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, 44(2):297–308.

    Article  Google Scholar 

  • El Kalmoun M. and Rüde U. 2003. A variational multigrid for computing the optical flow. In T. Ertl, B. Girod, G. Greiner, H. Niemann, H.-P. Seidel, E. Steinbach, and R. Westermann, (eds.), Vision, Modelling and Visualization, pp. 577–584. IOS Press.

Download references

Author information

Authors and Affiliations

  1. Mathematical Image Analysis Group, Faculty of Mathematics and Computer Science, Saarland University, Building E1 1, 66041, Saarbrücken, Germany

    Andrés Bruhn & Joachim Weickert

  2. Computer Vision, Graphics, and Pattern Recognition Group, Department of Mathematics and Computer Science, University of Mannheim, 68131, Mannheim, Germany

    Timo Kohlberger & Christoph Schnörr

Authors
  1. Andrés Bruhn
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Joachim Weickert
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Timo Kohlberger
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Christoph Schnörr
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andrés Bruhn.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bruhn, A., Weickert, J., Kohlberger, T. et al. A Multigrid Platform for Real-Time Motion Computation with Discontinuity-Preserving Variational Methods. Int J Comput Vision 70, 257–277 (2006). https://doi.org/10.1007/s11263-006-6616-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-006-6616-7

Keywords

Search

Navigation