Abstract
This paper studies segmentation of multiple rigid-body motions in a 3-D dynamic scene under perspective camera projection. We consider dynamic scenes that contain both 3-D rigid-body structures and 2-D planar structures. Based on the well-known epipolar and homography constraints between two views, we propose a hybrid perspective constraint (HPC) to unify the representation of rigid-body and planar motions. Given a mixture of K hybrid perspective constraints, we propose an algebraic process to partition image correspondences to the individual 3-D motions, called Robust Algebraic Segmentation (RAS). Particularly, we prove that the joint distribution of image correspondences is uniquely determined by a set of (2K)-th degree polynomials, a global signature for the union of K motions of possibly mixed type. The first and second derivatives of these polynomials provide a means to recover the association of the individual image samples to their respective motions. Finally, using robust statistics, we show that the polynomials can be robustly estimated in the presence of moderate image noise and outliers. We conduct extensive simulations and real experiments to validate the performance of the new algorithm. The results demonstrate that RAS achieves notably higher accuracy than most existing robust motion-segmentation methods, including random sample consensus (RANSAC) and its variations. The implementation of the algorithm is also two to three times faster than the existing methods. The implementation of the algorithm and the benchmark scripts are available at http://perception.csl.illinois.edu/ras/.
Article PDF
Similar content being viewed by others
References
Agrawal, R., Gehrke, J., Gunopulos, D., & Raghavan, P. (1998). Automatic subspace clustering of high dimensional data for data mining applications. In Proceedings of ACM special interest group on management of data.
Anandan, P., & Avidan, S. (2000). Integrating local affine into global projective images in the joint image space. In Proceedings of European conference on computer vision.
Ballard, D. (1981). Generalizing the Hough transform to detect arbitrary patterns. Pattern Recognition, 13(2), 111–122.
Bartoli, A. (2001). Piecewise planar segmentation for automatic scene modeling. In Proceedings of IEEE conference on computer vision and pattern recognition.
Campbell, N. (1978). The influence function as an aid in outlier detection in discriminant analysis. Applied Statistics, 27(3), 251–258.
Campbell, N. (1980). Robust procedures in multivariate analysis I: robust covariance estimation. Applied Statistics, 29(3), 231–237.
Costeira, J., & Kanade, T. (1998). A multibody factorization method of independently moving objects. International Journal on Computer Vision, 29(3), 159–179.
Fischler, M., & Bolles, R. (1981). Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24, 381–85.
Frahm, J., & Pollefeys, M. (2006). RANSAC for (quasi-) degenerate data (QDEGSAC). In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 453–460).
Gnanadesikan, R., & Kettenring, J. (1972). Robust estimates, residuals, and outlier detection with multiresponse data. Biometrics, 28(1), 81–124.
Goh, A., & Vidal, R. (2007). Segmenting motions of different types by unsupervised manifold clustering. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 1–6).
Goshen, L., Shimshoni, I., Anandan, P., & Keren, D. (2005). Motion recovery by integrating over the joint image manifold. International Journal on Computer Vision, 65(3), 131–145.
Hampel, F., Ronchetti, E., Rousseeuw, P., & Stahel, W. (1986). Robust statistics: the approach based on influence functions. New York: Wiley.
Han, M., & Kanade, T. (2000). Reconstruction of a scene with multiple linearly moving objects. In Proceedings of IEEE conference on computer vision and pattern recognition (Vol. 2, pp. 542–549).
Harris, J. (1992). Algebraic geometry: a first course. Berlin: Springer.
Huber, P. (1981). Robust statistics. New York: Wiley.
Jollife, I. (2002). Principal component analysis (2nd edn.). Berlin: Springer.
Kanatani, K. (2002a). Estimating the number of independent motions for multibody segmentation. In Proceedings of Asian conference on computer vision (pp. 7–12).
Kanatani, K. (2002b). Motion segmentation by subspace separation: model selection and reliability evaluation. International Journal of Image and Graphics, 2(2), 179–197.
Kanatani, K., & Sugaya, Y. (2003). Multi-stage optimization for multi-body motion segmentation. In Proceedings of the Australia-Japan advanced workshop on computer vision.
Lang, S. (2002). Algebra. Berlin: Springer.
Lee, K., Ho, J., & Kriegman, D. (2005). Acquiring linear subspaces for face recognition under variable lighting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(5), 684–698.
Leonardis, A., Bischof, H., & Maver, J. (2002). Multiple eigenspaces. Pattern Recognition, 35(11), 2613–2627.
Ma, Y., Derksen, H., Hong, W., & Wright, J. (2007). Segmentation of multivariate mixed data via lossy coding and compression. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(9), 1546–1562.
Ma, Y., Yang, A., Derksen, H., & Fossum, R. (2008). Estimation of subspace arrangements with applications in modeling and segmenting mixed data. SIAM Review, 50(3), 413–458.
Ng, A., Weiss, Y., & Jordan, M. (2001). On spectral clustering: analysis and an algorithm. In Proceedings of neural and information processing systems.
Ozden, K., Schindler, K., & van Gool, L. (2007). Simultaneous segmentation and reconstruction of monocular image sequences. In Proceedings of IEEE international conference on computer vision.
Rao, S., Yang, A., Wagner, A., & Ma, Y. (2005). Segmentation of hybrid motions via hybrid quadratic surface analysis. In Proceedings of IEEE international conference on computer vision.
Rao, S., Tron, R., Vidal, R., & Ma, Y. (2008). Motion segmentation via robust subspace separation in the presence of outlying, incomplete, or corrupted trajectories. In Proceedings of IEEE conference on computer vision and pattern recognition.
Rousseeuw, P. (1984). Least median of squares regression. Journal of American Statistics Association, 79, 871–880.
Schindler, K., & Suter, D. (2005). Two-view multibody structure-and-motion with outliers. In Proceedings of IEEE conference on computer vision and pattern recognition.
Scholkopf, B., Smola, A., & Muller, K. (1998). Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation, 10, 1299–1319.
Shapiro, L. (1995). Affine analysis of image sequences. Cambridge: Cambridge University Press.
Shashua, A., & Levin, A. (2001). Multi-frame infinitesimal motion model for the reconstruction of (dynamic) scenes with multiple linearly moving objects. In Proceedings of IEEE international conference on computer vision (Vol. 2, pp. 592–599).
Souvenir, R., & Pless, R. (2007). Manifold Clustering. In International conference on computer vision.
Stewart, C. (1995). MINPRAN: a new robust estimator for computer vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17(10), 925–938.
Stewart, C. (1999). Robust parameter estimation in computer vision. SIAM Review, 41(3), 513–537.
Subbarao, R., & Meer, P. (2006). Beyond RANSAC: user independent robust regression. In CVPR workshop on 25 years of RANSAC.
Sugaya, Y., & Kanatani, K. (2003). Outlier removal for motion tracking by subspace separation. IEICE Transactions on Information and Systems, E86-D(6), 1095–1102.
Taubin, G. (1991). Estimation of planar curves, surfaces, and nonplanar space curves defined by implicit equations with applications to edge and range image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(11), 1115–1138.
Tipping, M., & Bishop, C. (1999). Mixtures of probabilistic principal component analyzers. Neural Computation, 11, 443–482.
Tong, W., Tang, C., & Medioni, G. (2004). Simultaneous two-view epipolar geometry estimation and motion segmentation by 4D tensor voting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(9), 1167–1184.
Torr, P. (1998). Geometric motion segmentation and model selection. Philosophical Transactions of the Royal Society of London, 356(1740), 1321–1340.
Torr, P., & Davidson, C. (2003). IMPSAC: synthesis of importance sampling and random sample consensus. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(3), 354–364.
Torr, P., & Zisserman, A. (2000). MLESAC: A new robust estimator with application to estimating image geometry. Computer Vision and Image Understanding, (78), 138–156.
Triggs, B. (1995). Matching constraints and the joint image. In Proceedings of IEEE international conference on computer vision.
Tron, R., & Vidal, R. (2007). A benchmark for the comparison of 3-D motion segmentation algorithms. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 1–8).
Vidal, R., & Hartley, R. (2008). Three-view multibody structure from motion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(2), 214–227.
Vidal, R., & Ma, Y. (2006). A unified algebraic approach to 2-D and 3-D motion segmentation and estimation. Journal of Mathematical Imaging and Vision, 25, 403–421.
Vidal, R., Ma, Y., & Sastry, S. (2005). Generalized principal component analysis (GPCA). IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(12), 1–15.
Vidal, R., Ma, Y., Soatto, S., & Sastry, S. (2006). Two-view multibody structure from motion. International Journal on Computer Vision, 68(1), 7–25.
Wang, H., & Suter, D. (2004). MDPE: a very robust estimator for model fitting and range image segmentation. International Journal on Computer Vision, 59(2), 139–166.
Weiss, Y. (1999). Segmentation using eigenvectors: a unifying view. In Proceedings of IEEE international conference on computer vision (pp. 975–982).
Yan, J., & Pollefeys, M. (2006). A general framework for motion segmentation: independent, articulated, rigid, non-rigid, degenerate and non-degenerate. In Proceedings of European conference on computer vision (pp. 94–106).
Yang, A. (2006). Estimation of subspace arrangements: its algebra and statistics. PhD Dissertation, University of Illinois at Urbana-Champaign.
Yang, A., Rao, S., Wagner, A., & Ma, Y. (2005). Segmentation of a piece-wise planar scene from perspective images. In Proceedings of IEEE conference on computer vision and pattern recognition.
Yang, A., Rao, S., & Ma, Y. (2006). Robust statistical estimation and segmentation of multiple subspaces. In CVPR workshop on 25 years of RANSAC.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Open Access This is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (https://creativecommons.org/licenses/by-nc/2.0), which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
About this article
Cite this article
Rao, S.R., Yang, A.Y., Sastry, S.S. et al. Robust Algebraic Segmentation of Mixed Rigid-Body and Planar Motions from Two Views. Int J Comput Vis 88, 425–446 (2010). https://doi.org/10.1007/s11263-009-0314-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11263-009-0314-1