Skip to main content
SpringerLink
Log in

Automatic Video Segmentation Based on Information Centroid and Optimized SaliencyCut

  • Regular Paper
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

We propose an automatic video segmentation method based on an optimized SaliencyCut equipped with information centroid (IC) detection according to level balance principle in physical theory. Unlike the existing methods, the image information of another dimension is provided by the IC to enhance the video segmentation accuracy. Specifically, our IC is implemented based on the information-level balance principle in the image, and denoted as the information pivot by aggregating all the image information to a point. To effectively enhance the saliency value of the target object and suppress the background area, we also combine the color and the coordinate information of the image in calculating the local IC and the global IC in the image. Then saliency maps for all frames in the video are calculated based on the detected IC. By applying IC smoothing to enhance the optimized saliency detection, we can further correct the unsatisfied saliency maps, where sharp variations of colors or motions may exist in complex videos. Finally, we obtain the segmentation results based on IC-based saliency maps and optimized SaliencyCut. Our method is evaluated on the DAVIS dataset, consisting of different kinds of challenging videos. Comparisons with the state-of-the-art methods are also conducted to evaluate our method. Convincing visual results and statistical comparisons demonstrate its advantages and robustness for automatic video segmentation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Soomro K, Idrees H, Shah M. Action localization in videos through context walk. In Proc. the 2015 IEEE Int. Conf. Computer Vision, December 2015, pp.3280-3288.

  2. Soomro K, Idrees H, Shah M. Predicting the where and what of actors and actions through online action localization. In Proc. the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016, pp.2648-2657.

  3. Liu B Y, He X M. Multiclass semantic video segmentation with object-level active inference. In Proc. the 2015 CVPR, June 2015, pp.4286-4294.

  4. Huang H Z, Fang X N, Ye Y F, Zhang S H, Rosin P L. Practical automatic background substitution for live video. Computational Visual Media, 2017, 3(3): 273-284.

    Article  Google Scholar 

  5. Liu T, Duan H B, Shang Y Y, Yuan Z J, Zheng N J. Automatic salient object sequence rebuilding for video segment analysis. Science China Information Sciences, 2018, 61(1): Article No. 012205.

  6. Zhang Y, Tang Y L, Cheng K L. Efficient video cutout by paint selection. Journal of Computer Science and Technology, 2015, 30(3): 467-477.

    Article  Google Scholar 

  7. Zhang C C, Liu Z L. Prior-free dependent motion segmentation using Helmholtz-Hodge decomposition based object-motion oriented map. Journal of Computer Science and Technology, 2017, 32(3): 520-535.

    Article  MathSciNet  Google Scholar 

  8. Ochs P, Brox T. Higher order motion models and spectral clustering. In Proc. the 2012 CVPR, June 2012, pp.614-621.

  9. Fragkiadaki K, Zhang G, Shi J. Video segmentation by tracing discontinuities in a trajectory embedding. In Proc. the 2012 CVPR, June 2012, pp.1846-1853.

  10. Xu C L, Xiong C M, Corso J J. Streaming hierarchical video segmentation. In Proc. the 12th European Conference on Computer Vision, October 2012, pp.626-639.

  11. Zhang D, Javed O, Shah M. Video object segmentation through spatially accurate and temporally dense extraction of primary object regions. In Proc. the 2013 CVPR, June 2013, pp.628-635.

  12. Wang W G, Shen J B, Porikli F. Saliency-aware geodesic video object segmentation. In Proc. the 2015 CVPR, June 2015, pp.3395-3402.

  13. Caelles S, Maninis K K, Pont-Tuset J, Leal-Taixé L, Cremers D, van Gool L. One-shot video object segmentation. In Proc. the 2017 CVPR, July 2017, pp.221-230.

  14. Zhang S H, Li R L, Dong X et al. Pose2Seg: Detection free human instance segmentation. In Proc. the 2019 CVPR, June 2019, pp.889-898.

  15. Perazzi F, Khoreva A, Benenson R, Schiele B, Sorkine-Hornung A. Learning video object segmentation from static images. In Proc. the 2017 CVPR, July 2017, pp.3491-3500.

  16. Perazzi F, Pont-Tuset J, McWilliams B, van Gool L, Gross M, Sorkine-Hornung A. A benchmark dataset and evaluation methodology for video object segmentation. In Proc. the 2016 CVPR, June 2016, pp.724-732.

  17. Huang Z J, Huang L C, Gong Y C et al. Mask scoring R-CNN. In Proc. the 2019 CVPR, June 2019, pp.6409-6418.

  18. Cheng M M, Mitra N J, Huang X L, Torr P H, Hu S M. Global contrast based salient region detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 37(3): 569-582.

    Article  Google Scholar 

  19. Achanta R, Shaji A, Smith K, Lucchi A, Fua P, Süsstrunk S. SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(11): 2274-2282.

    Article  Google Scholar 

  20. Mannan S K, Kennard C, Husain M. The role of visual salience in directing eye movements in visual object agnosia. Current Biology, 2009, 19(6): R247-R248.

    Article  Google Scholar 

  21. Hou X D, Harel J, Koch C. Image signature: Highlighting sparse salient regions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 34(1): 194-204.

    Google Scholar 

  22. Rother C, Kolmogorov V, Blake A. “GrabCut” interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics, 2004, 23(3): 309-314

    Article  Google Scholar 

  23. Papazoglou A, Ferrari V. Fast object segmentation in unconstrained video. In Proc. the 2013 IEEE Int. Conf. Computer Vision, December 2013, pp.1777-1784.

  24. Wang W G, Shen J B, Yang R G, Porikli F. Saliency-aware video object segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 40(1): 20-33.

    Article  Google Scholar 

  25. Seo H J, Milanfar P. Static and space-time visual saliency detection by self-resemblance. Journal of Vision, 2009, 9(12): Article No. 15.

  26. Guo C, Ma Q, Zhang L. Spatio-temporal saliency detection using phase spectrum of quaternion Fourier transform. In Proc. the 2008 CVPR, June 2008.

  27. Fu H, Cao X, Tu Z. Cluster-based co-saliency detection. IEEE Transactions on Image Processing, 2013, 22(10): 3766-3778.

    Article  MathSciNet  Google Scholar 

  28. Zhou F, Kang B S, Cohen M F. Time-mapping using space-time saliency. In Proc. the 2014 CVPR, June 2014, pp.3358-3365.

  29. Wang W, Shen J, Yang R, Porikli F. Saliency-aware video object segmentation. IEEE Trans. Pattern Anal. Mach. Intell., 2018, 40(1): 20-33.

    Article  Google Scholar 

  30. Perazzi F, Krähenbühl P, Pritch Y, Hornung A. Saliency filters: Contrast based filtering for salient region detection. In Proc. the 2012 CVPR, June 2012, pp.733-740.

  31. Tsai Y H, Yang M H, Black M J. Video segmentation via object flow. In Proc. the 2016 CVPR, June 2016, pp.3899-3908.

Download references

Author information

Authors and Affiliations

  1. College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, 518060, China

    Hui-Si Wu, Meng-Shu Liu, Lu-Lu Yin & Zhen-Kun Wen

  2. Department of Computing, The Hong Kong Polytechnic University, Hong Kong, 999077, China

    Ping Li

  3. Faculty of Information Technology, Macau University of Science and Technology, Macau, 999078, China

    Hon-Cheng Wong

Authors
  1. Hui-Si Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Meng-Shu Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lu-Lu Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ping Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhen-Kun Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hon-Cheng Wong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Hui-Si Wu or Zhen-Kun Wen.

Electronic supplementary material

ESM 1

(PDF 327 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, HS., Liu, MS., Yin, LL. et al. Automatic Video Segmentation Based on Information Centroid and Optimized SaliencyCut. J. Comput. Sci. Technol. 35, 564–575 (2020). https://doi.org/10.1007/s11390-020-0246-3

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-020-0246-3

Keywords

Search

Navigation