Skip to main content
SpringerLink
Log in

Segmentation & bitstream encoding of foreground objects in HEVC encoder for edge computing environment

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Segmentation of foreground object is pivotal in identifying the other finer details about objects in a scene. Our work serves to investigate ways to not only segment but encode the foreground objects in the HEVC encoder with minimal change in original HEVC encoder performance in terms of bitrate increase and encoding time. We achieve this by reutilizing the intermediate residual data at the encoder to segment the foreground activity in each frame and finally encoding the same in the final compressed bitstream using specific provisions of HEVC high level bitstream syntax. The method operates entirely in the HEVC encoder loop, taking residual data of the frame, it divides the entire frame into 8 × 8 target patches and applies the proposed algorithm Median of Discrete Variance (MoDV) to classify the target block of each frame of the video sequence as foreground or background. The foreground information in each frame is then encoded into the compressed bitstream by harnessing our proposed format of supplemental enhancement information (SEI) Network Abstraction Layer (NAL) units to tag the location of the foreground activity. Along with the segmentation accuracy, change in encoder performance is also measured to judge the various trade-offs. We conclude by testing its efficacy on a variety of videos with difference frame resolutions and background conditions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Akilan T, Wu QMJ, Zhang W (2019) Video foreground extraction using multi-view receptive field and encoder–decoder DCNN for traffic and surveillance applications. IEEE Transactions on Vehicular Technology 68(10):9478–9493

    Article  Google Scholar 

  2. Babu RV, Ramakrishnan KR (2007) Compressed domain video retrieval using object and global motion descriptors. Multimed Tools Appl 32:93–113. https://doi.org/10.1007/s11042-006-0048-9

    Article  Google Scholar 

  3. Babu RV, Tom M, Wadekar P (2014) A survey on compressed domain video analysis techniques. Multimed Tools Appl 75(2):1043–1078

    Article  Google Scholar 

  4. Baser M, Mittal M, Samaiya D (2020) Real time foreground segmentation for video sequences with dynamic background. In: 2020 IEEE 17th India council international conference (INDICON)

    Google Scholar 

  5. Chiranjeevi P, Sengupta S (2014) Neighbourhood supported model level fuzzy aggregation for moving object segmentation. IEEE Trans Image Process 23(2):645–657

    Article  MathSciNet  Google Scholar 

  6. Common conditions and software reference configurations JCT-VC, joint collaborative team on video coding (JCT-VC) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 doc. JCTVC-H1100 (2012) http://phenix.it-sudparis.eu/jct/doc_end_user/documents/12_Geneva/wg11/JCTVC-L1100-v1.zip

  7. Dey B, Kundu MK (2013) Robust background subtraction for network surveillance in H.264 streaming video. IEEE Transactions on Circuits and Systems for Video Technology 23(10):1695–1703

    Article  Google Scholar 

  8. Dey B, Kundu MK (2015) Efficient foreground extraction from HEVC compressed video for application to real-time analysis of surveillance ‘big’ data. IEEE Trans Image Process 24(11):3574–3585

    Article  MathSciNet  Google Scholar 

  9. F. H. H. Institute (n.d.) High Efficiency Video Coding (HEVC). [Online]. Available: https://hevc.hhi.fraunhofer.de/. Accessed 22 Dec 2020

  10. Fan J, Zhu X, Najarian K et al (2003) Accessing video contents through key objects over IP. Multimed Tools Appl 21:75–96. https://doi.org/10.1023/A:1025086200838

    Article  Google Scholar 

  11. H.265 ITU recommendations https://www.itu.int/rec/T-REC-H.265-201504-S/en (n.d.)

  12. Haines TSF, Xiang T (2014) Background Subtraction with DirichletProcess Mixture Models. IEEE Transactions on Pattern Analysis and Machine Intelligence 36(4):670–683

    Article  Google Scholar 

  13. Helle P, Oudin S, Bross B, Marpe D, Bici MO, Ugur K, Jung J, Clare G, Wiegand T (2012) Block merging for Quadtree-based partitioning in HEVC. IEEE Transactions on Circuits and Systems for Video Technology 22(12):1720–1731

    Article  Google Scholar 

  14. Huang T, Dong S, Tian Y (2014) Representing visual objects in HEVC coding loop. IEEE Journal on Emerging and Selected Topics in Circuits and Systems 4(1):5–16

    Article  Google Scholar 

  15. ISO/IEC JTC1 14496–2: Coding of audio-visual objects – Part 2: Visual (MPEG-4 visual version 1) (1999)

  16. Liang Y, Xu M, Ren J, Wang Z (2015) Learning to segment videos in HEVC compressed domain. In: 2015 International Conference on Wireless Communications & Signal Processing (WCSP)

    Google Scholar 

  17. Maddalena L, Petrosino A (2012) The SOBS algorithm: What are the limits? In: 2012 IEEE computer society conference on computer vision and pattern recognition workshops, Providence, RI

    Google Scholar 

  18. Meier T, Ngan KN (1998) Automatic segmentation of moving objects for video object plane generation. IEEE Transactions on Circuits and Systems for Video Technology 8(5):525–538. https://doi.org/10.1109/76.718500

    Article  Google Scholar 

  19. Moriyama M, Minemura K, Wong K (2015) Moving object detection in HEVC video by frame sub-sampling. In: 2015 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS)

    Google Scholar 

  20. Reddy V, Sanderson C, Lovell BC (2013) Improved foreground detection via block-based classifier cascade with probabilistic decision integration. IEEE Transactions on Circuits and Systems for Video Technology 23(1):83–93

    Article  Google Scholar 

  21. Sabirin H, Kim M (2012) Moving object detection and tracking using a Spatio-temporal graph in H.264/AVC Bitstreams for video surveillance. IEEE Transactions on Multimedia 14(3):657–668

    Article  Google Scholar 

  22. Samaiya D, Gupta KK (2018) Intelligent video surveillance for real time energy savings in smart buildings using HEVC compressed domain features. Multimed Tools Appl 77(21):29059–29076

    Article  Google Scholar 

  23. Sanches SRR, Oliveira C, Sementille AC, Freire V (2018) Challenging situations for background subtraction algorithms. Appl Intell 49(5):1771–1784

    Article  Google Scholar 

  24. Stauffer C, Grimson WEL (1999) Adaptive background mixture models for real-time tracking. In: Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149), Fort Collins, CO, USA, vol 2, pp 246–252. https://doi.org/10.1109/CVPR.(1999

    Chapter  Google Scholar 

  25. Sullivan GJ, Ohm J-R, Han W-J, Wiegand T (2012) Overview of the high efficiency video coding (HEVC) standard. IEEE Transactions on Circuits and Systems for Video Technology 22(12):1649–1668

    Article  Google Scholar 

  26. Wang Y, Jodoin P-M, Porikli F, Konrad J, Benezeth Y, Ishwar P (2014) CDnet 2014: An Expanded Change Detection Benchmark Dataset. In: Proc. IEEE Workshop on Change Detection (CDW-2014) at CVPR-2014, pp 387–394

    Google Scholar 

  27. Xu Y, Dong J, Zhang B, Xu D (2016) Background modeling methods in video analysis: a review and comparative evaluation. CAAI Transactions on Intelligence Technology 1(1):43–60

    Article  Google Scholar 

  28. Zhao L, He Z, Cao W, Zhao D (2018) Real-time moving object segmentation and classification from HEVC compressed surveillance video. IEEE Transactions on Circuits and Systems for Video Technology 28(6):1346–1357

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Birla Institute of Technology & Science, Pilani, Rajasthan, 333031, India

    Devesh Samaiya & Karunesh Kumar Gupta

Authors
  1. Devesh Samaiya
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Karunesh Kumar Gupta
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Devesh Samaiya.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Samaiya, D., Gupta, K.K. Segmentation & bitstream encoding of foreground objects in HEVC encoder for edge computing environment. Multimed Tools Appl 81, 18397–18416 (2022). https://doi.org/10.1007/s11042-022-12198-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-12198-3

Keywords

Search

Navigation