Fast depth map mode decision based on depth–texture correlation and edge classification for 3D-HEVC

https://doi.org/10.1016/j.jvcir.2017.03.004Get rights and content

Highlights

  • A fast depth map mode decision algorithm for 3D-HEVC is proposed.

  • The depth map-texture video correlation is exploited throughout fast mode detection.

  • The edge classification is employed in the procedure of intra/inter prediction.

  • Experimental results demonstrate the effectiveness of the proposed algorithm.

Abstract

The 3D extension of High Efficiency Video Coding (3D-HEVC) has been adopted as the emerging 3D video coding standard to support the multi-view video plus depth map (MVD) compression. In the joint model of 3D-HEVC design, the exhaustive mode decision is required to be checked all the possible prediction modes and coding levels to find the one with least rate distortion cost in depth map coding. Furthermore, new coding tools (such as depth-modeling mode (DMM) and segment-wise depth coding (SDC)) are exploited for the characteristics of depth map to improve the coding efficiency. These achieve the highest possible coding efficiency to code depth map, but also bring a significant computational complexity which limits 3D-HEVC from real-time applications. In this paper, we propose a fast depth map mode decision algorithm for 3D-HEVC by jointly using the correlation of depth map-texture video and the edge information of depth map. Since the depth map and texture video represent the same scene at the same time instant (they have the same motion characteristics), it is not efficient to use all the prediction modes and coding levels in depth map coding. Therefore, we can skip some specific prediction modes and depth coding levels rarely used in corresponding texture video. Meanwhile, the depth map is mainly characterized by sharp object edges and large areas of nearly constant regions. By fully exploiting these characteristics, we can skip some prediction modes which are rarely used in homogeneity regions based on the edge classification. Experimental results show that the proposed algorithm achieves considerable encoding time saving while maintaining almost the same rate-distortion (RD) performance as the original 3D-HEVC encoder.

Introduction

Three-dimensional (3D) video has received more and more attention by the development of 3D content acquisition and display technologies in recent years. Multi-view video plus depth (MVD) is one of the most promising 3D video representations to support depth perception of 3D scenes [1], [2]. In the MVD systems, a small number of captured texture video and its corresponding depth map are coded and the resulting bitstream packets are multiplexed into a 3D video bitstream [3], [4]. After decoding texture video and depth map, much more intermediate virtual views suitable for displaying the 3D content on a free-viewpoint display can be synthesized using a depth-image-based-rendering (DIBR) technique [5]. Especially, the additional intermediate virtual view quality (rendered by the DIBR) highly depends on the coding result of depth map. Thus, high efficient depth map coding is most crucial to realize the 3D video practical applications.

As a result, efficient depth map coding has been recently investigated by Joint Collaborative Team on 3D Video Coding Extension Development (JCT-3V) established by international organization for standardization (ISO) MPEG and the international telecommunication union (ITU) VCEG, and 3D extension of high efficiency video coding (3D-HEVC) is developed for the effective compression of depth map data [6]. Different from the conventional texture video compression [7], it is key point to preserve the depth sharp object edges rather than the depth map visual quality. Based on this characteristic, 3D-HEVC introduces several new prediction modes and coding tools to preserve the sharp object edges in depth map coding, such as the Depth Modeling Mode (DMM) [8], segment-wise depth coding (SDC) [9], and motion parameter inheritance (MPI) [10]. Meanwhile, a computationally expensive exhaustive quadtree coding structure of HEVC is also used in 3D-HEVC depth map coding. Hence, the 3D-HEVC mode decision is required to be checked all combinations of conventional HEVC prediction modes [11], [12], [13] with the additional new tools to find the one with least rate distortion cost in depth map coding. These techniques achieve the highest possible coding efficiency but require a very high computational complexity, which limit 3D-HEVC from the practical applications. Therefore, fast algorithms, which can reduce the complexity of depth map coding without sacrificing rate distortion (RD) performance, are extremely necessary for 3D-HEVC real-time applications.

A number of fast algorithms [14], [15], [16], [17], [18], [19] have been proposed to reduce the depth map computational complexity for previous video coding standards (such as H.264/AVC and its extension of multi-view video coding (MVC)), achieving significant time savings with acceptable video quality degradation. However, these fast algorithms are not directly applicable to the new standard 3D-HEVC, which high computational complexity is intrinsically related to the use of new prediction modes and coding tools in 3D-HEVC encoder. Recently, several studies on the reduction of depth map coding complexity have been reported for 3D-HEVC encoders in the literature. A fast depth map wedgelet partitioning scheme is presented in [20] based on adaptively utilizing the mode with minimal cost in rough mode decision of HEVC intra prediction. Fast depth mode decision algorithms are proposed in [21], [22] to early terminate the unnecessary prediction modes with full RD cost calculation in 3D-HEVC. A fast mode decision algorithm based on simplified edge detector is proposed in [23] to reduce the complexity of the 3D-HEVC depth intra prediction. Fast depth mode decision algorithms are employed in [24], [25] to selectively omit unnecessary DMM based on the pre-calculated RD costs of the HEVC intra modes and the edge classification. A fast depth intra mode decision is introduced in [26] to reduce the depth intra complexity in a smooth region. A fast algorithm is designed in [27] based on the early SKIP mode detection and the prediction size correlation based-mode decision to reduce 3D-HEVC encoding time for real-time applications. A flexible block partitioning is employed in [28] to efficiently represent the depth map smooth areas delimited by sharp edges. A fast depth map coding algorithm is proposed in [29] to reduce the computation complexity of the 3D-HEVC encoder by utilizing early Skip and early DIS scheme. A fast depth map quadtree structure determination scheme is designed in [30] to terminate the quadtree-based partition process of coding tree unit(CTU) as early as possible. A fast mode decision algorithm based on the grayscale similarity and inter-view correlation is proposed in [31] to reduce the complexity of depth map coding by skip unnecessary mode checking within the mode decision process. Two fast algorithms including the squared euclidean distance of variances (SEDV) and probability-based early depth intra mode decision (PBED) are presented in [32] to speeding up the most time-consuming intra mode processes in 3D-HEVC depth map coding. Effective early termination and intra mode decision algorithms are also developed in our previous work [33], [34] to reduce the depth map coding complexity of 3D-HEVC encoders. The aforementioned algorithms are well developed for depth map coding achieving significant time savings in 3D-HEVC. However, most of these fast algorithms have not adequately exploited the correlation of depth map and texture video. Additionally, the characteristics of the depth map are not fully studied. This situation results in a limited time saving. Furthermore, most of the previous fast depth map algorithms are not designed for the recent 3D-HEVC test model HTM-16.0 [4]. There is still some room for further reduce mode decision complexity of the 3D-HEVC depth map coding.

To further relieve the computation complexity of depth map mode decision, this paper proposes a fast depth map mode decision algorithm for 3D-HEVC encoders based on the depth–texture correlation and edge classification. The main idea of the proposed algorithm is that the correlation of depth map–texture video and the edge information of depth map are used to analyze the current depth map coding unit (CU) prediction mode and early skip unnecessary variable-size mode decision. It consists of four fast mode decision strategies: adaptive depth map coding levels determination, early depth map SKIP/Merge mode detection, fast depth map inter mode size decision and Selective depth map intra prediction. Extensive experimental results demonstrate that the proposed fast mode decision algorithm can significantly reduce the depth map encoding time of 3D-HEVC while maintaining almost the same RD performance as the original encoder.

The rest of this paper is organized as follows. The proposed fast depth map mode decision algorithm is detail in Section 2. Simulation results and conclusions are given in Sections 3 Experimental results, 4 Conclusion, respectively.

Section snippets

Adaptive depth map coding levels determination based on the depth–texture correlation

3D-HEVC inherits an advanced quadtree-based coding approach from HEVC, wherein a picture is divided into coding tree unit (CTU) [35]. The CTU can then be split into four CUs, and the CU is the basic unit of region splitting used for inter/intra prediction, which allows recursive subdividing into four equally sized blocks. A specified maximum coding level is set to limit the CU split recursion. In the joint model of 3D-HEVC, a complex RD optimization process is performed all the possible coding

Experimental results

In order to evaluate the efficiency of the proposed fast depth map mode decision algorithm for 3D-HEVC including four components, adaptive depth map coding levels determination, early depth map SKIP/Merge mode detection, fast depth map inter mode size decision and selective depth map intra prediction, they have been implemented on the recent 3D-HEVC reference software HTM 16.0 [37]. The proposed algorithm is evaluated with eight sequences recommended by JCT-3V Group with two resolutions (1024 × 

Conclusion

In this paper, we propose a fast depth map mode decision based on depth–texture correlation and edge classification to reduce the computational complexity of 3D-HEVC encoders, which includes four approaches, i.e., adaptive depth map coding levels determination, early depth map SKIP/Merge mode detection, fast depth map inter mode size decision and selective depth map intra prediction. It makes use of the correlation of depth map–texture video and the edge information of depth map to predict the

Acknowledgments

The authors would like to thank the editors and anonymous reviewers for their valuable comments. This work was supported in part by the National Natural Science Foundation of China under grant No. 61302118, 61401404, 61501407, 61572445, and 61502435, the Program for Science and Technology Innovation Talents in Universities of Henan Province under grant No. 17HASTIT022, the Funding Scheme of Young Key Teacher of Henan Province Universities under grant No. 2016GGJS-087, the Scientific and

References (40)

  • Z. Gu et al.

    Low complexity bi-partition mode selection for 3D video depth intra coding

    Displays

    (2015)
  • Y. Chen et al.

    Next-generation 3D formats with depth map support

    IEEE MultiMedia

    (2014)
  • K. Müller et al.

    3-D video representation using depth maps

    Proc. IEEE

    (2011)
  • K. Müller et al.

    3D high efficiency video coding for multi-view video and depth data

    IEEE Trans. Image Process.

    (2013)
  • Y. Chen, G. Tech, K. Wegner, S. Yea, Test model 11 of 3D-HEVC and MV-HEVC, in: Joint Collaborative Team on 3D Video...
  • P. Kau et al.

    Depth map creation and image based rendering for advanced 3DTV services providing interoperability and scalability

    Sign. Process. Image Commun.

    (2007)
  • G. Tech et al.

    Overview of the multiview and 3D extensions of high efficiency video coding

    IEEE Trans. Circ. Syst. Video Technol.

    (2016)
  • C. Yan et al.

    Parallel deblocking filter for HEVC on many-core processor

    Electron. Lett.

    (2014)
  • K. Muller et al.

    3D video coding with depth modeling modes and view synthesis optimization

  • F. Jager

    Simplified depth map intra coding with an optional depth lookup table

  • M. Winken et al.

    Motion vector inheritance for high efficiency 3D video plus depth coding

  • C. Yan et al.

    Highly parallel framework for HEVC motion estimation on many-core platform

  • C. Yan et al.

    A highly parallel framework for HEVC coding unit partitioning tree decision on many-core processors

    IEEE Sign. Process. Lett.

    (2014)
  • C. Yan et al.

    Efficient parallel framework for HEVC motion estimation on many-core processors

    IEEE Trans. Circ. Syst. Video Technol.

    (2014)
  • J.Y. Lee et al.

    A fast and efficient multi-view depth image coding method based on temporal and inter-view correlations of texture images

    IEEE Trans. Circ. Syst. Video Technol.

    (2011)
  • Q. Zhang et al.

    Low complexity multiview video plus depth coding

    IEEE Trans. Consum. Electron.

    (2011)
  • L. Shen et al.

    Inter mode selection for depth map coding in 3D video

    IEEE Trans. Consum. Electron.

    (2012)
  • G. Cernigliaro et al.

    Low complexity mode decision and motion estimation for H.264/AVC based depth maps encoding in free viewpoint video

    IEEE Trans. Circ. Syst. Video Techn.

    (2013)
  • L. Shen et al.

    Low complexity depth coding assisted by coding information from color video

    IEEE Trans. Broadcast.

    (2014)
  • J. Lei et al.

    Fast mode decision using inter-view and inter-component correlations for multiview depth video coding

    IEEE Trans. Ind. Infor.

    (2015)
  • Cited by (32)

    • Low complexity intra prediction mode decision for 3D-HEVC depth coding

      2018, Signal Processing: Image Communication
      Citation Excerpt :

      In addition, PBED helps to select the most promising mode and to make the early termination of using SDC based on the low complexity RD Cost in rough mode decision. In [13] authors proposed a low-complexity algorithm for depth map mode decision of 3D-HEVC based on the correlation of the depth map-texture video. Therefore, edge classification allows to define four procedures of intra/inter prediction, i.e. select adaptive coding levels in depth map coding, an early Skip/Merge mode method, fast depth map inter mode size decision and a selective depth map intra prediction.

    • Content-adaptive mode decision for low complexity 3D-HEVC

      2023, Multimedia Tools and Applications
    • Fast CU size decision algorithm for VVC intra coding

      2023, Multimedia Tools and Applications
    View all citing articles on Scopus

    This paper has been recommended for acceptance by Zicheng Liu.

    View full text