Fast depth map mode decision based on depth–texture correlation and edge classification for 3D-HEVC☆
Introduction
Three-dimensional (3D) video has received more and more attention by the development of 3D content acquisition and display technologies in recent years. Multi-view video plus depth (MVD) is one of the most promising 3D video representations to support depth perception of 3D scenes [1], [2]. In the MVD systems, a small number of captured texture video and its corresponding depth map are coded and the resulting bitstream packets are multiplexed into a 3D video bitstream [3], [4]. After decoding texture video and depth map, much more intermediate virtual views suitable for displaying the 3D content on a free-viewpoint display can be synthesized using a depth-image-based-rendering (DIBR) technique [5]. Especially, the additional intermediate virtual view quality (rendered by the DIBR) highly depends on the coding result of depth map. Thus, high efficient depth map coding is most crucial to realize the 3D video practical applications.
As a result, efficient depth map coding has been recently investigated by Joint Collaborative Team on 3D Video Coding Extension Development (JCT-3V) established by international organization for standardization (ISO) MPEG and the international telecommunication union (ITU) VCEG, and 3D extension of high efficiency video coding (3D-HEVC) is developed for the effective compression of depth map data [6]. Different from the conventional texture video compression [7], it is key point to preserve the depth sharp object edges rather than the depth map visual quality. Based on this characteristic, 3D-HEVC introduces several new prediction modes and coding tools to preserve the sharp object edges in depth map coding, such as the Depth Modeling Mode (DMM) [8], segment-wise depth coding (SDC) [9], and motion parameter inheritance (MPI) [10]. Meanwhile, a computationally expensive exhaustive quadtree coding structure of HEVC is also used in 3D-HEVC depth map coding. Hence, the 3D-HEVC mode decision is required to be checked all combinations of conventional HEVC prediction modes [11], [12], [13] with the additional new tools to find the one with least rate distortion cost in depth map coding. These techniques achieve the highest possible coding efficiency but require a very high computational complexity, which limit 3D-HEVC from the practical applications. Therefore, fast algorithms, which can reduce the complexity of depth map coding without sacrificing rate distortion (RD) performance, are extremely necessary for 3D-HEVC real-time applications.
A number of fast algorithms [14], [15], [16], [17], [18], [19] have been proposed to reduce the depth map computational complexity for previous video coding standards (such as H.264/AVC and its extension of multi-view video coding (MVC)), achieving significant time savings with acceptable video quality degradation. However, these fast algorithms are not directly applicable to the new standard 3D-HEVC, which high computational complexity is intrinsically related to the use of new prediction modes and coding tools in 3D-HEVC encoder. Recently, several studies on the reduction of depth map coding complexity have been reported for 3D-HEVC encoders in the literature. A fast depth map wedgelet partitioning scheme is presented in [20] based on adaptively utilizing the mode with minimal cost in rough mode decision of HEVC intra prediction. Fast depth mode decision algorithms are proposed in [21], [22] to early terminate the unnecessary prediction modes with full RD cost calculation in 3D-HEVC. A fast mode decision algorithm based on simplified edge detector is proposed in [23] to reduce the complexity of the 3D-HEVC depth intra prediction. Fast depth mode decision algorithms are employed in [24], [25] to selectively omit unnecessary DMM based on the pre-calculated RD costs of the HEVC intra modes and the edge classification. A fast depth intra mode decision is introduced in [26] to reduce the depth intra complexity in a smooth region. A fast algorithm is designed in [27] based on the early SKIP mode detection and the prediction size correlation based-mode decision to reduce 3D-HEVC encoding time for real-time applications. A flexible block partitioning is employed in [28] to efficiently represent the depth map smooth areas delimited by sharp edges. A fast depth map coding algorithm is proposed in [29] to reduce the computation complexity of the 3D-HEVC encoder by utilizing early Skip and early DIS scheme. A fast depth map quadtree structure determination scheme is designed in [30] to terminate the quadtree-based partition process of coding tree unit(CTU) as early as possible. A fast mode decision algorithm based on the grayscale similarity and inter-view correlation is proposed in [31] to reduce the complexity of depth map coding by skip unnecessary mode checking within the mode decision process. Two fast algorithms including the squared euclidean distance of variances (SEDV) and probability-based early depth intra mode decision (PBED) are presented in [32] to speeding up the most time-consuming intra mode processes in 3D-HEVC depth map coding. Effective early termination and intra mode decision algorithms are also developed in our previous work [33], [34] to reduce the depth map coding complexity of 3D-HEVC encoders. The aforementioned algorithms are well developed for depth map coding achieving significant time savings in 3D-HEVC. However, most of these fast algorithms have not adequately exploited the correlation of depth map and texture video. Additionally, the characteristics of the depth map are not fully studied. This situation results in a limited time saving. Furthermore, most of the previous fast depth map algorithms are not designed for the recent 3D-HEVC test model HTM-16.0 [4]. There is still some room for further reduce mode decision complexity of the 3D-HEVC depth map coding.
To further relieve the computation complexity of depth map mode decision, this paper proposes a fast depth map mode decision algorithm for 3D-HEVC encoders based on the depth–texture correlation and edge classification. The main idea of the proposed algorithm is that the correlation of depth map–texture video and the edge information of depth map are used to analyze the current depth map coding unit (CU) prediction mode and early skip unnecessary variable-size mode decision. It consists of four fast mode decision strategies: adaptive depth map coding levels determination, early depth map SKIP/Merge mode detection, fast depth map inter mode size decision and Selective depth map intra prediction. Extensive experimental results demonstrate that the proposed fast mode decision algorithm can significantly reduce the depth map encoding time of 3D-HEVC while maintaining almost the same RD performance as the original encoder.
The rest of this paper is organized as follows. The proposed fast depth map mode decision algorithm is detail in Section 2. Simulation results and conclusions are given in Sections 3 Experimental results, 4 Conclusion, respectively.
Section snippets
Adaptive depth map coding levels determination based on the depth–texture correlation
3D-HEVC inherits an advanced quadtree-based coding approach from HEVC, wherein a picture is divided into coding tree unit (CTU) [35]. The CTU can then be split into four CUs, and the CU is the basic unit of region splitting used for inter/intra prediction, which allows recursive subdividing into four equally sized blocks. A specified maximum coding level is set to limit the CU split recursion. In the joint model of 3D-HEVC, a complex RD optimization process is performed all the possible coding
Experimental results
In order to evaluate the efficiency of the proposed fast depth map mode decision algorithm for 3D-HEVC including four components, adaptive depth map coding levels determination, early depth map SKIP/Merge mode detection, fast depth map inter mode size decision and selective depth map intra prediction, they have been implemented on the recent 3D-HEVC reference software HTM 16.0 [37]. The proposed algorithm is evaluated with eight sequences recommended by JCT-3V Group with two resolutions (1024 ×
Conclusion
In this paper, we propose a fast depth map mode decision based on depth–texture correlation and edge classification to reduce the computational complexity of 3D-HEVC encoders, which includes four approaches, i.e., adaptive depth map coding levels determination, early depth map SKIP/Merge mode detection, fast depth map inter mode size decision and selective depth map intra prediction. It makes use of the correlation of depth map–texture video and the edge information of depth map to predict the
Acknowledgments
The authors would like to thank the editors and anonymous reviewers for their valuable comments. This work was supported in part by the National Natural Science Foundation of China under grant No. 61302118, 61401404, 61501407, 61572445, and 61502435, the Program for Science and Technology Innovation Talents in Universities of Henan Province under grant No. 17HASTIT022, the Funding Scheme of Young Key Teacher of Henan Province Universities under grant No. 2016GGJS-087, the Scientific and
References (40)
- et al.
Low complexity bi-partition mode selection for 3D video depth intra coding
Displays
(2015) - et al.
Next-generation 3D formats with depth map support
IEEE MultiMedia
(2014) - et al.
3-D video representation using depth maps
Proc. IEEE
(2011) - et al.
3D high efficiency video coding for multi-view video and depth data
IEEE Trans. Image Process.
(2013) - Y. Chen, G. Tech, K. Wegner, S. Yea, Test model 11 of 3D-HEVC and MV-HEVC, in: Joint Collaborative Team on 3D Video...
- et al.
Depth map creation and image based rendering for advanced 3DTV services providing interoperability and scalability
Sign. Process. Image Commun.
(2007) - et al.
Overview of the multiview and 3D extensions of high efficiency video coding
IEEE Trans. Circ. Syst. Video Technol.
(2016) - et al.
Parallel deblocking filter for HEVC on many-core processor
Electron. Lett.
(2014) - et al.
3D video coding with depth modeling modes and view synthesis optimization
Simplified depth map intra coding with an optional depth lookup table
Motion vector inheritance for high efficiency 3D video plus depth coding
Highly parallel framework for HEVC motion estimation on many-core platform
A highly parallel framework for HEVC coding unit partitioning tree decision on many-core processors
IEEE Sign. Process. Lett.
Efficient parallel framework for HEVC motion estimation on many-core processors
IEEE Trans. Circ. Syst. Video Technol.
A fast and efficient multi-view depth image coding method based on temporal and inter-view correlations of texture images
IEEE Trans. Circ. Syst. Video Technol.
Low complexity multiview video plus depth coding
IEEE Trans. Consum. Electron.
Inter mode selection for depth map coding in 3D video
IEEE Trans. Consum. Electron.
Low complexity mode decision and motion estimation for H.264/AVC based depth maps encoding in free viewpoint video
IEEE Trans. Circ. Syst. Video Techn.
Low complexity depth coding assisted by coding information from color video
IEEE Trans. Broadcast.
Fast mode decision using inter-view and inter-component correlations for multiview depth video coding
IEEE Trans. Ind. Infor.
Cited by (32)
Low complexity intra prediction mode decision for 3D-HEVC depth coding
2018, Signal Processing: Image CommunicationCitation Excerpt :In addition, PBED helps to select the most promising mode and to make the early termination of using SDC based on the low complexity RD Cost in rough mode decision. In [13] authors proposed a low-complexity algorithm for depth map mode decision of 3D-HEVC based on the correlation of the depth map-texture video. Therefore, edge classification allows to define four procedures of intra/inter prediction, i.e. select adaptive coding levels in depth map coding, an early Skip/Merge mode method, fast depth map inter mode size decision and a selective depth map intra prediction.
An adaptive CU size decision algorithm based on gradient boosting machines for 3D-HEVC inter-coding
2023, Multimedia Tools and ApplicationsContent-adaptive mode decision for low complexity 3D-HEVC
2023, Multimedia Tools and ApplicationsFast CU size decision algorithm for VVC intra coding
2023, Multimedia Tools and ApplicationsFast 3D-HEVC inter coding using data mining and machine learning
2022, IET Image ProcessingApplication of 3D-HEVC fast coding by Internet of Things data in intelligent decision
2022, Journal of Supercomputing
- ☆
This paper has been recommended for acceptance by Zicheng Liu.