1.

Introduction

With the huge increase of video traffic on the internet and the emergence of new video content, such as virtual reality, high frame rate, and 360 video with resolutions of up to 4K, 8K, and even 16K, several advanced video compression standards have been developed, mainly by the VCEG and MPEG groups. This collaboration aims to identify the future generation of video coding. Therefore, the joint video experts team, established by the VCEG and MPEG groups, standardized a new video codec called Versatile Video Coding (VVC)¹ in July of 2020. The VVC standard is used to improve the coding efficiency of the previous codec High-Efficiency Video Coding (HEVC)² by providing a significant gain of about 50% in bitrate with the same video quality. The latest VVC standard enhances the RD performance at the cost of an increase in computational complexity caused by the improvement tools added to the VTM reference software. The new advanced quadtree with nested multitype tree (QTMT) partition structure is one of the most complicated improvement techniques compared with the quadtree plus binary tree (QTBT) decision tree used in the JEM reference software and the quadtree (QT) structure employed in HEVC.³ In addition to the different square blocks supported by the QT scheme ($128\times 128$, $64\times 64$, $32\times 32$, $16\times 16,$ and $8\times 8$), the VVC encoder provides further rectangular blocks in the horizontal and vertical splitting directions for the BT and TT structures ($32\times 16$, $16\times 32$, $16\times 8$, $8\times 16$, $32\times 8$, $8\times 32$, $32\times 4$, $4\times 32$, $8\times 4$, and $4\times 8$) to improve the coding units (CUs) partitioning shapes’ flexibility. Thus, a fast QTMT decision algorithm has been developed to reduce the computational complexity at all intra (AI) configuration in the face of the large CU partition sizes. Several works on video coding have exploited the popularity of the deep learning (DL) field to achieve a trade-off between complexity reduction and compression efficiency. In the literature, the DL-based approach results outperform those of the machine learning-based methods and the statistical-based approaches in video processing. Recently, the DL algorithms based on the convolution neural network (CNN) have shown significant coding results in terms of complexity reduction while maintaining almost the RD performance. This work focuses on developing a fast CU partition method while reducing the QTMT structure complexity. Therefore, a deep QTMT partitioning algorithm based on the CNN is designed in this study to minimize the VVC complexity at AI configuration.

In this paper, five parts are introduced as follows. Section 2 studies the related works on QTBT and QTMT complexity reduction for the VVC standard. Section 3 reviews the QTMT partition decision process. In Sec. 4, the proposed intra QTMT algorithm applying several CNNs is described, with different network architectures being designed for the QT and BT structures in the horizontal and vertical directions. The experimental results of the current approach are discussed in Sec. 5. Finally, Sec. 6 concludes the present work.

2.

Related Works

Several works have been developed in recent years to overcome the VVC complexity and HEVC standards by providing a trade-off between bitrate, video quality, and encoding time using ML techniques and deep methods.

For HEVC, an effective block partition decision algorithm in intra-coding⁴ was implemented to accelerate the encoding process by developing an early determination of the CU mode decision and a bypass strategy for a large block size partitioning. This approach reached a maximum of 67% complexity reduction with almost the same bitrate and quality compared with the original HM 10.0 software. Another work based on the determination of the CU depth range and on early termination algorithms was presented by Shen et al.⁵ to remove the motion estimation (ME) on unnecessary block sizes. The results demonstrated that the introduced approach reduces the computational complexity with almost the same coding efficiency compared with the original HM 2.0 reference software. In Ref. 6, an adaptive inter-mode decision approach was presented to reduce the encoding time of the HEVC standard by integrating three strategies including the early SKIP mode decision, prediction size correlation-based mode decision, and RD cost correlation-based mode decision. The proposed algorithm reduced the computational complexity by 49% to 52% on average with negligible RD performance degradation.

For the VVC standard, the work presented in Ref. 7 introduced a fast intra multitype tree (MTT) algorithm based on machine learning. The advanced method applied the random forest models to predict the intra MTT partition and skip the unnecessary ones using visual perception. The achieved results prove that the algorithm saved an important encoding time while keeping an effective coding performance. Dong et al.¹ suggested an intra-fast mode partition approach for VVC using both mode selection and prediction termination called adaptive mode pruning (AMP) and mode-dependent termination (MDT). In fact, the AMP stage deletes nonpromising prediction modes for the mode selection using various classifiers, and the MDT step skips the unnecessary prediction of the remaining CU levels. Experimental results demonstrate that the approach reduced the encoding time by 52% on average with a Bjontegaard bitrate (BDBR) increase from 0.93% to 1.08%. Yang et al.² introduced a fast intra-QTMT partitioning algorithm. A process called “early termination” was developed to determine the QTMT modes using the decision tree. The algorithm was utilized to control the transition from the given depth to the next using the different characteristics of each CU. The proposed algorithm reduced computational complexity by 63% on average with a BDBR increase of 1.93% and a Bjontegaad peak signal-to-noise ratio (BDPSNR) decrease of 0.11 dB. Another fast decision algorithm was developed at different CU sizes by Chen et al.⁸ The approach implemented different SVM classifiers to determine the QTMT structure. In fact, the CU features were selected to identify the partitioning directions. Then, these features were applied to train the SVM models at different CU sizes. Finally, the trained models were implemented to predict the QTMT partition structure. The proposed algorithm provided a significant gain in coding time, reaching 51.01% with a loss of 1.54% in BDBR. In Ref. 9, Zhang et al. elaborated a fast partition decision approach for a given CU based on the directed acyclic graph-support vector machine (DAG-SVM) model to reduce the partition complexity. The features were first extracted by encoding the video sequences on VTM4.0. These features were then used to train the DAG-SVM model. Finally, the classifier was implemented on VTM4.0 to predict the optimal partitioning modes. The results indicated that the proposed method saved 54.74% in coding time with a loss of 0.93% in BDBR. A fast method based on the CU partition and the intra-mode prediction was developed by Zhang et al.¹⁰ This method included two strategies: a fast CU partition using the random forest classifier (RFC) model and a fast intra-prediction mode using the texture of regions. The algorithm was based on splitting a given CU with the RFC and analyzing its energy with four predefined prediction modes using the region features to skip unnecessary intra-prediction modes. The results indicated that the presented algorithm achieved a gain of 54.91% in encoding time accompanied by a loss of 0.93% in BDBR. In Ref. 11, Wu et al. introduced an SVM classifier for the intra-partition module. The algorithm removed the redundant partitions using the different textures of each block with several SVM classifiers being developed for different block sizes. Threshold values were set for each classifier to balance the coding complexity with the RD performance. This proposed approach helped to reduce the coding time from 30.78% to 63.16% with an increase from 1.10% to 2.71% in BDBR.

For the deep methods, several DL algorithms have been developed at the partitioning module using the neural network. Jin et al.¹² generated an intra QTBT partitioning approach based on the CNN. The classifier was implemented to predict the QTBT partitioning decision of each block sized $32\times 32$, minimizing the RD cost computational complexity. The presented method offers a gain of about 42% in coding time with a loss of about 0.69% in BDBR. Also, Amna et al.¹³ suggested a fast QT partition algorithm based on a CNN model by predicting the decision tree of each block instead of applying the rate distortion optimization (RDO) process. This approach developed a CNN architecture using the three partition levels of the QT structure for a better intra QTBT structure performance. Experimental results showed that this method decreased the coding time by 35% on average with a BDBR rise of 1.7%. Another work based on determining the adaptive CU partition was introduced in Ref. 14 using a neural network called pooling-variable CNN. The approach computed the residual CU gradient to select the appropriate QTMT decision. Three decisions were used: dividing the given CU, not dividing it, or applying the pooling CNN to make the partitioning decision tree. This algorithm reduced the computational complexity by 33% with an increase of 0.99% in BDBR. Tissier et al. developed another work based on a CNN to overcome the complexity of the partitioning module. The CNN architecture model was inspired by the ResNet network.¹⁵ The CNN was used to analyze the CUs texture sized $64\times 64$ and predict a vector of probabilities for the different boundaries of blocks $4\times 4$. From these probabilities, a split probability was derived and compared with a predefined threshold value. The proposed method minimized the coding complexity by about 51% with an increase of 1.45% in bitrate. In Ref. 16, a network called multi-stage exit-CNN (MSE-CNN) was created to predict the intra QTMT partitioning structure. The MSE-CNN model predicted the CU decision tree by avoiding the Rdcost function through a multithreshold decision. The approach achieved a gain of 44.65% and 66.88% in encoding time with a BDBR increase of 1.322% and 3.188%, respectively, using two different threshold values. In Ref. 17, Tech et al. elaborated a fast intra-partition approach for the VVC encoder. A CNN was created to reduce the number of partition modes tested while maintaining the RD performance. The neural network estimated the parameters that limited the horizontal and vertical MTT partitions for each block sized $32\times 32$. The proposed CNN was stimulated by the ResNet model¹⁵ with a luminance block input sized $32\times 32$. Experimental results showed a reduction in encoding time of about 50% with a loss of 0.9% in BDBR. In Ref. 18, a multibranch CNN approach based on intra QTMT partitioning was presented to deal with computational complexity. The approach aimed to predict the depth of the QTMT partitioning for each block of size $32\times 32$ by applying two phases. Indeed, The CNN was used to predict the QT depth and determine the TT mode decision. Then, these prediction depths were adopted to ignore some QTMT partition combinations. The outcomes showed that the presented method reached a gain of 42.34% in coding time and a loss of 0.71% in BDBR compared with VTM6.1. In Ref. 19, Abdallah et al. developed a fast intra QTMT decision tree based on a neural network called early-terminated hierarchical CNN (ETH-CNN). The proposed algorithm predicted the QT partition depth of the CU $64\times 64$ based on the partition decision of splitting or skipping the current CU. The current method reduced the encoding complexity by 32.96% accompanied by an increase in bitrate of 4.18% and a decrease of 0.18 dB in quality.

3.

Overview of QTMT Partitioning Structure

In the VVC standard, each image is first split into square CTU blocks. The CTU within the QTMT structure is set to $128\times 128$, which can be partitioned into different CU shapes. Each CU is presented under square blocks by the QT structure or rectangular blocks given by the added MTT partition structure.

The QT mode depths range from a $128\times 128$ initial depth denoted QT0 to $8\times 8$. In addition, the MTT decision mode includes BT and TT modes in horizontal and vertical splitting directions with block sizes ranging from $32\times 32$ to $4\times 4$. As detailed in Fig. 1, the CTU is first split into four square leaf nodes sized $64\times 64$ corresponding to a QT depth of 1 denoted QT1. Then, each $64\times 64$ CU is divided by the QT structure until reaching a $8\times 8$ minimum QT shape, where the QT depth ranges from 1 to 4.

Fig. 1

QTMT partition structures.

At the QT depth denoted QT2, the CU $32\times 32$ can be partitioned by the BT and TT modes while respecting the maximum MTT depth fixed at 4. At this level, these modes have a depth equal to zero denoted BT0 and TT0, respectively. For the BT partition, the BT depth is varied from 0 to 4, producing a horizontal binary tree (BTH) and vertical binary tree (BTV) block partitioning and respecting the minimum MTT size $4\times 4$ as presented in Fig. 1. Similarly, the TT depth range can reach TT4 for the TT partition, generating horizontal ternary tree (TTH) and vertical ternary tree (TTV) structures. In the case of the MTT partition structure starting, a rectangular block size is produced and the use of the QT structure is thus forbidden.

To select the optimal QTMT decision tree, the RDO process is applied for all CU sizes until the minimum size is obtained. The RDcost is first calculated for all block sizes. Then, the QTMT partition with the minimum RDcost is selected as an optimal decision tree.

4.

Proposed Intra-QTMT Partition Algorithm

In this paper, an overall approach based on a deep QTMT partitioning decision for intra-coding is introduced to overcome computational complexity caused by the RDO processing. The overall approach focuses mainly on both the QT structure and the BT partition in the horizontal and vertical directions. Therefore, a CNN, named ETH-CNN, used from the state-of-the-art method is implemented at the QT mode, and two other CNNs called CNN-BTH and CNN-BTV are proposed to predict the BTH and BTV mode decisions, respectively.

4.1.

Intra-Partition Algorithm Based on QT Structure

For the QT algorithm,¹⁹ a deep network called ETH-CNN²⁰ is implemented in the QTMT processing to reduce the computational complexity caused by the QT structure.

As shown in Fig. 2, the ETH-CNN architecture is developed according to the three QT partitioning levels $64\times 64$, $32\times 32$, and $16\times 16$. In each level, the luminance component $64\times 64$ is preprocessed through two layers to decrease the variation of the input samples and to generate the corresponding QT partition level. Then, the preprocessing output is convoluted with three convolution layers to extract the various features of all CUs. In the first convolution layer, the low features of the splitting block are generated using 16 filters. The two following layers are designed to determine the high block features with 24 and 32 filters, respectively. Next, the high extracted features of the three levels are concatenated into one vector, which passes through a fully connected module to predict the decision tree map of each CU sized $64\times 64$.

Fig. 2

ETH-CNN architecture.

The ETH-CNN is retrained under our extracted database using the original VTM3.0 reference software to construct the partition map, which generates the QT decision depth of all $64\times 64$ CUs as an output.

At each level, a splitting probability is predicted to decide whether or not to split the current block into square CUs. To boost the algorithm performance, a bithreshold decision is used at each level to balance the encoding complexity with the compression performance. At the three CUs sized $64\times 64$, $32\times 32$, and $16\times 16$, the predicted probability is compared with the predefined bithreshold as presented in Fig. 3. Two decisions are introduced: whether the current CU is divided into four square blocks directly without calculating the RDcost and checking the MTT modes or whether it skips the QT mode and allows for rectangular splitting. When the probability ranges between the bithreshold values, the RDcost metric is computed at the whole partition mode.

Fig. 3

Flowchart of QTMT partitioning based on ETH-CNN.

The proposed algorithm can reduce the encoding complexity of the QTMT module by applying the ETH-CNN to predict the QT depth instead of the RDO process.

4.2.

Intra-Partition Algorithm Based on BT Structure

For the BTH algorithm,²¹ a CNN named CNN-BTH is proposed to optimize the intra-BTH splitting decision. In this part, we mainly focus on the three BTHs sized $32\times 32$, $32\times 16$, and $32\times 8$ with BTH depths ranging from 0 to 2.

Therefore, our CNN-BTH architecture is developed under these three levels starting with an input block sized $32\times 32$ as shown in Fig. 4. The input samples pass through three layers named preprocessing layers, convolutional layers, and fully connected layers. After the preprocessing step, the resulted $32\times 32$, $32\times 16$, and $32\times 8$ BTH levels are convoluted with three layers to extract the different characteristics for all levels. After concatenating the generated features, three fully connected layers are applied to predict the BTH partition map of each $32\times 32$ CU.

Fig. 4

CNN-BTH network architecture.

In the first stage, an image database is collected from an open database²² and then coded with the original VTM3.0 reference software at the AI configuration to build the intended dataset. The image database is coded under four quantization parameters (QPs) with values 22, 27, 32, and 37 to generate the different BTH depths of each CU sized $32\times 32$.

In the next stage, the proposed CNN-BTH network is trained under our database depths. Four models are created and then implemented in the developed algorithm to predict the BTH partition depth at each block sized $32\times 32$. Figure 5 shows the BTH process at $32\times 32$, $32\times 16$, and $32\times 8$, where a partition probability is predicted at each block. When the probability is higher than the fixed threshold, the current CU is split directly into two rectangular blocks in the horizontal direction, skipping both the RDcost computation and QT, BTV, TTH, and TTV modes verification.

Fig. 5

Flowchart of QTMT partitioning based on CNN-BTH.

For the BTV algorithm,²¹ a new convolution model called CNN-BTV is created according to the three partition levels sized $32\times 32$, $16\times 32$, and $8\times 32$. Its architecture is similar to the CNN-BTH model as shown in Fig. 6, in which the output layer predicts the CU partition structure in the vertical direction producing 1, 2, and 4 partitioning probabilities at $32\times 32$, $16\times 32$, and $8\times 32$ CUs, respectively. Before integrating the CNN-BTV, it is necessary to train the split or not split decision at each level from our generated database. The latter contains all of the $32\times 32$ CUs BTV depths extracted from the original VTM3.0 algorithm. The trained CNN-BTV model is then implemented in the QTMT module to optimize the RDcost processing as shown in Fig. 7, for which a decision of whether splitting the current block directly or activating the RDO process is taken at the CUs sized $32\times 32$, $16\times 32$, and $8\times 32$.

Fig. 6

CNN-BTV network architecture.

Fig. 7

Flowchart of QTMT partitioning based on CNN-BTV.

4.3.

Overall Proposed QTMT Algorithm Based on CNN

In this work, an intra-partition decision tree algorithm is proposed to determine the QTMT structure based on the three CNNs called ETH-CNN, CNN-BTH, and CNN-BTV.

The used configuration of each deep neural network structure is detailed in Table 1, which presents the different input sizes, the number of filters, the kernel sizes for convolution, and the type of activation layers. As detailed in Table 1, each model includes three convolutional layers and three fully connected layers divided into two hidden fully connected layers and one output layer. It is noted that the activation function rectified linear units (ReLU)²³ is used at the convolutional and hidden fully connected processing, whereas the output fully connected unit is activated using the sigmoid function to generate the binary labels at each level. In fact, the ETH-CNN output contains 1, 4, and 16 binary labels at levels 1, 2, and 3, respectively, whereas the CNN-BTH and CNN-BTV outputs provide 1, 2, and 4 binary elements.

Table 1

Configuration of implemented networks ETH-CNN, CNN-BTH, and CNN-BTV.

Networks	Layers	Input size	Number of filters	Kernel size	Activation
ETH-CNN	Convolution 1	Level 1	16 × 16	16	4 × 4	ReLU
		Level2	32 × 32	16	4 × 4
		Level3	64 × 64	16	4 × 4
	Convolution 2	Level 1	4 × 4	24	2 × 2	ReLU
		Level2	8 × 8	24	2 × 2
		Level3	16 × 16	24	2 × 2
	Convolution 3	Level 1	2 × 2	32	2 × 2	ReLU
		Level2	4 × 4	32	2 × 2
		Level3	8 × 8	32	2 × 2
	Fully connected 1	Level 1	2688			ReLU
		Level2	2688
		Level3	2688
	Fully connected 2	Level 1	64			ReLU
		Level2	128
		Level3	256
	Fully connected 3	Level 1	48			Sigmoid
		Level2	96
		Level3	192
CNN-BTH	Convolution 1	Level 1	32 × 8	16	4 × 4	ReLU
		Level2	32 × 16	16	4× 4
		Level3	32 × 32	16	4 × 4
	Convolution 2	Level 1		24	2 × 2	ReLU
		Level2		24	2 × 2
		Level3		24	2 × 2
	Convolution 3	Level 1	4 × 1	32	1 × 1	ReLU
		Level2	4 × 2	32	1 × 1
		Level3	4 × 4	32	1 × 1
	Fully connected 1	Level 1	1568	—	—	ReLU
		Level2	1568
		Level3	1568
	Fully connected 2	Level 1	64			ReLU
		Level2	128
		Level3	256
	Fully connected 3	Level 1	48			Sigmoid
		Level2	96
		Level3	192
CNN-BTV	Convolution 1	Level 1	8 × 32	16	4 × 4	ReLU
		Level2	16 × 32	16	4 × 4
		Level3	32 × 32	16	4 × 4
	Convolution 2	Level 1	—	24	2 × 2	ReLU
		Level2	—	24	2 × 2
		Level3	—	24	2 × 2
	Convolution 3	Level 1	4 × 1	32	1 × 1	ReLU
		Level2	4 × 2	32	1 × 1
		Level3	4 × 4	32	1 × 1
	Fully connected 1	Level 1	1568	—	—	ReLU
		Level2	1568
		Level3	1568
	Fully connected 2	Level 1	64			ReLU
		Level2	128
		Level3	256
	Fully connected 3	Level 1	48			Sigmoid
		Level2	96
		Level3	192

After training, these CNN models are implemented in the VTM3.0 reference software to handle the QTMT partitioning complexity at the QT and BT structures as presented in the following part.

At the CU sized $64\times 64$, the ETH-CNN is implemented to predict the QT structure until arriving at the $8\times 8$ size. In addition to the ETH-CNN model, the CNN-BTH and CNN-BTV networks are integrated at the $32\times 32$ level to finally predict the QTMT decision tree at each CTU sized $128\times 128$. The ultimate QTMT decision is based on a bithreshold used at each level o achieve a trade-off between the RD performance and the complexity reduction. Therefore, different bithresholds named [$\alpha 1$, $\alpha 2$], [$\alpha 3$, $\alpha 4$], and [$\alpha 5$, $\alpha 6$] are used at $64\times 64$, $32\times 32$, and $16\times 16$, respectively, for the QT structure and at $32\times 32$, $32\times 16$, or $16\times 32$ and $32\times 8$ or $8\times 32$ for the BT structure, respectively. The proposed intra-QTMT approach is described in Fig. 8 and explained as follows:

• Step 1: The current $128\times 128$ CTU is split into four $64\times 64$ sub-blocks directly without calculating the RDcost. At each $64\times 64$ level, the ETH-CNN determines whether or not to divide the current CU by the QT structure according to a bithreshold [$\alpha 1$, $\alpha 2$].

• Step 2: At the $64\times 64$ level, if the probability predicted by the ETH-CNN named prob-qt $>\alpha 2$, the $64\times 64$ CU is split directly into four $32\times 32$ sub-blocks. If prob-qt $<\alpha 1$, the current CU is not divided via the QT structure. Because the maximum size of the MT structure is set to $32\times 32$, the partition processing is completed at this level. If $\alpha 1<\mathrm{prob}-\mathrm{qt}<\alpha 2$, the RDcost is computed at the CU $64\times 64$.

• Step 3: At the $32\times 32$ level, the three networks ETH-CNN, CNN-BTH, and CNN-BTV are integrated to predict the partition probabilities and are denoted prob-qt, prob-bth, and prob-btv, respectively. In this case, a comparison is applied to determine the best partition scheme at the $32\times 32$ CU. If the conditions prob-qt > prob-bth and prob-qt > prob-btv are verified, the partition processing is pursued through the ETH-CNN. If prob-bth > prob-qt and prob-bth > prob-btv, then the decision tree at $32\times 32$ is determined by the CNN-BTH network.

Fig. 8

Flowchart of CNNs-based QTMT partition.

If the previous conditions are not verified, the CNN-BTV network is applied to select the partition structure of the current CU.

5.

Experimental Results

6.

Conclusion

This paper presents an intra-fast mode decision algorithm for the VVC standard based on CNN to reduce the computational complexity of the QTMT partitioning process. A QTMT partition method is introduced to predict the QT depth at each CU $64\times 64$ and the BT depth at CU sized $32\times 32$ in the horizontal and vertical directions instead of the RDO processing using different CNNs. This approach is able to save a significant encoding time of up to 55.51% compared with the original VTM3.0 reference software, with a gain of 34.93% on average accompanied by an acceptable BDBR increase and a negligible video quality degradation. Our algorithm shows good results in encoding efficiency and complexity reduction compared with the state-of-the-art methods. To enhance the RD performance and the time saving of the proposed approach, we will mainly focus on adjusting the choice of the bithreshold values in future work.