A Novel Macroblock Level Rate Control Method for Stereo Video Coding

To compress stereo video effectively, this paper proposes a novel macroblock (MB) level rate control method based on binocular perception. A binocular just-notification difference (BJND) model based on the parallax matching is first used to describe binocular perception. Then, the proposed rate control method is performed in stereo video coding with four levels, namely, view level, group-of-pictures (GOP) level, frame level, and MB level. In the view level, different proportions of bitrates are allocated for the left and right views of stereo video according to the prestatistical rate allocation proportion. In the GOP level, the total number of bitrates allocated to each GOP is computed and the initial quantization parameter of each GOP is set. In the frame level, the target bits allocated to each frame are computed. In the MB level, visual perception factor, which is measured by the BJND value of MB, is used to adjust the MB level bit allocation, so that the rate control results in line with the human visual characteristics. Experimental results show that the proposed method can control the bitrate more accurately and get better subjective quality of stereo video, compared with other methods.


Introduction
Three-dimensional (3D) videos, including stereo video multiview video and, have currently been coming to the home through various approaches, such as Blu-Ray disc, cable and satellite transmission, terrestrial broadcast, and streaming and download through Internet [1]. Moreover, 3D video technologies have been gradually matured to be moved into mobile platforms, for example, 3D mobile phone and mobile 3D television [2]. However, there are many crucial technologies that need further research for stereo video coding. Rate control is a key issue in stereo/multiview video coding and transmission. Lu et al. [3] proposed a rate control scheme for multiview video coding (MVC), on the basis of the new quantitative measure for stereo video quality. And the macroblock (MB) quantization parameters (QPs) are modified according to the QP of the neighboring MB on the purpose of eliminating block effect in stereo video. Shao et al. proposed a rate control method for asymmetric stereo video coding [4]. They use a fixed threshold to quantize binocular psychovisual redundancy and establish the relationship between distortion and quantization for asymmetric stereo video coding. Zheng et al. [5] presented a rate control algorithm for MVC by analyzing the rate allocation proportion among different types of views, and the frame complexity was used to regulate the target bit for each frame. Liu et al. [6] presented a rate control technique for multiview video plus depth (MVD) based 3D video coding, and an image-stitching method was utilized to simultaneously encode video and depth. While these rate control algorithms in stereo/multiview video coding made useful explorations, however, these algorithms were only designed to frame layer bit allocation. In order to more finely rate control, it is necessary to develop the rate control techniques on the MB layer.
Moreover, the above algorithms did not take into account human visual characteristics well. Due to physiological and 2 The Scientific World Journal psychological mechanisms of human visual system (HVS), human eyes do not detect all of image or video distortion [7]. Thus, the concept of just-noticeable difference (JND) has played an important role in understanding HVS [8,9]. JND implies a visibility threshold in which human eyes can perceive changes in image values and depends on the luminance and contrast of local image region. Liu et al. [10] utilized a JND model to separate edge and textured regions of the image. Recently, several studies had concentrated on constructing visibility threshold for 3D image/video. Based on modeling and incorporating the binocular combination of injected noises, luminance masking, and contrast masking, binocular JND (BJND) was reported as the first model to measure the perceptible distortion of stereo images [11]. It was empirically demonstrated that human cannot realize distortion in stereo image if the distortion in one view image is less than the BJND value.
In this paper, we propose a novel MB level rate control method for stereo video coding based on visual perception. The proposed rate control method is performed in stereo video coding with four levels, namely, view level, group-ofpictures (GOP) level, frame level, and MB level. The BJND model based on the parallax matching is first exploited. Then, visual perception factor, designed based on BJND, is used to adjust the MB level bit allocation. So the rate control results in line with human visual characteristics. The rest of this paper is organized as follows. Section 2 describes the BJND model and Section 3 describes the proposed method. Section 4 shows experimental results, and finally, a conclusion is given in Section 5.

The BJND Model
To account for the mutual effect on the minimum distortion in one view that evokes perceptual differences in stereo image, the BJND model was proposed by Zhao et al. and measured with a set of psychophysical stimuli [11]. It has been demonstrated that the BJND is binocular combination of injected noises, luminance masking, and contrast masking in binocular viewing.
In [11], Zhao et al. did not take into account the disparity between left and right views of stereo image in their experiments, and they obtained BJND values under assumption about that the disparities between left and right views were zero. This assumption is not reasonable since in most cases the disparity of stereo image is not zero. In practice, binocular disparity is attributed to the primary visual cortical areas, and it is one of the most important stereo cues. Stereo matching itself has been a difficult task in computer vision, and a comprehensive review of stereo matching is given in [12].
Here, a state-of-the-art stereo matching technique in [13] is adopted.
To improve Zhao's model, a modified BJND model is presented with the consideration of binocular disparity. Given the left and right views of stereo image with the disparity image corresponding to left view, by incorporating the models of the binocular combination of injected noises, luminance, and contrast masking effects, a BJND at the right view is defined as follows: where and are the pixel coordinates, is the disparity at the position of ( , ) in right view, and is a contrast masking threshold. Note that BJND is dependent on the background luminance level bg , the edge height eh , and the noise amplitude of the corresponding pixel position in the left view. Here, the original sequence is used, so there is no noise in the left view; that is, = 0. is defined by where limit ( ) is a luminance masking function and ( ) is a fitting function. limit (bg ) and bg are calculated in The fitting function (bg ) of bg is defined as The edge height eh is given by (3) (4) Figure 1: The framework of the proposed MB level rate control method for stereo video coding.

The Proposed BJND-Based Rate Control Method
Here, we present a new rate control method for the stereo video based on the modified BJND. The framework of the proposed method is illustrated in Figure 1, with rate control in four levels, including view level, group-of-pictures (GOP) level, frame level, and MB level.

View Level Rate Control. Hierarchical B Pictures (HBP)
coding structure in stereo video is shown in Figure 2. There are two views to be encoded. It is seen that there is no interview predication in left view, and right view is encoded with  Time   I0  I0  B3  B2  B3  B1  B3  B3  B2   B3  B2  B3  B1  B3  B3  B2  P0  P0   T0 T1  T2 T3 T4  T5 T6 T7 T8 . . .
. . . unidirectional interview prediction from the reconstructed left view.
In [14], we obtained the proportion between left view and right view according to preencoding the first GOP. The target bit rate for each view is computed by where total denotes the total bit rate of left and right views. ( ) is bit allocation proportion for each view. is the coding order index of view, and the value of is 0 or 1, which represents the left view and right view, respectively; view ( ) is target bit rate for the th view.

GOP Level Rate Control.
Within the left view or the right view, the total number of bits allocated to each GOP is computed and the initial QP of each GOP is set in GOP 4 The Scientific World Journal level rate control. The total number of bits allocated for the ith GOP is computed by where is frame rate, gop denotes total number of frames in the current GOP, and ( −1, gop ) is actual buffer occupancy after encoding the ( −1)th GOP. The buffer occupancy should be kept at /8 after encoding each GOP. In the case of the constant bit rate (CBR), is updated frame by frame as follows: where ( , −1 ) is the actual encoding bits for the ( − 1)th frame.
The initial QP of the first GOP is a predefined QP, denoted as QP 0 . As shown in Figure 2, I0 and B1 frames in the first GOP of the left view and P0 and B1 frames in the first GOP of the right view are encoded by QP 0 . The first frame QP in other GOP is computed by where is total number of bits in encoding B frame in a GOP and Sum BQP is the sum of QPs for all B frames in the previous GOP.

Frame Level Rate Control.
Within each GOP, the target bits for each B frame are calculated according to the target buffer fullness level and the number of remaining bits in frame level rate control.
The actual buffer occupancy, buf ( , ), is computed as buf ( , ) = view ( ) where is a constant and its typical value is 0.75 [15]. ( , −1 ) is the target buffer level of the -1th frame in the ith GOP.
( ,2 ) is the initial value of target buffer level. ( , ) is the actual buffer occupancy after coding the jth B frame in the ith GOP.
Meanwhile, the number of remaining bits should also be considered when the target bit is computed.
where , ( − 1) is the number of the remaining B frames in the ith GOP. The target bits, ( , ), are a weighted combination of buf ( , ) and rem ( , ) and represented by where is a constant and its typical value is 0.5 [15]. .
With the current MB's BJND value and the average BJND value of removing the left, right, upper, and lower boundary MBs, the tolerable distortion degree, ( , , ), of the current MB in the whole frame is measured as follows: where and denote the number of MBs in a row and a column for each frame, respectively. The fluctuation of ( , , ) is large, thus, it is normalized and then added with 0.5 so as to get the perception factor weighting ( , , ), which is denote by where min{ } indicates the minimum function and max{ } indicates the maximum function. The perception factor ( , , ) is used to adjust the target bit allocation for the MB level. The larger BJND value represents more sensitive to distortion for human eye; that is, the value of ( , , ) is larger, so the MB should be allocated more bits. Conversely, the MB with smaller BJND value should be allocated smaller bits. Consider The Scientific World Journal

Experimental Results and Discussions
In order to evaluate the performance of the proposed MB level rate control method for stereo video coding based on visual perception, six representative stereo video sequences with 1024 × 768 spatial resolution, including BookArrival, Alt-Moabit, DoorFlowers, LeavingLaptop, Newspaper, and Kendo, are used in the experiments. BookArrival, AltMoabit, Door-Flowers, and LeavingLaptop are provided by Fraunhofer HHI [16]. Newspaper is provided by Gwangju Institute of Science and Technology [17]. Kendo sequence is captured by Nagoya University with the moved camera array [18].
In the experiments, we use the revised MVC software JMVC7.0 to implement the rate control methods. The test conditions of the six sequences are shown in Table 1. Two middle views (view 9 and view 10) for BookArrival, DoorFlowers, and LeavingLaptop, two middle views (view 5 and view 6) for AltMoabit and Newspaper, the two middle views (view 4 and view 5) for Kendo are used to simulate the left and right views of stereo video in Figure 2. In order to avoid the influence of BJND results caused by occlusion and exposure in the left and right views of stereo video, the left, right, upper, and lower MBs in each frame of the right view sequence are not processed. And the BJND results of BookArrival and AltMoabit are given in this paper, as shown in Figures 3 and 4, respectively. Figures 3(a) and 3(b) show the original left and the right view images. Figure 3(c) is the BJND map of the right view in pixel domain, and Figure 3(d) is the BJND mask on the basis of Figure 3(c) by MB processing. In the same way, Figure 4 shows the similar results. Table 1, three methods, Zheng's in [5], SMBRC, and the proposed method, are compared. SMBRC denotes the stereo video MB level rate control method; that is, the MB level monoscopic video JVT-G012 algorithm is extended to stereo video coding and implemented in JMVC. The detailed average control accuracies of three methods are shown in Table 2. In Table 2, the target bitrate and the actually controlled bitrate are the average value of two views. Rate control error ratio, RC , is used to measure the accuracy of the bitrate estimation and defined by

Rate Control Accuracy in Stereo Video Coding. Under the test conditions shown in
where target and actual denote the target bitrate and the actual coding bitrate, respectively. Zheng's method is a frame-level rate control method, while the proposed method and SMBRC are MB level rate control scheme, so they have better control accuracy. This can be evidently seen from Table 2. Table 2 also indicates that the absolute inaccuracy of the proposed stereo video rate control method is within 0.192%. It is obvious that the proposed method can more precisely control the bitrate in stereo video coding. Figure 5 shows the rate-distortion (R-D) performance comparison results of the three methods. In Figure 5, the average bitrate is the average value of two views, and the peak signal-to-noise ratio (PSNR) value is also the average value of two views. It is clear that, compared with Zheng's and SMBRC methods, the proposed method can achieve the better R-D performance under different target bitrates since more accurate binocular perception information is used in the proposed method.

Subjective Quality
Comparison. Different sequences possess different motion and scene characteristics; therefore the video frames which can best reflect the differences in the subjective quality are selected in accordance with the characteristics of each sequence. For the outdoor scene AltMoabit sequence, we select the 52th frame in view 6 as an example, shown in Figure 6. The bus and the words on the bus in the image are the areas of human visual attention in Figure 6, and more bits are allocated for these areas under the visual perception factor based on BJND. Therefore the subjective quality of the proposed method is better than the other methods.
In order to eliminate the limitation of only one video frame, we select the 43rd frame to 47th frame for the indoor scene LeavingLaptop sequence, shown in Figure 7. The details of the human body and head and the book are allocated more bitrates, so the subjective quality of the proposed method is better than the other methods. Moreover, for quantization parameter selection, if we select a small value QP, the subjective quality of the three methods is too good to reflect the difference. Hence, QP is set to 37.   The Scientific World Journal

Conclusion
A novel macroblock (MB) level rate control method for stereo video is proposed in this paper. The proposed method is performed with four levels, namely, view level, groupof-picture (GOP) level, frame level, and MB level. In the view level, different proportions of bitrates are allocated for left view and right view of stereo video according to the prestatistical rate allocation proportion. In the GOP level, the total number of bitrates allocated to each GOP is computed and the initial quantization parameter of each GOP is set. In the frame level, the target bits allocated to each frame are computed. In the MB level, the binocular justnoticeable difference (BJND) model based on the parallax matching is first given. Then, visual perception factor, which is measured by the ratio between the BJND value of the current MB and the average BJND value of removing the left, right, upper, and lower boundary MBs, is used to adjust the MB level bit allocation. Experimental results show that the proposed method can control the bitrate more accurately and the average of rate control error is only about 0.19%. Compared with other schemes, the proposed method gets better rate-distortion performance and subjective quality of stereo images.