A Motion-Compensated Overcomplete Temporal Decomposition for Multiple Description Scalable Video Coding

We present a new multiple-description coding (MDC) method for scalable video, designed for transmission over error-prone networks. We employ a redundant motion-compensated scheme derived from the Haar multiresolution analysis, in order to build temporally correlated descriptions in a t +2 D video coder. Our scheme presents a redundancy which decreases with the resolution level. This is achieved by additionally subsampling some of the wavelet temporal subbanbds. We present an equivalent four-band lifting implementation leading to simple central and side decoders as well as a packet-based reconstruction strategy in order to cope with random packet losses.


INTRODUCTION
With the increasing usage of the Internet and other besteffort networks for multimedia communication, there is a stringent need for reliable transmission. For a long time, the research efforts have been concentrated on enhancing the existing error correction techniques, but during the last decades an alternative solution has emerged and is gaining more and more popularity. This solution mainly answers the situation in which immediate data retransmission is either impossible (network congestion or broadcast applications) or undesirable (e.g., in conversational applications with very low delay requirements). We are referring to a specific joint sourcechannel coding technique known as multiple-description coding (MDC). A comprehensive presentation of MDC is given in [1].
The MDC technique leads to several correlated but independently decodable (preferably with equivalent quality) bitstreams, called descriptions, that are to be sent over as many independent channels. In an initial scenario, these channels have an on-off functioning: either the bitstream is flawlessly conveyed or it is considered unusable at the so-called side decoder end if an error had occurred during the transmission. According to this strategy, some amount of redundancy has to be introduced at the source level such that an acceptable reconstruction can be achieved from any of the bitstreams. Then, the reconstruction quality will be enhanced with every bitstream received.
The application scenario for MDC is different from the one of scalable coding, for example. Indeed, the robustness of a scalable system relies on the assumption that the information has been hierarchized and the base layer is received without errors (which can be achieved, e.g., by adding sufficient channel protection). However, if the base layer is lost, the enhancement layers cannot be exploited and nothing can be decoded. The MDC framework has a complementary approach, trying to cope with the channel failures, and thus allowing the decoding of at least one of the descriptions, when the other is completely lost. An ingredient enabling the success of an MDC technique is the path diversity, since its usage balances the network load and reduces the congestion probability.
In wireless networks, for instance, a mobile receptor can benefit from multiple descriptions if these arrive independently, for example on two neighbor access points; when moving between these access points, it might capture one or the other, and in some cases both. Another way to take advantage of MDC in a wireless environment is by splitting in 2 EURASIP Journal on Image and Video Processing frequency the transmission of the two descriptions: for example, a laptop may be equipped with two wireless cards (e.g., 802.11a and 802.11g), each wireless card receiving a different description. Depending on the dynamic changes in the number of clients in each network, one of them may become overloaded and the corresponding description may not be transmitted.
In wired networks, the different descriptions can be routed to a receiver through different paths by incorporating this information into the packet header [2]. In this situation, a description might contain several packets and the scenario of on-off channels might no longer be suitable. The system should, in this case, be designed to take into consideration individual or bursty packet losses rather than a whole description.
An important issue that concerned the researchers over the years is the amount of introduced redundancy. One has to consider the tradeoff between this redundancy and the resulting distortion. Therefore, a great deal of effort has been spent on defining the achievable performances with MDC ever since the beginning of this technique [3,4] and until recently, for example, [5]. Practical approaches to MDC include scalar quantization [6], correlating transforms [7], and frame expansions [8]. Our work belongs to the last category and we concentrate on achieving a tunable low redundancy while preserving the perfect reconstruction property of our scheme [9].
In this paper, we present an application of multipledescription coding to robust video transmission over lossy networks, using redundant wavelet decompositions in the temporal domain of a t + 2D video coder.
Several directions have already been investigated in the literature for MD video coding. In [10][11][12][13], the proposed schemes mainly involve the spatial domain in hybrid video coders such as MPEG/H.26x. A very good survey on MD video coding for hybrid coders is given in [14].
Only few works investigated the design of MDC schemes allowing to introduce source redundancy in the temporal domain, although the field is very promising. In [15], a balanced interframe multiple-description coder has been proposed starting from the popular DPCM technique. In [16], the reported MDC scheme consists in temporal subsampling of the coded error samples by a factor of 2 so as to obtain 2 threads at the encoder, which are further independently encoded using prediction loops that mimic the decoders (two side prediction loops and a central one).
Existing work for t + 2D video codecs with temporal redundancy addresses three-band filter banks [17,18] and temporal or spatiotemporal splitting of coefficients in 3D-SPIHT sytems [19][20][21]. Here, we focus on a two-description coding scheme for scalable video, where temporal and spatial scalabilities follow from a classical dyadic subband transform. The correlation between the two descriptions is introduced in the temporal domain by exploiting an oversampled motion-compensated filter bank. An important feature of our proposed scheme is its reduced redundancy which is achieved by an additional subsampling of a factor of two of the resulting temporal details. The remaining details are then distributed in a balanced manner between the two descriptions, along with the nondecimated approximation coefficients. The global redundancy is thus tuned by the number of temporal decomposition levels. We adopt a lifting approach for the temporal filter-bank implementation and further adapt this scheme in order to design simple central (receiving both descriptions) and side decoders.
This paper relies on some of our previous work which is presented in [22]. Here, we consider an improved version of the proposed scheme and detail its application to robust video coding. The approximation subbands which participate in each description are decorrelated by an additional motion-compensated transform, as it will be explained in Section 5. Moreover we consider two transmission scenarios. In the first one, we tackle the reconstruction when an entire description is lost or when both descriptions are received error-free, and in the second one we discuss signal recovery in the event of random packet losses in each description. For the random-loss case, we compare our results with a temporal splitting strategy, as in [2], which consists in partitioning the video sequence into two streams by even/odd temporal subsampling and reconstructing it at half rate if one of the descriptions is lost.
An advantage of our scheme is to maintain the scalability properties for each of the two created descriptions, allowing to go further than the classical on-off channel model for MDC and also cope with random packet losses on the channels.
The rest of the paper is organized as follows. In Section 2 we present the proposed strategy of building two temporal descriptions. Section 3 gives a lifting implementation of our scheme together with an optimized version well suited for Haar filter banks. We explain the generic decoding approach in Section 4. We then discuss the application of the proposed scheme to robust video coding in Section 5 and the resulting decoding strategy in Section 6. Section 7 gives the simulation results for the two scenarios: entire description loss and random packet losses in each description. Finally, Section 8 concludes the paper and highlights some directions for further work.

TEMPORAL MDC SCHEME
The strategy employed to build two temporal descriptions from a video sequence is detailed in this section. We rely on a temporal multiresolution analysis of finite energy signals, associated with a decomposition onto a Riesz wavelet basis.
Throughout the paper, we are using the following notations. The approximation subband coefficients are denoted by a and the detail subband coefficients by d. The resolution level associated with the wavelet decomposition is denoted by j, whereas J stands for the coarsest resolution. The temporal index of each image in the temporal subbands of the video sequence is designated by n and the spatial indices are omitted in this section in order to simplify the notations.
The main idea of the proposed scheme consists in using an oversampled decomposition in order to get two wavelet representations. The superscript symbols I and II distinguish Christophe Tillier et al. 3 the coefficients in the first basis from those corresponding to the second one. For example, d I j,n stands for the detail coefficient in representation I at resolution j and temporal index n. Then a secondary subsampling strategy is applied along with distributing the remaining coefficients into two descriptions. The redundancy is reduced by this additional subsampling to the size of an approximation subband (in terms of number of coefficients).
Let (h n ) n∈Z (resp., (g n ) n∈Z ) be the impulse responses of the analysis lowpass (resp., highpass) filter corresponding to the considered multiresolution decomposition.
For the first J − 1 resolution levels, we perform a standard wavelet decomposition, which is given by for the temporal approximation subband, and by for the detail one, where j ∈ {1, . . . , J − 1}.
We introduce the redundancy at the coarsest resolution level J by eliminating the decimation of the approximation coefficients (as in a shift-invariant analysis). This leads to the following coefficient sequences: Each of these approximation subbands is assigned to a description.
In the following, we need to indicate the detail subbands involved in the two descriptions. At the last decomposition stage, we obtain in the same manner as above two detail coefficient sequences (as in a nondecimated decomposition): Note that the coefficients in representation II are obtained with the same even-subsampling, but using the shifted versions of the filters h and g: h n−1 and g n−1 , respectively. In order to limit the redundancy, we further subsample these coefficients by a factor of 2, and we introduce the following new notations: At each resolution, each description will contain one of these detail subsets. Summing up the above considerations, the two descriptions are built as follows. Description 1. This description contains the even-sampled detail coefficients ( d I j,n ) n for j ∈ {1, . . . , J}, and (a I J,n ) n , where, using the same notation as in (5), Description 2. This description contains the odd-sampled detail coefficients (ď I j,n ) n for j ∈ {1, . . . , J − 1}, (ď II J,n ) n , and (a II J,n ) n , where, similarly to (6), Once again, we have not introduced any redundancy in the detail coefficients, therefore the overall redundancy factor (evaluated in terms of number of coefficients) stems from the last level approximation coefficients, that is, it is limited to The choice of the subsampled detail coefficients at the coarsest level in the second description is motivated by the concern of having balanced descriptions [9].

Two-band lifting approach
Since the first J − 1 levels are obtained from a usual wavelet analysis, in the following we will be interested mainly in the last resolution level. The corresponding coefficients in the two descriptions are computed as follows: where, for simplicity, we have denoted by x k the approximation coefficients at the (J −1)th level and we have omitted the subscript J. We illustrate our scheme in Figure 1, using a one-stage lifting implementation of the filter bank. The p and u operators in the scheme stand for the predict and update, respectively, and γ is a real nonzero multiplicative constant. Note that the lifting scheme allows a quick and memory-efficient implementation for biorthogonal filter banks, but especially it guarantees perfect reconstruction. For readability, we display a scheme with only two levels of resolution, using a basic lifting core.

Equivalent four-band lifting implementation
The two-band lifting approach presented above does not yield an immediate inversion scheme, in particular when using nonlinear operators, such as those involving motion estimation/compensation in the temporal decomposition of the  video. This is the motivation behind searching an equivalent scheme for which global inversion would be easier to prove. In the following, we build a simpler equivalent lifting scheme for the Haar filter bank, by using directly the four-band polyphase components of the input signal, instead of the two-band ones. Let these polyphase components of (x n ) n∈Z be defined as For the first description, the approximation coefficients can be rewritten from (9a), while the detail coefficients are still obtained with (9b), leading to Similarly, for the second description, we express the approximation subband from (9c) and keep the details from (9d): Note that the coefficients in the two descriptions can thus be computed with an oversampled six-band filter bank with a decimation factor of 4 of the input signal, which consequently amounts to a redundant structure.
In the sequel of this paper, we will focus on the Haar filter banks, which are widely used for the temporal decomposition in t + 2D wavelet-based video coding schemes.
To go further and find an equivalent scheme for the Haar filter bank, note that the two-band polyphase components of the input signal, x 2n = a J−1,2n and x 2n+1 = a J−1,2n+1 , are first filtered and then subsampled (see Figure 1). However, for the Haar filter bank, recall that the predict and update operators are, respectively, p = Id and u = (1/2) Id (and the constant γ = √ 2). Since these are both instantaneous operators, one can reverse the order of the filtering and downsampling operations. This yields the following very simple expressions for the coefficients in the first description: and in the second: In Figure 2, we schematize the above considerations.

RECONSTRUCTION
In this section, we give the general principles for decoders design considering the generic scheme in Figure 2. The next sections will discuss the application of the proposed scheme to robust video coding and more details will be given about the central and side decoders in the video coding schemes. Some structure improvements that lead to better reconstruction will also be presented.
In the generic case, our aim is to recover x n , the input signal, from the subsampled wavelet coefficients. The components involved in the basic lifting decomposition can be perfectly reconstructed by applying the inverse lifting schemes. However, since we have introduced redundancy, we benefit from additional information that can be exploited at the Christophe Tillier et al. reconstruction. Let us denote the recovered polyphase components of the signal by x (i) n .

Central decoder
We first discuss the reconstruction performed at the central decoder. The first polyphase component of x n is obtained by directly inverting the basic lifting scheme represented by the upper block in Figure 2. The polyphase components reconstructed from a I n and d I n are denoted by y (0) n and y (1) n . Thus, we obtain where [ a I n ] and [ d I n ] are the quantized versions of a I n and d I n , analogous notations being used for the other coefficients. Obviously, in the absence of quantization, we have x (0) n = y (0) n and x (1) n = y (1) n . Similarly, the third polyphase component is reconstructed by directly inverting the second two-band lifting block in Figure 2: where the polyphase components reconstructed fromǎ II n anď d II n are denoted by z (1) n and z (2) n . The second polyphase component of x n can be recovered as the average between the reconstructed subbands from the two previous lifting blocks: The last polyphase component of the input signal can be computed as the average between the reconstructions fromǎ I n and a II n . Using (13b) and (14a), we get

Side decoders
Concerning the side decoders, again from Figure 2, we note that from each description we can partially recover the original sequence by immediate inversion of the scheme. For instance, if we only receive the first description, we can easily reconstruct the polyphase components x (0) n , x (1) n from the first Haar lifting block. The last two polyphase components x (2) n and x (3) n are reconstructed by assuming that they are similar: Similarly, when receiving only the second description, we are able to directly reconstruct x (1) n , x (2) n from the second Haar lifting block, while x (0) n and x (3) n are obtained by duplicating a II n+1 :

APPLICATION TO ROBUST VIDEO CODING
Let us now apply the described method to robust coding of video sequences. The temporal samples are in this case the input frames, and the proposed wavelet frame decompositions have to be adapted to take into account the motion estimation and compensation between video frames, which is an essential ingredient for the success of such temporal decompositions. However, as shown in the case of critically sampled two-band and three-band motion-compensated filter banks [23][24][25], incorporating the ME/MC in the lifting scheme leads to nonlinear spatiotemporal operators. Let us consider the motion-compensated prediction of a pixel s in the frame x (1) n from the frame x (0) n and denote by v the forward motion vector corresponding to s. Writing now (13a)-(13c) in a lifting form and incorporating the motion into the predict/update operators yield One can also note that several pixels s i , i ∈ {1 . . . , N}, in the current frame x (1) n may be predicted by a single pixel in the reference frame x (0) n , which is called in this case multiple 6 EURASIP Journal on Image and Video Processing connected [26]. Then, for the pixels s i and their corresponding motion vectors v i , we have s 1 − v 1 = · · · = s i − v i = · · · = s N − v N . After noting that the update step may involve all the details d I n (s i ), i ∈ {1, . . . , N}, while preserving the perfect reconstruction property, we have shown that the update step minimizing the reconstruction error is the one averaging all the detail contributions from the connected pixels s i [27]. With this remark, one can write (21) as follows: and with similar notations for multiple connections in the second description: Since for video coding efficiency, motion prediction is an important step, we propose an alternative scheme for building the two descriptions, in which we incorporate the motion estimation/compensation in the computation of the second approximation sequence ( a I n , resp.,ǎ II n ). This scheme is illustrated in Figure 3. Per description, an additional motion vector field needs to be encoded. In the following, this scheme will be referred to as 4B 1MV. In the case of the 4B 1MV scheme, if we denote by u the motion vector predicting the pixel s in frame x (3) n−1 from x (2) n−1 and by w the motion vector predicting the pixel s in frame x (0) n from x (3) n−1 , the analysis equations for a I n andǎ II n can be written aš for the connected pixels (here, only the first pixel in the scan order is considered in the computation), anď for the nonconnected pixels. Furthermore, a careful analysis of the video sequences encoded in each description revealed that the two polyphase components of the approximation signals that enter each description are temporally correlated. This suggested us to come up with a new coding scheme, illustrated in Figure 4, where a motion-compensated temporal Haar transform is applied on a I n andǎ I n (resp., onǎ II n and a II n ). Compared to the original structure, two additional motion vector fields have to be transmitted. The scheme will thus be referred to as 4B 2MV. In Figure 5 the temporal transforms involved in two levels of this scheme are represented. One can note the temporal subsampling of the details on the first level and the redundancy at the second level of the decomposition.

CENTRAL AND SIDE VIDEO DECODERS
The inversion of (22a) and (22b) is straightforward by the lifting scheme, allowing us to reconstruct the first two polyphase components. Using the same notations as in Section 4, the reconstructed polyphase components from the first description are as follows: When analyzing the reconstruction of the connected pixels in the first two polyphase components, one can note that it corresponds to the inverse lifting using the average update step. A similar reasoning for the second description allows us to find the reconstruction of the sequence from the received Christophe Tillier et al.  Figure 4: Four-band lifting scheme with motion estimation and Haar transform on the approximation subbands. 16  framesǎ II n ,ď II n , and a II n . By inverting (23a) and (23b), we obtain For the nonconnected pixels, we have As it can be seen, x (1) n can be recovered from both descriptions, and the final central reconstruction is obtained as the mean of these values. Also, one can note that by knowing x (2) n−1 (resp., x (0) n ) from the first (resp., second) description, it is possible to reconstruct x (3) n−1 , by reverting to (24) and (25). As for the side decoders of the initial scheme, the solution for the first description is given by (27) and while for the second description it reads in addition to x (1) n and x (2) n obtained with (28). For the 4B 1MV scheme, the additional motion compensation involved in the computation of the approximation sequences requires reverting the motion vector field in one of the components. Thus, we have for the first side decoder and for the second one.

EURASIP Journal on Image and Video Processing
For the scheme 4B 2MV, the temporal Haar transform being revertible, no additional difficulties appear for the central or side decoders.
Note that the reconstruction by one central and two side decoders corresponds to a specific application scenario, in which the user receives the two descriptions from two different locations (e.g., two WiFi access points), but depending on its position, it can receive both or only one of the descriptions. In a more general scenario, the user may be in the reception zone of both access points, but packets may be lost from both descriptions (due to network congestion, transmission quality, etc.). In this case, the central decoder will try to reconstruct the sequence by exploiting the information in all the received packets. It is therefore clear that an important issue for the reconstruction quality will be the packetization strategy. Even though the complete description of the different situations which can appear in the decoding (depending on the type of the lost packets) cannot be done here, it is worth noting that in a number of cases, an efficient usage of the received information can be employed: for instance, even if we do not receive the spatiotemporal subbands of one of the descriptions, but only a packet containing its motion vectors, these vectors can be exploited in conjunction with the other description for improving the fluidity of the reconstructed video. We also take advantage of the redundancy existing at the last level to choose, for the frames which can be decoded from both descriptions, the version which has the best quality, and thus to limit the degradations appearing in one of the descriptions.

SIMULATIONS RESULTS
The Haar lifting blocks in Figure 4 are implemented by a motion-compensated lifting decomposition [23]. The motion estimation is performed using hierarchical variable size block-matching (HVBSM) algorithm with block sizes ranging from 64 × 64 to 4 × 4. An integer-pel accuracy is used for motion compensation. The resulting temporal subbands are spatially decomposed with biorthogonal 9/7 Daubechies wavelets over 5 resolution levels. Spatiotemporal coefficients and motion vectors (MVs) are encoded within the MC-EZBC framework [26,28], where MV fields are first represented as quad-tree maps and MV values are encoded with a zero-order arithmetic coder, in raster-scan order.
First, we have tested the proposed algorithm on several QCIF sequences at 30 fps. In Figure 6, we compare the ratedistortion performance of the nonrobust Haar scheme with that of the MDC central decoder on the "Foreman" video test sequence. The bitrate corresponds to the global rate for the robust codec (both descriptions). Three temporal decomposition levels have been used in this experiment (J = 3). We can observe that even the loss of one description still allows for acceptable quality reconstruction especially at low bitrates and also that the global redundancy does not exceed 30% of the bitrate. Figure 7 illustrates the central rate-distortion curves for different levels of redundancy and, together with Figure 6, shows the narrowing of the gap with respect to the nonre- The difference in performance between the two descriptions is a phenomenon appearing only if the scheme involves three or more decomposition levels, since it is related to an asymmetry in the GOF structure of the two descriptions when performing the decimation. Indeed, as illustrated in Figure 5, when the first description is lost, some of the motion information in the second description cannot be used Christophe Tillier et al.   to improve the reconstruction, while this does not happen when loosing the second description.
In Figures 8-9, we present the rate-distortion curves for the central and side decoders, in the absence of packet losses. The performance of the scheme without ME/MC in the computation of the approximation sequencesǎ I n and a II n is compared with the 4B 1MV and 4B 2MV schemes.
One can note that the addition of the ME/MC step in the computation ofǎ I n and a II n does not lead to an increase in the coding performance of the central decoder, since the expected gain is balanced by the need to encode an additional MV field. On the other hand, the final MC-Haar transform leads to much better results, since instead of two correlated approximation sequences, we now only have transformed subbands. For the side decoders however, the introduction of the motion-compensated average in the computation ofǎ I n and a II n leads to a significant improvement in coding performances (increasing with the bitrate from 1 to 2.5 dB), and the MC-Haar transform adds another 0.3 dB of improvement.
In a second scenario, we have tested our scheme for transmission over a packet loss network, like Ethernet. In this case, the bitstreams of the two descriptions are separated in packets of maximal size of 1500 bytes. For each GOP, separate packets are created for the motion vectors and for each spatiotemporal subband. If the packet with motion vectors is lost, or if the packet with the spatial approximation subband of the temporal approximation subband is lost, then we consider that the entire GOP is lost (it cannot be reconstructed).
We compare our scheme with a nonredundant MCTF one and also with another temporal MDC scheme, consisting in a temporal splitting of the initial video sequence. Odd and even frames are separated into two descriptions which are encoded with a Haar MCTF coder ( Figure 10 illustrates the motion vectors and temporal transforms for this structure).
The coding performance as a function of the packet loss rate is illustrated in Figures 11 and 12 for the "Foreman" and "Mobile" video test sequences at 250 kbs. As expected, when there is no loss, the nonredundant coding is better than both MDC schemes (which have comparable performances). However, as soon as the packet loss rate gets higher than 2%, our scheme overpasses by 0.5-1 dB the temporal splitting and the nonrobust coding by up to 4 dB.
Moreover, we have noticed that the MDC splitting scheme exhibits a flickering effect, due to the fact that a lost packet will degrade the quality of one over two frames. In our scheme, this effect is not present, since the errors in one description have limited influence thanks to the existing redundancies, and also to a different propagation during the reconstruction process. Figure 13 presents the influence of the average update operator, with gains of about 0.2 dB over the entire range of packet loss rates. Finally, we have compared in Figure 14 the rate-distortion curves of the temporal splitting and the proposed MDC schemes for a fixed packet loss rate (10%). One can note a difference of 0.5-1.3 dB at medium and high bitrates (150-1000 kbs) and slightly smaller at low bitrates (100 kbs). It is noticeable that the PSNR of the reconstructed sequence is not monotonically increasing with the bitrate: a stiff increase in PSNR until 250 kbs is followed by a "plateau" effect which appears at higher bitrates. This is due to the loss of the information in the spatial approximation of the temporal approximation subband. Indeed, for low bitrates, this spatiotemporal subband can be encoded into a single packet, so for uniform error distribution, the rate-distortion curve increases monotonically. At a given threshold (here, it happens at about 250 kbs for packets of 1500 bytes), the 10 EURASIP Journal on Image and Video Processing 16  approximation subband has to be coded into two packets. Moreover, we considered that if any of these two packets is lost, the GOF cannot be reconstructed. Therefore, we see a drop in performance. From this point, with the increasing bitrate, the performance improves till a new threshold where the subband needs to be encoded into three packets and so on. A better concealment scheme in the spatial domain, allowing to exploit even a partial information from this subband, would lead to a monotonic increase in performance.

CONCLUSION AND FUTURE WORK
In this paper, we have presented a new multiple-description scalable video coding scheme based on a motion-compensated redundant temporal analysis related to Haar wavelets.  The redundancy of the scheme can be reduced by increasing the number of temporal decomposition levels. Reversely, it can be increased either by reducing the number of temporal decomposition levels, or by using nondecimated versions of some of the detail coefficients. By taking advantage of the Haar filter bank structure, we have provided an equivalent four-band lifting implementation, providing more insight into the invertibility properties of the scheme. This allowed us to develop simple central and side-decoder structures which have been implemented in the robust video codec.
The performances of the proposed MDC scheme have been tested in two scenarios: on-off channels and packet losses, and have been compared with an existing temporal splitting solution.
Note that the presented scheme builds the descriptions in the temporal domain of the video, but it can be combined with structures introducing the redundancy in the spatial domain, for which many more solutions have been proposed in the literature. The increased flexibility thus achieved may be exploited to better adapt the packetization to different situations of network losses and also to improve the reconstruction at different levels.