EURASIP Journal on Applied Signal Processing 2005:6, 954–960 c ○ 2005 Hindawi Publishing Corporation Iterative List Decoding of Concatenated Source-Channel Codes

Whenever variable-length entropy codes are used in the presence of a noisy channel, any channel errors will propagate and cause significant harm. Despite using channel codes, some residual errors always remain, whose effect will get magnified by error propagation. Mitigating this undesirable effect is of great practical interest. One approach is to use the residual redundancy of variable length codes for joint source-channel decoding. In this paper, we improve the performance of residual redundancy source-channel decoding via an iterative list decoder made possible by a nonbinary outer CRC code. We show that the list decoding of VLC's is beneficial for entropy codes that contain redundancy. Such codes are used in state-of-the-art video coders, for example. The proposed list decoder improves the overall performance significantly in AWGN and fully interleaved Rayleigh fading channels.


INTRODUCTION
Variable-length codes (VLCs) for entropy coding are by now a central part of most data compression techniques, which are in turn essential for many communications applications, including text, voice, images, and video. While VLCs achieve significant compression, they also introduce dependencies in the data structure through their variable length, thus leading to error propagation in the decoded sequence.
One of the techniques that has been used to combat this undesirable effect is joint source-channel decoding. It is known that even the most efficient symbol-by-symbol compression (Huffman code) does not always achieve the entropy limit, therefore redundancy often remains in compressed data. This redundancy can, in principle, be used to assist the decoder.
Taking this argument one step further, it has been proposed to leave redundancy intentionally in entropy codes, for the purposes of resilience against channel noise. For example, the video coding standard H.263+ and its descendants use a reversible variable-length code (RVLC) [1] whose compression efficiency is less than Huffman codes. However, the RVLC allows bidirectional symbol-based decoding which is useful in the presence of channel errors. This approach has been generalized by designing entropy codes with prespecified minimum distance [2,3].
The error resilience of entropy codes can be used to "clean up" any residual errors from the traditional error control coding (see Figure 1). For example, in the case of RVLC, one may start decoding from the end of the sequence whenever an error is observed. This is a separable approach to decoding. However, we know today that serially concatenated codes offer significantly improved performance if the decoding operation is done jointly, via the soft-input soft-output (SISO) decoding algorithm. This principle has been applied to finite-alphabet source-channel codes by Bauer and Hagenauer [4,5], and further analyzed in [6,7].
In this paper, we propose an improvement over the method of Bauer and Hagenauer by introducing a list decoder for source-channel decoding, made possible by a nonbinary CRC outer code. We implement this list decoder via an iterative decoding procedure similar to serial concatenated codes ( Figure 2).
We briefly summarize and review the issues of iterative source-channel decoding in Section 2. We introduce list decoding of the concatenated code in Section 3. We present some analytical and experimental results in Section 4 and offer concluding remarks in Section 5.

SERIAL CONCATENATION OF VLC AND CHANNEL CODES
For the clarity of exposition, we first consider the system of Figure 2 in the absence of the CRC and list decoding component. The simplified system consists of an outer (VLC) code and an inner channel code, separated by an interleaver π. The source and channel codes are jointly (iteratively) decoded at the receiver. As mentioned previously, this method relies on residual redundancy in the VLC, in particular, sometimes redundancy is retained in the VLC on purpose, for example, in RVLCs. Thus, for the purposes of this section, we treat both codes in terms of their distance properties. We treat the outer code, C o , as a channel code. The key difficulty of the analysis, which requires a generalization of the well-known work of [8], is that VLCs are nonlinear.
The proceeding analysis closely follows that of [7]. Assume a sequence of K symbols is encoded, and the average length of the outer entropy code symbols is ave . Hence, the output bit sequence of C o has a variable length with size N min to N max . Code C o is partitioned in a way such that all codewords of C o with length N ∈ [N min , N max ] form a subcode denoted by C N . In other words, to avoid dealing with variable lengths, we partition the set of all composite codewords into sets of equal length [2]. We define the free distance of C o , d o f as the minimum of the free distances of C N 's. The number of inner codewords with output weight h and input weight is shown by A i ,h . Assume the outer subcode C N has A o (N) pairs of codewords with Hamming distance . Using the uniform interleaver notion of [8], and thanks to linearity of the inner code, the number of pairs of codewords of the overall concatenated code, with Hamming distance h, is The pairwise error probability (PEP) of a pair of codewords with Hamming distance h is P h = Q( 2hE s /N 0 ). Using (1), the PEP of the concatenated code is where d f is the free distance of the concatenated code, R i is the rate of the inner channel code, and Pr(N) is the probability of the codewords of C N . We note that the above union bound can be used with different choices of inner and outer codes, for example, a convolutional or turbo code as inner code [4,9,10], or Huffman code or RVLC as outer code. A similar development is possible for symbol error rate [7], which we do not present here for the sake of brevity. Iterative decoding of the concatenated source-channel code is performed via soft-input soft-output (SISO) modules for the inner and outer codes. For the outer code, the SISO module is performed over a bit-level trellis representation of VLC, similar to the one originally proposed by Balakirsky [11].

LIST DECODING OF SERIALLY CONCATENATED VLC AND CHANNEL CODES
A list decoder provides an ordered list of the L most probable sequences in maximum-likelihood sense. Then, an outer error detecting code, usually a cyclic redundancy check (CRC) code, verifies the validity of the candidates and selects the error-free sequence, if exists, among the candidates. Two variations of the list Viterbi algorithm (LVA) are reported in [12]. An ordinary ML (Viterbi) decoder makes an error whenever the codeword closest to the received waveform is an erroneous codeword. For the list decoder to make an error, the correct sequence must lie outside of the L nearest neighbors of the received sequence. This error is less probable than the corresponding error in the ML decoder.
In a list decoder, the distance between the received sequence and all the candidates determines the performance. Therefore, determining the exact performance is mathematically intractable. But it is possible to calculate the asymptotic coding gain, for example, see [12]. In the case of AWGN channel, a geometrical argument reveals that the asymptotic coding gain is G = 10 log(2L/(L + 1)) dB for a list of length L. However, the actual gain is often less due to the multiplicity of the set of L nearest neighbors, which is neglected in the analysis [12].

List decoding of variable-length codes
List decoders can also be applied for variable-length encoded sequences, given an appropriate trellis (e.g., the bit-level trellises mentioned earlier). Our list decoding is constructed with the help of a non binary CRC code, which verifies the validity of the L most probable paths in the VLC trellis. The alphabet set of the CRC code must cover all codewords of the VLC (size q). If q is a power of a prime, it is possible to construct a q-ary CRC code, otherwise the size of VLC should be extended to the nearest power of a prime. One can use the a priori knowledge that these additional symbols are never present in the data sequence, but only (possibly) present in the parity sequence.
The asymptotic error rate for a list of size L = 2 is based on a simple geometric construction due to Seshadri and Sundberg [12] (see Figure 3). When the three codewords are pairwise equidistant, it produces a worst-case error probability. In this case, the minimum-magnitude noise resulting in an error is shown by the vector terminating at the circumcenter of the triangle. This vector represents the effective minimum distance, denoted by d eff , which is larger than d free /2, explaining the list decoding gain, which is equal to 10 log(2L/(L + 1)) dB, as calculated in [12].
This value of asymptotic gain, however, ignores the multiplicities of the minimum distance, and in our case minimum-distance error event has high multiplicities. Therefore, we augment the asymptotic analysis of [12,13] for L = 2, 3 list decoder of VLCs so that multiplicities are taken into account. We denote by N free the multiplicity of the minimum-distance errors. 2 The number of codeword triplets at minimum-distance that include the transmitted codeword is N eff = N free (N free − 1)/2. Thus, for L = 2 and assuming an AWGN channel, coding gain is the difference ∆γ = γ 1 − γ 2 , where γ 1 and γ 2 are the two values of E b /N 0 such that Simulations show that the coding gain thus obtained is more accurate than results that ignore multiplicities, for example, [12,13] (see Section 5). The disadvantage is that the equation above does not admit a closed-form solution.
Similarly, worst-case analysis can be repeated for L = 3 list decoder to calculate d eff . To obtain a more realistic approximation of the coding gain, we consider the multiplicity of the worst case of the set of three codewords, which is N eff = N free (N free −1)(N free −2)/6, given N free ≥ 3. The coding gain is calculated in a similar way as L = 2.

Proposed iterative list decoder
We now introduce an approximated list decoder for the concatenation of VLCs and channel codes. Our proposed iterative list decoder is demonstrated in Figure 4. After the last iteration, the final soft-output sequence produced by the SISO is decoded by the list Viterbi algorithm. The trellis used in this final decoder is similar to the one used in SISO-VLC.
The asymptotic analysis of the list decoder of turbo codes in [13] shows that the coding gain of list turbo decoder is higher than the coding gain of convolutional list decoder. Specifically, due to the low probability of multiple free-distance error events in a turbo-encoded sequence, the asymptotic coding gain is determined by the second minimum distance, yielding higher gain [13]. For the case of serially concatenated VLCs and convolutional codes, we show experimentally in Section 4 that significant improvements in coding performance can be achieved.

Nonbinary CRC
Wicker [14] provides a comprehensive background on Galois fields, rings of polynomials on Galois fields, and the construction of cyclic codes. We give here a quick summary of the key results as well as the procedure for designing nonbinary CRCs. Cyclic codes are built using a generator polynomial on the underlying Galois field GF(q). If the number of symbols in our application is not a power of a prime, the next higher appropriate q must be chosen, since for a field GF(q), q must be either a prime or a power of a prime. Cyclic codes are built from a generator polynomial g(X) on GF(q). The codewords are all the multiples of g(X) modulo X n − 1, where g(X) is a degree-r polynomial that divides X n − 1.
CRC codes are shortened cyclic codes that can encode up to n − r information symbols. CRC codes have excellent error detection capability. The CRC code with a generator of degree r detects all burst errors of length r or less, and the probability that the CRC will not detect a random error is q −r . Due to the lack of a convenient way to calculate the error spectrum of a CRC code, ad hoc methods have been used for code design in the binary case.
Unfortunately the existing ad hoc techniques for binary CRC design are not particularly helpful for the q-ary case, but nevertheless, the general structural properties, error coverage, and burst error detection properties remain the same across different underlying Galois fields. Therefore, even though we cannot design CRC with specified minimum distance, still it is possible to arrive at codes that have very respectable error detection performance. For example, for the 5-ary code used in the next section, a possible choice for generator polynomial is the primitive polynomial X 8 + 4X 6 + X 4 + X 3 + X 2 + 3X + 3 which requires 8 parity symbols for data sequences up to 390617 symbols. The undetected codeword error probability for this code is only 2.56 × 10 −6 . Table 1 shows the 5-ary source used in our experiments and various codes designed for this source. C 1 is a Huffman code, C 2 is an RVLC for this source reported in [4], C 3 is a highredundancy code designed by us because we observed that

EXPERIMENTAL RESULTS
the free distance of the outer code is a crucial factor in performance, as seen by the asymptotic behavior of the multiplicities A h in (1). It is noteworthy that despite the differences, the trellises of the different codes have roughly the same order of complexity, due to sparseness of the VLC trellises. Table 2 shows the recursive convolutional codes employed as inner code in our schemes. In our experiments, a packet of K symbols is entropy-encoded, interleaved, channel encoded, and transmitted using binary phase-shift keying (BPSK) modulation over an AWGN channel or fully interleaved Rayleigh fading channel. Figure 5a shows union bounds 3 and simulation results for the concatenated code C 2 +CC 1 . The calculation of the multiplicities for a nonlinear, variable-length code is a lengthy and time-consuming process, thus we present truncated bounds calculated with the first 10 terms of the multiplicities of the outer code that are available in [4]. The decoding experiment was performed with 10 iterations, with packet lengths of 20 and 200.

Iterative decoding
We consider two outer VLCs: code C 2 with free distance 2 and code C 3 with free distance 3, to build codes C 2 +CC 1 and C 3 +CC 3 with overall rates 0.445 and 0.404, respectively. 4 The symbol error rate (SER) of the two concatenated codes is shown in Figure 5b for K = 2000 symbols. In a wide range of E b /N 0 , the code C 3 + CC 3 outperforms C 2 + CC 1 and demonstrates a sharper drop in error rate. Other simulations have shown that in terms of frame error rate (FER), C 3 + CC 3 provides significant coding gain, about 1.4 dB at FER = 10 −3 .
For C 2 + CC 1 , we noticed that the higher number of iterations does not provide much of coding gain. We use the density evolution technique to give insight into the progress of iterative decoder. After an experimental verification that the LLR histograms are indeed Gaussian, we evaluated the approximate density evolution for C 2 + CC 1 and C 3 + CC 3 ( Figure 6). The two lower curves in each plot correspond to   Figure 6: Approximate Gaussian density evolution of C 2 + CC 1 and C 3 + CC 3 , K =2000.
the iterative decoding threshold [15]. The code C 3 + CC 3 has lower threshold than C 2 + CC 1 (0.5 dB compared to 1.15 dB). Borrowing the notion of iterative decoder tunnel from [15], we observe that the wider tunnel of C 3 +CC 3 provides a fast convergence with a few iterations: the higher the channel E b /N 0 , the fewer the iterations needed for convergence. These observations are in agreement with Figure 5b.

Iterative list decoding
We first evaluated the accuracy of our analysis for the performance of list decoding, which takes multiplicities into account. We used code C 2 , with K = 200 symbols in the AWGN channel. The coding gain at FER = 10 −4 is calculated as 1 dB for L = 2 and 1.4 dB for L = 3. These values are a better   match to simulations (Figure 7) than the coding gain predicted by [12]. Consider the two codes C 2 + CC 1 and C 3 + CC 3 . Figure 8 presents the FER of the iterative list decoder at the first, second, and third iterations with L = 1, 3 in AWGN channel with K = 500. C 3 + CC 3 outperforms C 2 + CC 1 . Figure 9 reports the FER of the concatenated codes in a fully interleaved Rayleigh channel with K = 200. At this frame size, the difference between the two concatenated codes is less pronounced, but still C 3 +CC 3 has lower error rate (except in the first iteration). List decoding has higher coding gain under fully interleaved Rayleigh channel, because of added diversity arising from increased equivalent free distance of the code [12]. The coding gain of C 2 +CC 2 at the fifth iteration for L = 2 is about 1.5 dB in Rayleigh fading, and 0.75 dB with L = 5 in AWGN channel. We refer the interested reader to [6] for further results on this code.

CONCLUSION
We propose an iterative list decoder for VLC-based sourcechannel codes. The iterative decoding of source-channel codes is made possible by the residual redundancy in the source code. Some source coders, such as H.263+, include additional redundancy for error resilience, making a sourcechannel decoder more desirable. It is shown that the amount of the redundancy in the VLC plays an important role in the performance of the code, given a total rate constraint. The list decoder is made possible by a nonbinary CRC code which also provides a stopping criterion for the iterative decoder. At a given iteration of the iterative decoder, the proposed list decoder improves the overall performance of the system. Extensive experimental results are provided in AWGN and fully interleaved Rayleigh channels.