Asymptotic Comparison of ML and MAP Detectors for Multidimensional Constellations

A classical problem in digital communications is to evaluate the symbol error probability (SEP) and bit error probability (BEP) of a multidimensional constellation over an additive white Gaussian noise channel. In this paper, we revisit this problem for nonequally likely symbols and study the asymptotic behavior of the optimal maximum a posteriori (MAP) detector. Exact closed-form asymptotic expressions for SEP and BEP for arbitrary constellations and input distributions are presented. The well-known union bound is proven to be asymptotically tight under general conditions. The performance of the practically relevant maximum likelihood (ML) detector is also analyzed. Although the decision regions with MAP detection converge to the ML regions at high signal-to-noise ratios, the ratio between the MAP and ML detector in terms of both SEP and BEP approach a constant, which depends on the constellation and a priori probabilities. Necessary and sufficient conditions for asymptotic equivalence between the MAP and ML detectors are also presented.


I. INTRODUCTION
The evaluation of the symbol error probability (SEP) and bit error probability (BEP) of a multidimensional constellation over an additive white Gaussian noise (AWGN) channel is a classical problem in digital communications.This problem traces back to [1] in 1952, where upper and lower bounds on Research supported by the Swedish Research Council (under grant #2011-5950) and by the Ericsson's Research Foundation.
A. Alvarado is with the Optical Networks Group, Department of Electronic & Electrical Engineering, University College London, London WC1E 7JE, UK (email: alex.alvarado@ieee.org).
the SEP of multidimensional constellations based on the maximum likelihood (ML) detector were first presented.
When nonuniform signaling is used, i.e., when constellation points are transmitted using different probabilities, the optimal detection strategy is the maximum a posteriori (MAP) detector.The main drawback of MAP detection is that its implementation requires decision regions that vary as a function of the signal-to-noise ratio (SNR).Practical implementations therefore favor the (suboptimal) ML approach where the a priori probabilities are essentially ignored.For ML detection, the decision regions are the so-called Voronoi regions, which do not depend on the SNR, and thus, are simpler to implement.
Error probability analysis of constellations for the AWGN channel has been extensively investigated in the literature, see e.g., [2]- [7].In fact, this a problem treated in many-if not all-digital communication textbooks.To the best of our knowledge, and to our surprise, the general problem of error probability analysis for mutidimensional constellations with arbitrary input distributions and MAP detection has not been investigated in such a general setup.
As the SNR increases, the MAP decision regions tend towards the ML regions.Intuitively, one would then expect that both detectors are asymptotically equivalent, which would justify the use of ML detection.
In this paper, we show that this is not the case.MAP and ML detection give different SEPs and BEPs asymptotically, where the difference lies in the factors before the dominant Q-function expression.More precisely, the ratio between the SEPs with MAP and ML detection approaches a constant, and the ratio between their BEPs approaches another constant.These constants are analytically calculated for arbitrary constellations, labelings, and input distributions.To the best of our knowledge, this has never been previously reported in the literature.Numerical results support our analytical results and clearly show the asymptotic suboptimality of ML detection.This paper is organized as follows.In Section II, the model is introduced and in Section III, the error probability bounds are presented.The main results of this paper are given in Section IV. Conclusions are drawn in Section V.All proofs are deferred to Appendices.

A. System Model
The system model under consideration is shown in Fig. 1.We consider the discrete-time, real-valued, Fig. 1.System model under consideration.For a given length-m transmitted binary label ci, the received vector Y is processed by the MAP or ML detector.The estimated symbol X is then converted to an estimated binary label Ĉ.
where the transmitted symbol X belongs to a discrete constellation X = {x 1 , x 2 , . . ., x M } and Z is an N -dimensional vector, independent of X, whose components are independent and identically distributed Gaussian random variables with zero mean and variance σ 2 per dimension.The conditional channel transition probability is We assume that the symbols are distinct and that each of them is transmitted with probability Neither the constellation points nor their probabilities depend on σ.We use the set I = {1, . . ., M } to enumerate the constellation points.The average symbol energy is The Euclidean distance between x i and x j is defined as δ ij = x i − x j and the minimum Euclidean distance (MED) of the constellation as d = min i,j∈I:i =j δ ij .
For the BEP analysis, assuming that M is a power of two, we consider a binary source that produces length-m binary labels.These labels are mapped to symbols in X using a binary labeling, which is a one-to-one mapping between the M = 2 m different length-m binary labels and the constellation points.
The length-m binary labels have an arbitrary input distribution, and thus, the same distribution is induced on the constellation points.The binary label of x i is denoted by c i , where i ∈ I.The Hamming distance between c i and c j is denoted by γ ij .
At the receiver, we assume that (hard-decision) symbol-wise decisions are made.The estimated symbol is then mapped to a binary label to obtain an estimate on the transmitted bits. 1 For any received symbol y, the MAP decision rule is This decision rule generates MAP decision regions defined as for all j ∈ I. Similarly, the ML detection rule is which results in the decision regions Example 1: Consider the 32-ary constellation with the nonuniform input distribution in [9, Fig. 2, Table I]2 .The constellation is shown in Fig. 2, where the area of the constellation points is proportional to the corresponding probabilities.In Fig. 2, the MAP and ML decision regions in (4) and ( 6) are shown for three values of the noise variance.These results show how the MAP regions converge to the ML regions as the noise variance decreases.△

B. Error Probability
Throughout this paper, the SEP and BEP are denoted by P s (σ) and P b (σ), respectively. 3Furthermore, we are interested in the error probability (SEP and BEP) of both the MAP and ML detectors.To study these four error probabilities, we define the generic error probability function where the transition probability T ij (σ) is given by The expressions ( 7)-( 9) represent both the MAP and ML detectors, as well as both the SEP and BEP, as explained in the following.
The error probability with MAP detection is obtaining by using R j (σ) = R map j (σ) in ( 9), where R map j (σ) is given by ( 4).Similarly, the use of R j (σ) = R ml j (σ) in ( 9), where R ml j (σ) is given by ( 6), leads to the error probability with ML detection.
To study the SEP, h ij in (7) should be set to one, which gives the well-known expression Similarly, the BEP expression [10, Eq. ( 1)], [11, Eq. ( 1)] is obtained by using h ij = γij m in (7).The four cases discussed above are summarized in the first three columns of Table I.

III. ERROR PROBABILITY BOUNDS
Error probability calculations for arbitrary multidimensional constellations and finite SNR is difficult because the decision regions defining the transition probabilities T ij (σ) in ( 9) are in general irregular.
Therefore, to analytically study the error probability, bounding techniques are usually the preferred alternative.In this section, we present two lemmas that give upper and lower bounds on the transition probability and will then be used to upper-and lower-bound the SEP and BEP in Section IV. where Proof: See Appendix A.
Lemma 2: For any i, j ∈ I, j = i and any σ < τ ij , where ∆ ij (σ) is given by (13), and The results in Lemmas 1 and 2 can be combined with (7) to obtain upper and lower bounds on the error probability: and where ∆ ij (σ), r(σ), and τ ij are given by ( 13), (15), and (16), respectively.In the next section, it will be proved that both these bounds are tight for asymptotically high SNR.

A. Main Results
The following theorem gives an asymptotic expression for the error probability in (7), i.e., it describes the asymptotic behavior of the MAP and ML detectors, for both SEP and BEP.The results are given in terms of the input probabilities p i , the Euclidean distances between constellation points δ ij , the MED of the constellation d, and the Hamming distances between the binary labels of the constellation points γ ij .
Theorem 3: For any input distribution, where and where h ij and w ij are constants given in Table I and The following corollary shows that, at high SNR, the ratio between the error probability with MAP and ML detection approaches a constant.This constant is upper-bounded by one and shows the asymptotic suboptimality of ML detection.
Corollary 4: For any input distribution and for either SEP or BEP, where where h ij is a constant given in Table I.Furthermore, R ≤ 1 with equality if and only if p i = p j , ∀i, j :

C. Examples
Example 2: Consider the one-dimensional constellation with M = 3, (x 1 , x 2 , x 3 ) = (−1, 0, 1), and , where 0 < p 1 < 1/2.If p 1 = 1/3, an equally likely and equally spaced 3-ary constellation is obtained.If p 1 = 1/K, this constellation is equivalent to a constellation with K equally likely points, of which K − 2 are located at the origin; such a constellation was used in [14] to disprove the so-called strong simplex conjecture.
In Fig. 3 (a), the exact SEP with MAP and ML detection, which can be analytically calculated, is shown for different values of p 1 .There is a clear performance differences between the two detectors when p 1 = 1/3.According to Corollary 4, B map s = 4 p 1 (1 − 2p 1 ) and B ml s = 2(1 − p 1 ), i.e., R = 2 p 1 (1 − 2p 1 )/(1 − p 1 ).Fig. 3 (b) shows the ratio of the SEP curves and how these converge to R as σ → 0 (indicated by the horizontal dashed lines).For p 1 = 0.167 and p 1 = 0.444, the asymptote is the same (R = 0.8); however, their SEP performance is quite different (see Fig. 3 (a)).This can be explained using the results in Fig. 3 (c), where the solid line shows R. The two markers when R = 0.8 correspond to p 1 = 0.167 and p 1 = 0.444, which explains the results in Fig. 3  bounds are only defined when E s /σ 2 > 20.77 dB, due to the restrictions on σ in (18).These result show very small differences between the MAP and ML detectors.To see the asymptotic behavior more , where G i is the number of neighbors at MED from point i.We illustrate this concept with the following example.bounds in (18), and the asymptotic expression, all divided by Q(d/(2σ)), are included as green, cyan, and blue curves, respectively.In this case, the lower bounds for ML and MAP detection are identical, as are the asymptotes.For this specific constellation, G i = 2 for all i ∈ I, and hence, B map s = B ml s = 2, independently of p 1 and p 2 , which implies R = 1.△

V. CONCLUSIONS
In this paper, an analytical characterization of the asymptotic behavior of the MAP and ML detectors in terms of SEP and BEP for arbitrary multidimensional constellations over the AWGN channel was presented.The four obtained results from Theorem 3 and Table I can be summarized as where the relative error in all approximations approaches zero as σ → 0. The expressions for MAP and ML are equal if and only if p i = p j , ∀i, j : Somewhat surprisingly, the results in this paper are the first ones that address the problem in such a general setup.The theoretical analysis shows that for nonuniform input distributions, ML detection is in general asymptotically suboptimal.In most practically relevant cases, however, MAP and ML detection give very similar asymptotic results.

APPENDIX A PROOF OF LEMMA 1
For MAP detection, we have from ( 9) and ( 4) that where H map ij (σ) is the half-space determined by a pairwise MAP decision (see ( 3)-( 4)), i.e., Using ( 2), (28) can be expressed as where ∆ ij (σ) is given by ( 13) and d ij = x j − x i .The value of |∆ ij (σ)| is the shortest Euclidean distance between x i and the hyperplane defining the half-space H map ij (σ).For a geometric interpretation, see H map ij (σ) in Fig. 6.Using (29), (27) can be calculated as where (31) follows from ( 1)-( 2) by recognizing Y − X, d ij /δ ij as a zero-mean Gaussian random variable with variance σ 2 .
The proof for the ML case is analogous but starts from (6) instead of (4).If follows straightforwardly that H ml ij (σ) is also given by (29), where now ∆ ij (σ) = δ ij /2 defines a hyperplane half-way between x i and x j .

APPENDIX B PROOF OF LEMMA 2
To lower-bound (9) for MAP detection, we first ignore all the contributions of constellation points not at MED, i.e., we use Pr Y ∈ R map j (σ)|X = x i ≥ 0, for all j such that δ ij > d.This gives the first case in (14).
To obtain a nonzero lower bound on (9) for δ ij = d, we first define an N -dimensional hyphersphere centered at the mid-point x ij = 1 2 (x i + x j ) and of radius r(σ) defined in (15).Second, we inscribe an N -dimensional hyphercube with half-side r(σ)/ √ N this hyphersphere.And third, we rotate this hyphercube so that one of its sides is perpendicular to d ij .We denote this hypercube by C ij (σ).For a geometric interpretation when N = 2, see Fig. 6.
To lower-bound (9), we integrate f (y|x) in (2) over the intersection of the MAP region R map j (σ) and the hypercube C ij (σ), i.e., We will show that for sufficiently low values of σ, the integration region in (33) is an orthotope (hyperrectangle) with N − 1 sides of length 2r(σ)/ √ N and one side of length d 2 + r(σ) √ N − ∆ ij (σ), i.e., the hyperplane defining the half-space H map ij (σ) intersects the hypercube C ij (σ).The result in the second case in (14) follows immediately by integrating the Gaussian probability density function over this orthotope.For a geometric interpretation when N = 2, see x i and x j in Fig. 6.
For R map j (σ) ∩ C ij (σ) to be an orthotope, we need to prove that R map l (σ) ∩ C ij (σ) = ∅ for any i, j, l ∈ I such that l / ∈ {i, j} and δ ij = d.To prove this, it suffices to prove that for any y ∈ C ij (σ), for k ∈ {i, j} and l / ∈ {i, j}.The proof follows from the inequalities where (36) follows from (15).Finally, we obtain which proves (34) for any y ∈ C ij (σ).
The last step in the proof is to show that r > 0 and that the hyperplane defining the half-space H map ij (σ) intersects the hypercube C ij (σ).The first condition is obtained from (15): The second condition is satisfied when Using ( 15), (13), and some simple algebra, it is possible to show that (39) is equivalent to which proves the second condition.The proof is completed by noting that 2(1+ Because of this, (40) is stricter than (38), and thus, the inequality in (15) holds for σ < τ ij with τ ij given by ( 16).
The proof for ML detection follows from lower-bounding T ml ij (σ) as in (32) (by using R ml j (σ)), and by the fact that all the remaining steps in the proof above also hold when ∆ ij = d/2.

APPENDIX C PROOF OF THEOREM 3
To prove Theorem 3, we first use (7) to obtain As will become apparent later, the limit on the right hand side of (41) exists and, hence, so does the limit on the left hand side.To calculate the limit in the right hand side of (41), we will sandwich it using Lemmas 1 and 2.
For MAP detection, we first study the asymptotic behavior of the upper bound in Lemma 1 where (44) follows from l'Hôpital's rule.
Next, we study the asymptotic behavior of the lower bound in Lemma 2 for δ ij = d (the lower bound ( 14) is zero for δ ij > d).Assuming that all the limits exist, we obtain The first limit in (48) is the same as in (42)-(46).The second limit is zero, because r(σ) > 0 if σ < τ ij as shown in Lemma 2. The last limit in (48) is one because by (15), lim σ→0 r(σ) = d/(2(1 + √ 3)).
Hence, all limits exist and asymptotically, both lower and upper bounds converge to (46).Using this in (41) gives which completes the proof for MAP detection.
The proof for ML detection follows similar steps.Substituting ∆ ij (σ) = δ ij /2 from ( 13) into (42) yields The asymptotic expression for the lower bound in (48) holds unchanged in the ML case too.In this case, the first limit is given by (51), the second is zero, and the third is one.This combined with (41) completes the proof for ML detection.

(a) σ 2 Fig. 2 .
Fig.2.ML (dashed blue) and MAP (solid red) decision regions for the constellation in Example 1 and three values of the noise variance.The area of the constellation points is proportional to their probabilities.As the noise variance decreases, the MAP regions converge to the ML regions.

Fig. 3 .
Fig. 3. Results obtained for the constellation in Example 2: (a) SEP with MAP and ML detection, (b) ratio of SEPs and asymptote given by Corollary 4, and (c) asymptote as a function of the symbol probability p1.

clearly, Fig. 4 (
b) shows the ratio between the eight curves in Fig. 4 (a) and Q(d/(2σ)).It is clear that the simulated BEPs closely follow the upper bounds at these SNR values.These results also show that both the upper bounds and lower bounds converge to B map b = 0.1450 and B ml b = 0.1495 for MAP and ML detection, respectively, as predicted by Theorem 3. Unlike Fig. 4 (a), Fig. 4 (b) clearly shows the asymptotic difference between the MAP and ML detectors.△ Corollary 4 gives necessary and sufficient conditions for the asymptotic optimality of ML detection.A nonuniform distribution will in general give R < 1, although there are exceptions.Consider for example a constellation that can be divided into clusters, where all pairs of constellation points in different clusters are at distances larger than the MED.Then ML detection is asymptotically optimal (i.e., R = 1) if the probabilities of all constellations points within a cluster are equal, even if the clusters have different probabilities.In this special case, (20) yields B map s

Example 4 :Fig. 5 .
Fig. 5. Results obtained for the constellation in [15, Fig. 3 (d)]: (a) Constellation where the pairs of symbols at MED are marked with solid lines and the symbol probabilities are indicated by the point areas, and (b) asymptotic performance shown as the ratio between SEPs and Q(d/(2σ)).

Fig. 6 .
Fig. 6.Geometric representation of the proofs of Lemma 1 and 2 for a 2D constellation with MAP detection.The MAP decision regions are shown for xj, x l , and x k , the half-space in (29) for xj (H map ij (σ)), and the hypercube for x k (Cij(σ)).

TABLE I VALUES
OF X AND hij THAT USED IN (7)-(9) GIVE SEP AND BEP EXPRESSIONS FOR BOTH THE MAP AND ML DETECTORS.THE LAST COLUMN SHOWS THE VALUES OF wij FOR THE ASYMPTOTIC EXPRESSIONS IN SECTION IV.
Using h ij = h ji and δ ij = δ ji we obtain which holds with equality if and only if p i = p j , ∀i, j : δ ij = d.