On Symbol Error Performance of Probabilistic Shaping in Noise-Limited and Fading Channels

In this work, we present symbol error performance analysis of probabilistic shaping (PS) for wireless communications in noise-limited and fading channels. Two fading models considered are the Rayleigh and log-normal fading channels. The results are corroborated with simulation and compared with the conventional uniformly distributed input symbols. In all channel conditions, PS results in significant reductions in the SNR required to achieve a specific error probability compared to the conventional uniformly shaped symbols. For example, in a noise-limited channel, PS based quadrature amplitude modulation (QAM) signal results in SNR gains of 1.16 dB, 1.41 dB, and 1.52 dB compared to the uniformly distributed QAM symbols at equal entropy rates of 4, 6, and 8 bit/symbol and a symbol error ratio (SER) of $1\times 10^{-3}$ .


I. INTRODUCTION
E NHANCING spectral efficiency and optimising the use of available scarce resources are crucial requirements of any communication systems. Yet, in the conventional data transmission, in which each symbol (or constellation point) is transmitted with equal probability, the distribution of input symbols is not a perfect fit for the linear additive white Gaussian noise (AWGN) channel. Consequently, it does not allow for optimal utilisation of the channel capacity [1]. Moreover, in a wireless channel, the influence of fading and multipath propagation deteriorate the system performance [2].
The error probability of pulse-amplitude modulation (PAM) and quadrature amplitude modulation (QAM) in AWGN channels has been studied extensively [2], [3], [4]. Moreover, several works have also presented the performance of wireless communication systems in Rayleigh and lognormal fading conditions [5], [6], [7]. In [5], tight upper and lower bounds on the error probability of coded modulation schemes in Rayleigh fading channels are studied. The average error probability in fading channels for different modulation techniques has also been evaluated by approximating the sum of two exponential functions in [6]. For a multiple access cellular uplink network in Rayleigh fading, a tight upper bound symbol error probability is also derived in [7]. However, all these studies consider only uniformly distributed input symbols.
Optimising the distribution of source symbols through signal shaping is crucial for enhancing communication system performance. One shaping method that has garnered considerable attention is probabilistic shaping (PS). It involves adjusting the probabilities of occurrence of constellation points to create a Gaussian-like distribution [1], [8]. Thus, resulting in non-uniform distribution of data symbols. Therefore, by applying PS to QAM symbols, low-energy symbols are transmitted more frequently than high-energy symbols. This is realised by using a distribution matcher, such as a constant composition distribution matcher (CCDM), proposed in [9], which maps input information bits into non-equiprobable symbols with a Gaussian-like distribution. The Maxwell-Boltzmann distribution is commonly used to generate Gaussian-like signalling which has been shown theoretically to attain the ultimate shaping gain (π e/6 or 1.53 dB) [8], [10], [11]. PS is essentially an approach to trade energy efficiency for spectral efficiency.
PS has shown to be an optimum technique in optical fibre communications to achieve record-setting transmission rates and distances [1], [12], [13], [14], as well as in optical wireless communications [15], [16], [17]. Similarly, PS has been employed in wireless communications to maximise transmission capacity and improve system performance. For example, PS has contributed to a substantial improvement in transmission capacity and system performance for highspeed Terahertz photonic wireless communication systems in [18], [19], [20]. PS has also been applied to mitigate hardware distortions in [21]. Therefore, it is important to study the error performance of PS in wireless communication systems in the presence of fading. Related work in this domain include, designing PS with constellation rotation and component interleaving for diversity gain in [22].
In this paper, we study the theoretical symbol error performance of PS. For such a scheme, the maximum likelihood detection will not be applicable. Therefore, an optimum maximum a posteriori (MAP) detector is derived. The symbol error ratio (SER) of PS based uncoded QAM with modulation order, M, (PS-M-QAM) under Gaussian noiselimited condition is analysed, and analytical expressions are provided. The results are compared with the conventional uniformly distributed uncoded QAM symbols. We extend the PS approach to Rayleigh and log-normal fading channel conditions and derive expressions to estimate the PS performance in the presence of fading. The presented analysis is based on symbol error rate and therefore agnostic of the choice of bits-to-symbol distribution matcher. The findings from this study can be extended to investigate the PS bit error rate (BER). The BER of PS depends on the choice of distribution matcher used. Moreover, the distribution dematcher is sensitive to errors and it usually needs to be accompanied with forward error correction (FEC). That is, FEC encoding along with the PS distribution matcher in the transmitter and FEC decoder and distribution dematcher in the receiver. The order of shaping and coding processing steps in the transmitter and receiver is still an open and yet interesting research to investigate [12], [23].
We analyse the impact of a rate parameter, which is used to find an optimum probability mass function (PMF), on the entropy and the error performance of the system. We also compare the SNR gain using a rate parameter which provides an equal entropy between PS and uniformly dis-  /symbol between uniform 16-QAM,  64-QAM, and 256-QAM against higher order PS-M-QAM, PS QAM provides SNR gains of 1.16 dB, 1.41 dB, and 1.52 dB at the same SER. These results approach the ultimate shaping gain (1.53 dB) that can be achieved using a Gaussian distribution over uniform distribution at an equal entropy rate [8].
The rest of the paper is organised as follows: Section II presents a generation and detection of PS based symbols. In Section III, the error performance analysis of PS-QAM in an AWGN channel condition is presented. This is then extended to fading channel conditions in Section IV. Performance in log-normal fading condition is presented in Section IV-A while that of Rayleigh fading channel is presented in Section IV-B. Finally, conclusions are given in Section V.

II. GENERATION AND DETECTION OF PS SYMBOLS
In this section, the basics of generating PS based symbols using a Gaussian-like input distribution is presented and an optimum maximum a posteriori (MAP) detector is derived. In PS-M-QAM, to maximise the channel capacity of a Gaussian channel, symbols are chosen from a Gaussian-like input distribution [8], [10]. The common approach is to use the input distribution from the family of Maxwell-Boltzmann distribution whose 1-dimensional form is represented as [14], [24]: with λ ≥ 0 representing the rate parameter. It is used to search for the optimum PMF depending on the choice of entropy, H = − x m X p X (x m ) log 2 (p X (x m )) and average symbol energy, ξ av = x m X |x m | 2 p X (x m ). Appendix A explains the relationship between the rate parameter, variance of a Gaussian distribution and illustrates how the Maxwell-Boltzmann distribution approximates it. For λ = 0, the distribution is uniform while it becomes Gaussian distribution with reduced variance as λ increases. The PMF of an M-QAM constellation is the product of the respective constituent PAM probabilities.

B. DETECTION OF PS SYMBOLS
The optimum detection scheme for PS is based on the MAP detector. This is because the detection needs to take the priori symbol distribution into account unlike the maximum likelihood detector that does not [3]. In this section, the MAP detector for PS with Maxwell-Boltzmann distribution is discussed. The analysis is based on a one-dimensional M-PAM signal which can be easily extended to a two-dimensional QAM signal. Consider an AWGN channel such that the received symbols over a symbol duration are defined as Y = X +Z, where X is the transmitted input symbols and Z is an independent and identically distributed noise vector with zero-mean and variance, σ 2 n = N 0 /2 Gaussian random variable. Here, N 0 denotes the single-sided noise power spectral density. The input symbols, X take values from {x 1 , x 2 , . . . , x m } according to the priori PMF given by (1). The AWGN channel can be described by its conditional distribution, p Y|X (y|x m ) as [3]: The MAP detector is thus given by [3]: Consequently, using (1) and (2), the MAP detector in (3) can be evaluated as: The full derivation of (4) is provided in Appendix B. The decision region, D m for all 1 ≤ m ≤ M and 1 ≤ m ≤ M where m = m is given as: Note that from (5), there are at most M − 1 threshold points, r th where D m = D m+1 . In the following sections, the error performance analysis of PS using the MAP detector rule given by (4) is presented. In Section III, the error performance analysis in an AWGN channel condition is presented followed by fading channel conditions in Section IV.

III. ERROR PERFORMANCE IN AWGN CHANNEL
The symbol error probability of the detection scheme is determined by accumulating the product of the priori distribution and the probability of error that occurs when the received symbol is not in the D m given x m is transmitted. This can be written as: In the following subsections, the error probability expression is formulated for 4-PAM, which will be used as an example. Then a general SER expression of M-PAM for an arbitrary amplitude level M is derived.

A. ERROR PROBABILITY OF 4-PAM
The signalling diagram of 4-PAM with amplitude levels of Fig. 1. Here, E s is the symbol energy while the average symbol energy is denoted by ξ av . For the 4-PAM scheme, the decision boundaries are determined using (5) as: Thus, the decision thresholds are r th m = (−2, 0, 2) × (1 + N 0 λ) √ E s /ξ av for m = 1, 2, 3. Generally, the threshold points of M-PAM can be induced as: Note that unlike the uniform distribution symbols, the MAP threshold points depend on the noise level and the shaping rate parameter, λ. This is further illustrated in Fig. 2 considering the normalised third threshold point, As λ increases, the threshold point increases and moves to the outer constellation point. The increase is higher in noisy channel (E s /N 0 = 10 dB) compared to a channel with less noise (E s /N 0 = 16 dB).
Taking advantage of the symmetry in amplitude and using (6), the error probability can therefore be evaluated as shown in (9). In this analysis, an error detection has occurred if a symbol m is transmitted but has been detected in D m region when it should be in region D m . This is mathematically described in (6). For example, the first term in (9a), shown at the bottom of the next page describes the case for when m = 1 (x = −3 E s ξ av ) is transmitted. For correct detection, it should be received in region D 1 . However, if it is received in region D m for m = m (i.e., m = 2, 3, 4) then error is detected. Consequently, the range of threshold for erroneous detection is [r th 1 , ∞). Using the fact that p 2 = p 3 , as shown in Fig. 1, p 2 in the second term of (9c), shown at the bottom of the page is replaced by p 3 and (9c), shown at the bottom of the page is reformulated in a summation form as follows: Here, Q(·) denotes the Gaussian Q-function and 1/ξ av .

B. GENERAL SYMBOL ERROR PROBABILITY EXPRESSION OF PS-PAM
From the decision boundary expressions in (7), (8), and the SER of 4-PAM in (10), regular patterns can be drawn for general M-PAM signal. This general form of symbol error performance of PS-M-PAM is given as: Note that for equiprobable symbols, λ = 0 and p m = 1/M for all m and (11) reduces to the error probability of a conventional uniform M-PAM. To illustrate the impact of λ on SER given in (11), the SER of PS-16-PAM is shown in Fig. 3 for different λ values. The PS-16-PAM symbols are generated with entropy rates, H = {3, 3.2, 3.6, 3.92} bit/symbol using λ = {0.0331, 0.0250, 0.0132, 0.0047}. The SER of uniform 16-PAM (H = 4 bit/symbol) is also shown for comparison. The result shows that PS achieves better energy efficiency and the gain increases as the λ increases. However, it comes at the cost of reduced net information rate: as λ increases, H decreases. Thus, to maximise the PS gain with optimum entropy level, the knowledge of the channel SNR response of the system is required, from which optimum PMF and thus the entropy is determined by the choice of optimum λ. This optimisation process is realised by minimising the gap between the entropy of symbols to the pre-estimated channel capacity. An experimental study that demonstrates how PS achieves near channel capacity transmission is presented in [15], [17].
To evaluate the PS error performance independent of a choice of rate parameter, comparison at equal net entropy rates is crucial. To achieve this, a comparison of the SER performance at equal net entropy rates is carried out by using PS-M-PAM symbols against uniform symbols of lower order. The results are shown in Fig. 4 and have been validated with . As PS has lower H at the same modulation order, higher orders (e.g., 2M) will be used compared to uniform distribution, which uses M. In all cases, PS has better SER performance than the uniform signal with 1.13 dB, 1.37 dB, 1.43 dB SNR gains at SER = 10 −3 for H = {2, 3, 4} bit/symbol, respectively. This validates that the SNR gain is not necessarily associated with the entropy reduction. In fact, at equal entropy rates, PS approaches the ultimate 1.53 dB shaping gain. Higher SNR gain can be obtained by using PS-QAM which uses two orthogonal PS-PAM constellations. Generally, irrespective of the SER, a shaping gain can be found by evaluating λ which gives equal entropy, H so that the entropy of the PS, H PS = log 2 M and using this λ to evaluate the average symbol energy of PS symbols. Fig. 5 shows the shaping gain for increasing entropy values relative to the ultimate shaping gain π e/6 = 1.5329 dB. The gain increases with entropy until it approaches the ultimate gain asymptotically, as shown in the inset figure.

C. SYMBOL ERROR PROBABILITY OF PS-QAM
The error probability of M-QAM can be determined from the error probability of √ M-PAM with half the total energy as [3]: Using (12) and the half energy form SER of PS-M-PAM given by (11), the SER of PS-M-QAM is obtained as: where, For PS-4-QAM, all constellation points are in an equal constellation ring with same probabilities. Hence, λ = 0, = 1, and m = 1. Consequently, (13) reduces to: which is the same as the symbol error probability of uniformly distributed 4-QAM.
The SER of PS-QAM given by (13) corroborated with Monte Carlo simulation is shown in Fig. 6. For comparison, the SER of the conventional uniformly distributed QAM is also included. In this illustration, PS-M-QAM symbols with M = {16, 64, 256} are generated with entropy, H = {3.6, 5.4, 7.2} bit/symbol using rate parameter, λ = {0.1353, 0.0408, 0.0119}, respectively. The entropy of PS-M-QAM is thus set to be 0.9 × log 2 M of uniform counterparts. The performance improvement from PS over the uniformly distributed symbols is evident from the result. For all modulation orders shown and ranges of SNR, the PS outperforms uniform distribution. For instance, at SER = 10 −3 , about 2.12 dB, 2.89 dB, 3.45 dB SNR improvement is obtained from PS-16-QAM, PS-64-QAM, and PS-256-QAM over the corresponding uniform QAM symbols, respectively. Note that the amount of SNR improvement depends on the rate parameter, λ for each modulation order, M.
The SER performance comparison is also repeated at equal net entropy rates by using PS-M-QAM symbols against uniform symbols of lower order (M/4-QAM) and presented in Fig. 7. Entropy rates, H = 4 bit/symbol (PS-64-QAM and  In all cases, PS has better SER performance than the uniform signal with SNR gains of 1.16 dB, 1.41 dB, and 1.52 dB for H = {4, 6, 8} bit/symbol, respectively, at SER = 10 −3 . These shaping gains are higher than PS-PAM and achieve the ultimate shaping gain.

IV. ERROR PERFORMANCE IN FADING CHANNELS
In a fading wireless channel, the received instantaneous signal power is scaled by |h| 2 , where h, the channel coefficient, is a random variable. Thus, the instantaneous SNR per symbol is defined as γ = |h| 2 E s /N 0 , and the average SNR per symbol becomes γ = |h| 2 E s /N 0 . Due to the impact of fading on the received signal amplitude, the decision threshold of M-PAM given in (8) for noise-limited condition will now change to the generalised form given by: The effect of h and λ on the normalised third decision threshold point, r th 3 is demonstrated in Fig. 8. For this instance, SNR = 10 dB is considered. For equiprobable case (λ = 0), and strong attenuation (h = 0.5), the threshold is r th 3 = 1 as against r th 3 = 2 with h = 1. This figure shows that the threshold is no longer fixed but varies with h for any given λ and N 0 . In noise-limited case, this depends on the SNR and λ as it was shown in Fig. 2.
The average symbol error probability in the a fading channel then becomes [2]: where p γ (γ ) is a probability density function (PDF) of the instantaneous SNR which depends on the nature of the fading channel. P s (γ ) is evaluated by the SER expression defined in (6) and takes into account the impact of fading on decision thresholds given by (16). However, this leads to an expression which is not tractable. Therefore, in the following subsections, we present a suboptimum tractable SER of Rayleigh and log-normal fading channel conditions with suboptimum decision threshold that is based on (8). It should be noted that the analysis can be easily extended to other fading models.

A. LOG-NORMAL FADING CHANNEL
The average symbol error probability in the presence of lognormal fading can be evaluated using (17). The PDF of log-normal fading, p γ (γ ) is given by [25]: where μ = −σ 2 I /2 and σ γ are the mean and standard deviation of ln I, respectively. From (13) and (18), the SER of PS-M-QAM in log-normal fading channel can be evaluated, using a Gauss-Hermite quadrature integration [26, eq. (25.4.46)], as: where, n = I 0 exp ν n 2σ 2 Here, {ν n } and {w n } denote the zeros and weights of the N thorder Hermite polynomial with n = 1, 2, . . . , N, respectively. The accuracy of (19) depends on the order of the Hermite polynomial.
The SER performance comparison between PS-QAM in log-normal fading channel (expression (19)), validated with Monte Carlo simulation, and the corresponding uniformly distributed symbols is shown in Fig. 9.  SNR required due to fading at the same BER as the noiselimited case) is about 4.5 dB for both schemes. These results reveal that PS can efficiently improve system performance in a log-normal fading channel as well.
The SER performance comparison is repeated at equal net entropy rates by using entropy rates H = {4, 6, 8} bit/symbol which are used in the AWGN case. The result is presented in Fig. 10. In all cases, PS QAM provides SNR gains of 0.98 dB, 1.33 dB, and 1.47 dB for H = {4, 6, 8} bit/symbol, respectively, at SER = 10 −3 . However, the SNR gains are lower than the respective AWGN cases. Overall, the performance of PS in terms of the SNR gain across different fading variances is evaluated. Here, the SNR gain refers to the improvement in SNR achieved by using PS-based QAM instead of uniformly distributed QAM to achieve a specific BER. The results indicate that the performance of PS is superior to that of the uniform signal in all cases. Figure 11 presents the SNR gain required to achieve a SER of 10 −3 . The result shows that the highest gain is achieved at H = 8 bit/symbol, and despite the increase in fading variance, the gain remains fairly consistent. However, the gain is relatively lower for lower entropy, such as H = 4 bit/symbol, and as the fading variance increases, the gain decreases. This corresponds to the result in Fig. 5, which shows that the shaping gain is higher for larger H values and approaches the ultimate gain.

B. RAYLEIGH FADING CHANNEL
The PDF of the instantaneous SNR for the Rayleigh fading channel is given by [2]: Using (13) and (21) in (17), the symbol error probability of PS-M-QAM in Rayleigh fading channel is estimated as given in (22), shown at the bottom of the next page.   to the uniform distribution. At a particular SER of 10 −3 , for instance, PS-16-QAM, PS-64-QAM, and PS-256-QAM yield SNR gain of about 1.63 dB, 2.59 dB, and 3.27 dB compared to the respective uniform QAM symbols. While at the same SER, the fading penalty is about 21 dB for both schemes. These results demonstrate that PS can effectively improve system performance in a Rayleigh fading wireless channel.
The SER performance comparison at equal net entropy rates is presented in Fig. 13 using entropy rates H = {4, 6, 8} bit/symbol similar to AWGN case. In all entropy values, despite the SNR gains are much lower than the AWGN case, PS has better SER performance than the uniform signal with SNR gains of 0.27 dB, 1.02 dB, and 1.33 dB for H = {4, 6, 8} bit/symbol, respectively, at SER = 10 −3 .

V. CONCLUSION AND FUTURE WORK
In this work, the symbol error performance analysis of PS with MAP detection scheme has been presented. The analysis is also extended to Rayleigh and log-normal fading channels. The symbol error performance of PS is analysed and closed-form analytical expressions validated with Monte Carlo simulations are provided. This study presents a framework that demonstrates how PS trades energy efficiency (SNR required to achieve a given error performance level) for spectral efficiency and how the design parameters are VOLUME 4, 2023 chosen. These include analysing the effect of the shaping rate parameter, which set the PMF and hence the source entropy and spectral efficiency, on the error performance of the PS scheme. The result is then compared with the error performance of the traditional uniform distribution. We also compare the SNR gain using a rate parameter which provides an equal entropy between PS and uniformly distributed QAM symbols. Our results show that PS outperforms uniform distribution and it reduces the SNR required to achieve a specific error probability significantly. The findings of this study have important practical applications in communication systems. The gain in SNR resulting from the use of PS can significantly improve the noise or fading resilience of the system, leading to more reliable and efficient data transmission. This is particularly important in scenarios where power consumption is a critical factor, as PS can enable the design of more energy-efficient wireless communication systems. Alternatively, the increase in SNR can be utilised to improve the achievable information rate without the need to increase the transmitted signal power. This is especially relevant in high-speed applications, where the use of PS can allow for higher data rates and more efficient utilisation of the available bandwidth. Overall, the findings suggest that PS can offer significant benefits for the design and optimisation of communication systems, leading to more robust and efficient communication in various practical applications.
Another possible area of future research would be to investigate the bit error performance of the PS. The BER analysis of PS will be based on the choice of the distribution matcher and FEC. Combining shaping with coding with optimal rates will enhance the SNR gain further.

APPENDIX A MAXWELL-BOLTZMANN AND GAUSSIAN DISTRIBUTIONS
The Maxwell-Boltzmann distribution provides a Gaussianlike distribution for input symbols. It can be shown that the Maxwell-Boltzmann distribution approached the Gaussian distribution depending on the modulation order, M and the rate parameter, λ. Assume a Gaussian distribution given by: Since PS symbols are symmetric about x = 0, it is possible to use b = 0. For (1) to approach the Gaussian distribution in (24), we have: (25b) Figure 14 shows a PS-based 8-PAM symbol distribution generated with a rate parameter, λ = 0.0435 and its Gaussian fit, f (x) using a = 0.2393, b = 0, and c = 3.3906.

APPENDIX B DERIVATION OF MAP DETECTOR RULE
Substituting (1) and (2) into (3), the full derivation of the MAP detector rule given by (4) is presented as follows: