Deep Learning-Based Signal Detection for Rate-Splitting Multiple Access Under Generalized Gaussian Noise

In this paper, we propose a long short-term memory-based deep learning (DL) architecture for signal detection in uplink and downlink rate-splitting multiple access systems with multi-carrier modulation, over Nakagami-m fading and generalized Gaussian noise (GGN). The proposed DL detector completely eliminates the need for the use of successive interference cancellation (SIC), which suffers from disadvantages such as error propagation. In an orthogonal frequency division multiplexing setting, we show that the proposed DL detector outperforms the standard SIC receivers such as the least squares detector and the minimum mean-squared error receiver, and attains the performance of the optimal maximum likelihood detector, in terms of the symbol error rate (SER). Furthermore, we study the effects of the shaping parameter of GGN, hyperparameters of the DL network such as batch size and learning rate on the SER performance.


I. INTRODUCTION A. BACKGROUND
The sixth generation standard (6G) for mobile communications system, which is a successor to the currently-beingdeployed fifth generation (5G) standard, has already become one of the most sought out research interests across the globe [1], [2]. Few of the appealing promises of the 6G wireless communication systems include heterogeneous qualityof-service (QoS), ultra high reliability and high throughput with massive connectivity [3], [4]. To realize these promises, it is vital to utilize the wireless resources rationally, and to handle the interference effectively. Towards this end, rate-splitting multiple access (RSMA) is foreseen as a permissive technology, which is a multi-user, multi-antenna, non-orthogonal multiple access (MA) scheme [5], [6].
Rate-splitting (RS) was first studied by Carleial [7] for a two-user case, in which each user was equipped with a single antenna. An inner bound on the capacity region and successive cancellation decoding were proposed. Han and Kobayashi (HK) [8] improved this inner bound by making use of RS and simultaneous decoding, which led to a flexible and powerful interference management scheme. This is the central principle of RS on which several wireless networks-based applications can be built on. Further, such an interference management allows RS to bridge the two extreme strategies of completely decoding the interference and completely treating the interference as noise. As an extension to the HK scheme, where a given user's message is split into a single common and a single private part, a generalized rate-splitting scheme divides each user's message into 2 Q−1 streams, where Q is the number of users [9]. A stream order is generated to detail the order in which streams need to be decoded at each user. The data streams are precoded and superposed, which is then transmitted over the channel. Precoding can either be linear Most of the literature on RSMA focuses on the downlink transmission of data, with different variants of RSMA, namely 1-layer RS [9], [10], [11], [14], [16], [17], [25], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], 2-layer HRS [9], [20] and generalized RS [9], [43], [44]. Uplink RSMA, although not as widely explored as downlink RSMA, has been investigated in [45], [46], [47], [48], [49], [50]. Although the most popular receiver is SIC-based, several other receiver architectures have also been proposed in the literature. A turbo decoding receiver and a joint decoding receiver have been proposed in [43], [51], [52]. However, these designs largely rely on the channel, noise and interference models and associated assumptions, which could be highly application-specific [21]. Other overheads include delay and complexity, which exponentially rises with an increase in the modulation order and/or the number of users/streams [53]. Further, it should be noted that the performances of these receivers severely degrades with an increase in the errors due to channel estimation, and in case of imperfect channel state information at the transmitter (CSIT) [54].
On the other hand, deep learning (DL) techniques have received significant attention for MA in wireless communications in the last few years in a wide range of applications, including internet-of-things (IoT) and vehicular ad hoc networks (VANET) [55]. As an example, consider the use case in VANETs where users in a given geographical area communicate with a base station (BS) to find empty parking spaces. Using DL algorithms in such scenarios not only increases the reliability, but also reduces complexity and latency. Such DL approaches can be largely grouped into two methods, namely data-driven and model-driven methods. The data-driven approach aims to improve the performance in a specific problem setting, by improving the data quality and data governance. On the contrary, the model-driven approach aims to improve the performance by building new models with new algorithmic improvements. Among the recent MA techniques, DL algorithms have been largely applied and studied for NOMA, which is a special case of RSMA, for signal detection [56], [57], [58], [59], [60], [61], [62], [63], [64], [65], [66], [67], [68], [69], [70] and channel estimation [71], [72]. In [56], a DL-based multiuser detection scheme has been investigated for NOMA in a MIMO setting, where a low-complexity deep neural network (DNN) is proposed instead of the conventional SIC receiver. In [57], a long short term memory (LSTM)-based DL algorithm is proposed for symbol detection in NOMA. In [58], the authors extend the single-carrier system in [56] to an orthogonal frequency division multiplexing (OFDM) system, and propose a recurrent neural network (RNN) with a LSTM layer that replaces the conventional SIC receiver. In [59], [60], a DL detector using LSTM is proposed for joint detection of OFDM-NOMA symbols. LSTM is also proposed for DL-based detection in co-operative NOMA [61] and grant-free NOMA [63]. All the aforementioned works consider data-driven approaches. Model-driven DL algorithms are also explored for NOMA signal detection, particularly in [65], [67], [68], [69], where the SIC technique is retained. Although all the aforementioned DL models can replace the SIC technique, the approach discussed in [58] has been shown to attain the performance of optimal maximum-likelihood (ML) receiver. Further, it is established that RNNs perform well in a variety of applications including speech recognition, signal detection, language modeling and translation [73], and are effective in processing of sequence tasks. When the sequence is relatively long, which is usually the case in signal detection, LSTM -which is an enhanced version of RNN -is considered as it solves the crucial issues of vanishing gradient, exploding gradient and long-term dependence problem [74]. In general, using LSTM provides an accurate performance due its ability of capturing long-term dependencies [75]. Due to these advantages, several studies have considered LSTM-based receiver architectures for NOMA signal detection. However, to the best of our knowledge, LSTM techniques for signal detection in RSMA have not been studied in literature so far.
In general, only very few works in literature consider DL techniques employed in RSMA. In [76], a DL based approach has been considered for optimal power allocation among the common and private streams. In particular, employing DL in the design of RSMA receivers has very limited literature. The primary advantage that a DL-based receiver provides is that it is suitable for scenarios where the channel and noise parameters cannot be acccurately estimated [77], [78]. Very recently, a model-based DL (MBDL) receiver design has been proposed for RSMA systems in [79]. Here, the authors have proposed a DL-based RSMA receiver that combines the simple structure of an SIC-based receiver with the advantages that DL offers. It was shown that this receiver achieves the optimal performance of the maximum a-posteriori probability (MAP) receiver, and significantly outperforms the traditional SIC receiver in terms of symbol error rate (SER). However, this MBDL receiver, which is built particularly for downlink RSMA, needs to perform SIC. Further, the MBDL receiver architecture has two DNNs for each stream at each user end, which have to be individually trained. Additionally, the MBDL architecture is trained with uncoded data bits, and does not consider practical coded scenarios involving multicarrier modulation. Moreover, this receiver is built under the assumption of a Rayleigh fading channel with additive white Gaussian noise (AWGN) added at the receiver, which limits its applications.
The available literature on RSMA largely considers performance studies in the presence of AWGN at the receiver for the ease of analysis, which fundamentally describes the thermal noise at the receiver end [80], [81], [82], [83]. However, the AWGN model is known to be not accurate in several communication scenarios as it neglects the other noise sources. On the other hand, it has been shown that the generalized Gaussian distribution (GGD) is more suitable in practice. For instance, it has been shown in literature that the interference and noise closely follow GGD in ultra wide-band communication systems [84]. It has also been discussed that the GGD better approximates impulsive atmospheric noises in several communication scenarios [85], [86]. Moreover, additive noise that arises in non-standard wireless media, such as under-ice noise and sea-surface agitation noise in underwater acoustics can also be modeled using GGD [87], [88]. Therefore, GGD exhibits a more appropriate fit to the noise data collected over a varied range of physical channel conditions [89], [90]. The well-known Gaussian distribution and the Laplacian distribution are special cases of GGD. To the best of our knowledge, performance study of RSMA in the presence of additive white generalize Gaussian noise (AWGGN) for DL-based signal detection in RSMA has not been considered in the literature earlier.

C. CONTRIBUTIONS
In this work, we propose the DL-based receiver studied in [58] to overcome to detrimental effects of SIC in RSMA systems. The considered DL-based receiver is designed to decode the common and private streams at the receiver in a single shot, without the need for explicitly estimating the channel. This receiver employs a DNN which jointly performs both the channel estimation and signal detection. Further, we consider both downlink and uplink communication, with multiple antennas on the transmitter and receiver. Moreover, we consider coded transmission over uplink and downlink RSMA and in particular, we consider the orthogonal frequency division multiplexing (OFDM) scheme. Additionally, we consider the noise at receiver to be AWGGN. In the presence of Gaussian noise, we show that the proposed DL-based receiver outperforms the conventional SIC receivers, which are based on minimum mean-squared error (MMSE) and least squares (LS) criteria. In the presence of AWGGN, we show that the proposed detector achieves the optimal performance of the maximum likelihood (ML) receiver. Our contributions can be summarized as follows.
r We propose a LSTM-based DNN for signal detection in downlink and uplink RSMA for decoding both the common and private streams. With this, we establish the utility of the DL-based detector as an alternative to the SIC decoder.
r We evaluate the performance of RSMA in the presence of AWGGN over a Nakagami-m fading channel in an OFDM setup. r A detailed performance comparison of the proposed DL technique in terms of the symbol error rate (SER) is carried out with other standard decoders such as the ML, MMSE and LS detectors. Under AWGN -which is a special case of AWGGN -we show that the DL detector outperforms the MMSE and LS decoders, and attains the optimal performance of the ML detector.
r The SER performance of the proposed DL technique is compared with that of the optimal ML technique in the presence of AWGGN. The effect of the GGD parameter and the hyperparameters of the DL architecture on the SER performance are discussed.

D. ORGANIZATION
The remainder of this paper is structured as follows. We provide a background on the downlink and uplink signal detection in RSMA in Sections II-A and II-B respectively, and discuss the generalized Gaussian noise model in Section II-C. In Section III, we present the architecture and the working principle of the proposed DL-based detector. In Section IV, we present the SER performance comparison of the proposed detector with other standard detectors, and discuss the effect of the noise parameter and DL hyperparameters on the SER performance. Concluding remarks are provided in Section V. The notations used in this paper are listed in Table 1.

II. SYSTEM MODEL
Consider a BS equipped with N BS antennas which communicates with Q users with N MS antennas each, using the rate-splitting multiple access (RSMA) technique, over a Nakagami-m fading channel in the presence of additive white generalized Gaussian noise (AWGGN), as depicted in Fig. 1. We consider the design on the uplink and the downlink channels separately, which will be elaborated later. We consider OFDM as the underlying multi-carrier modulation technique, which uses N subcarriers. Let the Q users be denoted as UE-1, UE-2, . . . , UE-Q. Each of these users use common frequency resources to communicate with the BS over uplink and downlink, which is typical to the OFDM transmission. The RSMA encoding technique is employed on each of the OFDM subcarriers, which is further modulated and transmitted across the channel. In the following subsections, we discuss the uplink and downlink communication designs for a 1-layer RSMA system.

A. DOWNLINK RSMA
The system model for the RSMA system in downlink is as depicted in Fig. 2 to as the common message and M d pq is referred to as the private message of user q = {1, . . . , Q}. We generate the data for common stream, s d c by encoding the common parts of each user using a standard codebook. 1 The remainder of the message, that is the private part of the message of each user, is separately encoded into independent private streams of data, which is mathematically represented as M d pq of UE-q being encoded as s d pq . As a consequence, the resulting Q + 1 encoded signal streams, denoted by s d c , s d p1 , s d p2 , . . . , s d pQ , need to be transmitted. Therefore, the encoded signal streams are precoded linearly by the usage of a precoder matrix of size N t × (Q + 1) given by P = p c p 1 · · · p Q , where p c ∈ C N t ×1 represents the precoder for the common stream and p q ∈ C N t ×1 denotes the precoder for the private stream of UE-q, q = 1, . . . , Q. The design of precoders is done according to the method described in [9]. Let x d (k) ∈ C N t ×1 be the signal transmitted by the BS over the k th OFDM subcarrier, which is given by At the q th user, the received signal red at each user given by y d q (k) ∈ C N r ×1 can be written as where H q (k) ∈ C N r ×N t is the fading channel between the BS and q th user over the k th subcarrier, which is assumed to be a Nakagami-m random variable independent and identically distributed (i.i.d.) across users and time, and n q (k)∈ C N r ×1 is the AWGGN vector with i.i.d. elements, which have zero mean and variance σ 2 q . More details on the AWGGN noise model is provided in Section II-C. Typically, at each user-equipment, successive interference cancellation (SIC) technique is employed to decode the common message and its private messages, and also to alleviate the interference.

B. UPLINK RSMA
The system model for the considered RSMA system in uplink is as shown in Fig. 3. In RSMA uplink transmission, we assume that there is a synchronized communication amongst the users and the BS. Each user divides its intended messages into two parts, {M u cq , M u pq }, q = 1, . . . , Q. 2 These messages are independently encoded into separate data streams, the common stream s u cq and the private stream s u pq . Even in this case, we consider the users to have same common messages, which 1 In this work, it is assumed that all users have same common message, in both uplink and downlink. This assumption holds in several applications, e.g., VANETs [55], [91], cryptography [92], [93], antenna selection in MIMO [94], [95], and in random coding channels [96], [97], [98]. 2 In uplink RSMA, each part of the message can be treated as a virtual user [6]. We use the terms common and private for ease of understanding and comparing with downlink RSMA. has applications in VANETs [55], [91] and cryptography [92], [93]. Different power allocation factors are allotted to each of the streams and superposed together to be transmitted to the BS. Let x u q (k) ∈ C N t ×1 be the signal transmitted by each user over the k th OFDM subcarrier, which is given by which is added at the BS synchronously. Therefore, the signal which is received at the BS represented by y u (k) ∈ C N r ×1 is given by where H q (k) is the channel attenuation coefficient between the BS and q th user over the k th subcarrier, which is modeled by i.i.d. Nakagami-m random variables, and n(k)∈ C N r ×1 is the AWGGN vector with i.i.d. elements, which have zero mean and variance σ 2 . From the received signal y u (k), the BS is required to extract 2QN MS data streams, which is usually performed by employing SIC. The aforementioned signal detection scheme based on the SIC technique, in case of both downlink and uplink, is not only complex, but also time-consuming. As mentioned earlier, another major drawback of SIC is the error propagation. To overcome these disadvantages, we propose an alternate signal detection scheme that makes use of the deep learning (DL) framework, which is detailed next in Section III. The proposed DL block replaces the traditional SIC block and can decode all the data streams together-at-once, in both uplink and downlink RSMA transmission.

C. NOISE MODEL
Consider the equations (2) and (4), where n q and n represent the AWGGN added at the receiver. The entries of the noise vector are assumed to be independent and identically distributed, each having a PDF given by [89] where (.) is the gamma function, β ∈ R + denotes the shaping parameter and = 0 /σ 2 q represents the noise power normalization coefficient, where 0 = (3/β )/ (1/β ). It is important to note that some of the commonly used noise models are special cases of the AWGGN. To give an example, when β = 2, the PDF in (5) reduces to the PDF of the Gaussian noise [99], and when β = 1, we can obtain the PDF of the Laplacian noise [100]. It has been reported in literature that for any given noise variance, Gaussian noise is worst additive noise that can be considered for wireless channels [101]. Thus, for practical reasons, we assume β to be less than or equal to 2, i.e., 0 < β ≤ 2.
It should also be noted that as the value of β increases, the tail of the GGD decreases faster, which implies that the noise level is lower. Consequently, as as the value of β decreases, the impact of GGD on the received signal becomes more. For a receiver that performs the maximum likelihood detection, the transmitted common and private streams can be decoded according to [84] as where is the set of signal constellations for the underlying modulated symbols.

III. DEEP LEARNING-BASED SIGNAL DETECTION
In this section, we present the details on the architecture of the proposed DL-based signal detection block. We transmit the RSMA signal as multi-carrier modulated packets, specifically OFDM packets, where each packet consists of D = 3 symbols, which comprises of two pilot symbols and one data symbol. 3 The pilot symbols considered in the packet are considered to be fixed sequences. 4 Moreover, each of the three symbols of the packet are digitally modulated. We further assume that there are N OFDM subcarriers, and consider the signal to be transmitted over an arbitrary k th OFDM subcarrier, k = 1, . . . , N. We then perform OFDM operations which include computing inverse discrete Fourier transform (IDFT) and appending cyclic prefix (CP) as the guard interval. To effectively alleviate the inter-symbol interference (ISI), the length of the channel impulse response should be greater than the length of the CP. At the receiver, after OFDM demodulation is performed, the demodulated signal forms the feature vector and is fed to a long short-term memory (LSTM)-based deep neural network model. The length of this feature vector, considering samples from both real and imaginary parts, is equal to 2DNN t N r . Recall that N r = N MS in case of downlink and N r = N BS in case of uplink. Note that, for simplicity, we henceforth consider Q = 2 users with the same common message, although it can be easily extended to an arbitrary   [58]. The details on the layers incorporated in deep neural network (DNN) architecture is depicted in Fig. 4. The DNN considered has five layers. The input layer is of the dimension of the feature vector. The hidden layers comprises of an LSTM layer, a fully connected layer and a softmax layer, whose dimensions are given in Table 2. It is worth noting that the LSTM layer, which is a type of recurrent neural network (RNN), is the main component of our DNN block. The LSTM layer is used to exploit data-time dependencies, and is generally used for classification of sequential data [102]. We build the LSTM layer by considering subcarriers as time steps. The softmax layer makes use of a softmax function, which is used as activation function to the input data and the outputs obtained are values between 0 and 1. We use a classification layer for the output layer, with the number of nodes equal to the number of label classes. In order to decode the information from multiple antennas in one-go, the outputs obtained are formed into groups, where each group has the information corresponding to a particular antenna from which the symbols were transmitted from. The number of groups is equal to the number of transmit antennas.
Considering QPSK modulation as the digital modulation scheme for the symbols in the OFDM packet, the downlink DNN block deployed at the users will have 4 2 Table 2. We first generate the training data samples for a considered channel and noise profile and then train the DNN in an offline mode where it learns the channel and noise attributes. Once the DNN is trained accurately, it is deployed at the receiver, where it maps the received symbol to a corresponding transmitted symbol in an online mode. By doing this, we effectively decode all the symbols in one shot.
In the case of downlink transmission, the proposed DNN block is deployed at each of the user terminals. The network is trained and then deployed at the user terminals to directly decode the intended user signal instead of adopting the conventional SIC method. The signal received at the BS, according to (2), is OFDM modulated and stored as a training sample for the model. The input feature vector Y is a collection of the real and imaginary parts of all the OFDM packets received at each user. Using this, the network is trained to decode the common message, s d c , and the respective user's private message s d pq , q = 1, 2. Similarly in the case of uplink transmission, a single DNN block is deployed at the BS which will decode the common stream and the private data streams of all the users. The training sample, now, will be generated from (4). We assume that the common messages of both users are equal, that is, s u c = s u c1 = s u c2 . The trained network will decode the three data streams, s u c , s u p1 and s u p2 . It is clear that the proposed model has successfully eliminated the need for SIC and can decode all data streams in one shot, with less complexity and overhead.

IV. RESULTS AND DISCUSSION
In this section, we evaluate the performance of the proposed DL-based receiver for the RSMA signal. We consider an OFDM signal with 64 subcarriers, and 3 symbols per OFDM packet. An OFDM packet consists of two pilot symbols and one data symbol. We consider the pilot symbols to be same for each user for simplicity. Note that this model also assumes the common messages of both the users to be equal, and hence there is one common stream and two private data streams to be decoded together. Further, we set N BS = N MS = 1, unless specified otherwise. The observed trends are similar for arbitrary N BS and N MS . Other parameters considered for the simulation are given in Table 3.
We use MATLAB with the deep learning toolbox for building and training our DNN block. To understand how DL effectively manages the interference and noise, we keep the channel profile static during the training (offline) and testing (online) processes. We construct our ML receiver on the assumption of perfect channel state information at the transmitter (CSIT) and the performance of the maximum likelihood receiver is considered as the benchmark with which we compare the performance of the proposed DL-based receiver.
As mentioned earlier, we evaluate the symbol error rate (SER) performance of RSMA system in both uplink and downlink transmissions under the assumption of AWGGN. The shaping parameter β has been varied from 0.5 to 2, for practical reasons mentioned in Section II-C. We first set the value of β = 2, which reduces the noise model to additive white Gaussian noise (AWGN), and evaluate the performance of our DL-based receiver against the traditional SIC-based receivers namely the optimal ML, LS and MMSE receivers. We provide these results under Section IV-A. Then, we consider the effect of the shaping parameter β and analyze how β affects the SER performance. These results are presented under Section IV-C. Further, we present the effect of the variation of hyperparameters of the DNN block, namely the batch size and learning rate, under the assumption of AWGN at the receiver, and study their impact on the SER performance in Sections IV-D and IV-E, respectively. The time taken for generating training samples, offline training and online testing for different hyperparameters -specifically, the learning rate (LR) and batch size (B) -are tabulated in Table 4. The simulations were carried out on a computer with Intel(R) Core(TM) i5 − 10300 CPU with 8 GB RAM and 1 TB Nvidia GeForce GTX 1650 Ti GPU. Furthermore, we compare the performance of the proposed detector with a multi-antenna model-based DL receiver design [79] in Section IV-F. Fig. 5(a) and (b) show the performance of the RSMA DLbased detector with downlink transmission in terms of SER for decoding the common stream and private data streams of UE-1, respectively. For the sake of brevity, we omit the results of decoding of the common stream and private data streams of UE-2, where a similar set of conclusions can be drawn. This performance is compared with ML, LS and MMSE receivers. The implementation of LS and MMSE estimations are based on [103]. It can be seen that DL-based RSMA detector outperforms both the SIC-based receivers built on LS and MMSE, and nearly attains the performance of ML receiver at all SNR values. The complexity of the proposed DL-based receiver is also significantly reduced at the user equipments, which expands the bandwidth of operations and applications for the users. A similar comparison is done for the uplink scenario, and the corresponding plots are shown in Fig. 5(c) and (d), where we can observe a similar trend. Fig. 6(a) and (b) show the performance of the DL-based detector for decoding common and private data streams of UE-1 for downlink and uplink transmissions, respectively. The trends in these figures indicate that the SER performance is significantly enhanced in both uplink and downlink scenarios, with an increase in the number of transmitting and receiving antennas. Such a trend is expected, since an increase in the number of antennas N BS or N MS results in an increase in diversity.

C. EFFECT OF PARAMETER β
To understand the effects of β on the SER performance, we carry out a performance analysis for various values of beta between 0.5 and 2. We compare the SER performance of the DL-based receiver with the considered benchmark, that is the optimal ML receiver. Other SIC receivers are omitted, since LS and MMSE receivers perform inferior compared to the ML receiver. Fig. 7(a) and (b) show the performance of the DLbased detector for decoding the common stream and private data streams of UE-1, respectively on downlink transmission. It is clear that as the value of β increases, its effect on the SER performance decreases. This is expected because with an increase in the value of β, the tail of the GGD becomes tighter. This trend is similar to that observed in the SIC-based detection performance for NOMA discussed in [104]. The   results corresponding to the case of uplink are provided in Fig. 7(c) and (d), where a similar trend is observed.

D. EFFECT OF BATCH SIZE
We consider different choices of batch sizes for both downlink and uplink transmissions, with a constant LR parameter of 0.01, and present the obtained results in Figs. 8 and 9, respectively. Note that since smaller batch sizes result in more epochs, the network takes longer duration to be trained. However, the network can be trained faster in case of larger batch sizes, but at the expense of accuracy. Also, note that a larger batch size converges slowly as compared to a smaller batch size [105]. Further, these trends can also be observed from the convergence times discussed in Table 5 respectively.

E. EFFECT OF LEARNING RATE
The variation in performance of the DL-based detector are obtained for different learning rates in downlink and uplink   transmissions, for a fixed batch size of 4000 for downlink and 16000 for uplink, which are plotted in Figs. 10 and 11, respectively. Note that lower LRs take a longer time to converge but at a higher accuracy, while higher LRs reach convergence quicker, but it results in a suboptimal performance. This trend is observed in Table 5. Along with batch size, the design of LR is crucial. Hence, the trade-off between the choices of batch size and learning rates constitutes an important design in the DL-based RSMA receiver.

F. COMPARISON WITH MBDL RECEIVER
More recently, the authors in [79] proposed a model-based DL (MBDL) receiver design for decoding RSMA signals in the presence of SIC. The MBDL unifies DL and SIC frameworks, where as SIC is completely eliminated in our design. The MBDL receiver is designed specifically for a downlink communication system, where a BS equipped with N t antennas communicates with the two users. For the same set of assumptions and channel, noise and system models, we use our DNN block for training the network and then test the SER performance. Performance comparison of the proposed DL block with the MBDL receiver is presented in Fig. 12, and the associated parameters are chosen from Fig. 8(a) in [79]. Here, we see that the performance of the MBDL receiver is slightly superior, since [79] considers an uncoded transmission for a single carrier system, as opposed to our receiver which considers a multi-carrier modulation setup. This trend is similar to the observation that a single carrier system performs better than a multi-carrier system as shown in Fig. 8 of [106].

V. CONCLUSION AND FUTURE WORK
We proposed a DL-based signal detection technique for downlink and uplink RSMA system over a Nakagami-m fading channel with AWGGN at the receiver. The SER performance of the considered DL detector is compared against the conventional LS, MMSE and optimal ML receivers. It was observed that the DL receiver outperforms LS and MMSE receivers, and attains the performance of the ML detector. Further, we studied to examine the effect of the shaping parameter on the SER performance and show that as β increases, its impact on SER decreases. Additionally, we also studied the tradeoff between batch size and learning rate of the DL architecture on the performance. Note that the architecture proposed in this work can be readily extended to other channel models, depending on the application. As a part of the future work, the following ideas can be explored. Design of a spectral-efficient DL architecture which reduces the pilot overhead is an important topic of interest. Extension of this work to a general case where users have different common messages is currently under investigation. The various LSTM models explored for NOMA can also be extended to RSMA, if feasible.