Disentangled self-attention neural network based on information sharing for click-through rate prediction

With the exponential growth of network resources, recommendation systems have become successful at combating information overload. In intelligent recommendation systems, the prediction of click-through rates (CTR) plays a crucial role. Most CTR models employ a parallel network architecture to successfully capture explicit and implicit feature interactions. However, the existing models ignore two aspects. One limitation observed in most models is that they focus only on the interaction of paired term features, with no emphasis on modeling unary terms. The second issue is that most models input characteristics indiscriminately into parallel networks, resulting in network input oversharing. We propose a disentangled self-attention neural network based on information sharing (DSAN) for CTR prediction to simulate complex feature interactions. Firstly, an embedding layer transforms high-dimensional sparse features into low-dimensional dense matrices. Then, the disentangled multi-head self-attention learns the relationship between different features and is fed into a parallel network architecture. Finally, we set up a shared interaction layer to solve the problem of insufficient information sharing in parallel networks. Results from experiments conducted on two real-world datasets demonstrate that our proposed method surpasses existing methods in predictive accuracy.


INTRODUCTION
In recent years, deep learning has been widely used in areas such as computer vision and natural language processing and has achieved remarkable popularity and application (Aldarmaki et al., 2022;Tong & Wu, 2022;Reddy, 1976).Deep neural networks may dynamically modify their internal weights and biases in response to changes in input data, enhancing their robustness and accuracy (Santarsiero, Gori & Alonzo, 2019).Deep learning becomes a valuable model for evaluating online user reaction rate issues like advertising click-through rates (CTR) based on the learning mentioned above capability.
CTR prediction plays a vital role in industrial online advertising and recommendation systems (Graepel et al., 2010;Lu, 2021), and its purpose is to determine whether to recommend the item to users based on the likelihood of users clicking on the item.In CTR prediction, feature interaction is a common technique that combines different features to form new features to capture the interaction and nonlinear relationships between features.Utilizing effective methods for encoding feature interactions is often essential for enhancing the predictive accuracy of click-through rate models (Gao et al., 2023).
The manual selection of feature combinations for early-stage feature interaction is a time-consuming process that demands substantial human and financial resources, significantly depleting valuable assets.To address this issue, Factorization Machines (FM) (Rendle, 2010) represented each feature using latent factor vectors, and the pairwise feature interactions were modeled by taking the inner product of these latent vectors.The FNN (Liu, 2023) employed DNNs to learn and represent the intricate relationships among features, enabling them to uncover complex patterns and dependencies.The product-based neural network (PNN) (Qu et al., 2016) was a feature interaction model that leveraged the concept of inner product to capture interactions between features.One area of concern for both FNN and PNN models is their emphasis on high-order feature interactions, often overlooking the importance of low-order interactions.In order to better model the interaction of high-order and low-order features, Google proposed the model of Wide & Deep (WDL) (Cheng et al., 2016) in 2016, which combined the linear model with DNNs to improve the model generalization ability and take into account the memory ability.Deep & Cross (DCN) (Wang et al., 2017) and DeepFM (Guo et al., 2017) not only overcame the problem of focusing only on high-order feature interaction but also required no manual feature engineering.
According to literature (Zhang et al., 2021), CTR prediction models were classified into two types by combining explicit and implicit features of network modeling, namely parallel network architecture and stacked network architecture.DESTINE (Xu et al., 2021) was a stacked network architecture that calculated high-order feature interactions by stacking multiple disentangled self-attention layers.However, the stacked network architecture is essentially a linear model, which may not capture the higher-order interaction relationships between more complex features, limiting the expressive power of the model.Most current models adopt a parallel network architecture, where one network focuses on explicit feature interactions and the other on implicit feature interactions.In a parallel network architecture, explicit and implicit interactions are only combined at the final layer, with no information sharing occurring at the intermediate layers.This limitation diminishes the strength of interaction signals between them (Chen et al., 2021).Moreover, most models input features indiscriminately into parallel networks, resulting in excessive sharing of network inputs.
We discover two issues with existing models in the preceding description.Firstly, modeling unary terms is not considered in feature interaction but emphasizes the interaction between features.Secondly, it ignores the defect of no interaction between the layers of the parallel network architecture.To address the first problem, we introduce a disentangled self-attention layer.This layer divides the self-attention mechanism into two parts: the paired term is used to model the specific interaction between two features, and the unary term is used to model the influence of one feature on other features (Xu et al., 2021;Yin et al., 2020).For the second problem, we propose a shared interaction layer to solve the problem of insufficient information sharing in the parallel network.We set up two modules in the shared interaction layer to enhance the interaction signals in parallel networks.One module distinguishes the feature distribution, and the other fuses the output features.Precisely, this article models the interaction between features with a single-layer disentangled self-attention mechanism.Different from the unary term in DESTINE, to better capture the difference and variation range between features, we map the unary term to different subspaces and obtain multi-head feature representations.The disentangled self-attention layer considers features from a global perspective and focuses on essential feature representations.Then, the obtained feature representation is fed into the shared interaction layer, which consists of two introduced modules.One is the decomposition module, and the other is the sharing module.The decomposition module is used to distinguish feature distribution in different networks by field control networks, and the sharing module can capture the layered interaction signals in parallel networks (Singh et al., 2022).
This article makes three main contributions, as follows: • We introduce a disentangled self-attention mechanism and define paired terms and unary terms.It allows us to think about feature representation from a global perspective and focus on the crucial features.
• This article introduces two modules in the shared interaction layer to improve the interaction signals between parallel networks.One module distinguishes the feature distribution, and the other fuses the parallel network's features.
• Extensive experiments were conducted on two datasets to demonstrate the superior accuracy and lower loss rate of the proposed method compared to existing prediction methods.

RELATED WORK
Improving click-through rate (CTR) prediction is a topic of ongoing research and interest among researchers and academics (Singh et al., 2022).The prediction accuracy constantly improves from the early linear regression (LR) and FM to the current DNN, DeepFM.
In this section, we primarily concentrate on developing the CTR prediction model and discuss the approach for feature interaction.It also briefly introduces knowledge related to the disentangled self-attention mechanism.

Click-through rate prediction
Predicting whether a user will click on the recommended item is an essential problem in recommendation systems (Sangaiah et al., 2023;Guo et al., 2022;Aljunid & Huchaiah, 2022).In CTR prediction models, commonly used algorithms include linear regression (LR), gradient boosted decision tree (GBDT), FM, FFM, DeepFM, WDL, deep Interest Network (DIN), deep interest evolution network (DIEN), and others.LR served as a linear model primarily designed for sparse feature processing, while GBDT was a tree-based model proficient in managing non-linear features.FM, FFM, and DeepFM were models based on factorization machines that could handle high-dimensional and sparse features.WDL, DIN, and DIEN combined linear models and deep learning models to handle both low-dimensional and high-dimensional features.To further model automated feature interaction learning, the HoAFM (Tao et al., 2020) established intersectional features that were expressive and informative by stacking multiple cross layers.The co-action network (CAN) (Cai et al., 2021) was an effective CTR prediction model, which believed that there was no information sharing among the feature combinations of previous models, and thus introduced a dynamic pluggable feature interactive learning Unit Co-Action Unit, which realized the expression of feature combination information.

Feature interaction
The efficacy of learning feature interactions has been demonstrated in the click-through rate prediction tasks.FM was proposed mainly to capture interactions between features through factorization.Subsequently, several FM variants, such as FFM (Juan et al., 2016), AFM (Xiao et al., 2017), FmFM (Sun et al., 2021) and FwFM (Pan et al., 2018) were proposed.In recent years, some researchers have modeled higher-order feature interactions.Parallel network architecture and stacked network architecture are two classic types of click-through rate prediction models.Figure 1 shows the classic models DeepFM and NFM in two network architectures.Stacked architecture models, such as NFM (He & Chua, 2017), DIN (Zhou et al., 2018), andDIEN (Zhou et al., 2019), are representative examples.In NFM, secondorder feature interactions were stacked upon deep neural networks to model higher-order features.DIN and DIEN extracted interest representations from historical behavior and utilized attention mechanisms to model the relationship between user interests and the target item.
The parallel network architecture model captures interaction signals from explicit and implicit features and then fuses the information at the output layer.Notable models in this category include DeepFM, DCN, DeepCrossing (Shan et al., 2016), xDeepFM (Lian et al., 2018), and AutoInt (Song et al., 2019).In the parallel architecture model's implicit feature part, feature extraction primarily relied on deep neural networks (DNNs).DeepFM utilized the factorization machine (FM) structure for learning in the explicit feature interaction component.DCN proposed a cross-network approach to capture the interactions between features.The AutoInt model efficiently captured the non-linear relationships between features by adaptively learning the interaction weights between each pair of features.

Disentangled self-attention mechanism
In natural language processing tasks, attention mechanisms are extensively employed, including machine translation, text summarization, question-answering systems (Choi et al., 2016;Yang et al., 2016).The attention mechanism generally assigns weights to each input item to reflect their importance in the target task (Ali, Zhu & Zakarya, 2021).Transformers (Vaswani et al., 2017) introduced a self-attention mechanism that enabled the model to focus on different positions in the input sequence simultaneously without The disentangled self-attention mechanism is a method for introducing decoupling based on the self-attention mechanism.It first appeared in the field of computer vision, and the two terms were used to capture the edge and the center of an image, respectively (Yin et al., 2020).The disentangled self-attention mechanism is divided into two parts: the paired term and the unary term.Paired terms are used to model a specific interaction between two features, and unary terms are used to model the effect of one feature on the other.DESTINE computed higher-order feature interactions by stacking multiple disentangled self-attention layers.It does not allocate the features of the unary term to distinct subspaces.In addition, we think that the stacked network architecture is essentially a linear model, which may not capture the higher-order interaction relationships between more complex features, limiting the expressive power of the model.

METHODS
In this article, we introduce paired and unary terms in the disentangled self-attention layer, enabling us to consider the feature representation from a global perspective and focus on essential features.To address the issue of inadequate information sharing in parallel networks, we introduce two modules within the shared interaction layer.Specifically, as shown in Fig. 2, the DSAN has the following four parts: the embedding layer, the disentangled self-attention layer, the shared interaction layer, and the output layer.Firstly, the embedding layer receives the input features and converts them into compact, low-dimensional embedding vectors.Secondly, the embedded features are fed into a disentangled self-attention layer to focus on the important feature representations.Thirdly, upon acquiring these significant representations of different features, they are channeled into the shared interaction layer to govern feature distribution.Finally, the feature representations are fed into the output layer for prediction.

Problem definition
The main objective of CTR prediction is to assist advertisers or e-commerce companies in estimating the likelihood of users clicking on recommended items.It helps select the most valuable items for recommendation in the recommendation system, thereby maximizing its effectiveness and commercial value.Let us assume that the entire dataset D = x 1 ,y 1 , x 2 ,y 2 ,..., x N ,y N consists of N examples, where each sample x i consists of M user and item feature fields, and its associated label y i ∈ {0,1} is the ground truth ith sample.CTR prediction is the probability of a user clicking on a recommendation using the formula ŷ = f (x i ) under the given feature vector x i .

Embedding layer
In vision or natural language processing, the input data is often images or text signals with spatial or temporal relevance.However, input features in the recommendation system are usually sparse, and there is no apparent spatiotemporal correlation.Because CTR predictions contain discrete categorical variables that cannot be directly used for numerical computations, feature embedding is essential to forecasting click-through rates.Suppose the entire dataset consists of N examples, each sample consisting of M user and item feature fields.Some of these features are categorical data, while others are numerical data.
The most often used approach for categorical data is feature embedding, which involves converting each sparse vector into a low-dimensional dense vector.To illustrate, if the one-hot vector of the ith field is designated as x i , the related embedding matrix is written as w i .The formulation of the e i can be represented as follows.
where x i is a one-hot vector and w i is the embedding matrix of the ith sparse feature.By applying a conversion process, the numerical feature x j can also be transformed into the same low-dimensional space.
e j = w j x j where w j is an embedding vector and x j is a scalar value.According to the above method, the embedding layer compresses a high-dimensional sparse vector into a low-dimensional dense vector.The equation for the embedding layer is expressed as: E = e 1 ;e 2 ;...e i ;...e j ;...e M where '';'' represents the matrix stacked operation, and M represents the number of feature fields.

Disentangled self-attention layer
Self-attention is a mechanism that enables the model to assign varying degrees of importance to different positions in the input sequence, facilitating accurate predictions.In traditional self-attention, a single attention head is used to compute the attention weights between all pairs of positions in the input sequence.In multi-headed self-attention, the input sequence is mapped to multiple attention subspaces separately, and independent self-attention computations are performed in each subspace.Each attention head can learn different attention patterns and dependencies to capture multiple levels of information in the input sequence.Building upon the principles of multi-head self-attention, this article introduces the disentangled multi-head self-attention layer, which is further divided into paired terms and unary terms.
For the input feature x i , it is transformed into a dense embedding vector e i by embedding search.After obtaining the low-dimensional representation of each feature, We use the dot product attention approach to simulate higher-order interactions between features.We define four different matrices w (h)  q , w (h) k1 , w (h) k2 and w (h) v , then multiply e i by each of them, and the four vectors are represented as follows: i are obtained by the four weights w (h) q , w (h) k1 , w (h) k2 and w v ∈ R d ×d respectively.d is the dimension of the filed embedding, and d is the dimension of the attention.
It has been demonstrated in previous visual learning tasks that the standard self-attention mechanism is detrimental to feature learning.The disentangled self-attention mechanism is used in this article, and paired and unary terms are disentangled utilizing activation functions and embedding matrices.For the paired terms, in order to eliminate the offset information between features and enhance the expressiveness of the model, we obtain the multi-head feature representation of Q (h)  i and K (h) i , and then subtract their average values.Similar to the ordinary self-attentive mechanism, the two feature representations above are multiplied together and then passed through the softmax activation function to obtain the vector of attentional weights P (h)  i for the paired terms.For the unary terms, we first map the features into different subspaces, from which multiple feature representations are learned to obtain the matrix K (h)  i .Then, the mean value i is calculated and multiplied by the matrix K (h)  i − µ (h) k1 to obtain the characteristic representations.Finally, the attentional weight vector U (h)   i is obtained by the softmax activation function.After we have the paired and unary terms, we add the two terms and dot them with V (h)  i .The calculation formula is as follows: where σ 4 (•) and σ 5 (•) are the softmax activation function, k1 e i take average of the Q (h) i and the K (h) i vectors, respectively, and M is the total number of features for users and items.Then, we connect all the attention heads with the following formula: Z = Head (1)  ;Head (2) ;...;Head (h) where h is the number of attention heads, and Z stacks up all the features after getting the attention head.w 1 means the weight matrix, and b 1 denotes the bias.
To preserve the information contained within the original embedded vector, a residual structure is incorporated into the network.Furthermore, normalization and ReLU activation function procedures are implemented before the output feature representation to increase the model's stability, convergence speed, and performance.
where w r ∈ R d ×d is a linear projection matrix to avoid dimension mismatch, ϕ(•) is the ReLU activation function.

Sharing interaction layer
There are two primary modules in the shared interaction layer: the decomposition module and the sharing module.The decomposition module distinguishes feature distribution in different networks by field control networks.The shared module establishes a link between the hierarchical attention and the deep network and captures the interaction signals between the two networks.These two modules are lightweight with minimal time and space complexity, and they can be easily applied to the CTR model of parallel network architecture.
Decomposition module.The existing CTR models provide all features equally across the two networks.The decomposition modules distinguishes features and place them in different networks in parallel architectures.Within the parallel network architecture, we employ Deep Neural Networks (DNNs) to perform implicit high-level modeling, complemented by hierarchical attention mechanisms for explicit modeling.In the aforementioned disentangled self-attention layer, we distinguish the importance of different features and generate the feature representation L o ∈ R M ×d .Next, we pass L o as input into the shared interaction layer.Then, we use the decomposition module to get DNN networks and hierarchical attention networks.
where g i and g i denote the gating weight for the ith field, and • denotes the Hadamard Product of two vectors.σ 2 and σ 3 are softmax activation function, w 2 and w 3 are learnable parameter, γ 2 and γ 3 are hyperparameter.
Lian et al. ( 2018) proves that each hidden layer of CrossNet is a scalar multiple of x 0 and interacts in a bit-wise manner, which may not capture some higher-order feature interaction patterns.In this study, the explicit feature interaction component employs hierarchical attention to build vector-wise level feature interactions, while implicit feature interaction is acquired through the fully connected layer.C l is the explicit high-order interaction part, and D l is the implicit high-order interaction part.To obtain the l + 1-th explicit high-order feature interactions C l+1 , we aggregate the lth layer named C l .The formula for attention aggregation is as follows: Sharing module.Traditional parallel network architectures independently handle explicit and implicit feature interactions, deferring information fusion to the final layer.Consequently, these traditional methods do not effectively reflect the interrelationships between parallel networks, thereby attenuating the interaction signals between explicit and implicit feature interactions.To tackle this challenge, we introduce three methods to capture the signals.We have also evaluated and compared the performance of the three methods in the experimental section.1.We assume that C l and D l are two feature representations in a parallel network architecture.The first feature fusion, denoted as F HP , can be expressed as where • is the Hadamard Product, and it takes their element-wise product.2. It may not be possible to effectively model sparse feature interaction using the Hadamard Product or inner product.Therefore, we combine the inner product and the Hadamard Product to learn feature interaction.We named the second feature fusion F IH .This interaction function is denoted as , where a i l are the shared parameters in the lth layer and • denotes the regular inner product.3. The third approach of feature fusion concatenates two vectors.In order to keep the vector dimension of output M × d, we design a feedforward layer with an activation function.The formula can be expressed as , where w k and b k are the weight and bias parameters for the the lth layer, respectively.Recurrently applying formulas in the shared interaction layer can generate an lth layer vector representation.The two vector representations obtained at the lth layer are C l ∈ R M ×d and D l ∈ R M ×d respectively.They are combined through addition before undergoing dimensional transformation.Finally, the final prediction result is obtained through a layer of linear functions.
where w c and b c are parameters of weight and bias, respectively, and ŷ ∈ (0,1) is the predicted label, and σ is the activation function.

Output layer
The loss function is utilized in the click-through rate prediction task to calculate the difference between the model forecast and the actual click-through rate situation.The model output is a probability value between 0 and 1, and the real value can only be 0 or 1. Binary cross-entropy measures the loss by calculating the difference between the predicted values and the actual values.The loss function seeks to minimize cross-entropy in the training process so that the predicted result can accurately match the actual click situation.Our loss function is J , which is defined as follows: where y i and ŷi represent the real value and predicted value, respectively, and y i is the true label of the ith sample.We use a gradient descent algorithm to update model weights.

EXPERIMENT AND ANALYSIS
In this section, we conduct experiments to verify the validity of the proposed DSAN model.Firstly, we describe the experimental settings, which include datasets, baseline models, etc. Subsequently, we compare the proposed model with the baseline model and provide a detailed explanation of the reasons behind the superior performance of our model.Finally, we perform ablation experiments to verify that each component is practical.

Experimental settings
Datasets.We evaluate DSAN on two publicly available datasets, namely Criteo (https://www.kaggle.com/c/criteo-display-ad-challenge)and Avazu (https://www.kaggle.com/c/avazu-ctr-prediction).The Criteo dataset is a widely used public data set for predicting click-through rates of Internet ads.The dataset contains data on billions of advertisements displayed and clicks on anonymous websites monthly.Each data point comprises 13 digital and 26 category characteristics marked as clicked or unclicked.Avazu contains 10-day mobile ad click logs with 23 categories, including domains, types, and others.We remove the id field from the sample because it is useless in click-through rate prediction.We convert the timestamp field into hour, weekday, and is_weekend.The category characteristics of the two datasets are hashed to protect user privacy.The size and characteristics of the dataset are shown in Table 1.
Data preprocessing.We use the same dataset processing method as in Zhu et al. (2021), dividing it into training, validation, and test sets in the ratio of 8:1:1.We treat the categorical features in the two datasets differently.For the Criteo dataset, we replace the features with less than or equal to ten with ''OOV'' and set the embedding dimension to 16, and then for the rare features with less than or equal to two, the embedding size is set to 40.For the Avazu dataset, we replace the infrequent features that occur less than three times with ''OOV'' and set the embedding dimension to 16.For the features that occur less than two times, we set the embedding size to 40.Finally, we output the preprocessed file in HDF5 format.
Experimental details.All models are implemented using Pytorch.All the experiments are conducted on an NVIDIA RTX 3090 GPU with 24GB of memory.The batch size for all models in both data sets is 1024.The default learning rate for all models is 0.001.We have discovered that dropout and regularization weights influence the model's performance.Therefore, we carefully adjust them between 0 and 1.In addition to this, we also fine-tuned the number of attention heads and the number of network layers of the models to get the best results.We execute an early stop strategy, stopping training when two consecutive Logloss metrics on the verification set increase.We investigate the performance of the models when the embedding dimensions are 16 and 40.
Evaluation metrics.In the experiment, we evaluate the performance of all methods using two famous metrics.These two metrics are widely used in CTR prediction evaluations: AUC and Logloss.AUC: We present a quantitative evaluation of the model's performance using the ROC curve.The evaluation takes into account the sorting order of positive and negative instances.We randomly select positive and negative samples and use a trained classifier to predict both samples.The AUC measures the model's performance, with higher values indicating better performance.Suppose the data set has Q positive and G negative samples.The AUC formula is defined as follows: 1,P positive > P negative 0.5,P positive = P negative 0,P positive < P negative Logloss: Logloss is a commonly used metric for evaluating the performance of classification models and is particularly suitable for binary classification problems.It calculates the loss based on the difference between the probability value predicted by the model and the actual label.It is defined as shown in the output layer J .
Baseline models.We compare the proposed method with state-of-the-art approaches designed explicitly for CTR tasks.Here is an introduction to the benchmark model.
• LR (Richardson, Dominowska & Ragno, 2007): It can only learn the first-order feature interaction, which cannot represent the interaction between features.
• FM (Rendle, 2010): It can effectively handle high-dimensional sparse features by modeling the interaction between features and having good generalization capabilities.But it can't simulate higher-order feature interactions.
• Wide & Deep (WDL) (Cheng et al., 2016): It is a hybrid architecture that combines the power of deep neural networks for learning intricate patterns with the memorization capability of a comprehensive linear model, enabling accurate click-through rate prediction by capturing both generalization and specific feature interactions.
• NFM (He & Chua, 2017): It is a neural network-based extension of Factorization Machines that leverages deep learning techniques to capture feature interactions and improve click-through rate prediction in various applications.
• Deep & Cross (DCN) (Wang et al., 2017): It is a neural network architecture incorporating cross-network operations to capture high-order feature interactions (He et al., 2016), enabling accurate click-through rate prediction by balancing depth-wise representation learning and explicit feature interactions.
• DeepFM (Guo et al., 2017): It is a hybrid approach combining deep neural networks and factorization machines, leveraging their complementary strengths to capture high-order feature interactions and low-rank representations, enabling accurate click-through rate prediction in large-scale recommendation systems.
• xDeepFM (Lian et al., 2018): It is an advanced deep learning architecture integrating cross-network operations and deep neural networks, effectively capturing intricate feature interactions and hierarchical representations.
• FiBiNet (Huang, Zhang & Zhang, 2019): It introduces two modules: Bililinear Feature Interaction and SENet.SENet is a powerful mechanism that selectively recalibrates feature representations by learning channel-wise attention weights.The bilinearinteraction layer performs element-wise product and linear transformation operations to capture intricate feature interactions.
• InterHAt (Li et al., 2020): It incorporates hierarchical self-attention mechanisms to capture feature interactions at different levels, improving click-through rate prediction by effectively modeling the importance and dependencies among features in recommendation systems.
• DESTINE (Xu et al., 2021): It stacks multiple disentangled self-attention mechanisms to model the interaction of higher-order features, which decouples the unary feature importance calculation from the second-order feature interactions.
• DeepLight (Deng et al., 2021): It introduces a parallel network architecture that combines DNN and FwFM and conducts analysis compared to DeepFM and xDeepFM.This approach aims to address the challenges of increased service latency and high memory usage when delivering real-time services in production.
• CowClip (Zheng et al., 2022): It develops the adaptive column-wise clipping to address the model's training speed, reducing the training time from 12 h to 10 min on a single V100 GPU.
• FinalMLP (Mao et al., 2023): It proposes seamlessly integrating feature gating and interaction aggregation layers into an upgraded dual-stream MLP model.In other words, by combining these two MLPs, it can achieve improved performance.

Performance comparison
In this section, we report the model's performance on two datasets.The DSAN and the best baseline are emphasized in bold and underlined formats.The ''Previously Reported'' column shows the best results for both datasets that we found in our existing work.The ''#Params'' indicates the number of parameters used in each model.Table 2 presents the performance of models, and we can draw the following conclusions: • In the ''Previously Reported'' column, the performance of InterHAt is inferior to LR on both datasets.It could be attributed to differences in how the models preprocess the datasets and the splitting ratios.This article evaluates all models using the same evaluation protocol and preprocessing methods for result comparability.In addition, we perform a paired t -test to verify the statistical significance of the relative improvement of DSAN.
• The performance obtained on Avazu datasets is essentially the same or even higher than the reported results.For example, InterHAt and DCN receive better results on Avazu datasets.The experimental results show that LR and FM shallow models perform worse literature (Zhang et al., 2022), it points out that feature sparsity is the cause of one epoch.
Improving model performance by improving data sparsity may be a worthwhile research topic.
• The performance of the Criteo dataset is generally in line with the reported results, but in some aspects, it is slightly lower than expected.The lack of detailed hyperparameter information provided in the article resulted in us not obtaining the best combination of hyperparameter tuning and feature engineering.However, InterHAt and NFM models demonstrate superior performance on Criteo datasets, underscoring the effectiveness of our approach to data preprocessing and hyperparameter optimization.We can also observe that the Criteo dataset has better feature representation at an embedding dimension of 16, while the Avazu dataset obtains better results at higher embedding dimensions.This difference may stem from the complexity and characteristics of the datasets themselves, leading to variations in the optimal choice of embedding size.
• Tuning model parameters is one of the crucial steps in deep learning.We adjust the models regarding embedding size, number of attention heads, and network layers.We tune the embedding size to { 16, 24, 32, 40, 48 } for both datasets to investigate the effect of embedding dimensions on the model.The number of network layers in deep learning is tuned between 0 and 4. To further illustrate the need for tuning, Table 3 shows the results of the three benchmark models before and after tuning.The ''Reported'' reflects the findings of the existing research; ''Rerun'' denotes the outcome pre-adjustment; and ''Retuned'' signifies the outcome post-adjustment.We can observe that while DeepLight and FinalMLP didn't achieve the reported performance on the Criteo dataset, they slightly surpassed the previous performance on the Avazu dataset.This variance might arise from the distinct characteristics and complex relationships among features in these datasets.It indicates that the characteristics of the data influence a model's performance across different datasets, and it requires specific adjustments tailored to each dataset to achieve better performance.

Performance analysis
In this section, we examine the performance of the model.We compare the proposed disentangled self-attention layer with the traditional self-attention approach and investigate the performance of three variants of the sharing module.

Disentangled self-attention layer performance analysis
The disentangled self-attention mechanisms are a method that introduces decoupling based on self-attention mechanisms.We devise two comparative approaches to assess the efficacy of the disentangled self-attention mechanism proposed in this article, and we compare it with the disentangled self-attention layer in DESTINE.Method one, DSAN alt SA, combines the common self-attention mechanism with the shared interaction layer.The second method, DSAN alt DS, combines the disentangled self-attention mechanism proposed in Xu et al. (2021) with the shared interaction layer.According to the experimental results in Fig. 3, our method DSAN outperforms the two methods above on both datasets.Compared with DSAN alt SA, the AUC of this  article improved by 0.9% and 0.6% on the Criteo and Avazu datasets, respectively.The results show that the disentangled self-attention mechanism proposed in this article is superior to the common self-attention mechanism.Compared with DSAN alt DS, DSAN improved the AUC by 0.6% and 0.3% on the two datasets, Criteo and Avazu, respectively.This result indicates differences in the design of disentangled self-attention layers between DESTINE and this article.In this article, we successfully capture the differences and variation range between the unary term characteristics of the disentangled self-attention layer, thus improving the model's performance.

Different variant of sharing module
We investigate the effects of three different variants in the shared module.To capture the signal between different networks of the parallel architecture, we express the Hadamard Product of two vectors as DSAN-HP, which is also the way used in the article, and thus as DSAN.The inner product and Hadamard Product feature fusion are denoted as DSAN-IH, and the way the two vectors are connected and passed through the feed-forward neural network is denoted as DSAN-CN.As shown in Fig. 4, DSAN performs the best.It is due to the Hadamard Product being in the same position as the elements and not involving any weights or coefficients.

Ablation study
We conduct experiments to verify the efficacy of individual components within the DSAN model and understand their respective significance.Each experiment entails the removal of a single component while keeping the remaining components unchanged, ensuring a focused evaluation of their relative importance.The DSAN w/o DA denotes the removal of disentangled multi-head self-attention, implying only a shared interaction layer in the model.DSAN w/o PT is the removal of paired terms in disentangled multi-head self-attention, and DSAN w/o UT is the removal of unary terms in disentangled multi-head self-attention.DSAN w/o SI indicates the removal of the part of the two modules that share the interaction layer, which means that only the DNN network and the attention aggregation exist in the parallel structure.
Based on Table 4, it is evident that eliminating any component from DSAN results in a decline in model performance.Compared to DSAN w/o DA, DSAN improves the AUC values by 1.2% and 0.9% on both the Criteo and Avazu datasets, respectively.This progress demonstrates the efficacy of the introduced disentangled self-attention layer in bolstering the model's accuracy.To verify the effectiveness of the unary and pairwise terms presented in the disentangled self-attention layer, we design DSAN w/o PT and DSAN w/o UT.Compared to the DSAN w/o SI, the DSAN model exhibited 1.2% and 0.6% improvements in AUC values on the Criteo and Avazu datasets, respectively.This demonstrates that integrating decomposition and sharing modules into a parallel network architecture is effective.

CONCLUSION
Click-through rate prediction plays an important role in recommender systems.However, the existing models ignore two aspects.One limitation observed in most models is that they focus only on the interaction of paired term features, with no emphasis on modeling unary terms.The second issue is that most models input characteristics indiscriminately DSAN uses the disentangled self-attention layer to learn the ambiguity of feature interactions and then inputs into the shared interaction layer.We designed two modules in the shared interaction layer.One module distinguishes the feature distribution, and the other fuses the parallel network's features.
Our proposed model achieves better off-line AUC and Logloss than other models.How to construct an interpretable click-through rate prediction network will be investigated in future work.Furthermore, we observed a one-epoch phenomenon related to click-through rate prediction during the experiment.Addressing this in future research could enhance the performance of click-through rate prediction.

Figure 1
Figure 1 Parallel architectures based on CTR prediction.

Figure 2
Figure 2 DSAN architecture based on information sharing.The layer with dashed lines at the bottom is the disentangled self-attention layer, while the layer above it, also enclosed in a dashed box, is the shared interaction layer.Full-size DOI: 10.7717/peerjcs.1764/fig-2 j ∈ {1,...,M } where w j l and b j l are the weight and bias parameters on the jth field in the lth layer.

Table 4 Ablation experiments.
, resulting in network input oversharing.In this work, we propose the DSAN model for click-through rate prediction.