Improved DeepFM Recommendation Algorithm Incorporating Deep Feature Extraction

Abstract

In recent years, deep learning has been applied to the field of recommendation, which can learn complex user interaction features and make better recommendations. However, deep learning only focuses on the interaction of high-order features and neglects the low-order features. The DeepFM model combines the linear FM (Factorization Machines) model and the deep DNN (Deep Neural Network) model to realize the interactive learning of low-order and high-order features, but it does not take into account that user interests will change dynamically with time. When the data sparsity is high, it cannot be effectively recommended. Based on this, an improved DeepFM recommendation algorithm that combines depth feature extraction was proposed, named fDeepFM. Firstly, the word features are transformed into low-dimensional dense vectors through the Embedding layer. Then Doc2Vec is combined to mine item features with context, and the two are stitched together as the input to the FM model and DNN model. Subsequently, user features are input to the GRU (Gated Cyclic Unit) model according to different cycles to mine user features. Finally, the results of the FM model, DNN model, and GRU model are combined by linear stitching as the overall output of the fDeepFM model. Experiments were carried out on Movielens-20M and Amazon data sets. The experimental results showed that MAE, RMSE, F1-score, and AUC on the Movielens-20M data set were optimized by 1.69%, 2.4%, 1.67%, and 2.28%, respectively; On the Amazon dataset, MAE, RMSE, F1-score, and AUC are optimized by 3.2%, 3.86%, 1.63%, and 2.2% respectively compared with DeepFM.

1. Introduction

With the development of the Internet, there are various service platforms on the network, such as shopping, music, news, and short videos. These platforms urgently need to find a way to help users quickly find items suitable for themselves. A recommendation system was born based on this situation, which can find the information that users are interested in from among the huge amount of data when their needs are not clear. The recommendation system solves the data explosion problem, and now personalized recommendation has become the trend of modern society.

In recommendation systems, the CTR (Click-Through Rate) prediction task is used to estimate the probability of users clicking on items, so it is important that recommendation algorithms improve the prediction accuracy of CTR. Rendle first proposed the FM model [1], a machine learning algorithm based on matrix decomposition to solve the problem of feature combination in a coefficient matrix. Juan added the field concept to the FM model and proposed the FFM (Field-aware Factorization Machines) model [2]. However, the weights of each cross feature in FFM are the same, and based on this, Xiao et al. introduced an attention mechanism to assign different weights to different cross features and proposed the AFM model [3]. Pan et al. proposed the FwFm model [4], a model similar to the FFM model in that it considers the interaction weights between different fields, but it assigns a uniform weight. These two models are all second-order feature intersection models, which are prone to the problem of combinatorial explosion. Hence He et al. added deep neural networks to the FM model to improve the expressiveness of the model and proposed the NFM model [5]. Tao et al. proposed the HoAFN model [6] because the AFM model cannot extract higher-order features. However, all the above methods are linear improvements based on the FM model, and the model results are still unsatisfactory. The PNN model [7] was proposed by the team at Shanghai Jiao Tong University at the ICDM conference. It can obtain the cross information between features in a more targeted way. Cheng et al. proposed the classical Wide & Deep model [8], which combines linear models with deep learning, but Wide partially relies on manual feature screening and cannot extract high-order and low-order features simultaneously. Guo et al. proposed the DeepFM model [9] based on the Wide & Deep model, which combines high-order and low-order features while alleviating the problem of insufficient low-order feature representation and easy overfitting of high-order features. It has attracted people’s attention. Wang et al. combined the advantages and features of the DeepFM model based on the FwFm model, considered the changes in the external environment and internal perception, and constructed the FG_DRFwFm model for learning the interaction of high- and low-order features of multiple features. Experiments showed better recommendation results [10]. Wang et al. introduced the attention mechanism into the DeepFM model and proposed the DIFMN model, which can better consider the diversity of user interests [11]. Inspired by the DeepFM model, Chen et al. proposed a DCFM model by combining Factorization Machine crossover networks and deep neural networks, combining user and commodity characteristics in many aspects [12]. Yu et al. improved the parameters of the DeepFM model and combined the sensors of loMT to form a healthcare system for disease prediction, which performed better in terms of accuracy and time [13].

The time factor is often an important factor in influencing the user’s choice, and the items preferred in the same period are likely to be the same type of items. The effective extraction of the item and user features has a great impact on the recommendation effect. However, the above algorithms and models do not effectively extract the features of items and do not take into account the change in users’ interests over time. To address these problems, this paper proposes an improved DeepFM recommendation algorithm incorporating deep feature extraction. The proposed method has the following contributions:

• Word features are trained by the Embedding layer, text description features are processed by Doc2Vec to get feature vectors containing contextual semantics, and the two are stitched to item features, making the extracted features more detailed;
• Introducing the GRU model to learn users’ changing interests using its supplementary memory mechanism;
• Training FM models, DNN models, and GRU models in parallel. It not only learns the combination of high and low-order features but also considers the changing pattern of user interest over time.

This paper is structured as follows. Section 2 introduces the model of fDeepFM. Section 3 makes experiments to verify the effect of the fDeepFM model. Section 4 concludes the paper and discusses future work.

2. Improved DeepFM Recommendation Algorithm Incorporating Deep Feature Extraction

For the recommendation algorithm, the effective extraction of item features is an important factor affecting the recommendation effect. The user’s interest changes with time. Recommending what the user liked in the past often makes the recommendation meaningless, so the recommendation algorithm should also consider timeliness. However, data sparsity is a common problem for recommendation datasets. In general, the word vector is sparse after One-hot coding, especially for textual item descriptions, and One-hot only exacerbates the sparsity. Therefore, a new model is designed in this paper, named fDeepFM.

The algorithm model is shown in Figure 1. The model is divided into three parts, the FM model, the DNN model, and the GRU model. Firstly, item features are divided into word features and text description features. The word features are trained through the Embedding layer to alleviate the sparsity of data, the text description features are combined with Doc2Vec [14] to process the item description information, and the two are spliced to form new vector representation of the item as input to the FM model and DNN model. Secondly, the GRU model can analyze the characteristics of the long-term sequence to predict the user behavior at the next moment, so the user’s historical behavior records are trained according to different cycles through GRU [15] model to get the user’s feature representation. Finally, the three parts of the model are trained in parallel, a fully connected layer is added to the output features, and the final results are output after the activation function is applied. The model can improve the effect of feature learning and learn the feature changes of the user’s historical interest.

2.1. Distributed Vector Representation Incorporating Contextual Semantics

Data sets within the domain of recommendation algorithms are often sparse, and sparse data can lead to ineffective recommendations. Therefore, separate feature extraction for the text of the item description class and the words of the item name class can extract features in more detail. Then the two are stitched together to form a feature representation of the item.

The vectorized representation of item features used to use a bag-of-words model, such as One-hot coding, TF-IDF, and so on. But this model does not consider lexical and sequential issues. It considers each word independent, which leads to the loss of word order features and ignores semantic features. However, the word vector model is a model that considers word position relations, which can make up for the shortcomings of the bag-of-words model and better characterize words.

Doc2Vec is an unsupervised deep learning algorithm that can be trained without fixed sentence length and can learn from long texts to get fixed-length feature representations. And it takes into account the word order relationship of the context during training, so the word vector obtained from training will contain a semantic understanding of word order.

This algorithm uses jieba splitting to split the item text description $i_c$, $i_c=i_c^1+i_c^2+\cdots +i_c^m$, $m$ is the number of splitting words, then inputs to the PV-DM model of Doc2Vec for training, as shown in Figure 2.

The algorithm obtains the document vectors by training a neural network that predicts the probability distribution of words in a paragraph. It gives randomly sampled words from the passage and generates word vectors $W$ for each word and document vectors $D$ for each document, then trains the weights of the Softmax hidden layer and finally trains new samples of content descriptions using gradient descent until convergence. Finally, a low-dimensional vector representation $i_d$ of item features is obtained.

For phrases such as item names, $i_w^j\left(1\le j\le k\right)$ are trained with word embeddings. The vectors after One-hot are sparse and large dimensional, so the Embedding layer needs to be trained to transform them into dense vectors. The concept of field in FFM is introduced here, where features of the same nature are grouped into the same field, the dimensionality of the input can be different for different fields, but the output should be the same to obtain the item feature representation $i_w$, $i_w=i_w^1+i_w^2+\cdots i_w^k$, $k$ is the number of features. The final item feature vector $i$ obtained by stitching $i_d$ and $i_w$.

2.2. DeepFM Model

The DeepFM model consists of two parallel neural networks, an optimization of the Wide & Deep model. It inherits the advantages of the Wide & Deep model and replaces the LR model of the Wide part with an FM model. The model structure is shown in Figure 3. It has a Factorization Machine (FM) part on the left and a Deep Neural Network (DNN) part on the right, forming end-to-end training and avoiding the work of manual feature extraction engineering. The FM part and the DNN part share feature vector parallel training and learn low-order feature crossover and high-order feature crossover simultaneously, which has good memory characteristics and generalization ability.

The FM part is used to extract the low-order features of the input data, which is based on the linear regression model to introduce cross-terms. It trains the weights of the combined features by learning a hidden weight vector for each feature. The inner product of the two vectors as a weight can not only alleviate the data sparsity problem but also effectively learn the feature combination. The calculation formula is as follows:

$$y\left(x\right)={\omega }_0+{\displaystyle \sum }_{i=1}^n{\omega }_i{x}_i+{\displaystyle \sum }_{i=1}^n{\displaystyle \sum }_{j=i+1}^n{\omega }_{ij}{x}_i{x}_j$$

(1)

where ${\omega }_{ij}$ is the weight of the feature combination $x_i{x}_j$.

On the right side is the DNN (Deep Neural Networks) part, which forms end-to-end training and avoids the manual extraction of feature engineering. The DNN network is a feed-forward neural network for learning higher-order feature crossover, and in the fully connected layer, let the feature vectors be combined to form higher-order features.

The input of the DNN layer is

$$a^{\left(0\right)}=\left[e_1,e_2,\cdots ,e_m\right]$$

where $e_i$ is the $i$th feature vector, $m$ is the total number of features, and then $a^{\left(0\right)}$ is fed back to the DNN, and the forward processing is as follows:

$$a^{\left(l+1\right)}=\sigma \left(W^{\left(l\right)}{a}^{\left(l\right)}+b^{\left(l\right)}\right)$$

(2)

where $\sigma$ denotes the activation function, $a^{\left(l\right)}$ denotes the output of the l-layer, $W^{\left(l\right)}$ denotes the l-layer weight matrix, and $b^{\left(l\right)}$ denotes the bias matrix.

The final output of the DNN model is shown in Equation (3)

$$y_{DNN}=\sigma \left(W^{\left|H\right|+1}{a}^H+b^{\left|H\right|+1}\right)$$

(3)

where $H$ denotes the number of network layers of the model.

The output of the final model is the sum of the two

$$y=sigmoid\left(y_{FM}+y_{DNN}\right)$$

(4)

where $y_{FM}$ is the output of the FM model part and $y_{DNN}$ is the output of the DNN model.

2.3. Representation of User Characteristics Incorporating Temporal Factors

A basic user representation can be obtained based on the user’s historical interaction items, but the sequence of historical items that the user has interacted with is temporal in nature. In general, items that the user has recently interacted with have more influence on the user’s choice than items that the user has previously interacted with. The GRU model can be used to process and predict sequential data. It will influence the subsequent output based on the previous memory information, and there is no gradient disappearance. Therefore this paper uses the GRU model to train to get the vector representation of the user. The model is shown in Figure 4.

The overall representation of user characteristics can be obtained based on the historical items $v$ that users have interacted with, defined as $v=\left\{v_u^1,v_u^2,\cdots v_u^n,\right\}$. Human behavior generally varies with weekly and monthly cycles; thus, a historical sequence of user interactions is divided according to weeks and months, then input into the GRU model separately, as shown in Equation (5)

$$v^T=v_1^T+v_2^T+\cdots +v_k^T$$

(5)

where $v_j^T$ denotes the set of item sequences that have interacted in the $j$th cycle with $T$ as the period.

The GRU model obtains the state of the gating $r$ and the gating $z$ based on the state $h_{t-1}$ passed down from the previous node, and the input $x_t$ from the current node. The gating $r$ is the control reset gating. The gating $z$ is the control update gating. The equations are as follows:

$$r=\sigma \left(W_r\cdot \left[h_{t-1},x_t\right]\right)$$

(6)

$$z=\sigma \left(W_z\cdot \left[h_{t-1},x_t\right]\right)$$

(7)

where $\sigma$ is the sigmoid function that controls the data in the range of (0, 1) and $W$ is the parameter where the closer the gating $z$ is to 1 indicates the more data to stay.

After getting the gating signal, the reset gate is used to get the reset data $h_{t-1}^{\prime }=h_{t-1}\odot r$, $\odot$ is the Hadamard product, Then $h_{t-1}^{\prime }$ and the input $x_t$ are spliced, and deflate the data to the (−1, 1) by the function $\tan \mathrm{h}$ to get $h^{\prime }$.

$$h^{\prime }=\tan \mathrm{h}\left(W_{h^{\prime }}\cdot \left[h_{t-1}^{\prime },x_t\right]\right)$$

(8)

where $W_{h^{\prime }}$ is the parameter to be learned. The final update of memory requires both forgetting and remembering steps. The formula is shown in Equation (9).

$$h_t=\left(1-z\right)\odot h_{t-1}+z\odot h^{\prime }$$

(9)

This step excludes some dimensions of the previous step and adds some of the current stage input. $\left(1-z\right)\odot h_{t-1}$ means removing some unimportant information in the dimension $h_{t-1}$, $z\odot h^{\prime }$ means selectively retaining the information of the current node. The final obtained $h_t$ is the user vector representation.

The user representation with a weekly cycle is $u_{T1}$, the user feature representation with a monthly cycle is $u_{T2}$, and finally the two are stitched to obtain the user feature representation $u$.

2.4. The Output of the fDeepFM Models

The output of the fDeepFM model is stitched together from the output of three parts. The formula is expressed as follows:

$$y_{fDeepFM}=sigmoid\left(y_{FM}+y_{DNN}+y_{GRU}\right)$$

(10)

After stitching, the output features are added to a fully connected layer, and the final result is output after the activation function. The model mines the relationship between users and items for recommendation. It has good memory properties as well as generalization ability and can effectively alleviate the problem of data sparsity.

3. Experimental Verification

3.1. Description of the Dataset

This subsection validates the proposed algorithm with experimental data from the Amazon product data and MovieLens-20M datasets. The Amazon dataset was collected between July 1995 and March 2013 and contains about 35 million datasets, including user information, item information, and review information. We conduct experiments on a subset named Electronics, which contains 192,403 users, 63,001 items, 801 categories, and 1,689,188 samples. User behaviors in this dataset are rich, with more than 5 reviews for each user and item. In this paper, the user’s rating of each item is considered positive. The MovieLens-20M evaluation dataset was collected between January 1995 and March 2015. This dataset contains 138,493 users for 27,278 movies, 20,000,263 ratings, and 465,564 tags. Scoring is based on a 5-point scale, we label the samples with a rating of 4 and 5 to be positive and the rest to be negative. Since this dataset does not have information such as movie summaries, thus this paper uses a web crawler to crawl movie content data from IMDB. The sources website of movies in this dataset, which include movie summaries, storylines, etc. Punctuation, special symbols, and some stopwords are removed from the text. For content text data, data pre-processing work such as word separation and word frequency analysis is performed using jieba to collect useful content information.

The user’s choice is mostly related to his gender, age, occupation, etc., and the rating level also indicates the user’s preference for the item. To make the classification effect more accurate when training the model, for the MovieLens-20M dataset, the selected user dimensions include user ID, user age, occupation information, user characteristics, and user rating. The movie dimensions are movie ID, movie summary, movie type, and storyline synopsis. Similarly, on the Amazon dataset, the selected user dimensions include user ID, user age, occupation information, user characteristics, and user rating. The product dimensions are product ID, product name, product details, and product price.

3.2. Evaluation Indicator

In this paper, the Root Mean Square Error (RMSE) and Mean Absolute Error (MAE) between the actual and predicted scores are used to measure the computational errors of different algorithms. The smaller the value of the evaluation index is, the smaller the prediction error and the higher the accuracy rate, and the formula is defined as follows:

$$RMSE=\sqrt{\frac{1}{m}{\displaystyle \sum }_{i=1}^m{\left(y_i-{\widehat{y}}_i\right)}^2}$$

(11)

$$MAE=\frac{1}{m}{\displaystyle \sum }_{i=1}^m\left|y_i-{\widehat{y}}_i\right|$$

(12)

The F1-Score value is used to measure the recommendation effectiveness of different algorithms. The higher the value, the higher the recommendation accuracy. The F1-Score is calculated from Precision and Recall and is defined by the following formulas:

$$Precision=\frac{TP}{TP+FP}$$

(13)

$$Recall=\frac{TP}{TP+FN}$$

(14)

$$F1-Score=\frac{2\cdot Precision\cdot Recall}{Precision+Recall}$$

(15)

Among them, TP (True Positive) means the positive class is a positive class. FN (False Negative) means missed report. A positive class is decided to be a false class. FP (False Positive) means false report, false class is decided as a positive class. TN (True Negative) means a false class is a false class.

AUC (Area Under Curve) is used to measure the sorting ability of the algorithm. AUC is the area of the ROC (Receiver Operating Characteristic) curve and the coordinate axis. The larger the value, the better the recommendation effect. The formula is defined as follows:

$$AUC=\frac{\sum I\left(P_p,P_n\right)}{M\ast N}$$

(16)

$$\sum I\left(P_p,P_n\right)=\left\{\begin{array}{c}1,p_p>p_n\\ 0.5,p_p=p_n\\ 0,p_p<p_n\end{array}\right.$$

(17)

where $M$ denotes the number of positive samples, $N$ denotes the number of negative samples, and $M\ast N$ denotes the logarithm of samples.

The experiments randomly divide the dataset into a training set (80%), test set (10%), and validation set (10%) in the ratio of 8:1:1, using the training set to train the model. And use the experimental results of the algorithm on the validation set to evaluate the prediction effect of the algorithm. The vector length of Doc2Vec is 100 dimensions; the embedding dimension is set to 64. The number of feature dimensions of GRU is 5, and the number of neuronal nodes is 32. The number of hidden layers in DNN is 3, with 32 neuronal nodes in each layer, and the layer is fully connected and calculated without activation function. The Batch Size is set to 256 and 1024 on the two datasets, the learning rate is set to 0.001, the optimizer uses Adam, and the activation function is used when the epoch is set to 7. In the output layer, the activation function is ReLU, the loss function adopts using Logarithmic loss function, and to prevent over-fitting of the experimental model training, the L2 regularization factor is used.

3.3. Performance Comparison Experiments of Different Algorithms

This paper chooses to compare experiments with other similar FM family recommendation algorithms: AFM, NFM, ONN, DeepFM, and xDeepFM.

AFM: The AFM model is based on the FM model by introducing an attention mechanism to learn the importance of different second-order crossover features;

NFM: The NFM model uses a Bi-interaction Pooling structure for feature crossover learning as a way to improve the ability of FM to capture multi-order interaction information between features;

ONN [16]: the ONN model distinguishes between different Embedding layer operations on the same feature and proposes a new Embedding model;

xDeepFM [17]: xDeepFM proposes a new neural model called a compressed interaction network to replace the FM part in DeepFM.

On the MovieLens-20M dataset, the evaluation metric values of each algorithm are shown in

Table 1. Numerical comparison of evaluation indexes of the algorithm in MovieLens-20M data set.

Algorithm	MAE	RMSE	F1-Score	AUC
AFM	0.7451	0.9442	0.8523	0.7861
NFM	0.7367	0.9395	0.8475	0.7794
ONN	0.7309	0.9321	0.8584	0.7843
DeepFM	0.7334	0.9354	0.8623	0.7950
xDeepFM	0.7256	0.9240	0.8738	0.8074
fDeepFM	0.7165	0.9114	0.8790	0.8178

, and the training time of models is shown in Figure 5. On the Amazon dataset, the evaluation metric values of each algorithm are shown in

Table 2. Numerical comparison of evaluation indexes of the algorithm in Amazon data set.

Algorithm	MAE	RMSE	F1-Score	AUC
AFM	1.5723	1.7452	0.6853	0.6165
NFM	1.5358	1.6736	0.6754	0.5952
ONN	1.5007	1.6513	0.6881	0.6193
DeepFM	1.5048	1.6519	0.7023	0.6221
xDeepFM	1.4921	1.6434	0.7148	0.6360
fDeepFM	1.4728	1.6133	0.7186	0.6441

.

From Table 1 and Table 2, it can be seen that because the Amazon dataset is more sparse, all kinds of algorithms generally work better on the MovieLens-20M dataset. AFM introduces the attention mechanism but does not learn the higher-order features, so the results of each index are poor. ONN, DeepFM, xDeepFM, and fDeepFM all modify the Embedding layer to improve the model expression in the shallow layer. Then use DNN to learn higher-order features, which makes the algorithm increasingly more effective. The algorithm of fDeepFM works best because the item features are divided into word-like features and text description-like features for extraction separately. This makes the item feature content information integrate more emotion and contextual semantics and introduces the GRU model to learn the change of user interest, and over time the recommendation effect is improved. So the MAE value, RMSE value, F1-Score value, and AUC value are improved by 1.69%, 2.4%, 1.67%, and 2.28%, respectively, on the MovieLens-20M dataset compared with DeepFM; on the Amazon dataset, the MAE value, RMSE value, F1-Score, and AUC values were optimized by 3.2%, 3.86%, 1.63%, and 2.2%, respectively, compared to DeepFM.

From Figure 5, it can be seen that the AFM and NFM training time is short, but the AFM time is the shortest, indicating that the addition of an attention mechanism can shorten the training time. ONN takes the longest to train because this model sacrifices complexity to improve recommendation accuracy. The training time of DeepFM, xDeepFM, and fDeepFM increases sequentially, and the fDeepFM increases the training time compared with the DeepFM model, but its accuracy is improved.

3.4. Comparison Experiments of Algorithm Effects under Different Scales of Sparsity

To verify the performance of each algorithm under the different scales of data sparsity, in this paper, using MovieLens-20M datasets, the original data is divided into a training set and test set according to different ratios. The training set accounts for 80%, 70%, 60%, 50%, and 40%, respectively, and each type of algorithm is tested to obtain the value of RMSE. The test results are shown in

Table 3. Comparison experiments of algorithm effects under a different scale of sparsity.

Algorithm	80%	70%	60%	50%	40%
AFM	0.9442	0.9573	0.9668	0.9831	1.1892
NFM	0.9395	0.9490	0.9592	0.9778	1.0323
ONN	0.9321	0.9489	0.9634	0.9790	1.0528
DeepFM	0.9354	0.9499	0.9684	0.9743	1.0044
xDeepFM	0.9240	0.9391	0.9430	0.9577	0.9733
fDeepFM	0.9114	0.9273	0.9322	0.9481	0.9694

.

The above table shows that the recommendation performance of each algorithm decreases as the percentage of the training set decreases. Among which, the AFM model has the worst effect because it is a shallow model and cannot alleviate the problem of data sparsity well. The other models have a better effect, which indicates that the addition of a neural network can alleviate the effect of data sparsity on the algorithm. The fDeepFM algorithm has the best effect, which indicates it can effectively alleviate the problem of data sparsity.

4. Conclusions

This paper proposes an improved DeepFM recommendation algorithm incorporating deep feature extraction to address the problem that the DeepFM model does not consider the effect of time on user selection and has low recommendation accuracy under sparse data. The model makes full use of the attributes of items and user features and mines the low-order features and high-order features for training. At the same time, it can also learn the changing pattern of user interest. Through comparison experiments with several algorithms of the same series on MovieLens-20M and Amazon datasets, this algorithm has the best results in four evaluation indexes, MAE, RMSE, F1-Score, and AUC. This proves that the proposed methods alleviate the data sparsity problem and also effectively improve the accuracy of the recommendation algorithm. The next step in the research is to study the application of deep learning algorithms in recommendation systems.