Paper—Two Models Based on Social Relations and SVD++ Method for Recommendation System Two Models Based on Social Relations and SVD++ Method for Recommendation System

Recently, Recommender Systems (RSs) have attracted many researchers whose goal is to improve the performance of the prediction accuracy of recommendation systems by alleviating RSs drawbacks. The most common limitations are sparsity and the cold-start user problems. This article proposes two models to mitigate the effects of these limitations. The proposed models exploit five sources of information: rating information, which involves two sources, namely explicit and implicit, which can be extracted via users’ ratings, and two types of social relations: explicit and implicit relations, the last source is confidence values that are included in the first model only. The whole sources are combined into the Singular Value Decomposition plus (SVD++) method. First, to extract implicit relations, each non-friend pair of users, the Multi-Steps Resource Allocation (MSRA) method is adopted to compute the probability of being friends. If the probability has accepted value which exceeds a threshold, an implicit relationship will be created. Second, the similarity of explicit and implicit social relationships for each pair of users is computed. Regarding the first model, a confidence value between each pair of users is computed by dividing the number of common items by the total number of items which have also rated by the first user of this pair. The confidence values are combined with the similarity values to produce the weight factor. Furthermore, the weight factor, explicit, and implicit feedback information are integrated into the SVD++ method to compute the missing prediction values. Additionally, three standard datasets are utilized in this study, namely Last. Fm, Ciao, and FilmTrust, to evaluate our models. The experimental results have revealed that the proposed models outperformed state-of-the-art approaches in terms of accuracy. Keywords—Recommendation system, SVD++, social relations, data sparsity, cold-start 70 http://www.i-jim.org Paper—Two Models Based on Social Relations and SVD++ Method for Recommendation System


Introduction
Recommendation Systems (RSs) play a critical role in providing interested users' items among the mass of information that is available in online applications. Technically, a recommender system can be categorized in various fields, such as machine learning, text classification, and information retrieval [1] [2]. The main aim of the RS is to find the relevant items by understanding users' behavior and predicting items that might be of interest to the user [3]. Recently, Collaborative Filtering (CF) has become one of the most common prototypes and most widely successful technologies in RS [4]. CF involves two types: memory-based and model-based. User-based and item-based are two techniques of the memory-based method, and both compute the similarity among users/items [5]. On the other hand, the model-based approach utilizes machine learning techniques to analyze users' activity and behavior in a particular online application, the system predicts items for that user according to his behavior [1]. However, CF faces a set of challenges, such as data sparsity, cold-start, and scalability. As data arise, which reaches thousands of users and items that remain without ratings, such a phenomenon is called a data sparsity problem. Moreover, the cold-start problem is the issue of the lack of ability to make predictions for new items or users. Furthermore, there is no history information on cold-starts of users/items; thus, RSs cannot handle this problem [5]. To curtail such shortcomings, many data mining methods have been adopted, including clustering CF and matrix factorization (MF) [6], MF can find the best latent factor in the user-item matrix. MF includes many methods, such as Singular Value Decomposition SVD [7], Probabilistic Matrix Factorization (PMF) [8] and [9], Non-negative Matrix Factorization (NMF) [10], and SVD++ [1]. Many studies have been made to overcome further the problems of RSs. These studies exploited different techniques and integrated various sources such as social relations, context, and personal information to enhance the accuracy of the RS. Social relations have widely been used recently in the RS field [11] and [12], in which the user can return to his/her friends' relationships before deciding to accept or reject a particular item. Several social factors were applied, such as reliability-based Trust-Aware Collaborative Filtering (TACF) [13], explicit relationships Social Recommendation [14], explicit implicit relationships using Probabilistic Matrix Factorization (EISR) [9], and MSRA-SVD [5]. Explicit relations are the main source of social information. Here, a user can find his interesting items by exploiting the preferences items of his/her explicit friends. Despite the explicit relations contributing to alleviate the data sparsity problem, the problem still remains because most of the users have a limited number of relations. Accordingly, another factor can be adopted which has massive information called implicit relations [15].
In real life, users have similar preferences to each other even when there are no direct relations between them in the network. Many methods have utilized explicit relations and others have exploited implicit relations to overcome the data sparsity problem, but still require further sufficient prediction accuracy. It is poor practice to depend only on direct connections in the social network, as proposed by [14], and indirect relations with one step of the users' neighbourhood only [9]. The limitation of [9] is neglecting information of other steps of the indirect connection which may impli-cate vital information. In addition, social interactions among users are established based on trust relations which can be exploited as extra factors to reduce the sparsity problem [13]. Moreover, the limitations of [5] ignored the implicit feedback information, and the trust relations of the users were not exploited.
In this study, the foremost goal increases the accuracy by overcoming the previous limitations, two models are proposed. Both methods adopted four sources of information: implicit relations which are identified by Multi-Steps Resource Allocation (MSRA), explicit social relations, explicit feedback, and implicit feedback, which can be extracted via rating information. These sources are integrated into the SVD++ method to create the MSRA-SVD++ model. Finally, the confidence factor between users, which is the fifth source of information, it is utilized in the first proposed model only, which is applied and combined with the similarity values to produce the weight-SVD++ model.

Material and Methods
In this section, the methods and sources that are used in this research are explained. SVD++ method is demonstrated in the next subsection. Whereas the link prediction technique is explained in 2.2. Social relations are clarified in subsection 2.3. Finally, the two proposed models are highlighted in subsections 2.4 and 2.5.

SVD++ method
The SVD++ method, which is utilized by many researchers, is derived from SVD. However, the SVD method exploits explicit information (votes or ratings) that are available in user-item matrices and ignores the implicit information such as clicking, browsing, and purchasing [1]. SVD++ proposed by [16]. For example, assume a 2D user-item matrix (M), wherein the rows p indicates the users, and the columns q represents the items. The rating value where the users can rate the items can be expressed as ru,v. The purpose of the RS is to predict items for users that might interest them. In general, the dataset is huge and many users leave the items without a rating, thus the RS faces a data sparsity problem. However, SVD++ decomposes the user-item matrix into two matrices, p and q, which are orthogonal matrices to represent the users and items, respectively. The decomposition process converts the high dimensional useritem matrix that contains all the data into low-dimensional data pu,f and qi,f, where u is the user number, i the item number, and f indicates the dimension after reduction. In other words, SVD++ reduces the size of matrices p and q to a specific value and finds the best low-dimensional data representation of the user-item matrix by removing any unimportant data. In addition, SVD++ is an expanded version of SVD. The difference is that the implicit feedback is taken into consideration in SVD++, whereas the SVD method depends only on users' ratings (explicit feedback). Subsequently, a binary matrix is constructed to represent the implicit feedback, where the new matrix is separated into left and right orthogonal matrices. The following formula shows the prediction of SVD++: where is the average value of the whole dataset, and are the observed deviations from the mean value of user u and item i, respectively. | | is the number of items rated by user u, | | is the number of users that rated a specific item, | | is the number of items that were rated by a set of users, and is the left orthogonal matrix of the implicit matrix. [17] proposed two regularization parameters ( 1 and 2) which can be added to | | and | | to compute the user bias and item bias. Eqs (2) and (3), are adopted in this study and modified thus:

Link prediction
Social networks encompass a large amount of information, including explicit information such as direct relations (trust relations) and indirect relations (friend's relations). Another kind of information is implicit relations. Regarding the implicit relations, the link prediction technique is exploited to predict missing information in a social network. Many methods are utilized to find the implicit information by predicting new links in a social network, such as Resource Allocation Index (RAI) [15] and Multi-Steps Resource Allocation (MSRA) [18]. According to [5], MSRA achieved better results than RAI, CN ad JC such that it can predict implicit relations with an acceptance ratio of the accuracy (for more details, review reference [5]). MSRA accommodates a multi-step process, but in this work two-steps resource transmission is employed to achieve the implicit relations, thus: ᴦ( ) and ᴦ( ) indicate the sets of the friends of x and y, respectively. Z1 refers to ᴦ( ) ∩ ᴦ( ) and Z2 to ᴦ( 1) ∩ ᴦ( )/ 1.

Social relations
Social relations provide solutions to overcome the data sparsity problem in RS. In reality, users prefer to ask their friends before deciding on a certain action. The same aspect can be employed in RS. Recently, many applications allow how users to establish their friends' relationships and the RS can exploit these relations to enhance the performance. Several studies have utilized social relations to boost prediction accuracy. [14] utilized explicit social relations. For each pair of users, the similarity technique was applied to calculate the similarity between them. Finally, the model was combined with MF. Similarly, in EISR was proposed by [9], explicit and implicit relations were exploited. The same scenario of the previous study was followed, but here implicit relations were added as an extra factor. In [5], the implicit and explicit relations were used, but the difference is using MSRA algorithm instead of RAI algorithm, then the similarity was computed and all the social factors with rating information were incorporated into SVD method. Subsequently, the last two studies were based on implicit relations to represent implicit friends' relationships, and these relations concluded by link prediction technique that finds missing relations in a social network. Another kind of social relations is confidence statements, which can be divided into explicit trust and implicit trust statements [19]. Unlike friend relations, a trust statement means directed relations, such as when user x trusts user y, it is not necessary for user y trusts user x. In some datasets, an explicit trust may be unavailable; therefore, the implicit trust can be estimated from a user-item matrix, by monitoring the attitude of the users through their ratings that lead to conclude new resource information, which may increase the prediction accuracy. The following formula proposed by [20] shows the calculation of implicit trust.
Where is the number of items that are rated by user u. , is the set of common items rated by u and v.
According to [13], the trust statement can be integrated into the rated information to determine the sparsity problem. Thus, the adjusted weight between users u and v is computed thus: Where the , denotes the similarity between users u and v, which computes using the Pearson Correlation Coefficient (PCC) as the following.
Here, , is the rating value of item j that is rated by user u. ̅ indicates the mean rating of the user x, and n refers to the items that are rated by both users u and v. The similarity value between u and v ranges over [-1,1], so a higher value means high similarity. Although the preceding methods improved the RS performance, it still lacks adequate accuracy. The following subsections propose how to overcome this limitation.

Weight-SVD++ model
Five sources of information are used to feed the matrix factorization technique, namely the rating file that includes explicit and implicit feedback information, explicit relations, implicit relations, and confidence values. The rating information and explicit social relations sources are obtainable directly from the rating file and social file respectively, while the implicit relations and confidence values sources are computed. Regarding the implicit relations, MSRA is adapted to discover the implicit relations of users within the social network. The user not only can be affected by his friends' taste but also by other users, even if there is a distance between him and them in the network, and especially when those users are friends of his/her friends. Therefore, MSRA is utilized to apply a link prediction technique and predict missing relations that may contain vital information.
In addition, confidence values can be exploited to reduce the sparsity problem and consequently increase the prediction accuracy. After acquiring the implicit relations via MSRA, a binary user-user matrix is created, one's values refer to explicit/implicit relations. Afterward, the similarity of the explicit relations and implicit relations is computed using the Pearson correlation coefficient (PCC) which is uses rating values of the users, as shown in Eq. (10). The output after applying the similarity is another user-user matrix with the computed similarity values.
Furthermore, the confidence value is computed among all users using Eq. (8); a user-user matrix is created that has confidence values. Later, the weight values are computed by integrating the user-user matrix of the similarity values, and the user-user matrix of the confidence values, by Eq. (9). Eventually, the weight values and rating values are integrated into the SVD++ method to produce the weight-SVD++ model, details of which are shown in Figure 1. The optimization technique is adopted using Stochastic Gradient Descent (SGD) method, which is used to regularize the square error as follows: Where βw is a constant value that is used to determine the influence degrees of the weight values. F(i) denotes the numbers of users that have weights of user i; and W(i,f) refers to the weight value between users i and f. λ is the constant value that determines the degree of the constraint. These equations are utilized to find the best value for vectors , , , and vectors. The following equations show the updating values after each loop: Here, is the learning rate. Then, the prediction value is computed according to Eq. 1. Algorithm 1 shows the implementation steps of the weight-SVD++ model.

MSRA-SVD++ model
The second model involves four sources of information, namely explicit feedback, implicit feedback, explicit relations, and implicit relations. Explicit and implicit feedback information can be derived from the rating matrix, whereas the explicit relations are available directly in the social file. Implicit relations can be computed using the MSRA algorithm. The difference between the first model and the current model is the utilization of the confidence values. In this model, the similarity values of the implicit and explicit relations are adopted only. In general, the same equations, which are used in the previous model, are applied here with a simple difference in Eq. (11), which is modified as follows: where and are constant values greater than zero that used to control the impact of the explicit and implicit relationships, respectively, F(i) denotes the numbers of explicit friends and F*(i) indicates implicit relationships of user i; and sim(x,y) is the similarity value between users x and y. λ is the constant value that determines the degree of the constraint.
Algorithm 2: MSRA-SVD++ model: 1. Read the rating file and create a user-item matrix 2. Read the social file and create a user-user matrix as explicit relations 3. Apply the MSRA algorithm using Eqs. (6) and (7), then create a user-user matrix of implicit relations 4. Compute the similarity between each pair of users who have an explicit relation or an implicit relation using Eq. (10) 5. Initialize random latent factor matrices for vectors p and q 6. Compute bi and bu vectors using Eqs. (4) and (5) 7. Compute the regularization square error using Eqs. (18) and (12) 8. Update pu, qi, yj, bu, bi by using Eqs.

Results and Discussion
The two detailed implementations of the two models are presented in this section, including the dataset description, evaluation metrics, parameter settings, the benchmarking of the experiments, and the impact of using social relations.

Dataset description
In this study, three standard datasets are used: Last. Fm, Ciao, and FilmTrust. Last. Fm is an online music system that enables users to create their relationships. This dataset was released for the first time in 2011 at the HetRec workshop [21], Last.FM has 2100 users and 18745 items. In this study, users with greater than four relationships are included. Thus, the number of users is 1123 and 17892 items. Last. Fm does not have rating values but instead, it has listening counts number and Eq. (19), proposed by [22], is applied to map these values from 1 to 5 to present the rating values.
Here, r is the rating integer value, denotes the listening count, and ⌊ ⌋ denotes the operation of rounding towards zero.
The second dataset is Ciao, which was gathered by [23] and involves of 7375 users and 99746 items. The rating value includes integer values 1-5. A subset of the Ciao dataset is utilized in this study to avoid memory consuming in which the items with ratings of less than two are ignored and users that have one or more ratings with at least one relationship are selected. Therefore, the size of the Ciao dataset became 6767 users and 22229 items. Finally, FilmTrust is the third dataset to be applied. It is a movie rating and trust social relations dataset. FilmTrust was produced by [24]. The ratings are float values [0.5 -4] with 0.5 increments. Table 1 depicts the statistical of these datasets.

Evaluation metrics
Two well-known metrics are utilized to evaluate the performance of the proposed method, namely the Mean Absolute Error (MAE) and the Root Mean Square Error (RMSE), which are defined by the following formulas: where N is the number of the whole prediction, , is the real rated value in the user-item matrix for user u and item i, and ̅ , is the computed value of the proposed models. High accuracy results mean the values of both metrics are close to zero.
In the literature, the most similar studies of this study are TrustMF, SocReg, MSRA-SVD, and UE-SVD++. Thus, for equitable comparison, the proposed models were tested under the same conditions, such as the number of iterations, explicit social relations, the dimensional reduction value, and the number of users. Table 2 shows the parameters setup. Two testing views are displayed in this study. The first view includes all users who have at least one relationship (for Ciao and Last.FM) and one rating (for all datasets), the users were distributed into training and testing parts randomly. The second view is the Cold-start users, which means users with ratings of less than five were included in the test part for the Ciao and FilmTrust datasets, whereas only three ratings of 20% of all users in the Last. Fm dataset are randomly selected.

5-fold
Cross-validation was adopted in this experiment. The datasets were divided randomly into five folds in which four-folds had been used for training (80%) and one for testing (20%). To guarantee test all the data, each execution had been performed five times and the average was computed to produce the outcome of each implementation. Moreover, the experiment was implemented several times and the mean was produced to display the eventual result. Tables 3 and 4 depict the accuracy of the results for all users and cold-start users, respectively. The accuracy results were computed through the MAE and RMSE metrics. For both metrics, high values indicated low accuracy. On the other hand, if the values had been close to zero, the results would have been highly accurate. Two results were displayed in both tables when the number of dimensions (d) is 5 and 10. The proposed models were compared with state-of-the-art studies, with PMF, SVD++, and UE-SVD++ that used only rated information and the other studies that utilized social relations. All these studies are shown in Tables 3 and 4. Boldface implies the best results and (*) indicates the second-best results. Tables 3 and 4 show the influences of social relations to improve the prediction process. Accordingly, comparing the results of PMF and SVD++ against TrustMF, SocReg, and MSRA-SVD models depict how social relations have a vital rule to boost accuracy. Moreover, the effect of the social relations in cold-start users (Table  4) is greater than that in all users (Table 3). To clarify, cold start users have insufficient rating history to utilize it in the prediction process, whereas for all users, there is reasonable rating history that can be exploited to predict new items for them, and using social relations assist the prediction, but to smaller degree.
According to Table 3, the proposed models outperformed all the previous approaches for both models (weight-SVD++ and MSRA-SVD++) and the UE-SVD++ has the closest results in FilmTrust dataset. However, MSRA-SVD has the nearest results to our proposed models in Ciao and Last. Fm datasets. Moreover, MSRA-SVD++ achieved the best performance such that it outperformed our first model (weight-SVD++) in the FilmTrust dataset. Although the weight-SVD++ model uses many sources of information to reduce the sparsity, the accuracy results were lower than the MSRA-SVD++ results. To justify the reason, the weight-SVD++ model uses confidence values as additional information sources to find relations between users. Additionally, dimension reduction methods such as SVD++ achieve good performance when datasets have acceptable sparsity ratios. In contrast, when the sparsity is reduced to a minimum level (as performed by weight-SVD++), the results will be affected negatively especially with high-density datasets, such as FilmTrust. Nonetheless, the weight SVD++ model nearly outperformed the MSRA-SVD++ model in terms of MAE and RMSE in the Ciao and Last.FM datasets because these datasets have higher sparsity than FilmTrust, so reducing the sparsity using social relations and confidence values achieved higher accuracy. For example, the improvement that is achieved by the MSRA-SVD++ model comparing with the second-best results in Ciao is about 0.63% and 1.11% for MAE and RMSE respectively when d = 5, and it is about 0.57% for MAE and 1% for RMSE when d = 10. Likewise, the improvement in FilmTrust is approximately 1.4% For MAE and 0.07% for RMSE when d equals 5, Also it is 1.24% and 0.32% for MAE and RMSE respectively when d equals 10. Table 4 presents the results of cold-start users, who have less than five ratings. Table 4 reveals that MSRA-SVD++ once again exceeded the weight-SVD++ model in FilmTrust. It accomplished the best results in terms of MAE and RMSE, and the improvement that is achieved by comparing MSRA-SVD++ over the best-second results, when d=10, is about 3.03% and 3.38% for MAE and RMSE in FilmTrust, 3.29% for MAE and 3.83% for RMSE in FilmTrust, and in Last. Fm dataset the improvement of around 5.29% for MAE and 0.79% for RMSE. Although MSRA-SVD and our present studies are adopted MSRA algorithm to extract implicit relations, the present study has significant improvement in terms of accurate results. Two different reasons caused this improvement. First, adding 1 and 2 during the computation of user-bias and item-bias, respectively as shown in Eqs. (4) and (5). Second, the proposed method used implicit feedback information source as one of the five sources employed in this study, whereas, MSRA-SVD ignored this source.
Additionally, the weight-SVD++ model is superior to MSRA-SVD++ for Ciao and Last.FM datasets in terms of RMSE and MAE such that the improvement that is achieved by weight-SVD++ over MSRA-SVD++ is around 0.03% and 0.04% when d = 5 in terms of MAE and RMSE, respectively in Ciao. The same superiority can be shown in the Last. Fm dataset, which is 0.57% for MAE and 0.14 for RMSE.
In summary of the above results, the weight-SVD++ model can achieve better performance when the dataset has less density such that the density is 0.0012% and 0.0026% for Ciao and Last.FM respectively. Furthermore, with high-density datasets such as FilmTrust, which is 1.14%, extra sources of information have an adverse impact; thus, the MSRA-SVD++ model produces higher accuracy results than the weight-SVD++ model. Subsequently, for the dataset with high sparsity, the cold-start user problem can be alleviated using social relations with confidence values, which indicates a vital source of information and it has more effectiveness than the dataset with a lower sparsity ratio. Accordingly, the proposed methods enhanced the results in terms of accuracy and overcame the data sparsity and cold-start user problems by exploiting four/five sources of information: explicit social relations, implicit social relations, explicit rating values, implicit rating feedback (for MSRA-SVD++ model), with confidence values (for weight-SVD++ model). All the results confirm that the proposed method succeeded to enhance the recommendation system performance.    , also small beta values lead to low accuracy. The best beta values to produce the highest accuracy are different from one dataset to another based on the sparsity ratio and the level of similarity among users. For example, the best beta value, in general, is 0.001 for Figure 3, where the result is 0.9476. In contrast, the best beta value in Figure 4 is 0.002, the result is 0.4575. Likewise, the beta value is different in Figure 5, the best result is reported when beta equals 0.005, the result is 0.7995.

Conclusion
This paper proposed two models by integrating into the first model: rating information, which indicates explicit and implicit ratings, social relations that involve explicit and implicit social relations, also confidence values that are computed by measuring the common rating items of each pair of users. Regarding the second model, the rating and social relations were exploited without confidence values. Both models were tested via three standard datasets, which were utilized to evaluate the performance of our approach. The results revealed that the dimension reduction method presents better performance when it works with an acceptable level of density than working with high density (preferring sparsity over density). Additionally, the comprehensive experimental outcome showed that social relations have further influence when it was applied to cold-start users, who needed extra information to find their preferences. Therefore, the impact of social relations for the cold-start users is more obvious than for those who have reasonable rating history in the same dataset. Moreover, the effectiveness of the social relations increases for the low-density ratio datasets, where social relations contribute to enriching the dataset with more information that can support the prediction process. For further development, the confidence equation in the weight-SVD++ model can be modified to keep the density ratio at an acceptable level, or applying another similarity equation. Eventually, the clustering model may improve accuracy by separating users into different communities.