Fast algorithms to evaluate collaborative filtering recommender systems

doi:10.1016/j.knosys.2015.12.025

Knowledge-Based Systems

Volume 96, 15 March 2016, Pages 96-103

https://doi.org/10.1016/j.knosys.2015.12.025 Get rights and content

Abstract

Before deploying a recommender system, its performance must be measured and understood. So evaluation is an integral part of the process to design and implement recommender systems. In collaborative filtering, there are many metrics for evaluating recommender systems. Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) are among the most important and representative ones. To calculate MAE/RMSE, predicted ratings are compared with their corresponding true ratings. To predict item ratings, similarities between active users and their candidate neighbors need to be calculated. The complexity for the traditional and naive similarity calculation corresponding to user u and user v is quadratic in the number of items rated by u and v. In this paper, we explore the mathematical regularities underlying the similarity formulas, introduce a novel data structure, and design linear time algorithms to calculate the similarities. Such complexity improvement shortens the evaluation time and will finally contribute to increasing the efficiency of design and development of recommender systems. Experimental results confirm the claim.

Introduction

Recommender systems are popular facilities widely deployed to address the challenge of overwhelming information. They are used to seek interesting information to targeted users from a large volume of data. Typical domains where we can see their real-world applications include E-commerce [1], [2], E-government [3], [4], Social Network [5], [6], [7], Academia [8], [9], [10], [11], Entertainment [12], [13], Telecom [14], [15], and so on. Readers can see the comprehensive progress in this topic in a recently published survey paper [16].

There are three basic methods to generate recommendations: collaborative filtering (CF), content-based filtering, and a hybrid approach. CF recommendation aims to produce a list of interesting items to active users based on the preferences of their like-minded neighborhood. Content-based filtering approaches utilize a series of discrete features of items, e.g., the genres, directors, and actors of movies, to generate recommendations. These two approaches are often combined to make hybrid recommender systems [17], [18].

A CF recommender system generally works in three steps. First, it calculates the similarities (cosine similarity, adjusted cosine similarity, Pearson correlation similarity, et al.) between the active user and other users. Second, it selects the active user’s nearest neighbors based on the similarities obtained from the first step. Third, it recommends a list of top-k items by aggregating the nearest neighbors’ preferences.

CF is generally believed to be one of the most successful techniques applied in recommender systems. A flowchart to illustrate the process of designing a recommender system is shown in Fig. 1. It should be emphasized that evaluation is an important and inalienable part of designing a good CF algorithm. There are many metrics to evaluate the performance of recommender systems. Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) are among the most popular ones.

Given the N actual/predicted rating pairs (r_{u, i}, p_{u, i}), where u refers to a user and i to an item, the MAE of the N pairs is evaluated as: $M A E = \frac{| \sum_{i = 1}^{N} (p_{u, i} - r_{u, i}) |}{N}$ and the RMSE of the N pairs is evaluated as: $R M S E = \sqrt{\frac{\sum_{i = 1}^{N} {(p_{u, i} - r_{u, i})}^{2}}{N}}$

Note that lower MAE and RMSE values indicate more accurate predictions, signifying better performance of recommender systems.

To evaluate MAE/RMSE, four steps need to be conducted:

(1)
Divide the dataset into a test set (acting as active users, similarly hereinafter) and a training set.
(2)
Compute and select top-k nearest neighbors in the training set based on a similarity metric, for a rated item of the current test user in the test set.
(3)
Aggregate the ratings of the top-k nearest neighbors to calculate a prediction for the item.
(4)
Repeat Step (2) and Step (3) until the prediction for every rated item of every user in the test set is calculated, and then compute MAE and/or RMSE for the test users.

Essentially, the major effort of our work is to shorten the time spent in the similarity calculation of Step (2).

A flowchart to illustrate the design process of recommender systems is illustrated in Fig. 1. As we have pointed out, evaluation is an inalienable part of the design process. Since the scale of datasets is large, evaluation generally costs much time, which is mechanically spent on the running of recommender systems and is with a low technology content. Shortening the evaluation time can save the designers from waiting for the evaluation done, let them determine the performance more quickly, and focus on other work with a high technology content.

CF research papers have published few technical details about their evaluation processes. We have reviewed the major open source recommender systems which use neighborhood-based CF, such as Mahout¹, LensKit², Crab³, Python-recsys⁴, and easyrec⁵, and we find that their MAE/RMSE evaluation procedures are either flawed or they ignore efficiency.

In particular, given a user u in the test set and a user v in the training set, two major problems in evaluating MAE/RMSE (computing the similarities specifically) are described as follows:

(1)
Incorrect method: The similarity between u and v is calculated only once, which is then used to help generate the |I_u| predictions⁶. This means in the above Step (2), in case of each rated item, the similarities between u and v are considered equal and are calculated only once, however, they are not equal indeed! The method is illustrated in Fig. 2a.
A correct method to evaluate MAE/RMSE should include the calculation of predictions for every item rated by user u. For each prediction, there should be a unique similarity between u and v. In the end, the difference between the predicated rating and the true rating can be used to evaluate the MAE/RMSE.
The reason why the similarity should be calculated individually is that for each predicted rating, the corresponding true rating in u should be treated as unknown.
(2)
Correct but inefficient method: The similarities between u and v are considered separately, varying for each rated item in u. For each item rating prediction, the similarity is calculated once, so there are |I_u| similarity calculations. Each time the calculation is with complexity of $O (| I_{u} | + | I_{v} |),$ so the whole complexity for the |I_u| calculations is $O (| I_{u} | \times (| I_{u} | + | I_{v} |))$ . The method is illustrated in Fig. 2b.

The objective of this work is to design efficient algorithms for computing similarity in the process of evaluating recommender systems, whose complexity is of $O (| I_{u} | + | I_{v} |)$ . In other words, the |I_u| times of similarity calculations should be done in one iteration of I_u and I_v respectively.

Most work of recommender systems to date has focused on response time, accuracy and scalability. There has been little attention paid to speeding up the evaluation performances of recommender systems. However, evaluation is a crucial step for designing and tuning recommender systems.

The contributions of this work are as follows.

Firstly, the need for more efficient algorithms to evaluate recommender systems is identified, which could facilitate their design and implementation.

Secondly, the underlying mathematical regularities behind the similarity formula are investigated to help design more efficient algorithms to evaluate MAE/RMSE.

This work can shorten the time in designing and implementing recommender system algorithms.

The rest of the paper is organized as follows. Section 2 presents the related work. Section 3 introduces the basic knowledge on CF recommendation. Section 4 explores the mathematical regularities underlying the similarity formulas and explains why the proposed methods are more efficient. Then the algorithms are articulated in Section 5. The experimental methods and results are shown in Section 6. Finally, Section 7 concludes the paper and future work is suggested.

Section snippets

Related work

Recommender systems are becoming more and more popular with the onset of the World Wide Web and big data. Designing recommender systems requires overcoming challenges of accuracy, scalability, cold start, and privacy [19], [20], [21], [22].

Herlocker et al. [23] comprehensively review the key decisions in evaluating CF recommender systems. They demonstrate the importance of recommender system evaluation and they claim that accuracy is a major metric.

An accuracy metric measures how well a

Basics on collaborative filtering recommendation

Recommender system techniques can be classified into three categories: content-based filtering, collaborative filtering, and hybrid filtering. Content-based filtering produces recommendations for active users according to the similarities between items. Collaborative filtering, however, provides recommendations based on the preferences of other like-minded users. Hybrid filtering combines these two methods to produce recommendations [29]. All such recommender systems are based on user-item

Methodologies

This section first gives a formal definition for the recommender system problem presented above. It then presents theorems which prove that MAE/RMSE can be evaluated with time complexity O(|I_u| × |I_v|), for each of the three user similar metrics presented in the previous section.

Definition 1 Recommendation Problem

There are a set of users $U = {u_{1}, u_{2}, \dots, u_{n}}$ and a set of items $I = {i_{1}, i_{2}, \dots, i_{m}}$ . Users’ preferences for items are represented using a n × m ratings matrix $M = {(R_{1}, R_{2}, \dots, R_{n})}^{T}$ . Each user u rates a set of items $I_{u} = {i_{j_{1}}, i_{j_{2}}, \dots, i_{j_{k}}},$

Algorithms

To calculate MAE/RMSE, it is necessary to maintain a data structure NNVector (Fig. 4), which stores the lists of the nearest neighbors, one for each rated item in the test set of users.

The NNVector in its essence is an array where each element is a list of the nearest neighbors. It is obvious that the size of the NNVector corresponding to user u is determined by the number of his/her rated items, |I_u|.

A traditional and naive method to judge if user v is user u’s neighbor is shown in Algorithm 1

Experiments

The contributions of this study pertain to the reduction of computation time for the evaluation of the MAE and RMSE metrics. From the findings of mathematical regularities of the formulas, their complexity is reduced from $O (| I_{u} | \times (| I_{u} | + | I_{v} |))$ to $O (| I_{u} | + | I_{v} |)$ . Experiments corroborate this. Note that this study does not cover the precision, scalability and cold start problems in recommender systems. Therefore there is no difference in the computational results, except for the speed they are

Conclusions

To address the challenges of the big data era, the efficiency of developing recommender system algorithms must be improved. Algorithm evaluation is an important part in the development process. In this paper, we explored the mathematical regularities in the MAE/RMSE computations and developed efficient algorithms to evaluate them. This work is an integral part of the process for the design and implementation of recommender system algorithms.

However, in the big data era, as the size of data

Acknowledgments

The study is partially supported by the Hubei Provincial Natural Science Foundation of China under Grant no. 2015CFB450, the Fundamental Research Founds for National University, China University of Geosciences (Wuhan), the enterprise-funded latitudinal research project under Grant No. 2014196221, and the Guangzhou Research Infrastructure Development Fund under Grant No. 2012224–12.

References (31)

M. Al-Hassan et al.
A semantic enhanced hybrid recommendation approach: a case study of e-government tourism service recommendation system
Decision Support Syst.
(2015)
S.K. Lee et al.
Collaborative filtering with ordinal scale-based implicit ratings for mobile music recommendations
Inf. Sci.
(2010)
Z. Zhang et al.
A hybrid fuzzy-based personalized recommender system for telecom products/services
Inf. Sci.
(2013)
J. Lu et al.
Recommender system application developments: a survey
Decis. Support Syst.
(2015)
J. Bobadilla et al.
Recommender systems survey
Knowl.-based Syst.
(2013)
J.B. Schafer et al.
Recommender systems in e-commerce
Proceedings of the 1st ACM Conference on Electronic Commerce
(2000)
J. Wang et al.
Opportunity model for e-commerce eecommendation: Right product, right time
Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval
(2013)
J. Lu et al.
Bizseeker: a hybrid semantic recommendation system for personalized government-to-business e-services
Internet Res.
(2010)
H. Kautz et al.
Referral web: combining social networks and collaborative filtering
Commun. ACM
(1997)
X. Liu et al.
Soco: A social network aided context-aware recommender system
Proceedings of the 22nd International Conference on World Wide Web
(2013)

P. Kazienko et al.

Multidimensional social network in the social recommender system

IEEE Trans. Syst., Man Cybern., Part A: Syst. Hum.

(2011)

Q. He et al.

Context-aware citation recommendation

Proceedings of the 19th International Conference on World Wide Web

(2010)

D.W. McDonald

Evaluating expertise recommendations

Proceedings of the 2001 International ACM SIGGROUP Conference on Supporting Group Work

(2001)

K.D. Bollacker et al.

Discovering relevant scientific literature on the web

IEEE Intell. Syst. Appl.

(2002)

K. Sugiyama et al.

Scholarly paper recommendation via users recent research interests

Proceedings of the 10th Annual Joint Conference on Digital Libraries

(2010)

Cited by (82)

Survey of similarity functions on neighborhood-based collaborative filtering
2021, Expert Systems with Applications
Today, recommender systems play a vital role in the acceleration of searches by internet users to find what they are interested in. Among the strategies proposed for recommender systems, collaborative filtering has received due attention regarding its simplicity and efficiency. The key factor for the success of this strategy returns to the similarity calculation methods that affect the accuracy of its recommendations. Regarding the large volume of articles published in the field of collaborative filtering for the development of recommender systems, it is necessary to provide a comprehensive review of the similarity functions and their efficiency presented in the field. Of course, several surveys have already been published to investigate the similarity functions proposed for collaborative filtering, but these articles either have looked briefly at these functions or reviewed a few numbers of them. In this study, the effort was to provide a comprehensive study on the similarity functions proposed for collaborative filtering with a special focus on the rating-based and neighbor-based approaches. After a brief explanation of each similarity function, some popular evaluation metrics were used for their evaluation using the MovieLens datasets. The comparative evaluation results presented in this article provide a highly useful reference for researchers in this field to choose their appropriate similarity function.
An efficient and accurate recommendation strategy using degree classification criteria for item-based collaborative filtering
2021, Expert Systems with Applications
Citation Excerpt :
These aim to achieve an efficient recommendation while also improving the search cost to online users. Zhang et al. (2016) explored the mathematical regularities underlying similarity formulas and designed linear time algorithms to reduce time spent in similarity calculation. Liu and Wu Liu and Wu (2016) introduced a novel latent factor model, latent collaborative relations (LCR), to provide fast recommendations for active users while retaining accuracy and coverage.
An efficient and accurate recommender system provides online users with a variety of personalized recommendation services, thus effectively improving the satisfaction and experience of users. However, there is a trade-off between the accuracy and efficiency in recommender systems. Accordingly, this study introduces a recommendation strategy to address this limitation. The extended degree classification criteria are first proposed to assign items to more fine-grained classes. Later, item similarity measure is deployed to quickly evaluate similarities between items within the same class, which greatly reduces the runtime of similarity calculation. To obtain a better recommendation result, a Hellinger distance (HD) based item similarity is presented to calculate item similarity from the perspective of rating probability distribution. Additionally, a sigmoid function is considered in the HD similarity to emphasize the importance of the co-rated items and effectively distinguish differences between a pair of items. The experimental results on two benchmark datasets show that the proposed similarity method using the classification criteria has better performance in both accuracy and efficiency compared to other methods. Also, the results verify the effectiveness of the proposed classification criteria, especially the runtime of item-based CF method is reduced by at least 61% while maintaining a relatively stable or higher accuracy.
Improving neighbor-based collaborative filtering by using a hybrid similarity measurement
2020, Expert Systems with Applications
Memory-based collaborative filtering is one of the recommendation system methods used to predict a user’s rating or preference by exploring historic ratings, but without incorporating any content information about users or items. It can be either item-based or user-based. Taking item-based Collaborative Filtering (CF) as an example, the way it makes predictions is accomplished in 2 steps: first, it selects based on pair-wise similarities a number of most similar items to the predicting item from those that the user has already rated on. Second, it aggregates the user’s opinions on those most similar items to predict a rating on the predicting item. Thus, similarity measurement determines which items are similar, and plays an important role on how accurate the predictions are. Many studies have been conducted on memory-based CFs to improve prediction accuracy, but none of them have achieved better prediction accuracy than state-of-the-art model-based CFs. In this paper, we proposed a new approach that combines both structural and rating-based similarity measurement. We found that memory-based CF using combined similarity measurement can achieve better prediction accuracy than model-based CFs in terms of lower MAE and reduce memory and time by using less neighbors than traditional memory-based CFs on MovieLens and Netflix datasets.
Personalized E-Learning Recommender System Based on Autoencoders
2023, Applied System Innovation
Efficient Hybrid Movie Recommendation System Framework Based on A Sequential Model
2023, International Journal of Intelligent Systems and Applications in Engineering
The Role of Cognitive Absorption in Recommender System Reuse
2023, Sustainability (Switzerland)

View all citing articles on Scopus

View full text

Fast algorithms to evaluate collaborative filtering recommender systems

Abstract

Introduction

Section snippets

Related work

Basics on collaborative filtering recommendation

Methodologies

Algorithms

Experiments

Conclusions

Acknowledgments

Decision Support Syst.

Inf. Sci.

Inf. Sci.

Decis. Support Syst.

Knowl.-based Syst.

Recommender systems in e-commerce

Proceedings of the 1st ACM Conference on Electronic Commerce

Opportunity model for e-commerce eecommendation: Right product, right time

Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval

Bizseeker: a hybrid semantic recommendation system for personalized government-to-business e-services

Internet Res.

Referral web: combining social networks and collaborative filtering

Commun. ACM

Soco: A social network aided context-aware recommender system

Proceedings of the 22nd International Conference on World Wide Web