I’m Like You, Just Not In That Way: Tag Networks to Improve Collaborative Filtering

4.1 Spam filtering

Because CiteULike publishes its data set on a daily basis, it has become an excellent resource for data in collaborative filtering. Unfortunately, it has also become an excellent target for spam. Unscrupulous marketers have seeded the data set with phony article descriptions designed to lead users to online offers. Many recent publications have used the data from CiteULike without acknowledging the amount of this material which is included in the daily data dumps. Before enacting experiments on the data set, we set out to scrub the underlying repository of extraneous spam data.

CiteULike publishes two data files: one which includes an entry for each documentid/userid/tag combination¹⁰, and one which associates these documentids with external URLs (called a linkout file)¹¹. CiteULike does not store full documents, only document descriptions. This URL information is therefore used to link a document description to its associated content. This content is usually the full text of a journal article, although any existing web page may be used.

Because setting up the target web page for an external URL requires some significant effort, we can be reasonably certain that any documentid which is associated with an external URL in the linkout file is not spam. As a first step to removing spam from the repository we simply removed all documents which did not have associated linkout entries. This made for a dramatic change in the dataset: based solely on the linkout test, roughly one third of the users, two thirds of the documents, and three quarters of the tags used in the corpus are associated with spam entries. We were somewhat surprised by this result, and so decided to verify the finding using an alternative method. A direct examination of those documents deemed questionable by the linkout test was undertaken. It is possible to query CiteULike by documentid directly. Legitimate documentid's produce their corresponding descriptions, even when these descriptions do not contain a linkout entry. Documents which correspond to spam entries, however, fail to produce a document description at all. By querying the site by documentid directly, we can determine whether an individual document is spam. Note that this procedure is different from the linkout test, where we were checking for the existence of an external URL associated with a given description. Here we check whether the description itself exists or not.

Our questionable data set contains over half a million documents. Given that the time required to query each of these would be prohibitive, we adopt a more aggressive approach: we query a given document, and if it is found to be spam we remove it as well as all the users who have included it in their saved documents. We assume that few non-spammers will include spam entries in their saved articles, and that spammers have an incentive to include the same spam document in more than one of their "false" profiles. A manual examination of the resulting data sets (potentially good and still questionable) verified the approach: tags selected at random from the questionable set proved suspicious (e.g. "free", "cash", "enlargement"), while tags selected at random from the potentially good set matched tags from known good profiles.

After performing this process on data determined to be questionable by the naive linkout file method, we found only 282 potentially good users, 2820 potentially good documents, and 10907 potentially good tag groups. These represented 1.0%, 0.3%, and 0.9% of the original data set, respectively. In other words, using just the linkout file to filter spam articles is a method which performs very well.

4.2 Data set characteristics

Figures 1(a)–(c) show that spam removal tends to change the characteristics of the data set, which is to be expected. More importantly, removing spam from the data brings to light a crucial network property that can be used to dramatically improve the results of user-based collaborative filtering.

Figure 1. CiteULike dataset characteristics before/after filtering.

We plot the number of papers belonging to a given number of users, tags, and tag groups respectively, then plot a regression line through the data. All data are plotted in log/log format (base 10), and number of papers is plotted on the x-axis. The slope and mean squared error of the resulting regression line are displayed.

We alluded to a fundamental problem of accuracy: user-based collaborative filtering, because it does not take into account the fact that we trust certain people in certain areas, tends to return suggestion lists which are not very accurate. While these suggestion lists do contain relevant items, they contain a large number of irrelevant items. In Figure 1 we see that this is indeed the case. In each figure, the magnitude of the slope of the regression line indicates the skewness of the distribution: lower magnitudes indicate fatter tails. Shown in Figure 1(a), for plots of number of papers per user, this value is -1.26 and -1.51 for raw and filtered data, respectively. This indicates relatively fat tails for the number of papers by user. In other words, a large proportion of users in the data set have saved a large number of papers to their profiles. Yet the data set is sparse; the number of papers shared by users is small. Therefore a suggestion list based on user similarity will contain a relatively large number of papers, many of these not included on the basis of this similarity.

We can see a similar situation develop when we examine papers by tag. Tag in this case refers to the semantic token; any papers tagged with the term "biology" for example, would be grouped together. Here the slope of the regression line is -1.67 or -1.5, indicating fat tails. If we attempt to compare based on tag, then it seems we will also return a large number of extraneous documents.

Turning to tag groups, however, we notice a change. A tag group in this case refers to just the set of papers which a user has grouped together using a tag; the semantic content of the tag itself is ignored. Here the slope of the regression line after filtering is -2.38, indicating much thinner tails. According to this, a comparison based on tag group should result in far fewer irrelevant documents. Moreover, we should get better results by ignoring the semantic content of the tags altogether, and just using them to define areas of trust.

A further result emerges from the analysis of tag groups. Not only does the slope of the regression line decrease to a value of -2.38, so does the mean squared error of its fit to the data. In other words, the distribution of papers by number of tag groups follows a power law much more closely after filtering. This fact might be useful in estimating the prevalence of spam in an arbitrary data set which uses tag groups.

4.3 Other preprocessing

In order to apply the algorithms below to the data set, some additional preprocessing was required. In that singlet documents (documents saved by only one user) are impossible to analyze and represent only noise for purposes of investigation, these were removed. Additionally, because the investigation below depends upon tag group documents, we also removed tag groups which contained only one document. These two steps can depend on one another (i.e. removing tag groups with one document might create additional situations where a document is owned by only one user and vice- versa), so we ran both procedures iteratively until the number of documents converged to a representative set. Finally, we removed documents which were saved to the repository without any associated tags. Our final data set consisted of 4612 users, 32085 documents, and 40704 tag groups.

5.1 Evaluation metrics

In order to evaluate the performance of our algorithm, we employ a set of metrics commonly used to validate leave-n-out suggestion algorithms. In these situations, we remove a target set of n items which we attempt to recover. Recall measures the percentage of items in the target set which appear in our suggestion list. Precision measures the percentage of the suggestion list which is made up of target set items. In a realistic suggestion environment, we care not only about how many items in the suggestion list match target list items, but also at what position in the list these matches occur⁴. We therefore measure accuracy, which represents the average position of a target item in the suggestion list. A common metric for evaluation which combines recall, precision, and accuracy is mean average precision, or MAP. Average precision is the sum of the precision at each relevant item in the suggestion list divided by the total number of relevant items in the collection. If we average this value over all suggestion lists, we get the MAP for the suggestion algorithm. A perfect MAP score, then, must recall all relevant documents and place them at the top of the suggestion list for a perfect score of 1.0. A final metric which we use in evaluation we call satisfaction, and is defined as the percentage of users in the set for whom at least one suggestion is found to match an item in the target set. Satisfaction is useful in isolating the percentage of cases which represent complete failures of the algorithm.

5.2 User collaborative filtering

To see how user-based collaborative filtering performs on the CiteULike data set, and to establish a baseline for subsequent experiments, we apply a conventional form of user-based collaborative filtering. The procedure is as follows:

1) Select a user at random who has saved at least 6 items; call this the target user.

2) Remove half of the documents belonging to this user; call this the target set.

3) After removing the target set documents from the target user document vector, find the k users in the repository whose document vectors are most similar to its remaining document vector using cosine similarity.

4) The documents belonging to the document vectors of these k users, but not currently contained in the target user's document vector, represent the suggestion list. Score the suggestion list according to criteria described in section 5.1.

The above procedure depends upon a single parameter k: the size of the user neighborhood that is used to generate the suggestion documents. Previous studies on this data set suggest that generic user-based collaborative filtering algorithms perform best at values of k between 5 and 12⁹. Table 2 lists results for values of k in (5,8,12). We see that satisfaction (percentage of users who received at least one good suggestion) is fairly high, but accuracy is dismal (recall lower accuracy scores are better). This agrees with our earlier assumptions about user-based filtering in the context of trust. The algorithm returns suggestions from all of a particular user's interests - not just those areas in which his interests align with the target user's. Because of this, a large number of the returned suggestions are irrelevant, and accuracy suffers.

5.3 Tag-based filtering

We now apply the tag-based algorithm that implicitly specifies the interest area of a target user by comparing tag groups. We start with the natural approach detailed above; instead of comparing users to other users, we compare user interests to other user interests. The procedure is as follows:

1) Select a tag group at random which contains at least 6 items; call this the target group.

2) Remove half of the documents belonging to this group; call this the target set.

3) After removing the target set documents from the target group document vector, find the k tag groups in the repository whose document vectors are most similar to its remaining document vector using cosine similarity.

4) The documents belonging to the document vectors of these k tag groups, but not currently contained in the target group's document vector, represent the suggestion list. Score the suggestion list according to criteria described in section 5.1.

Again, we run the procedure for values of k in (5,8,12).

Results are listed in Table 3. The assumptions relating to trust appear again to hold. Here we have paid a nominal price in satisfaction for a huge improvement in accuracy. The improvement in accuracy is sufficient to raise the overall MAP score between 25% and 65% depending on k.

7.1 A Hybrid approach

The network methods we seek to apply here depend upon tag group comparisons over the entire tag group set. Attempting to analyze the entire network of tag groups would be either computationally intractable or at the very least extremely expensive. In order to produce a viable algorithm, we must find some way to limit the number of tag groups, and the resulting tag group network structure.

Our hybrid approach can be broken down into two steps. In the first step, we calculate a neighborhood of users based upon overall document similarity (as in User-Collaborative Filtering, above). Once this neighborhood is established, we can filter the suggestion results by employing tag-based filtering. Here, instead of comparing over the entire data set, we limit our comparisons to tag groups included in the initial user neighborhood.

1) Select a tag group at random which contains at least 6 items; call this the target group.

2) Remove half of the documents belonging to this group; call this the target set.

3) For the user associated with the target group, run the procedures detailed in section 5.2 to produce a neighborhood of k similar users.

4) After removing the target set documents from the target group document vector, find the k tag groups in this user neighborhood whose document vectors are most similar to its remaining document vector using cosine similarity.

5) The documents belonging to the document vectors of these k tag groups, but not currently contained in the target group's document vector, represent the suggestion list. Score the suggestion list according to criteria described in section 5.1.

Here, we have limited the set of tag groups on which we compare to those in the initial user neighborhood. This approach is similar to⁸, except that the process is reversed. In⁸, a set of tags is employed to limit the neighborhood under which user similarity is calculated. Here, a set of users is employed to limit the neighborhood under which tag similarity is calculated. Additionally, our method does not make explicit use of tagging semantics, which are required in Zanardi's model in order to generate a tag neighborhood⁸.

The results of this modification are detailed in Table 4 at user neighborhood (k) values in (5,8,12).

Clearly, this method does not outperform the simple tag-based approach above. What is interesting, though, is that it does not suffer terribly. In other words, while limiting tag groups to a neighborhood of similar users does not improve the algorithm, it doesn't hamper it considerably, even when we consider relatively small user neighborhood sizes (e.g. 5). Under these conditions, the resulting tag group graph becomes a feasible target of network methods.

We also see that the naive user-based algorithm implemented above still offers considerably higher satisfaction and recall. In other words, the user neighborhood established under this naive approach contains a substantial number of documents that are not being found under the tag-based approach; there is still room for improvement.

7.2 Mutual clustering coefficient

Now that we have established a neighborhood under which to apply network approaches feasibly, we return to the problem sketched above. Our goal is to find those tag groups in G that are not found using the basic tag- based algorithm. Because the user-based algorithm of section 5.2 delivers much higher levels of satisfaction and recall, we can be fairly confident that some of the tag groups in G lie within the neighborhood we've created.

Generally, this problem can be thought of in terms of clustering: what we are attempting to do is cluster k and G so that we can draw from documents in G even though G itself does not contain non-hidden documents from t. Many different attempts at graph clustering have been proposed for problems ranging from sociology to biology. Here, we first employ a straightforward graph metric which has been used to identify protein interactions in small-world networks¹².

The clustering coefficient of a graph measures its cohesiveness. For an individual vertex, we count the number of links among its immediate neighbors and divide by the total number of possible links in this set. An average of this value for all vertices gives us the clustering coefficient. The mutual clustering coefficient extends this idea by measuring how many neighbors are shared between two vertices. For two vertices, we use a cumulative hypergeometric to generate a p value which corresponds to the likelihood of this or a more extreme shared configuration occuring by chance.

Formally, for tag groups a and b, we take the neighborhood of each (N(a), N(b)), the number of neighbors in common (N(a) ∩ N(b)), and the total number of tag groups in the graph (Total) we calculate:

The summation represents the p value cited above. Taking the negative log allows us to compare different p values more intuitively.

Higher mutual clustering coefficients signify a greater likelihood that a link should exist between two tag groups. This is useful in the situation we're addressing; a large number of shared neighbors among t and G should imply the existence of a similarity where the cosine algorithm has been unable to find one for reasons cited above.

We apply this measure to our graph, this time generating a neighborhood k by finding the tag groups with the highest mutual clustering coefficients to the target tag group t. Results for running this procedure for various k are listed in Table 5.

The algorithm delivers in one sense: we've gained in recall and satisfaction. However, we've paid a steep price in accuracy which serves to bring our overall MAP score down significantly. A closer look at our algorithm suggests a reason why: whereas the naive tag-based similarity network weighs document suggestions based directly on link weights, here we use link topology. In some sense, each individual similarity link here counts equally in the overall assessment of similarity. Once any type of similarity is encoded as a link, the weight of that link is ignored in the topological calculation above. Recall that the main strength of the naive tag-based algorithm was its ability to filter according to tag group size. This is precisely the information that is lost in the topological assessment above.

It is possible that the above algorithm could be modified to introduce a notion of threshold similarity and take link weight into account. That is, if we only include those links which are sufficiently powerful, we might recapture some of the lost information. Doing so would introduce another parameter to the model, though, one whose value appears on first glance to be somewhat subjective.

It is also possible that a version of mutual clustering coefficient which takes link weights into account directly might be applied. There have been efforts to establish a weighted version of the clustering coefficient, but to date no weighted version of the mutual clustering coefficient has been developed.

7.3 A random walk

In our discussion of recent developments above, we noted a method which has been used in networks to leverage the putative transitivity of trust relationships. A notion of spreading activation is used in Massa to model trust in online networks¹³. Most models of spreading activation resemble Hopfield networks. In these, nodes are binary threshold units which can be activated if input to the node exceeds some threshold. Nodes are connected with weighted links, and the weight of a given link determines how much input it provides to its target once it has been activated.

While a spreading activation model can be used to capture important qualities of a network based on transitive principles, its main benefit lies in its simplicity. Moreover, it is unclear why a binary mechanism better models a real-world notion of trust than would a conti- nous mechanism. Here, instead of a spreading activation model we employ a model which simulates a random walk on the similarity graph. This graph model allows for continuous similarity values in (0,1) and can be extended to an arbitrary number of steps.

We begin with the basic graph described above, which details the similarity of every tag group to every other within our user neighborhood. From this we generate n tag neighborhoods. Our first neighborhood is identical to the one we computed for the naive tag-based algorithm: it is generated by reading the top k similarities from the t row of the adjacency matrix. If we remove the (t,t) entry, and normalize the matrix to be row stochastic, this first set of tag groups can be seen as the most likely destinations in a one-step random walk from the t node in the similarity graph. Successive neighborhoods are simple matrix products: if we multiply the matrix generated in step n by the original (step 0), the t row gives us the probability of moving from t to each other tag group in the network in n + 1 steps. The probability of moving to a given tag group other than the target in n + 1 steps is added to the probability of moving to it in steps 1 through n and the tag groups with the highest probability scores overall are selected for the final neighborhood. Each document in a tag group receives that tag group's probability score (these are cumulative for each document across tag groups). The final suggestion list orders documents according to these scores.

With this algorithm, we have two parameters - one for the size of the initial user neightborhood, k, and one for the size of the final tag group neighborhood (with the highest probability scores) which we call z. We can also walk the graph for an arbitrary number of steps n. We try the algorithm for n = 2 and n = 3.

We first experiment with various values of (k,z) where n = 2 (Table 6). Here a random walk appears to perform better than mutual clustering. MAP scores for k = z are close to what they are for the hybrid model without network analysis, and accuracy scores are also roughly on a par. Unfortunately, we have not achieved a significant improvement in satisfaction or recall at k = z. When we move up to larger tag neighborhoods, we see that the accuracy diminishes to such an extent that the overall MAP score is decreased. While any practical algorithm for suggestion would not be based solely on this score, the benefit of MAP is that it puts a premium on accuracy - a metric which can be argued is more important than recall in a real system. As mentioned above, users are unlikely to peruse a list of suggestions beyond the first 20 or so items.

Our next results detail a walk with various values of (k,z) where n = 3 (Table 7). Our results for a 3- step random walk appear slightly worse than those for the 2-step. Satisfaction and recall are roughly similar, but accuracy is slightly lower for the 3-step walk. This is probably due to the fact that our 3-step algorithm tends to weigh tag groups which are further out in the similarity graph a fraction higher than those closer to the target. As a result, those documents are slightly overweighted, and will appear higher in the suggestion list than is probably warranted.

The results above show that we can achieve good recall without leaving the initial user neighborhood. However, doing so often involves increasing the size of the tag group neighborhood significantly, and with such an increase comes a decrease in accuracy. While the accuracy of both random walk models is significantly better than that of the naive user-based algorithm, high levels of recall (e.g. over .4) are associated with low list position (e.g. over 30). Again, in a practical situation this is probably not a reasonable tradeoff.

We believe there exists a remedy, however. Overall, our strategy has been to look at the data set in sets of successively smaller neighborhoods: we first confine the results to a group of similar users, then confine this set according to tag groups. The former strategy does not appear to severely limit the number of good recommendations that can be made, and the latter strategy helps to remove items which do not relate to the target user's area of interest. If we continue this approach, and look to characterize the resulting tag group network in terms of its items, it is possible that we can improve the accuracy of recommendations and so improve the overall MAP score. If, for example, we look to find those items which are most important for maintaining cohesiveness of the tag graph, it would make sense that these items appear higher in the suggestion list. Approaches similar to this one are reserved for future study.