ABSTRACT
Interaction data such as clicks and dwells provide valuable signals for learning and evaluating personalized models. However, while models of personalization typically distinguish between clicked and non-clicked results, no preference distinctions within the non-clicked results are made and all are treated as equally non-relevant.
In this paper, we demonstrate that failing to enforce a prior on preferences among non-clicked results leads to learning models that often personalize with no measurable gain at the risk that the personalized ranking is worse than the non-personalized ranking. To address this, we develop an implicit preference-based framework that enables learning highly selective rankers that yield large reductions in risk such as the percentage of queries personalized. We demonstrate theoretically how our framework can be derived from a small number of basic axioms that give rise to well-founded target rankings which combine a weight on prior preferences with the implicit preferences inferred from behavioral data.
Additionally, we conduct an empirical analysis to demonstrate that models learned with this approach yield comparable gains on click-based performance measures to standard methods with far fewer queries personalized. On three real-world commercial search engine logs, the method leads to substantial reductions in the number of queries re-ranked (2x - 7x fewer queries re-ranked) while maintaining 85-95% of the total gain achieved by the standard approach.
- E. Agichtein, E. Brill, and S. Dumais. Improving web search ranking by incorporating user behavior information. In Proc. SIGIR, 2006. Google ScholarDigital Library
- E. Agichtein, E. Brill, S. Dumais, and R. Ragno. Learning user interaction models for predicting web search result preferences. In Proc. SIGIR, 2006. Google ScholarDigital Library
- P. Bennett et al. Modeling the impact of short- and long-term behavior on search personalization. In Proc. SIGIR, 2012. Google ScholarDigital Library
- P. N. Bennett, F. Radlinski, R. W. White, and E. Yilmaz. Inferring and using location metadata to personalize web search. In Proc. SIGIR, 2011. Google ScholarDigital Library
- C. Burges, R. Ragno, and Q. Le. Learning to rank with non-smooth cost functions. In Proc. NIPS, 2006.Google Scholar
- C. J. C. Burges, K. M. Svore, P. N. Bennett, A. Pastusiak, and Q. Wu. Learning to rank using an ensemble of lambda-gradient models. JMLR, 14, 2011.Google Scholar
- O. Chapelle and Y. Zhang. A dynamic bayesian network click model for web search ranking. In Proc. WWW, 2009. Google ScholarDigital Library
- A. Chuklin, P. Serdyukov, and M. de Rijke. Click model-based information retrieval metrics. In Proc. SIGIR, 2013. Google ScholarDigital Library
- K. Collins-Thompson. Reducing the risk of query expansion via robust constrained optimization. In Proc. CIKM, 2009. Google ScholarDigital Library
- K. Collins-Thompson, P. N. Bennett, F. Diaz, C. Clarke, and E. M. Voorhees. Trec 2013 web track overview. In TREC '13, 2013.Google Scholar
- N. Craswell, O. Zoeter, M. Taylor, and B. Ramsey. An experimental comparison of click position-bias models. In Proc. WSDM, 2008. Google ScholarDigital Library
- B. T. Dinçer, C. Macdonald, and I. Ounis. Hypothesis testing for the risk-sensitive evaluation of retrieval systems. In Proc. SIGIR, 2014. Google ScholarDigital Library
- A. Dong et al. Time is of the essence: Improving recency ranking using twitter data. In Proc. WWW, 2010. Google ScholarDigital Library
- G. E. Dupret and B. Piwowarski. A user browsing model to predict search engine click data from past observations. In Proc. SIGIR, 2008. Google ScholarDigital Library
- H. Fang and C. Zhai. An exploration of axiomatic approaches to information retrieval. In Proc. SIGIR, 2005. Google ScholarDigital Library
- H. Fang and C. Zhai. Semantic term matching in axiomatic approaches to information retrieval. In Proc. SIGIR, 2006. Google ScholarDigital Library
- S. Fox et al. Evaluating implicit measures to improve web search. ACM Trans. Inf. Syst., 23 (2), Apr. 2005. Google ScholarDigital Library
- S. Gollapudi and A. Sharma. An axiomatic approach for result diversification. In Proc. WWW, 2009. Google ScholarDigital Library
- F. Guo, C. Liu, A. Kannan, T. Minka, M. Taylor, Y.-M. Wang, and C. Faloutsos. Click chain model in web search. In Proc. WWW, 2009. Google ScholarDigital Library
- K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of ir techniques. ACM Trans. Inf. Syst., 20 (4), Oct. 2002. Google ScholarDigital Library
- T. Joachims. Optimizing search engines using clickthrough data. In Proc. SIGKDD, 2002. Google ScholarDigital Library
- T. Joachims, L. Granka, B. Pan, H. Hembrooke, F. Radlinski, and G. Gay. Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search. ACM Trans. Inf. Syst., 25 (2), Apr. 2007. Google ScholarDigital Library
- T.-Y. Liu. Learning to rank for information retrieval. Foundations and Trends in Information Retrieval, 3 (3), 2009.Google Scholar
- F. Radlinski and T. Joachims. Query chains: Learning to rank from implicit feedback. In Proc. SIGKDD, 2005. Google ScholarDigital Library
- F. Radlinski, M. Kurup, and T. Joachims. How does clickthrough data reflect retrieval quality. In CIKM '08, 2008. Google ScholarDigital Library
- K. Raman, T. Joachims, P. Shivaswamy, and T. Shnabel. Stable coactive learning via perturbation. In Proc. ICML, 2013.Google Scholar
- X. Shen, B. Tan, and C. Zhai. Context-sensitive information retrieval using implicit feedback. In Proc. SIGIR, 2005. Google ScholarDigital Library
- P. Shivaswamy and T. Joachims. Online structured prediction via coactive learning. In Proc. ICML, 2012.Google Scholar
- M. Shokouhi. Learning to personalize query auto-completion. In Proc. SIGIR, 2013. Google ScholarDigital Library
- M. Shokouhi, R. W. White, P. Bennett, and F. Radlinski. Fighting search engine amnesia: Reranking repeated results. In Proc. SIGIR, 2013. Google ScholarDigital Library
- A. Sieg, B. Mobasher, and R. Burke. Web search personalization with ontological user profiles. In Proc. CIKM, 2007. Google ScholarDigital Library
- K. M. Svore, M. N. Volkovs, and C. J. Burges. Learning to rank with multiple objective functions. In Proc. WWW, 2011. Google ScholarDigital Library
- J. Teevan, S. T. Dumais, and D. J. Liebling. To personalize or not to personalize: modeling queries with variation in user intent. In Proc. SIGIR, 2008. Google ScholarDigital Library
- J. Teevan, D. J. Liebling, and G. Ravichandran Geetha. Understanding and predicting personal navigation. In Proc. WSDM, 2011. Google ScholarDigital Library
- Y. Ustinovskiy and P. Serdyukov. Personalization of web-search using short-term browsing context. In Proc. CIKM, 2013. Google ScholarDigital Library
- Y. Ustinovskiy, G. Gusev, and P. Serdyukov. An optimization framework for weighting implicit relevance labels for personalized web search. In Proc. WWW, 2015. Google ScholarDigital Library
- L. Wang, P. N. Bennett, and K. Collins-Thompson. Robust ranking models via risk-sensitive optimization. In Proc. SIGIR, 2012. Google ScholarDigital Library
- Q. Wu, C. Burges, K. Svore, and J. Gao. Adapting boosting for information retrieval measures. Journal of Information Retrieval, 2009. Google ScholarDigital Library
- B. Xiang, D. Jiang, J. Pei, X. Sun, E. Chen, and H. Li. Context-aware ranking in web search. In Proc. SIGIR, 2010. Google ScholarDigital Library
Index Terms
Implicit Preference Labels for Learning Highly Selective Personalized Rankers
Recommendations
Personalized re-ranking for recommendation
RecSys '19: Proceedings of the 13th ACM Conference on Recommender SystemsRanking is a core task in recommender systems, which aims at providing an ordered list of items to users. Typically, a ranking function is learned from the labeled dataset to optimize the global performance, which produces a ranking score for each ...
A large-scale evaluation and analysis of personalized search strategies
WWW '07: Proceedings of the 16th international conference on World Wide WebAlthough personalized search has been proposed for many years and many personalization strategies have been investigated, it is still unclear whether personalization is consistently effective on different queries for different users, and under different ...
Improving Ranking Consistency for Web Search by Leveraging a Knowledge Base and Search Logs
CIKM '15: Proceedings of the 24th ACM International on Conference on Information and Knowledge ManagementIn this paper, we propose a new idea called ranking consistency in web search. Relevance ranking is one of the biggest problems in creating an effective web search system. Given some queries with similar search intents, conventional approaches typically ...
Comments