Abstract
The Probability Ranking Principle (PRP) ranks search results based on their expected utility derived solely from document contents, often overlooking the nuances of presentation and user interaction. However, with the evolution of Search Engine Result Pages (SERPs), now comprising a variety of result cards, the manner in which these results are presented is pivotal in influencing user engagement and satisfaction. This shift prompts the question: How does the PRP and its user-centric counterpart, the Interactive Probability Ranking Principle (iPRP), compare in the context of these heterogeneous SERPs? Our study draws a comparison between the PRP and the iPRP, revealing significant differences in their output. The iPRP, accounting for item-specific costs and interaction probabilities to determine the “Expected Perceived Utility” (EPU), yields different result orderings compared to the PRP. We evaluate the effect of the EPU on the ordering of results by observing changes in the ranking within a heterogeneous SERP compared to the traditional “ten blue links”. We find that changing the presentation affects the ranking of items according to the (iPRP) by up to 48% (with respect to DCG, TBG and RBO) in ad-hoc search tasks on the TREC WaPo Collection. This work suggests that the iPRP should be employed when ranking heterogeneous SERPs to provide a user-centric ranking that adapts the ordering based on the presentation and user engagement.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Azzopardi, L., Thomas, P., Craswell, N.: Measuring the utility of search engine result pages: an information foraging based measure. In: 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018, pp. 605–614, June 2018. https://doi.org/10.1145/3209978.3210027, https://dl.acm.org/doi/10.1145/3209978.3210027
Azzopardi, L., Zuccon, G.: An analysis of theories of search and search behavior. In: ICTIR 2015 - Proceedings of the 2015 ACM SIGIR International Conference on the Theory of Information Retrieval (2015). https://doi.org/10.1145/2808194.2809447
Bota, H., Zhou, K., Jose, J.M.: Playing your cards right: the effect of entity cards on search behaviour and workload. In: CHIIR 2016 - Proceedings of the 2016 ACM Conference on Human Information Interaction and Retrieval, pp. 131–140, March 2016. https://doi.org/10.1145/2854946.2854967, https://dl.acm.org/doi/10.1145/2854946.2854967
Chierichetti, F., Kumar, R., Raghavan, P.: Optimizing two-dimensional search results presentation. In: Proceedings of the 4th ACM International Conference on Web Search and Data Mining, WSDM 2011, pp. 257–266 (2011). https://doi.org/10.1145/1935826.1935873
Cutrell, E., Guan, Z.: What are you looking for? An eye-tracking study of information usage in Web search. In: Conference on Human Factors in Computing Systems - Proceedings (2007). https://doi.org/10.1145/1240624.1240690
Dziadosz, S., Chandrasekar, R.: Do thumbnail previews help users make better relevance decisions about web search results? In: SIGIR Forum (ACM Special Interest Group on Information Retrieval) (2002). https://doi.org/10.1145/564437.564446
Fuhr, N.: A probability ranking principle for interactive information retrieval. Inf. Retrieval 11(3), 251–265 (2008). https://doi.org/10.1007/s10791-008-9045-0
Guo, Q., Agichtein, E.: Beyond dwell time: estimating document relevance from cursor movements and other post-click searcher behavior. In: WWW 2012 - Proceedings of the 21st Annual Conference on World Wide Web, pp. 569–578 (2012). https://doi.org/10.1145/2187836.2187914, https://dl.acm.org/doi/10.1145/2187836.2187914
Joho, H., Jose, J.M.: A comparative study of the effectiveness of search result presentation on the Web. In: Lalmas, M., MacFarlane, A., Rüger, S., Tombros, A., Tsikrika, T., Yavlinsky, A. (eds.) Advances in Information Retrieval, ECIR 2006. LNCS, vol. 3936. Springer, Heidelberg (2006). https://doi.org/10.1007/11735106_27
Kim, Y., Hassan, A., White, R.W., Zitouni, I.: Modeling dwell time to predict click-level satisfaction. In: WSDM 2014 - Proceedings of the 7th ACM International Conference on Web Search and Data Mining, pp. 193–202 (2014). https://doi.org/10.1145/2556195.2556220, https://dl.acm.org/doi/10.1145/2556195.2556220
Maxwell, D., Azzopardi, L., Moshfeghi, Y.: A study of snippet length and informativeness behaviour, performance and user experience. In: SIGIR 2017 - Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 135–144, August 2017. https://doi.org/10.1145/3077136.3080824
Rele, R.S., Duchowski, A.T.: Using eye tracking to evaluate alternative search results interfaces. In: Proceedings of the Human Factors and Ergonomics Society (2005). https://doi.org/10.1177/154193120504901508
Robertson, S.E.: The probability ranking principle in IR (1977). https://doi.org/10.1108/eb026647
Smucker, M.D., Clarke, C.L.A.: Time-based calibration of effectiveness measures. In: SIGIR 2012 - Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 95–104 (2012). https://doi.org/10.1145/2348283.2348300
Teevan, J., et al.: Visual snippets: summarizing web pages for search and revisitation. In: Conference on Human Factors in Computing Systems - Proceedings (2009). https://doi.org/10.1145/1518701.1519008
Tombros, A., Sanderson, M.: Advantages of query biased summaries in information retrieval. In: SIGIR Forum (ACM Special Interest Group on Information Retrieval) (1998). https://doi.org/10.1145/290941.290947
Wang, Y., et al.: Beyond ranking: optimizing whole-page presentation. In: WSDM 2016 - Proceedings of the 9th ACM International Conference on Web Search and Data Mining, pp. 103–112, February 2016. https://doi.org/10.1145/2835776.2835824, https://app.litmaps.com
Webber, W., Moffat, A., Zobel, J.: A similarity measure for indefinite rankings. ACM Trans. Inf. Syst. 28(4), 1–38 (2010). https://doi.org/10.1145/1852102.1852106
Zhang, Y., Zhai, C.: Information retrieval as card playing: a formal model for optimizing interactive retrieval interface. In: SIGIR 2015 - Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (2015). https://doi.org/10.1145/2766462.2767761
Acknowledgements
We want to thank the reviewers for their insightful suggestions and feedback and all the participants who took part in the study. The work reported here is funded by the DoSSIER project under European Union’s Horizon 2020 research and innovation program, Marie Skłodowska-Curie grant agreement No. 860721.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Pathak, K., Azzopardi, L., Halvey, M. (2024). Ranking Heterogeneous Search Result Pages Using the Interactive Probability Ranking Principle. In: Goharian, N., et al. Advances in Information Retrieval. ECIR 2024. Lecture Notes in Computer Science, vol 14609. Springer, Cham. https://doi.org/10.1007/978-3-031-56060-6_7
Download citation
DOI: https://doi.org/10.1007/978-3-031-56060-6_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-56059-0
Online ISBN: 978-3-031-56060-6
eBook Packages: Computer ScienceComputer Science (R0)