Abstract
Reinforcement learning has been widely used in recommender systems in order to optimize users’ long-term utilities. An accurate and explainable user simulator is crucial for reinforcement learning based recommendation, as an online interactive environment is often unavailable. On short video platforms, it is very important to keep users on the platform as long as possible in each session. Thus, session-based user utilities depend on two factors: how much users like every single video (video preference) and the number of videos watched (video views) in each session. To this end, the simulator should simultaneously model the user’s degree of liking for each video and video views. However, most previous studies on the short video recommendation only paid attention to the former. In this work, we propose KESWA, a Knowledge-Enhanced Session-Wide Attention method for short video user simulation. KESWA fuses information foraging theory with a deep learning model for both video preference and video views modeling, providing an explainable prediction for users’ staying and leaving behavior. Comparative experiments demonstrate that KESWA provides a better simulation of video views compared with existing models. Meanwhile, reinforcement learning agents can achieve higher session-based user utilities trained by KESWA than by other user simulators.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Azzopardi, L., Thomas, P., Craswell, N.: Measuring the utility of search engine result pages: an information foraging based measure. In: The 41st international ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 605–614 (2018)
Cai, Q., et al.: Constrained reinforcement learning for short video recommendation. arXiv preprint arXiv:2205.13248 (2022)
Chen, X., Li, S., Li, H., Jiang, S., Qi, Y., Song, L.: Generative adversarial user model for reinforcement learning based recommendation system. In: International Conference on Machine Learning, pp. 1052–1061. PMLR (2019)
Fujimoto, S., Meger, D., Precup, D.: Off-policy deep reinforcement learning without exploration. In: International Conference on Machine Learning, pp. 2052–2062. PMLR (2019)
Li, D., Li, X., Wang, J., Li, P.: Video recommendation with multi-gate mixture of experts soft actor critic. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1553–1556 (2020)
Li, J., Ren, P., Chen, Z., Ren, Z., Lian, T., Ma, J.: Neural attentive session-based recommendation. In: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pp. 1419–1428 (2017)
Liu, S., Chen, Z., Liu, H., Hu, X.: User-video co-attention network for personalized micro-video recommendation. In: The World Wide Web Conference, pp. 3020–3026 (2019)
Liu, Y., Lyu, C., Liu, Z., Tao, D.: Building effective short video recommendation. In: 2019 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pp. 651–656. IEEE (2019)
Pirolli, P., Card, S.: Information foraging in information access environments. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 51–58 (1995)
Rendle, S., Freudenthaler, C., Gantner, Z., Schmidt-Thieme, L.: BPR: Bayesian personalized ranking from implicit feedback. arXiv preprint arXiv:1205.2618 (2012)
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)
Shi, J.C., Yu, Y., Da, Q., Chen, S.Y., Zeng, A.X.: Virtual-Taobao: virtualizing real-world online retail environment for reinforcement learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 4902–4909 (2019)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Zheng, G., et al.: DRN: a deep reinforcement learning framework for news recommendation. In: Proceedings of the 2018 World Wide Web Conference, pp. 167–176 (2018)
Zou, L., et al.: Pseudo Dyna-Q: a reinforcement learning framework for interactive recommendation. In: Proceedings of the 13th International Conference on Web Search and Data Mining, pp. 816–824 (2020)
Acknowledgement
This work was supported by the National Social Science Major Program under grant number 20 &ZD161.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Yang, Z., Liu, H. (2023). Staying or Leaving: A Knowledge-Enhanced User Simulator for Reinforcement Learning Based Short Video Recommendation. In: Kashima, H., Ide, T., Peng, WC. (eds) Advances in Knowledge Discovery and Data Mining. PAKDD 2023. Lecture Notes in Computer Science(), vol 13937. Springer, Cham. https://doi.org/10.1007/978-3-031-33380-4_30
Download citation
DOI: https://doi.org/10.1007/978-3-031-33380-4_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-33379-8
Online ISBN: 978-3-031-33380-4
eBook Packages: Computer ScienceComputer Science (R0)