Skip to main content
SpringerLink
Log in

An integrated model based on deep multimodal and rank learning for point-of-interest recommendation

  • Published:
World Wide Web Aims and scope Submit manuscript

Abstract

Modeling point-of-Interest (POI) for recommendations is vital in location-based social networks, yet it is a challenging task due to data sparsity and cold-start problems. Most existing approaches incorporate content features into a probabilistic matrix factorization model using unsupervised learning, which results in inaccuracy and weak robustness of recommendations when data is sparse, and the cold-start problems remain unsolved. In this paper, we propose a deep multimodal rank learning (DMRL) model that improves both the accuracy and robustness of POI recommendations. DMRL exploits temporal dynamics by allowing each user to have time-dependent preferences and captures geographical influences by introducing spatial regularization to the model. DMRL jointly learns ranking for personal preferences and supervised deep learning models to create a semantic representation of POIs from multimodal content. To make model optimization converge more rapidly while preserving high effectiveness, we develop a ranking-based dynamic sampling strategy to sample adverse or negative POIs for model training. We conduct experiments to compare our DMRL model with existing models that use different approaches using two large-scale datasets obtained from Foursquare and Yelp. The experimental results demonstrate the superiority of DMRL over the other models in creating cold-start POI recommendations and achieving excellent and highly robust results for different degrees of data sparsity.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. https://developer.foursquare.com/

References

  1. Guo, L., Yin, H., Wang, Q., Chen, T., Zhou, A., Hung, N.Q.V.: Streaming Session-based Recommendation. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. Pp.1569–1577. ACM Press, Anchorage, AK, USA (2019). https://doi.org/10.1145/3292500.3330839

  2. Chen, T., Yin, H., Chen, H., Yan, R., Nguyen, Q.V.H., Li, X.: AIR: Attentional Intention-Aware Recommender Systems. In: Proceedings of the 35th International Conference on Data Engineering. pp. 304–315. IEEE Press, Macao, China(2019). https://doi.org/10.1109/ICDE.2019.00035

  3. Wang, S., Hu, L., Wang, Y., Cao, L., Sheng, Q.Z., Orgun, M.: Sequential Recommender Systems: Challenges, Progress and Prospects. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp. 6332–6338. AAAI Press, Macao (2019)

    Google Scholar 

  4. Liu, Y., Pham, T.-A.N., Cong, G., Yuan, Q.: An experimental evaluation of point-of-interest recommendation in location-based social networks. Proc. VLDB Endow. 10, 1010–1021 (2017). https://doi.org/10.14778/3115404.3115407

    Article  Google Scholar 

  5. Li, J., Lu, K., Huang, Z., Shen, H.T.: Two Birds One Stone: On both Cold-Start and Long-Tail Recommendation. In: Proceedings of the 2017 ACM on Multimedia Conference, pp. 898–906. ACM Press, California (2017)

    Chapter  Google Scholar 

  6. Gao, H., Tang, J., Hu, X., Liu, H.: Content-aware point of interest recommendation on location-based social networks. In: Proceedings of the 29th AAAI conference on artificial intelligence. pp. 1721–1727.AAAI Press, Austin, Texas (2015)

  7. Guo, H., Li, H., He, M., Zhao,X.Y., Liu, G.Q., Xu GD.: Joint Factor Model with Content, Social, Location for Heterogeneous Point-of-Interest Recommendation. In: Proceedings of the International Conference on Knowledge Science, Engineering and Management. pp. 613–627. Springer Press, Passau, Germany (2016)

  8. Yin, H., Zhou, X., Cui, B., Wang, H., Zheng, K., Nguyen, Q.V.H.: Adapting to user interest drift for POI recommendation. IEEE Trans. Knowl. Data Eng. 28, 2566–2581 (2016). https://doi.org/10.1109/TKDE.2016.2580511

    Article  Google Scholar 

  9. Wang, H., Wang, N., Yeung, D.-Y.: Collaborative deep learning for recommender systems. In: Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1235–1244. ACM Press, Sydney (2015)

    Chapter  Google Scholar 

  10. Kim, D., Park, C., Oh, J., Lee, S., Yu, H.: Convolutional matrix factorization for document context-aware recommendation. In: Proceedings of the 10th ACM conference on recommender systems, pp. 233–240. ACM Press, Boston (2016)

    Chapter  Google Scholar 

  11. Zhang, Y., Yin, H., Huang, Z., Du, X., Yang, G., Lian, D.: Discrete deep learning for fast content-aware recommendation. In: Proceedings of the eleventh ACM international conference on web search and data mining, pp. 717–726. ACM Press, Marina Del Rey (2018)

    Chapter  Google Scholar 

  12. Yin, H., Wang, W., Wang, H., Chen, L., Zhou, X.: Spatial-aware hierarchical collaborative deep learning for POI recommendation. IEEE Trans. Knowl. Data Eng. 29, 2537–2551 (2017). https://doi.org/10.1109/TKDE.2017.2741484

    Article  Google Scholar 

  13. Liu, Q., Wu, S., Wang, L., Tan, T.: Predicting the Next Location: A Recurrent Model with Spatial and Temporal Contexts. In: Proceedings of the 30th AAAI Conference on Artificial Intelligence, pp. 194–200. AAAI Press, Phoenix (2016)

    Google Scholar 

  14. Liao, J., Liu, T., Liu, M., Wang, J., Wang, Y., Sun, H.: Multi-context integrated deep neural network model for next location prediction. IEEE Access. 6, 21980–21990 (2018). https://doi.org/10.1109/ACCESS.2018.2827422

    Article  Google Scholar 

  15. Cheng, C., Yang, H., King, I., Lyu, M.R.: Fused Matrix Factorization with Geographical and Social Influence in Location-Based Social Networks. In: Proceedings of the 26th AAAI Conference on Artificial Intelligence, pp. 17–23. AAAI Press, Toronto (2012)

    Google Scholar 

  16. Lian, D., Zhao, C., Xie, X., Sun, G., Chen, E., Rui, Y., GeoMF: Joint geographical modeling and matrix factorization for point-of-interest recommendation. In: Proceedings of the 20th ACM SIGKDD international conference on knowledge discovery and data mining, pp. 831–840. ACM Press, New York (2014)

    Chapter  Google Scholar 

  17. Liu, Y., Wei, W., Sun, A., Miao, C.: Exploiting Geographical Neighborhood Characteristics for Location Recommendation. In: Proceedings of the 23rd ACM International Conference on Information and Knowledge Management, pp. 739–748. ACM Press, Shanghai (2014)

    Google Scholar 

  18. Wang, H., Shen, H., Ouyang, W., Cheng, X.: Exploiting POI-Specific Geographical Influence for Point-of-Interest Recommendation. In: Proceedings of the 27th International Joint Conference on Artificial Intelligence. pp. 3877–3883. IJCAI Press, Stockholm (2018)

  19. Liu, B., Xiong, H., Papadimitriou, S., Fu, Y., Yao, Z.: A general geographical probabilistic factor model for point of interest recommendation. IEEE Trans. Knowl. Data Eng. 27, 1167–1179 (2015). https://doi.org/10.1109/TKDE.2014.2362525

    Article  Google Scholar 

  20. Li, X., Jiang, M., Hong, H., Liao, L.: A time-aware personalized point-of-interest recommendation via high-order tensor factorization. ACM Trans. Inf. Syst. 35, 1–23 (2017). https://doi.org/10.1145/3057283

    Article  Google Scholar 

  21. Yao, L., Sheng, Q.Z., Wang, X., Zhang, W.E., Qin, Y.: Collaborative location recommendation by integrating multi-dimensional contextual information. ACM Trans. Internet Technol. 18, 1–24 (2018). https://doi.org/10.1145/3134438

    Article  Google Scholar 

  22. Liu, B., Xiong, H.: Point-of-interest recommendation in location based social networks with topic and location awareness. In: Proceedings of the 2013 SIAM international conference on data mining, pp. 396–404. SIAM, Philadelphia (2013)

    Google Scholar 

  23. Li, X., Xu, G.D., Chen, E.H., Li, L.: MARS: A multi-aspect Recommender system for Point-of-Interest. In: Proceedings of the IEEE 31st International Conference on Data Engineering, pp. 1436–1439. IEEE press, South Korea (2015)

    Google Scholar 

  24. Wang, H., Fu, Y., Wang, Q., Yin, H., Du, C., Xiong, H.: A location-sentiment-aware recommender system for both home-town and out-of-town users. In: Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135–1143. ACM Press, Halifax (2017)

    Chapter  Google Scholar 

  25. Xie, M., Yin, H., Wang, H., Xu, F., Chen, W., Wang, S.: Learning Graph-based POI Embedding for Location-based Recommendation. In: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pp. 15–24. ACM Press, Indianapolis (2016)

    Chapter  Google Scholar 

  26. Zhang, F., Yuan, N.J., Lian, D., Xie, X., Ma, W.-Y.: Collaborative Knowledge Base embedding for recommender systems. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 353–362. ACM Press, San Francisco (2016)

    Chapter  Google Scholar 

  27. Wang, S., Wang, Y., Tang, J., Shu, K., Ranganath, S., Liu, H.: What your images reveal: exploiting visual contents for point-of-interest recommendation. In: Proceedings of the 26th international conference on world wide web, pp. 391–400. ACM Press, Perth (2017)

    Chapter  Google Scholar 

  28. Li, P., Burges, C.J.C., Wu, Q.: Learning to Rank Using Classification and Gradient Boosting. In: Advances in Neural Information Processing Systems 20. p. 10 (2008)

  29. Jing He, L.L., Xin Li: Category-aware Next Point-of-Interest Recommendation via Listwise Bayesian Personalized Ranking. In: Proceedings of the 26th International Joint Conference on Artificial Intelligence. pp. 1837–1843. IJCAI Press, Melbourne (2017)

  30. Rendle, S., Freudenthaler, C., Gantner, Z., Schmidt-Thieme, L.: BPR: Bayesian personalized ranking from implicit feedback. In: Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence, pp. 452–461. AUAI Press, Montreal (2009)

    Google Scholar 

  31. Rendle, S., Freudenthaler, C.: Improving pairwise learning for item recommendation from implicit feedback. In: Proceedings of the 7th ACM international conference on web search and data mining, pp. 273–282. ACM Press, New York (2014)

    Chapter  Google Scholar 

  32. Feng, S., Li, X., Zeng, Y., Cong, G., Chee, Y.M., Yuan, Q.: Personalized Ranking Metric Embedding for Next New POI Recommendation. In: Proceedings of the 24th International Joint Conference on Artificial Intelligence, pp. 2069–2075. AAAI Press, New York (2015)

    Google Scholar 

  33. Li, X., Cong, G., Li, X.-L., Pham, T.-A.N., Krishnaswamy, S.: Rank-GeoFM: A Ranking based Geographical Factorization Method for Point of Interest Recommendation. In: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 433–442. ACM Press, Santiago (2015)

    Chapter  Google Scholar 

  34. Zhao, S., Zhao, T., Yang, H., Lyu, M.R., King, I.: STELLAR: Spatial-Temporal Latent Ranking for Successive Point-of-Interest Recommendation. In: Proceedings of the 30th AAAI Conference on Artificial Intelligence, pp. 315–322. AAAI Press, Phoenix (2016)

    Google Scholar 

  35. Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., Ng, A.Y.: Multimodal Deep Learning. In: Proceedings of the 28th International Conference on Machine Learning. pp. 689–696. ACM Press, Bellevue (2011)

  36. Zheng, W.-L., Liu, W., Lu, Y., Lu, B.-L., Cichocki, A.: EmotionMeter: A Multimodal Framework for Recognizing Human Emotions. IEEE Trans. Cybern. 1–13 (2018). doi:https://doi.org/10.1109/TCYB.2018.2797176

  37. Wang, B., Yang, Y., Xu, X., Hanjalic, A., Shen, H.T.: Adversarial Cross-Modal Retrieval. In: Proceedings of the 2017 ACM on Multimedia Conference, pp. 154–162. ACM Press, Mountain View (2017)

    Chapter  Google Scholar 

  38. Oramas, S., Nieto, O., Sordo, M., Serra, X.: A Deep Multimodal Approach for Cold-start Music Recommendation. ArXiv170609739 Cs. 32–37 (2017). doi:https://doi.org/10.1145/3125486.3125492

  39. Zhao, Z., Yang, Q., Lu, H., Weninger, T., Cai, D., He, X., Zhuang, Y.: Social-aware movie recommendation via multimodal network learning. IEEE Trans. Multimed. 20, 430–440 (2018). https://doi.org/10.1109/TMM.2017.2740022

    Article  Google Scholar 

  40. Wang, X., Zhao, Y.-L., Nie, L., Gao, Y., Nie, W., Zha, Z.-J., Chua, T.-S.: Semantic-based location recommendation with multimodal venue semantics. IEEE Trans. Multimed. 17, 409–419 (2015). https://doi.org/10.1109/TMM.2014.2385473

    Article  Google Scholar 

  41. He, R., McAuley, J.: VBPR: visual Bayesian personalized ranking from implicit feedback. In: Proceedings of the 30th AAAI conference on artificial intelligence. pp. 144–150. AAAI Press, Phoenix (2016)

  42. Srivastava, N., Mansimov, E., Salakhutdinov, R.: Unsupervised learning of video representations using LSTMs. In: Proceedings of the 32nd international conference on machine learning. pp. 843–852. ACM Press, Lille (2015)

  43. Yin, H., Chen, H., Sun, X., Wang, H., Wang, Y., Nguyen, Q.V.H.: SPTF: A Scalable Probabilistic Tensor Factorization Model for Semantic-Aware Behavior Prediction. In: 2017 IEEE International Conference on Data Mining, pp. 585–594. IEEE, New Orleans (2017)

    Chapter  Google Scholar 

  44. Liu, Y., Yang, J.: Improving ranking-based recommendation by social information and negative similarity. Procedia Comput. Sci. 55, 732–740 (2015). https://doi.org/10.1016/j.procs.2015.07.164

    Article  Google Scholar 

  45. Yang, D., Zhang, D., Zheng, V.W., Yu, Z.: Modeling user activity preference by leveraging user spatial temporal characteristics in LBSNs. IEEE Trans. Syst. Man Cybern. Syst. 45, 129–142 (2015). https://doi.org/10.1109/TSMC.2014.2327053

    Article  Google Scholar 

  46. Yin, H., Zou, L., Nguyen, Q.V.H., Huang, Z., Zhou, X.: Joint Event-Partner Recommendation in Event-based Social Networks. In: Proceedings of the 34th IEEE International Conference on Data Engineering. pp.929-940. IEEE Press, Paris, France (2018)

  47. Wang, C., Blei, D.M.: Collaborative topic modeling for recommending scientific articles. In: Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 448–456. ACM Press, San Diego (2011)

    Chapter  Google Scholar 

  48. Mnih, A., Salakhutdinov, R.R.: Probabilistic matrix factorization. In: Proceedings of the 20th international conference on neural information processing systems. pp. 1257–1264. MIT Press, Vancouver (2007)

Download references

Acknowledgments

This work was supported in part by the National Key R&D Program of China 2020YFB1807800???2020YFB1807804, in part by the National Natural Science Foundation of China under Grants 62071067,62001054 and 61771068, in part by the Beijing Municipal Natural Science Foundation under Grant 4182041, and in part by the National Postdoctoral Program for Innovative Talents under Grant BX20200067, and in part by the Ministry of Education and China Mobile Joint Fund MCM20180101, and in part by the Beijing University of Posts and Telecommunications-China Mobile Research Institute Joint Innovation Center.

Author information

Authors and Affiliations

  1. State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China

    Jianxin Liao, Tongcun Liu, Jingyu Wang & Yulong Wang

  2. School of Information Technology and Electrical Engineering, University of Queensland, Brisbane, Australia

    Hongzhi Yin & Tong Chen

Authors
  1. Jianxin Liao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tongcun Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hongzhi Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jingyu Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yulong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Jianxin Liao or Tongcun Liu.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liao, J., Liu, T., Yin, H. et al. An integrated model based on deep multimodal and rank learning for point-of-interest recommendation. World Wide Web 24, 631–655 (2021). https://doi.org/10.1007/s11280-021-00865-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11280-021-00865-8

Keywords

Search

Navigation