Skip to main content
SpringerLink
Log in

Style-aware adversarial pairwise ranking for image recommendation systems

  • Regular Paper
  • Published:
International Journal of Multimedia Information Retrieval Aims and scope Submit manuscript

Abstract

The vulnerability of Machine Learning (ML) models to adversarial attack and their prominence pose security issues, notably in image recommendation systems. The adversarial training method is an excellent strategy for improving the generalization capacity of ML models by creating attacks in the embedding space during training. While there has been a plethora of testing on image recommendation system vulnerabilities and defenses, iterative adversarial training methodologies have received little attention. Furthermore, when browsing visual images, consumers are more interested in the content and how well the image style matches the content. However, when compared to image content, the impact of image styles on the adversarial recommendation community has rarely been examined. In this work, we propose a robust Adversarial Content and Style Bayesian Personalized Ranking (ACSBPR) approach that leverages content and style features for image recommendation. The ACSBPR technique makes three significant contributions: (1) Incorporate content and style features jointly for image recommendation. (2) Present a multi-objective pairwise ranking with Dynamic Negative Sampling to optimize the system and anticipate consumer preferences. (3) To reduce the influence of the attack, we train the ACSBPR objective function using minimax iterative adversarial training. Extensive investigations on the Flickr dataset demonstrate that our strategy achieves better performance when compared to state-of-the-art image recommendation models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Algorithm 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

References

  1. Rendle S, Freudenthaler C, Gantner Z, Schmidt-Thieme L (2009) BPR: Bayesian personalized ranking from implicit feedback. In: Proceedings of the 25th conference on uncertainty in artificial intelligence, pp 452–461

  2. He X, Zhang H, Kan MY, Chua TS (2016) Fast matrix factorization for online recommendation with implicit feedback. In: Proceedings of the 39th international ACM SIGIR conference on research and development in information retrieval, pp 549–558

  3. Kabbur S, Ning X, Karypis G (2013) Fism: factored item similarity models for top-n recommender systems. In: The 19th ACM SIGKDD international conference on knowledge discovery and data mining, pp 659–667

  4. Xue F, He X, Wang X, Xu J, Liu K, Hong R (2019) Deep item-based collaborative filtering for top-n recommendation. ACM Trans Inf Syst 37(3):1–25

    Article  Google Scholar 

  5. He R, McAuley J (2016) VBPR: visual Bayesian personalized ranking from implicit feedback. In: Proceedings of the 30th AAAI conference on artificial intelligence, pp 144–150

  6. He R, Lin C, Wang J, Mcauley J (2016) Sherlock: sparse hierarchical embeddings for visually-aware one-class collaborative filtering. In: Proceedings of the 25th international joint conference on artificial intelligence, pp 3740–3746

  7. He R, McAuley J (2016) Ups and downs: modeling the visual evolution of fashion trends with one-class collaborative filtering. In: Proceedings of the 25th international conference on world wide web, pp 507–517

  8. Kang W C, Fang C, Wang Z, McAuley J J (2017) Visually-aware fashion recommendation and design with generative image models. In: IEEE international conference on data mining, pp 207–216

  9. Yu W, Zhang H, He X, Chen X, Xiong L, Qin Z (2018) Aesthetic-based clothing recommendation. In: Proceedings of the 2018 world wide web conference on world wide web, pp 649–658

  10. Liu Q, Wu S, Wang L (2017) DeepStyle: learning user preferences for visual recommendation. In: Proceedings of the 25th international conference on world wide web, pp 841–844

  11. He M, Zhang S, Meng Q (2019) Learning to style-aware Bayesian personalized ranking for visual recommendation. IEEE Access 7:14198–14205

    Article  Google Scholar 

  12. Luo S, Chen L, Wu L (2019) Learning from visual content and style: an image-enhanced recommendation model. CCF Trans Pervasive Comput Interact 1(4):275–284

    Article  Google Scholar 

  13. Deldjoo Y, Noia TD, Daniele Malitesta D, Merra FA (2022) Leveraging content-style item representation for visual recommendation. In: 44th European conference on IR research, pp 84–92

  14. Huang L, Joseph AD, Nelson B, Rubinstein BIP, Tygar JD (2011) Adversarial machine learning. In: Proceedings of the 4th ACM workshop on security and artificial intelligence, pp 43–58

  15. Akhtar N, Mian AS (2018) Threat of adversarial attacks on deep learning in computer vision: a survey. IEEE Access 6:14410–14430

    Article  Google Scholar 

  16. Zhang WE, Sheng QZ, Alhazmi A, Li C (2020) Adversarial attacks on deep-learning models in natural language processing: a survey. ACM Trans Intell Syst Technol 11(3):1–41

    Google Scholar 

  17. Deldjoo Y, Noia TD, Merra FA (2020) Adversarial machine learning in recommender systems (AML-RecSys). In: The 13th ACM international conference on web search and data mining, pp 869–872

  18. Goodfellow IJ, Shlens J, Szegedy C (2015) Explaining and harnessing adversarial examples. arXiv:1412.6572

  19. Kurakin A, Goodfellow IJ, Bengio S (2017) Adversarial examples in the physical world. arXiv:1607.02533

  20. Madry A, Makelov A, Schmidt L, Tsipras D, Vladu A (2017) Towards deep learning models resistant to adversarial attacks. arXiv:1706.06083

  21. He X, He Z, Du X, Chua TS (2018) Adversarial personalized ranking for recommendation. In: The 41st international ACM SIGIR conference on research and development in information retrieval, pp 355–364

  22. Anelli VW, Bellogín A, Deldjoo Y, Noia T D, and Merra F A (2021) Msap: multi-step adversarial perturbations on recommender systems embeddings. In: Proceedings of the thirty-fourth international Florida artificial intelligence research society. https://doi.org/10.32473/flairs.v34i1.128443

  23. Tang J, Du X, He X, Yuan F, Tian Q, Chua TS (2018) Adversarial training towards robust multimedia recommender system. IEEE Trans Knowl Data Eng 32(5):1–1

    Google Scholar 

  24. Di Noia T, Malitesta D, Merra F A (2020) TAaMR: targeted adversarial attack against multimedia recommender systems. In: DSN Workshops, pp 1–8

  25. Agyemang P, Wu Z, Liu K, Gong S (2022) Robust multi-objective visual Bayesian personalized ranking for multimedia recommendation. Appl Intell 52(4):3499–3510

    Article  Google Scholar 

  26. Zhang W, Chen T, Wang J, Yu Y (2013) Optimizing top-n collaborative filtering via dynamic negative item sampling. In: The 36th international ACM SIGIR conference on research and development in information retrieval, pp 785–788

  27. Hu Y, Koren Y, Volinsky C (2008) Collaborative filtering for implicit feedback datasets. In: Proceedings of the 8th IEEE international conference on data mining, pp 263–272

  28. He X, Liao L, Zhang H, Nie L, Hu X, Chua TS (2017) Neural collaborative filtering. In: Proceedings of the 26th international conference on world wide web, pp 173–182

  29. Ding J, Feng F, He X, Yu G, Li Y, Jin D (2018) An improved sampler for Bayesian personalized ranking by leveraging view data. In: Companion of the the web conference 2018 on the web conference, pp 13–14

  30. Wang X, He X, Nie L, Chua TS (2017) Item silk road: recommending items from information domains to social users. In: Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval, pp 185–194

  31. Al-Halah Z, Stiefelhagen R, Grauman K (2017) Fashion forward: forecasting visual style in fashion. In: IEEE international conference on computer vision, pp 388–397

  32. Han X, Wu Z, Jiang YG, Davis LS (2017) Learning fashion compatibility with bidirectional LSTMs. In: Proceedings of the 2017 ACM on multimedia conference, pp 1078–1086

  33. Li Y, Cao L, Zhu J, Luo J (2017) Mining fashion outfit composition using an end-to-end deep learning approach on set data. IEEE Trans Multimed 19(8):1946–1955

    Article  Google Scholar 

  34. Kiapour MH, Yamaguchi K, Berg AC, Berg TL (2014) Hipster wars: discovering elements of fashion styles. In: Proceedings of the European conference on computer vision, pp 472–488

  35. Veit A, Kovacs B, Bell S, Mcauley J, Bala K, Belongie S (2015) Learning visual clothing style with heterogeneous dyadic co-occurrences. In: Proceedings of the IEEE international conference on computer vision, pp 4642–4650

  36. Simoserra E, Ishikawa H (2016) Fashion style in 128 floats: joint ranking and classification using weak data for feature extraction. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 298–307

  37. Deldjoo Y, Noia TD, Merra FA (2019) Assessing the impact of a user-item collaborative attack on class of users. arXiv:1908.07968

  38. Anelli VW, Deldjoo Y, Noia TD, Sciascio ED, Merra FA (2020) Sasha: semantic-aware shilling attacks on recommender systems exploiting knowledge graphs. In: The semantic web-17th international conference, pp 307–323

  39. Fang M, Gong NZ, Liu J (2020) Influence function based data poisoning attacks to top-n recommender systems. In: The web conference (WWW), pp 3019–3025

  40. Fang M, Yang G, Gong NZ, Liu J (2018) Poisoning attacks to graph-based recommender systems. In: Proceedings of the 34th annual computer security applications conference, pp 381–392

  41. Lam SK, Riedl J (2004) Shilling recommender systems for fun and profit. In: Proceedings of the 13th international conference on world wide web, pp 393–402

  42. O’Mahony MP, Hurley NJ, Kushmerick N, Silvestre GCM (2004) Collaborative recommendation: a robustness analysis. ACM Trans Internet Technol 4(4):344–377

    Article  Google Scholar 

  43. Chen H, Li J (2019) Adversarial tensor factorization for context aware recommendation. In: Proceedings of the 13th ACM conference on recommender systems, pp 363–367

  44. Yuan F, Yao L, Benatallah B (2019) Adversarial collaborative neural network for robust recommendation. In: Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval, pp 1065–1068

  45. Du Y, Fang M, Yi J, Xu C, Cheng J, Tao D (2019) Enhancing the robustness of neural collaborative filtering systems under malicious attacks. IEEE Trans Multimed 21(3):555–565

    Article  Google Scholar 

  46. Rafailidis D, Crestani F (2019) Adversarial training for review-based recommendations. In: Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval, pp 1057–1060

  47. Agyemang P, Zhao X, Fang L, Wu Z (2021) Ownership recommendation via iterative adversarial training. Neural Process Lett 54(1):637–655

    Google Scholar 

  48. Agyemang P, Wu Z, Liu K, Gong S (2022) Personalized recommendation: from clothing to academic. Multimed Tools Appl 81(10):14573–14588

    Article  Google Scholar 

  49. Rendle S, Freudenthaler C (2014) Improving pairwise learning for item recommendation from implicit feedback. In: 7th ACM international conference on web search and data mining, pp 273–282

Download references

Acknowledgements

This research was supported by Zhejiang Provincial Natural Science Foundation of China under Grant No. LZ22F010005.

Author information

Authors and Affiliations

  1. College of Information Engineering, Zhejiang University of Technology, Hangzhou, 310023, People’s Republic of China

    Zhefu Wu, Song Zhang, Agyemang Paul & Luping Fang

Authors
  1. Zhefu Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Song Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Agyemang Paul
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Luping Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Luping Fang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, Z., Zhang, S., Paul, A. et al. Style-aware adversarial pairwise ranking for image recommendation systems. Int J Multimed Info Retr 12, 22 (2023). https://doi.org/10.1007/s13735-023-00295-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s13735-023-00295-4

Keywords

Search

Navigation