skip to main content
research-article
Open Access

Practitioners Versus Users: A Value-Sensitive Evaluation of Current Industrial Recommender System Design

Published:11 November 2022Publication History
Skip Abstract Section

Abstract

Recommender systems are playing an increasingly important role in alleviating information overload and supporting users' various needs, e.g., consumption, socialization, and entertainment. However, limited research focuses on how values should be extensively considered in industrial deployments of recommender systems, the ignorance of which can be problematic. To fill this gap, in this paper, we adopt Value Sensitive Design to comprehensively explore how practitioners and users recognize different values of current industrial recommender systems. Based on conceptual and empirical investigations, we focus on five values: recommendation quality, privacy, transparency, fairness, and trustworthiness. We further conduct in-depth qualitative interviews with 20 users and 10 practitioners to delve into their opinions about these values. Our results reveal the existence and sources of tensions between practitioners and users in terms of value interpretation, evaluation, and practice, which provide novel implications for designing more human-centric and value-sensitive recommender systems.

References

  1. Himan Abdollahpouri, Gediminas Adomavicius, Robin Burke, Ido Guy, Dietmar Jannach, Toshihiro Kamishima, Jan Krasnodebski, and Luiz Pizzato. 2020a. Multistakeholder recommendation: Survey and research directions. User Modeling and User-Adapted Interaction , Vol. 30, 1 (2020), 127--158.Google ScholarGoogle ScholarCross RefCross Ref
  2. Himan Abdollahpouri and Robin Burke. 2019. Multi-stakeholder recommendation and its connection to multi-sided fairness. arXiv preprint arXiv:1907.13158 (2019).Google ScholarGoogle Scholar
  3. Himan Abdollahpouri, Robin Burke, and Bamshad Mobasher. 2017. Recommender Systems as Multistakeholder Environments. In Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization. Association for Computing Machinery, New York, NY, USA, 347--348.Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Himan Abdollahpouri, Masoud Mansoury, Robin Burke, and Bamshad Mobasher. 2020b. The Connection Between Popularity Bias, Calibration, and Fairness in Recommendation. In Fourteenth ACM Conference on Recommender Systems. Association for Computing Machinery, New York, NY, USA, 726--731.Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Esma A"imeur, Gilles Brassard, José M Fernandez, and Flavien Serge Mani Onana. 2008. A lambic: a privacy-preserving recommender system for electronic commerce. International Journal of Information Security, Vol. 7, 5 (2008), 307--334.Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Tamara Alsheikh, Jennifer A Rode, and Siân E Lindley. 2011. (Whose) Value-Sensitive design: a study of long-distance relationships in an Arabic cultural context. In Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work. Association for Computing Machinery, New York, NY, USA, 75--84.Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Oscar Alvarado, Vero Vanden Abeele, David Geerts, and Katrien Verbert. 2019. ?I really don't know what ?thumbs up' means": Algorithmic Experience in Movie Recommender Algorithms. In IFIP Conference on Human-Computer Interaction. Springer, 521--541.Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, et al. 2019. Guidelines for Human-AI Interaction. In Proceedings of the 2019 CHI conference on human factors in computing systems. Association for Computing Machinery, New York, NY, USA, 1--13.Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Morgan G Ames. 2016. Learning consumption: Media, literacy, and the legacy of One Laptop per Child. The information society , Vol. 32, 2 (2016), 85--97.Google ScholarGoogle Scholar
  10. Chris Anderson and Mia Poletto Andersson. 2004. Long tail. (2004).Google ScholarGoogle Scholar
  11. Arda Antikacioglu and R Ravi. 2017. Post processing recommender systems for diversity. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, New York, NY, USA, 707--716.Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Joeran Beel, Marcel Genzmehr, Stefan Langer, Andreas Nürnberger, and Bela Gipp. 2013. A comparative analysis of offline and online evaluations and discussion of research paper recommender system evaluation. In Proceedings of the international workshop on reproducibility and replication in recommender systems evaluation. Association for Computing Machinery, New York, NY, USA, 7--14.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Shlomo Berkovsky, Ronnie Taib, and Dan Conway. 2017. How to recommend? User trust factors in movie recommender systems. In Proceedings of the 22nd International Conference on Intelligent User Interfaces. Association for Computing Machinery, New York, NY, USA, 287--300.Google ScholarGoogle Scholar
  14. Arnaud Berlioz, Arik Friedman, Mohamed Ali Kaafar, Roksana Boreli, and Shlomo Berkovsky. 2015. Applying Differential Privacy to Matrix Factorization. In Proceedings of the 9th ACM Conference on Recommender Systems. Association for Computing Machinery, New York, NY, USA, 107--114.Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Alex Beutel, Jilin Chen, Tulsee Doshi, Hai Qian, Li Wei, Yi Wu, Lukasz Heldt, Zhe Zhao, Lichan Hong, Ed H Chi, et al. 2019. Fairness in Recommendation Ranking through Pairwise Comparisons. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. Association for Computing Machinery, New York, NY, USA, 2212--2220.Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Alan Borning, Batya Friedman, Janet Davis, and Peyina Lin. 2005. Informing public deliberation: Value sensitive design of indicators for a large-scale urban simulation. In ECSCW 2005. Springer Netherlands, Dordrecht, 449--468.Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Alan Borning and Michael Muller. 2012. Next steps for value sensitive design. In Proceedings of the SIGCHI conference on human factors in computing systems. Association for Computing Machinery, New York, NY, USA, 1125--1134.Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Robin Burke, Nasim Sonboli, and Aldo Ordonez-Gauger. 2018. Balanced neighborhoods for multi-sided fairness in recommendation. In Conference on Fairness, Accountability and Transparency. PMLR, 202--214.Google ScholarGoogle Scholar
  19. Pablo Castells, Neil J Hurley, and Saul Vargas. 2015. Novelty and diversity in recommender systems. In Recommender systems handbook. Springer, 881--918.Google ScholarGoogle Scholar
  20. Òscar Celma and Perfecto Herrera. 2008. A New Approach to Evaluating Novel Recommendations. In Proceedings of the 2008 ACM conference on Recommender systems. Association for Computing Machinery, New York, NY, USA, 179--186.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Dong-Kyu Chae, Jihoo Kim, Duen Horng Chau, and Sang-Wook Kim. 2020. AR-CF: Augmenting Virtual Users and Items in Collaborative Filtering for Addressing Cold-Start Problems. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. Association for Computing Machinery, New York, NY, USA, 1251--1260.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Abhijnan Chakraborty, Gourab K Patro, Niloy Ganguly, Krishna P Gummadi, and Patrick Loiseau. 2019. Equality of voice: Towards fair representation in crowdsourced top-k recommendations. In Proceedings of the Conference on Fairness, Accountability, and Transparency. Association for Computing Machinery, New York, NY, USA, 129--138.Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Allison JB Chaney, Brandon M Stewart, and Barbara E Engelhardt. 2018. How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility. In Proceedings of the 12th ACM Conference on Recommender Systems. Association for Computing Machinery, New York, NY, USA, 224--232.Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Shuo Chang, F Maxwell Harper, and Loren Gilbert Terveen. 2016. Crowd-based Personalized Natural Language Explanations for Recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems. Association for Computing Machinery, New York, NY, USA, 175--182.Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Chaochao Chen, Ziqi Liu, Peilin Zhao, Jun Zhou, and Xiaolong Li. 2018a. Privacy preserving point-of-interest recommendation using decentralized matrix factorization. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32.Google ScholarGoogle ScholarCross RefCross Ref
  26. Laming Chen, Guoxin Zhang, and Eric Zhou. 2018b. Fast greedy map inference for determinantal point process to improve recommendation diversity. In Advances in Neural Information Processing Systems. 5622--5633.Google ScholarGoogle Scholar
  27. Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. 2016. Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems. 7--10.Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Peizhe Cheng, Shuaiqiang Wang, Jun Ma, Jiankai Sun, and Hui Xiong. 2017. Learning to Recommend Accurate and Diverse Items. In Proceedings of the 26th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 183--192.Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Juliet Corbin and Anselm Strauss. 2014. Basics of qualitative research: Techniques and procedures for developing grounded theory. Sage publications.Google ScholarGoogle Scholar
  30. Henriette Cramer, Vanessa Evers, Satyan Ramlal, Maarten Van Someren, Lloyd Rutledge, Natalia Stash, Lora Aroyo, and Bob Wielinga. 2008. The effects of transparency on trust in and acceptance of a content-based art recommender. User Modeling and User-adapted interaction , Vol. 18, 5 (2008), 455.Google ScholarGoogle Scholar
  31. Alexei Czeskis, Ivayla Dermendjieva, Hussein Yapit, Alan Borning, Batya Friedman, Brian Gill, and Tadayoshi Kohno. 2010. Parenting from the pocket: Value tensions and technical directions for secure and private parent-teen mobile safety. In Proceedings of the sixth symposium on usable privacy and security. 1--15.Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Janet Davis and Lisa P Nathan. 2015. Value Sensitive Design: Applications, Adaptations, and Critiques. Handbook of Ethics, Values, and Technological Design: Sources, Theory, Values and Application Domains (2015), 11--40.Google ScholarGoogle Scholar
  33. Michael A DeVito, Darren Gergle, and Jeremy Birnholtz. 2017. "Algorithms ruin everything" # RIPTwitter, Folk Theories, and Resistance to Algorithmic Change in Social Media. In Proceedings of the 2017 CHI conference on human factors in computing systems. Association for Computing Machinery, New York, NY, USA, 3163--3174.Google ScholarGoogle Scholar
  34. Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. 2006. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography Conference. Springer, 265--284.Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Michael D Ekstrand, Mucun Tian, Ion Madrazo Azpiazu, Jennifer D Ekstrand, Oghenemaro Anuyah, David McNeill, and Maria Soledad Pera. 2018. All the cool kids, how do they fit in?: Popularity and demographic biases in recommender evaluation and effectiveness. In Conference on fairness, accountability and transparency. PMLR, 172--186.Google ScholarGoogle Scholar
  36. Michael D Ekstrand and Martijn C Willemsen. 2016. Behaviorism is not enough: better recommendations through listening to users. In Proceedings of the 10th ACM conference on recommender systems. Association for Computing Machinery, New York, NY, USA, 221--224.Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Motahhare Eslami, Karrie Karahalios, Christian Sandvig, Kristen Vaccaro, Aimee Rickman, Kevin Hamilton, and Alex Kirlik. 2016. First I "like" it, then I hide it: Folk Theories of Social Feeds. In Proceedings of the 2016 CHI conference on human factors in computing systems. Association for Computing Machinery, New York, NY, USA, 2371--2382.Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Motahhare Eslami, Kristen Vaccaro, Min Kyung Lee, Amit Elazari Bar On, Eric Gilbert, and Karrie Karahalios. 2019. User attitudes towards algorithmic opacity and transparency in online reviewing platforms. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1--14.Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Mary Flanagan, Daniel C Howe, and Helen Nissenbaum. 2005. Values at play: Design tradeoffs in socially-oriented game design. In Proceedings of the SIGCHI conference on human factors in computing systems. Association for Computing Machinery, New York, NY, USA, 751--760.Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Batya Friedman. 1996. Value-sensitive Design. Interactions, Vol. 3, 6 (1996), 16--23.Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Batya Friedman and David G Hendry. 2019. Value sensitive design: Shaping technology with moral imagination. MIT Press.Google ScholarGoogle Scholar
  42. Batya Friedman, Peter H Kahn, Alan Borning, and Alina Huldtgren. 2013. Value Sensitive Design and Information Systems. In Early Engagement and New Technologies: Opening up the Laboratory. Springer, 55--95.Google ScholarGoogle Scholar
  43. Batya Friedman and Peter H Kahn Jr. 1992. Human agency and responsible computing: Implications for computer system design. Journal of Systems and Software , Vol. 17, 1 (1992), 7--14.Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Batya Friedman, Peter H Kahn Jr, Jennifer Hagman, Rachel L Severson, and Brian Gill. 2006 a. The watcher and the watched: Social judgments about privacy in a public place. Human-Computer Interaction , Vol. 21, 2 (2006), 235--272.Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Batya Friedman, Ian Smith, Peter H Kahn, Sunny Consolvo, and Jaina Selawski. 2006 b. Development of a privacy addendum for open source licenses: Value Sensitive Design in industry. In International Conference on Ubiquitous Computing. Springer Berlin Heidelberg, Berlin, Heidelberg, 194--211.Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Zuohui Fu, Yikun Xian, Ruoyuan Gao, Jieyu Zhao, Qiaoying Huang, Yingqiang Ge, Shuyuan Xu, Shijie Geng, Chirag Shah, Yongfeng Zhang, et al. 2020. Fairness-aware explainable recommendation over knowledge graphs. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 69--78.Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Chen Gao, Chao Huang, Dongsheng Lin, Depeng Jin, and Yong Li. 2020. DPLCF: Differentially Private Local Collaborative Filtering. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. Association for Computing Machinery, New York, NY, USA, 961--970.Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Florent Garcin, Boi Faltings, Olivier Donatsch, Ayar Alazzawi, Christophe Bruttin, and Amr Huber. 2014. Offline and Online Evaluation of News Recommender Systems at Swissinfo.Ch. In Proceedings of the 8th ACM Conference on Recommender systems. Association for Computing Machinery, New York, NY, USA, 169--176.Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Mouzhi Ge, Carla Delgado-Battenfeld, and Dietmar Jannach. 2010. Beyond accuracy: evaluating recommender systems by coverage and serendipity. In Proceedings of the Fourth ACM Conference on Recommender Systems. Association for Computing Machinery, New York, NY, USA, 257--260.Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. David Gefen. 2002. Customer loyalty in e-commerce. Journal of the association for information systems, Vol. 3, 1 (2002), 2.Google ScholarGoogle ScholarCross RefCross Ref
  51. Mitchell L Gordon, Kaitlyn Zhou, Kayur Patel, Tatsunori Hashimoto, and Michael S Bernstein. 2021. The disagreement deconvolution: Bringing machine learning performance metrics in line with reality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1--14.Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Satya Keerthi Gorripati, Anupama Angadi, and Pedada Saraswathi. 2021. Chapter 7 - Recommender systems: security threats and mechanisms. In Security in IoT Social Networks, , Fadi Al-Turjman and B.D. Deebak (Eds.). Academic Press, 149--178.Google ScholarGoogle Scholar
  53. Ulrike Gretzel and Daniel R Fesenmaier. 2006. Persuasion in recommender systems. International Journal of Electronic Commerce, Vol. 11, 2 (2006), 81--100.Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. DeepFM: a factorization-machine based neural network for CTR prediction. arXiv preprint arXiv:1703.04247 (2017).Google ScholarGoogle Scholar
  55. Casper Hansen, Christian Hansen, Jakob Grue Simonsen, Stephen Alstrup, and Christina Lioma. 2020. Content-aware neural hashing for cold-start recommendation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. Association for Computing Machinery, New York, NY, USA, 971--980.Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. Jaron Harambam, Dimitrios Bountouridis, Mykola Makhortykh, and Joris Van Hoboken. 2019. Designing for the better by taking users into account: A qualitative evaluation of user control mechanisms in (news) recommender systems. In Proceedings of the 13th ACM Conference on Recommender Systems. Association for Computing Machinery, New York, NY, USA, 69--77.Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. Chen He, Denis Parra, and Katrien Verbert. 2016. Interactive recommender systems: A survey of the state of the art and future research challenges and opportunities. Expert Systems with Applications , Vol. 56 (2016), 9--27.Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural collaborative filtering. In Proceedings of the 26th international conference on world wide web. Association for Computing Machinery, New York, NY, USA, 173--182.Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé, Miro Dudik, and Hanna Wallach. 2019. Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need? Association for Computing Machinery, New York, NY, USA, 1--16.Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. Rong Hu and Pearl Pu. 2011. Enhancing recommendation diversity with organization interfaces. In Proceedings of the 16th international conference on intelligent user interfaces. Association for Computing Machinery, New York, NY, USA, 347--350.Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. Dietmar Jannach, Paul Resnick, Alexander Tuzhilin, and Markus Zanker. 2016. Recommender systems-beyond matrix completion. Commun. ACM, Vol. 59, 11 (2016), 94--102.Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. Arjan JP Jeckmans, Michael Beye, Zekeriya Erkin, Pieter Hartel, Reginald L Lagendijk, and Qiang Tang. 2013. Privacy in recommender systems. In Social media retrieval. Springer, 263--281.Google ScholarGoogle Scholar
  63. Michael Jugovac and Dietmar Jannach. 2017. Interacting with recommenders-overview and research directions. ACM Transactions on Interactive Intelligent Systems (TiiS), Vol. 7, 3 (2017), 1--46.Google ScholarGoogle ScholarDigital LibraryDigital Library
  64. Marius Kaminskas and Derek Bridge. 2016. Diversity, serendipity, novelty, and coverage: a survey and empirical analysis of beyond-accuracy objectives in recommender systems. ACM Transactions on Interactive Intelligent Systems (TiiS), Vol. 7, 1 (2016), 1--42.Google ScholarGoogle ScholarDigital LibraryDigital Library
  65. Bart P Knijnenburg and Martijn C Willemsen. 2015. Evaluating recommender systems with user experiments. In Recommender systems handbook. Springer, 309--352.Google ScholarGoogle Scholar
  66. Vinod Krishnan, Pradeep Kumar Narayanashetty, Mukesh Nathan, Richard T Davies, and Joseph A Konstan. 2008. Who Predicts Better? Results from an Online Study Comparing Humans and an Online Recommender System. In Proceedings of the 2008 ACM Conference on Recommender Systems. Association for Computing Machinery, New York, NY, USA, 211--218.Google ScholarGoogle Scholar
  67. Johannes Kunkel, Tim Donkers, Lisa Michael, Catalin-Mihai Barbu, and Jürgen Ziegler. 2019. Let me explain: impact of personal and impersonal explanations on trust in recommender systems. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1--12.Google ScholarGoogle ScholarDigital LibraryDigital Library
  68. Shyong Lam, Dan Frankowski, and John Riedl. 2006. Do you trust your recommendations? An exploration of security and privacy issues in recommender systems. In International Conference on Emerging Trends in Information and Communication Security. Springer, 14--29.Google ScholarGoogle ScholarDigital LibraryDigital Library
  69. Hui Li, Qiaowei Shen, and Yakov Bart. 2018. Local market characteristics and online-to-offline commerce: An empirical analysis of Groupon. Management Science, Vol. 64, 4 (2018), 1860--1878.Google ScholarGoogle ScholarDigital LibraryDigital Library
  70. Tong Li, Mingyang Zhang, Hancheng Cao, Yong Li, Sasu Tarkoma, and Pan Hui. 2020. " What Apps Did You Use?": Understanding the Long-term Evolution of Mobile App Usage. In Proceedings of The Web Conference 2020. Association for Computing Machinery, New York, NY, USA, 66--76.Google ScholarGoogle ScholarDigital LibraryDigital Library
  71. Benedikt Loepp, Tim Donkers, Timm Kleemann, and Jürgen Ziegler. 2019. Interactive recommending with Tag-Enhanced Matrix Factorization (TagMF). International Journal of Human-Computer Studies , Vol. 121 (2019), 21--41. Advances in Computer-Human Interaction for Recommender Systems.Google ScholarGoogle ScholarCross RefCross Ref
  72. Pasquale Lops, Marco De Gemmis, and Giovanni Semeraro. 2011. Content-based recommender systems: State of the art and trends. Recommender systems handbook (2011), 73--105.Google ScholarGoogle Scholar
  73. Jie Lu, Dianshuang Wu, Mingsong Mao, Wei Wang, and Guangquan Zhang. 2015. Recommender system application developments: a survey. Decision Support Systems , Vol. 74 (2015), 12--32.Google ScholarGoogle ScholarDigital LibraryDigital Library
  74. Alexandra Luccioni and Yoshua Bengio. 2020. On the morality of artificial intelligence [Commentary]. IEEE Technology and Society Magazine , Vol. 39, 1 (2020), 16--25.Google ScholarGoogle ScholarCross RefCross Ref
  75. Noëmi Manders-Huits. 2011. What values in design? The challenge of incorporating moral values into design. Science and engineering ethics , Vol. 17, 2 (2011), 271--287.Google ScholarGoogle Scholar
  76. Masoud Mansoury, Himan Abdollahpouri, Mykola Pechenizkiy, Bamshad Mobasher, and Robin Burke. 2020. Feedback loop and bias amplification in recommender systems. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management. Association for Computing Machinery, New York, NY, USA, 2145--2148.Google ScholarGoogle ScholarDigital LibraryDigital Library
  77. Paolo Massa and Paolo Avesani. 2007. Trust-aware recommender systems. In Proceedings of the 2007 ACM conference on Recommender systems. Association for Computing Machinery, New York, NY, USA, 17--24.Google ScholarGoogle ScholarDigital LibraryDigital Library
  78. Sean M McNee, John Riedl, and Joseph A Konstan. 2006. Being Accurate is Not Enough: How Accuracy Metrics Have Hurt Recommender Systems. In CHI'06 extended abstracts on Human factors in computing systems. Association for Computing Machinery, New York, NY, USA, 1097--1101.Google ScholarGoogle Scholar
  79. Frank McSherry and Ilya Mironov. 2009. Differentially Private Recommender Systems: Building Privacy into the Netflix Prize Contenders. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, New York, NY, USA, 627--636.Google ScholarGoogle ScholarDigital LibraryDigital Library
  80. Rishabh Mehrotra, James McInerney, Hugues Bouchard, Mounia Lalmas, and Fernando Diaz. 2018. Towards a Fair Marketplace: Counterfactual Evaluation of the Trade-off between Relevance, Fairness & Satisfaction in Recommendation Systems. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management. Association for Computing Machinery, New York, NY, USA, 2243--2251.Google ScholarGoogle ScholarDigital LibraryDigital Library
  81. Martijn Millecamp, Nyi Nyi Htun, Cristina Conati, and Katrien Verbert. 2020. What's in a User? Towards Personalising Transparency for Music Recommender Interfaces. In Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization (Genoa, Italy) (UMAP '20). Association for Computing Machinery, New York, NY, USA, 173--182.Google ScholarGoogle ScholarDigital LibraryDigital Library
  82. Jessica K Miller, Batya Friedman, Gavin Jancke, and Brian Gill. 2007. Value tensions in design: the value sensitive design, development, and appropriation of a corporation's groupware system. In Proceedings of the 2007 International ACM Conference on Supporting Group Work. Association for Computing Machinery, New York, NY, USA, 281--290.Google ScholarGoogle ScholarDigital LibraryDigital Library
  83. Lisa P Nathan, Batya Friedman, Predrag Klasnja, Shaun K Kane, and Jessica K Miller. 2008. Envisioning systemic effects on persons and society throughout interactive system design. In Proceedings of the 7th ACM conference on designing interactive systems. Association for Computing Machinery, New York, NY, USA, 1--10.Google ScholarGoogle ScholarDigital LibraryDigital Library
  84. Zahra Nazari, Christophe Charbuillet, Johan Pages, Martin Laurent, Denis Charrier, Briana Vecchione, and Ben Carterette. 2020. Recommending Podcasts for Cold-Start Users Based on Music Listening and Taste. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. Association for Computing Machinery, New York, NY, USA, 1041--1050.Google ScholarGoogle ScholarDigital LibraryDigital Library
  85. Thao Ngo, Johannes Kunkel, and Jürgen Ziegler. 2020. Exploring Mental Models for Transparent and Controllable Recommender Systems: A Qualitative Study. In Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization. Association for Computing Machinery, New York, NY, USA, 183--191.Google ScholarGoogle ScholarDigital LibraryDigital Library
  86. Tien T Nguyen, Pik-Mai Hui, F Maxwell Harper, Loren Terveen, and Joseph A Konstan. 2014. Exploring the filter bubble: the effect of using recommender systems on content diversity. In Proceedings of the 23rd international conference on world wide web. Association for Computing Machinery, New York, NY, USA, 677--686.Google ScholarGoogle ScholarDigital LibraryDigital Library
  87. John O'Donovan and Barry Smyth. 2005 a. Trust in recommender systems. In Proceedings of the 10th international conference on intelligent user interfaces. 167--174.Google ScholarGoogle ScholarDigital LibraryDigital Library
  88. John O'Donovan and Barry Smyth. 2005 b. Trust in Recommender Systems. In Proceedings of the 10th international conference on intelligent user interfaces. Association for Computing Machinery, New York, NY, USA, 167--174.Google ScholarGoogle ScholarDigital LibraryDigital Library
  89. Michael P O'Mahony, Neil J Hurley, and Guénolé CM Silvestre. 2005. Recommender systems: Attack types and strategies. In AAAI. 334--339.Google ScholarGoogle Scholar
  90. Pearl Pu and Li Chen. 2007. Trust-Inspiring Explanation Interfaces for Recommender Systems. Knowledge-Based Systems , Vol. 20, 6 (2007), 542--556.Google ScholarGoogle ScholarDigital LibraryDigital Library
  91. Pearl Pu, Li Chen, and Rong Hu. 2011. A User-Centric Evaluation Framework for Recommender Systems. In Proceedings of the Fifth ACM Conference on Recommender Systems. Association for Computing Machinery, New York, NY, USA, 157--164.Google ScholarGoogle ScholarDigital LibraryDigital Library
  92. Paul Resnick, Neophytos Iacovou, Mitesh Suchak, Peter Bergstrom, and John Riedl. 1994. Grouplens: An Open Architecture for Collaborative Filtering of Netnews. In Proceedings of the 1994 ACM conference on Computer Supported Cooperative Work. 175--186.Google ScholarGoogle ScholarDigital LibraryDigital Library
  93. Paul Resnick and Hal R Varian. 1997. Recommender systems. Commun. ACM, Vol. 40, 3 (1997), 56--58.Google ScholarGoogle ScholarDigital LibraryDigital Library
  94. Samantha Robertson, Tonya Nguyen, and Niloufar Salehi. 2021. Modeling Assumptions Clash with the Real World: Transparency, Equity, and Community Challenges for Student Assignment Algorithms. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1--14.Google ScholarGoogle ScholarDigital LibraryDigital Library
  95. Chaofeng Sha, Xiaowei Wu, and Junyu Niu. 2016. A Framework for Recommending Relevant and Diverse Items.. In IJCAI, Vol. 16. 3868--3874.Google ScholarGoogle Scholar
  96. Katie Shilton. 2013. Values levers: Building ethics into design. Science, Technology, & Human Values , Vol. 38, 3 (2013), 374--397.Google ScholarGoogle ScholarCross RefCross Ref
  97. Rashmi Sinha and Kirsten Swearingen. 2002. The role of transparency in recommender systems. In CHI'02 extended abstracts on Human factors in computing systems. Association for Computing Machinery, New York, NY, USA, 830--831.Google ScholarGoogle Scholar
  98. Donnavieve Smith, Satya Menon, and K. Sivakumar. 2005. Online Peer and Editorial Recommendations, Trust, and Choice in Virtual Markets. Journal of Interactive Marketing , Vol. 19 (06 2005), 15 -- 37.Google ScholarGoogle ScholarCross RefCross Ref
  99. Nasim Sonboli, Jessie J Smith, Florencia Cabral Berenfus, Robin Burke, and Casey Fiesler. 2021. Fairness and Transparency in Recommendation: The Users' Perspective. In Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization. Association for Computing Machinery, New York, NY, USA, 274--279.Google ScholarGoogle ScholarDigital LibraryDigital Library
  100. Anselm Strauss and Juliet M Corbin. 1997. Grounded theory in practice. Sage.Google ScholarGoogle Scholar
  101. Xiaoyuan Su and Taghi M Khoshgoftaar. 2009. A survey of collaborative filtering techniques. Advances in artificial intelligence , Vol. 2009 (2009).Google ScholarGoogle Scholar
  102. Kirsten Swearingen and Rashmi Sinha. 2001. Beyond algorithms: An HCI perspective on recommender systems. In ACM SIGIR 2001 workshop on recommender systems, Vol. 13. Citeseer, 1--11.Google ScholarGoogle Scholar
  103. Kirsten Swearingen and Rashmi Sinha. 2002. Interaction design for recommender systems. In Designing Interactive Systems, Vol. 6. Citeseer, 312--334.Google ScholarGoogle Scholar
  104. Katarzyna Tarnowska, Zbigniew W Ras, and Lynn Daniel. 2019. Recommender system for improving customer loyalty. Vol. 55. Springer.Google ScholarGoogle Scholar
  105. Nava Tintarev and Judith Masthoff. 2007. A survey of explanations in recommender systems. In 2007 IEEE 23rd international conference on data engineering workshop. IEEE, 801--810.Google ScholarGoogle ScholarDigital LibraryDigital Library
  106. Nava Tintarev and Judith Masthoff. 2011. Designing and evaluating explanations for recommender systems. In Recommender systems handbook. Springer, 479--510.Google ScholarGoogle Scholar
  107. Nava Tintarev and Judith Masthoff. 2015. Explaining recommendations: Design and evaluation. In Recommender systems handbook. Springer, 353--382.Google ScholarGoogle Scholar
  108. Saúl Vargas and Pablo Castells. 2011. Rank and relevance in novelty and diversity metrics for recommender systems. In Proceedings of the fifth ACM conference on recommender systems. Association for Computing Machinery, New York, NY, USA, 109--116.Google ScholarGoogle ScholarDigital LibraryDigital Library
  109. Patricia Victor, Martine De Cock, and Chris Cornelis. 2011. Trust and recommendations. In Recommender systems handbook. Springer, 645--675.Google ScholarGoogle Scholar
  110. Amy Voida, Lynn Dombrowski, Gillian R Hayes, and Melissa Mazmanian. 2014. Shared Values/conflicting Logics: Working around E-government Systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 3583--3592.Google ScholarGoogle ScholarDigital LibraryDigital Library
  111. Suhang Wang, Jiliang Tang, and Huan Liu. 2015. Toward Dual Roles of Users in Recommender Systems. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management. Association for Computing Machinery, New York, NY, USA, 1651--1660.Google ScholarGoogle ScholarDigital LibraryDigital Library
  112. Mark Wilhelm, Ajith Ramanathan, Alexander Bonomo, Sagar Jain, Ed H Chi, and Jennifer Gillenwater. 2018. Practical diversified recommendations on youtube with determinantal point processes. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management. Association for Computing Machinery, New York, NY, USA, 2165--2173.Google ScholarGoogle ScholarDigital LibraryDigital Library
  113. Jill Palzkill Woelfer, Amy Iverson, David G Hendry, Batya Friedman, and Brian T Gill. 2011. Improving the safety of homeless young people with mobile phones: Values, form and function. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1707--1716.Google ScholarGoogle ScholarDigital LibraryDigital Library
  114. Yuanbo Xu, Yongjian Yang, En Wang, Jiayu Han, Fuzhen Zhuang, Zhiwen Yu, and Hui Xiong. 2020. Neural Serendipity Recommendation: Exploring the Balance between Accuracy and Novelty with Sparse Explicit Feedback. ACM Transactions on Knowledge Discovery from Data (TKDD), Vol. 14, 4 (2020), 1--25.Google ScholarGoogle ScholarDigital LibraryDigital Library
  115. Qian Yang, Aaron Steinfeld, Carolyn Rosé, and John Zimmerman. 2020. Re-examining whether, why, and how human-AI interaction is uniquely difficult to design. In Proceedings of the 2020 chi conference on human factors in computing systems. Association for Computing Machinery, New York, NY, USA, 1--13.Google ScholarGoogle ScholarDigital LibraryDigital Library
  116. Daisy Yoo, Alina Huldtgren, Jill Palzkill Woelfer, David G Hendry, and Batya Friedman. 2013. A value sensitive action-reflection model: evolving a co-design space with stakeholder and designer prompts. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 419--428.Google ScholarGoogle ScholarDigital LibraryDigital Library
  117. Shuo Yu, Feng Xia, and Huan Liu. 2019. Academic team formulation based on Liebig's barrel: Discovery of anticask effect. IEEE Transactions on Computational Social Systems, Vol. 6, 5 (2019), 1083--1094.Google ScholarGoogle ScholarCross RefCross Ref
  118. Justin Zhan, Chia-Lung Hsieh, I-Cheng Wang, Tsan-Sheng Hsu, Churn-Jung Liau, and Da-Wei Wang. 2010. Privacy-preserving collaborative recommender systems. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), Vol. 40, 4 (2010), 472--476.Google ScholarGoogle ScholarDigital LibraryDigital Library
  119. Jingjing Zhang and Shawn P Curley. 2018. Exploring Explanation Effects on Consumers' Trust in Online Recommender Agents. International Journal of Human-Computer Interaction, Vol. 34, 5 (2018), 421--432.Google ScholarGoogle ScholarCross RefCross Ref
  120. Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. 2019. Deep learning based recommender system: A survey and new perspectives. ACM Computing Surveys (CSUR) , Vol. 52, 1 (2019), 1--38.Google ScholarGoogle ScholarDigital LibraryDigital Library
  121. Yongfeng Zhang and Xu Chen. 2018. Explainable recommendation: A survey and new perspectives. arXiv preprint arXiv:1804.11192 (2018).Google ScholarGoogle Scholar
  122. Yu Zheng, Chen Gao, Liang Chen, Depeng Jin, and Yong Li. 2021. DGCN: Diversified Recommendation with Graph Convolutional Networks. In Proceedings of the Web Conference 2021. Association for Computing Machinery, New York, NY, USA, 401--412.Google ScholarGoogle ScholarDigital LibraryDigital Library
  123. Tao Zhou, Zoltán Kuscsik, Jian-Guo Liu, Matúvs Medo, Joseph Rushton Wakeling, and Yi-Cheng Zhang. 2010. Solving the apparent diversity-accuracy dilemma of recommender systems. Proceedings of the National Academy of Sciences, Vol. 107, 10 (2010), 4511--4515.Google ScholarGoogle ScholarCross RefCross Ref
  124. Haiyi Zhu, Bowen Yu, Aaron Halfaker, and Loren Terveen. 2018b. Value-sensitive algorithm design: Method, case study, and lessons. Proceedings of the ACM on Human-Computer Interaction, Vol. 2, CSCW (2018), 1--23.Google ScholarGoogle ScholarDigital LibraryDigital Library
  125. Ziwei Zhu, Xia Hu, and James Caverlee. 2018a. Fairness-aware Tensor-based Recommendation. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management. Association for Computing Machinery, New York, NY, USA, 1153--1162.Google ScholarGoogle ScholarDigital LibraryDigital Library
  126. Ziwei Zhu, Shahin Sefati, Parsa Saadatpanah, and James Caverlee. 2020. Recommendation for new users and new items via randomized training and mixture-of-experts transformation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. Association for Computing Machinery, New York, NY, USA, 1121--1130.Google ScholarGoogle ScholarDigital LibraryDigital Library
  127. Cai-Nicolas Ziegler, Sean M McNee, Joseph A Konstan, and Georg Lausen. 2005. Improving recommendation lists through topic diversification. In Proceedings of the 14th international conference on World Wide Web. Association for Computing Machinery, New York, NY, USA, 22--32.Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Practitioners Versus Users: A Value-Sensitive Evaluation of Current Industrial Recommender System Design

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader