Abstract
In this paper, we present an evaluation of a social adaptive website in the domain of cultural events, iCITY DSA, which provides information about cultural resources and events that promote the cultural heritage in the city of Turin. Using this evaluation, our objective was to investigate the actual usage of a social adaptive website, in an effort to discover the real behavior of users, the unforeseen correlations among user actions and the consequent interactive behavior, the accuracy of both system and social recommendations and their impact on the users themselves, and the role of tagging in the user modeling process. The major contributions of the paper are manifold: insights into user interactions with social adaptive systems; guidelines for future designs; evaluation of the tagging activity and tag meanings in relation to the application domain and thus their impact on the representation of the user model; and a demonstration of how a combination and interplay of evaluation methodologies (e.g., quantitative and qualitative) can enhance our comprehension of evaluation data.
Similar content being viewed by others
References
Abel, F., Herder, E., Houben, G.-J., Henze, N., Krause, D.: Cross-system user modeling and personalization on the social web. Special issue on Personalization in SocialWeb-Based Systems, User Modeling and User-Adapted Interaction, 23(2–3) (2013)
Bakeman R., Gottman J.M.: Observing Interaction: An Introduction to Sequential Analysis. Cambridge University Press, Cambridge (1997)
Barker T., Jones S., Britton C., Messer D.: The use of a co-operative student model of learner characteristics to configure a multimedia application. User Model. User-Adapt. Interact. 12(2–3), 207–241 (2002)
Brusilovsky P.: Methods and techniques of adaptive hypermedia. User Model. User-Adapt. Interact. 6(2–3), 87–129 (1996)
Brusilovsky, P., Karagiannidis, P., Sampson, D.: The benefits of layered evaluation of adaptive applications and services. In: Weibelzahl, D.N.C.S. (ed.) Empirical Evaluation of Adaptive Systems, Proceedings of Workshop At the Eighth International Conference on User Modeling, UM2001, pp. 1–8 (2001)
Brusilovsky, P., Chavan, G., Farzan, R.: Social adaptive navigation support for open corpus electronic textbooks. In: DeBra, P., Nejdl, W. (eds.) Proceedings of Adaptive Hypermedia and Adaptive Web-Based Systems, AH04, vol. 3137 of Lecture Notes in Computer Science, pp. 176–189. Springer, Berlin (2004)
Carmagnola F., Cena F., Console L., Cortassa O., Gena C., Goy A., Torre I., Toso A., Vernero F.: Tag-based user modeling for social multi-device adaptive guides. User Model. User-Adapt. Interact. 18(5), 497–538 (2008)
Carmagnola, F., Cena, F., Console, L., Grillo, P., Perrero, M., Simeoni, R., Vernero, F.: Supporting content discovery and organization in networks of contents and users. Multimedia Syst. 17, 199–218 (2011a). ISSN 0942-4962, http://10.1007/s00530-010-0219-4
Carmagnola, F., Cena, F., Gena, C.: User model interoperability: a survey. User Model. User-Adapt. Interact. 21, 285–331 (2011b). ISSN 0924-1868. http://10.1007/s11257-011-9097-5
Cena, F., Gena, C., Modeo, S.: How to communicate recommendations? Evaluation of an adaptive annotation technique. In: Costabile, M.F., Paternò, F. (eds.) INTERACT, vol. 3585 of Lecture Notes in Computer Science, pp. 1030–1033. Springer, Berlin (2005). ISBN 3-540-28943-7
Cena, F., Carmagnola, F., Cortassa, O., Gena, C., Wang, Y., Stash, N., Aroyo, L.: Tag interoperability in cultural web-based applications. In: Brusilovsky, P., Davis, H.C. (eds.) Hypertext, pp. 221–222. ACM, Pittsburgh (2008). ISBN 978-1-59593-985-2
Chin, D.N.: Empirical evaluation of user models and user-adapted systems. User Model. User-Adapt. Interact. 11(1–2), 181–194 (2001). ISSN 0924-1868
Damiano R., Gena C., Lombardo V., Nunnari F., Pizzo A.: A stroll with carletto: adaptation in drama-based tours with virtual characters. User Model. User-Adapt. Interact. 18(5), 417–453 (2008)
deGemmis, M., Lops, P., Semeraro, G., Basile, P.: Integrating tags in a semantic content-based recommender. In: Proceedings of the 2008 ACM conference on Recommender systems, RecSys ’08, pp. 163–170. ACM, New York (2008). ISBN 978-1-60558-093-7. http://doi.acm.org/10.1145/1454008.1454036
Diederich, J., Iofciu, T.: Finding communities of practice from user profiles based on folksonomies. In: Proceedings of the 1st International Workshop on Building Technology Enhanced Learning solutions for Communities of Practice (TEL-CoPs’06), October 2006
Fidel, R., Crandall, M.: Users’ perception of the performance of a filtering system. In: Proceedings of the 20th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’97, pp. 198–205. ACM, New York (1997). ISBN 0-89791-836-3. http://doi.acm.org/10.1145/258525.258568
Freyne, J., Berkovsky, S., Daly, E.M., Geyer, W.: Social networking feeds: recommending items of interest. In: Proceedings of the Fourth ACM Conference on Recommender Systems, RecSys’10, pp. 277–280. ACM, New York (2010). ISBN 978-1-60558-906-0. http://doi.acm.org/10.1145/1864708.1864766
Gena C.: Methods and techniques for the evaluation of user-adaptive systems. Knowl. Eng. Rev. 20(1), 1–37 (2005)
Gena, C., Weibelzahl, S.: Usability engineering for the adaptive web. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.) The Adaptive Web, vol. 4321 of Lecture Notes in Computer Science, pp. 720–762. Springer, Berlin (2007). ISBN 978-3-540-72078-2
Good, N., Schafer, J.B., Konstan, J.A., Borchers, A., Sarwar, B.M., Herlocker, J.L., Riedl, J.: Combining collaborative filtering with personal agents for better recommendations. In: Proceedings of the Sixteenth National Conference on Artificial Intelligence and Eleventh Conference on Innovative Applications of Artificial Intelligence, AAAI/IAAI 1999, pp. 439–446, 18–22 July 1999, Orlando, Florida. AAAI Press/The MIT Press, Menlo Park/Cambridge (1999)
Guy, I., Zwerdling, N., Carmel, D., Ronen, I., Uziel, E., Yogev, S., Ofek-Koifman, S.: Personalized recommendation of social software items based on social relations. In: RecSys ’09: Proceedings of the third ACM Conference on Recommender Systems, pp. 53–60. ACM, New York (2009). ISBN 978-1-60558-435-5. http://10.1145/1639714.1639725.
Herlocker, J.L., Konstan, J.A., Terveen, L., Riedl, J.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1), 5–53 (2004). ISSN 1046-8188. http://10.1145/963770.963772.
Höök, K.:Evaluating the utility and usability of an adaptive hypermedia system. In: Proceedings of International Conference on Intelligent User Interfaces, IUI97, pp. 179–186. ACM Press, New York (1997). http://10.1145/238218.238320.
Jameson, A.: Adaptive interfaces and agents. In: The human-computer interaction handbook: fundamentals, Evolving Technologies and Emerging Applications, pp. 305–330. Lawrence Erlbaum Associates Inc., Hillsdale (2003). ISBN 0-8058-3838-4.
Karagiannidis, C., Sampson, D.G. : Layered evaluation of adaptive applications and services. In: Proceedings of the International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, AH00, pp. 343–346. Springer, London (2000). ISBN 3-540-67910-3
Kay, J.: Scrutable adaptation: Because we can and must. In: Wade, V.P., Ashman, H., Smyth, B. (eds.) AH, vol. 4018 of Lecture Notes in Computer Science, pp. 11–19. Springer, Berlin (2006). ISBN 3-540-34696-1
Keppel, G., Saufley, W.H., Tokunaga, H.: Introduction to Design and Analysis: A Student’s Handbook. W.H. Freeman and Co., New York (1998)
Kim, H.-N., ElSaddik, A.: Exploring social tagging for personalized community recommendations. Special issue on Personalization in SocialWeb-Based Systems, User Modeling and User-Adapted Interaction, 23(2–3) (2013)
Kobsa A., Koenemann J., Pohl W.: Personalized hypermedia presentation techniques for improving online customer relationship. Knowl. Eng. Rev. 16(2), 111–115 (2001)
Kyriacou, D., Davis, H.C., Tiropanis, T.: A (multi‘domain’sional) scrutable user modelling infrastructure for enriching lifelong user modelling. In: Proceedings of Lifelong User Modelling Workshop AT Conference UMAP 2009, Trento, pp. 46–54, 2009
Loizou, S.K., Dimitrova, V.: Adaptive notifications to support knowledge sharing in close-knit virtual communities. Special issue on Personalization in SocialWeb-Based Systems, User Modeling and User-Adapted Interaction, 23(2–3) (2013)
Marlow, C., Naaman, M., Boyd, D., Davis, M.: Ht06, tagging paper, taxonomy, flickr, academic article, to read. In: 17th Conference on Hypertext and Hypermedia HYPERTEXT ’06, pp. 31–40. ACM, New York (2006)
McLaughlin, M.R., Herlocker, J.L.: A collaborative filtering algorithm and evaluation metric that accurately model the user experience. In: Sanderson, M., Jrvelin, K., Allan, J., Bruza, P. (eds.) SIGIR 2004: Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Sheffield, UK, 25–29 July 2004, pp. 329–336. ACM, New York (2004)
Michlmayr, E., Cayzer, S.: Learning user profiles from tagging data and leveraging them for personal(ized) information access. In: Proceedings of the Workshop on Tagging and Metadata for Social Information Organization, 16th International World Wide Web Conference, 2007
Nakamoto, R.Y., Nakajima, S., Miyazaki, J., Uemura, S., Kato, H., Inagaki, Y.: Reasonable tag-based collaborative filtering for social tagging systems. In: Proceedings of the 2nd ACM Workshop on Information Credibility on the Web, WICOW ’08, pp. 11–18. ACM, New York (2008). ISBN 978-1-60558-259-7. doi:10.1145/1458527.1458533
Nauerz, A., Bakalov, F., Welsch, M., Konig-Ries, B.: New tagging paradigms for content recommendation in web 2.0 portals. In: Proceedings of International Workshop on Adaptation and Personalization for Web 2.0 (AP-WEB 2.0 2009), pp. 143–147, 2009
Nielsen, J.: Risks of quantitative studies. http://www.useit.com/alertbox/20040301.html. Accessed Jan 2009, March 2004
Niu, W.T., Kay, J.: Pervasive personalisation of location information: Personalised context ontology. In: Proceedings of the 5th International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, AH ’08, pp. 143–152. Springer, Berlin (2008). ISBN 978-3-540-70984-8
Nonnecke, B., Preece, J., Andrews, D.: What lurkers and posters think of each other. In: Proceedings of the Proceedings of the 37th Annual Hawaii International Conference on System Sciences (HICSS’04)—Track 7—vol. 7, HICSS ’04, p. 70195.1. IEEE Computer Society, Washington (2004). ISBN 0-7695-2056-1. http://dl.acm.org/citation.cfm?id=962755.963094
Paramythis, A., Weibelzahl, S.: A decomposition model for the layered evaluation of interactive adaptive systems. In: User Modeling, vol. 3538 of Lecture Notes in Computer Science, pp. 438–442. Springer, Berlin (2005)
Paramythis, A., Totter, P., Stephanidis, P.: A modular approach to the evaluation of adaptive user interface. In: Weibelzahl, S., Chin, D., Weber, G. (eds.) Proceedings of the First Workshop on Empirical Evaluation of Adaptive Systems, pp. 9–24, Sonthofen, Germany (2001)
Paramythis, A., Weibelzahl, S., Masthoff, J.: Layered evaluation of interactive adaptive systems: framework and formative methods. User Model. User-Adapt. Interact. 20, 383–453 (2010). ISSN 0924-1868. http://10.1007/s11257-010-9082-4
Pirolli, P., Kairam, S.: A knowledge-tracing model of learning from a social tagging system. Special issue on Personalization in SocialWeb-Based Systems, User Modeling and User-Adapted Interaction, 23(2–3) (2013)
Preece J.J., Nonnecke B., Andrews D.: The top five reasons for lurking: improving community experiences for everyone. Comput. Hum. Behav. 20(2), 201–223 (2004)
Prieur, C., Cardon, D., Beuscart, J.-S., Pissard, N., Pons, P.: The strength of weak cooperation: a case study on flickr. CoRR. abs/0802.2317 (2008)
Rizzo, P., Lee, H., Shaw, E., Johnson, W.L., Wang, N., Mayer, R.E.: A semi-automated wizard of oz interface for modeling tutorial strategies. In: Proceedings of International Conference on User Modeling, UM 2005, pp. 174–178. doi:10.1007/11527886
Sarwar, B., Karypis, G., Konstan, J., Reidl, J.: Item-based collaborative filtering recommendation algorithms. In: Proceedings of the 10th International Conference on World Wide Web, pp. 285–295. ACM Press, New York (2001)
Shapira, B., Rokach, L., Freilichman, S.: Utilizing facebook single and cross domain data for recommendation systems. Special issue on Personalization in SocialWeb-Based Systems, User Modeling and User-Adapted Interaction, 23(2–3) (2013)
Shepitsen, A., Gemmell, J., Mobasher, B., Burke, R.: Personalized recommendation in social tagging systems using hierarchical clustering. In: Proceedings of the 2008 ACM conference on Recommender Systems, RecSys ’08, pp. 259–266. ACM, New York (2008). ISBN 978-1-60558-093-7. http://doi.acm.org/10.1145/1454008.1454048
Sinha, R.R., Swearingen, K.: Comparing recommendations made by online systems and friends. In: DELOS Workshop: Personalisation and Recommender Systems in Digital Libraries, 2001
Strauss, A., Corbin, J.: Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. SAGE Publications, Thousand Oaks (1998). ISBN 0803959400
Szomszor, M., Alani, H., Cantador, I., O’Hara, K., Shadbolt, N.: Semantic modelling of user interests based on cross-folksonomy analysis. In: Sheth, A.P., Staab, S., Dean, M., Paolucci, M., Maynard, D., Finin, T.W., Thirunarayan, K. (eds.) International Semantic Web Conference, vol. 5318 of Lecture Notes in Computer Science, pp. 632–648. Springer, Berlin (2008). ISBN 978-3-540-88563-4
Totterdell, P., Boyle, E.: Adaptive user interfaces. In: Browne, D., Totterdell, A., Norman, D. (eds.) The Evaluation of Adaptive Systems, pp. 161–194. Academic Press, London (1990)
van Setten, M., Brussee, R., van Vliet, H., Gazendam, L., van Houten, Y., Veenstra, M.: On the importance of “Who Tagged What”. In: Workshop on the Social Navigation and Community Based Adaptation Technologies, pp. 552–561, Dublin, Ireland, 2006. http://www.sis.pitt.edu/%7Epaws/SNC_BAT06/crc/vansetten.pdf
Vig, J., Sen, S., Riedl, J.: Navigating the tag genome. In: Pu, P., Pazzani, M.J., André, E., Riecken, D. (eds.) IUI, pp. 93–102. ACM, New York (2011). ISBN 978-1-4503-0419-1
Wang Y., Stash N., Aroyo L., Gorgels P., Rutledge L., Schreiber G.: Recommendations based on semantically enriched museum collections. J. Web Semant. 6(4), 283–290 (2008)
Wang, Y., Zhang, J., Vassileva, J.:Personalized recommendation of integrated social data across social networking sites. In: Proceedings of SASweb Workshop, at UMAP 2010, pp. 19–30, 2010
Weibelzahl, S.: Evaluation of adaptive systems. In: User Modeling, vol. 2109 of Lecture Notes in Computer Science, pp. 292–294. Springer, Berlin (2001)
Weibelzahl, S.: Evaluation of adaptive systems. Dissertation. University of Trier, Trier (2003)
Weibelzahl, S., Lauer, C.:Framework for the evaluation of adaptive CBR-systems. In: Reimer, U., Schmitt, S., Vollrath, I. (eds.) Proceedings of the 9th German Workshop on Case-Based Reasoning (GWCBR01), Aachen: Shakerm, pp. 254–263 (2001)
Weibelzahl, S., Weber, G.: A database of empirical evaluations of adaptive systems. In: Proceedings of Workshop Lernen–Lehren–Wissen–Adaptivity (LLWA 01), Research Report in Computer Science Nr. 763, University of Dortmund, pp. 302–306, 2001
Zanardi, V., Capra, L.: Social ranking: uncovering relevant content using tag-based recommender systems. In: Pu, P., Bridge, D.G., Mobasher, B., Ricci, F. (eds.) Proceedings of International Conference on Recommender Systems, RecSys08, pp. 51–58. ACM, New York (2008). ISBN 978-1-60558-093-7
Zhen, Y., Li, W.-J., Yeung, D.-Y.: Tagicofi: tag informed collaborative filtering. In: Bergman, L.D., Tuzhilin, A., Burke, R.D., Felfernig, A., Schmidt-Thieme, L. (eds.) RecSys, pp. 69–76. ACM, New York (2009). ISBN 978-1-60558-435-5
Zimmermann, A., Lorenz, A.: Listen: a user-adaptive audio-augmented museum guide. User Model. User-Adapt. Interact. 18, 389–416 (2008). ISSN 0924-1868. doi:10.1007/s11257-008-9049-x. http://portal.acm.org/citation.cfm?id=1455853.1455878
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Gena, C., Cena, F., Vernero, F. et al. The evaluation of a social adaptive website for cultural events. User Model User-Adap Inter 23, 89–137 (2013). https://doi.org/10.1007/s11257-012-9129-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11257-012-9129-9