Skip to main content
Log in

The evaluation of a social adaptive website for cultural events

  • Original Paper
  • Published:
User Modeling and User-Adapted Interaction Aims and scope Submit manuscript

Abstract

In this paper, we present an evaluation of a social adaptive website in the domain of cultural events, iCITY DSA, which provides information about cultural resources and events that promote the cultural heritage in the city of Turin. Using this evaluation, our objective was to investigate the actual usage of a social adaptive website, in an effort to discover the real behavior of users, the unforeseen correlations among user actions and the consequent interactive behavior, the accuracy of both system and social recommendations and their impact on the users themselves, and the role of tagging in the user modeling process. The major contributions of the paper are manifold: insights into user interactions with social adaptive systems; guidelines for future designs; evaluation of the tagging activity and tag meanings in relation to the application domain and thus their impact on the representation of the user model; and a demonstration of how a combination and interplay of evaluation methodologies (e.g., quantitative and qualitative) can enhance our comprehension of evaluation data.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Abel, F., Herder, E., Houben, G.-J., Henze, N., Krause, D.: Cross-system user modeling and personalization on the social web. Special issue on Personalization in SocialWeb-Based Systems, User Modeling and User-Adapted Interaction, 23(2–3) (2013)

  • Bakeman R., Gottman J.M.: Observing Interaction: An Introduction to Sequential Analysis. Cambridge University Press, Cambridge (1997)

    Book  Google Scholar 

  • Barker T., Jones S., Britton C., Messer D.: The use of a co-operative student model of learner characteristics to configure a multimedia application. User Model. User-Adapt. Interact. 12(2–3), 207–241 (2002)

    Article  MATH  Google Scholar 

  • Brusilovsky P.: Methods and techniques of adaptive hypermedia. User Model. User-Adapt. Interact. 6(2–3), 87–129 (1996)

    Article  Google Scholar 

  • Brusilovsky, P., Karagiannidis, P., Sampson, D.: The benefits of layered evaluation of adaptive applications and services. In: Weibelzahl, D.N.C.S. (ed.) Empirical Evaluation of Adaptive Systems, Proceedings of Workshop At the Eighth International Conference on User Modeling, UM2001, pp. 1–8 (2001)

  • Brusilovsky, P., Chavan, G., Farzan, R.: Social adaptive navigation support for open corpus electronic textbooks. In: DeBra, P., Nejdl, W. (eds.) Proceedings of Adaptive Hypermedia and Adaptive Web-Based Systems, AH04, vol. 3137 of Lecture Notes in Computer Science, pp. 176–189. Springer, Berlin (2004)

  • Carmagnola F., Cena F., Console L., Cortassa O., Gena C., Goy A., Torre I., Toso A., Vernero F.: Tag-based user modeling for social multi-device adaptive guides. User Model. User-Adapt. Interact. 18(5), 497–538 (2008)

    Article  Google Scholar 

  • Carmagnola, F., Cena, F., Console, L., Grillo, P., Perrero, M., Simeoni, R., Vernero, F.: Supporting content discovery and organization in networks of contents and users. Multimedia Syst. 17, 199–218 (2011a). ISSN 0942-4962, http://10.1007/s00530-010-0219-4

  • Carmagnola, F., Cena, F., Gena, C.: User model interoperability: a survey. User Model. User-Adapt. Interact. 21, 285–331 (2011b). ISSN 0924-1868. http://10.1007/s11257-011-9097-5

  • Cena, F., Gena, C., Modeo, S.: How to communicate recommendations? Evaluation of an adaptive annotation technique. In: Costabile, M.F., Paternò, F. (eds.) INTERACT, vol. 3585 of Lecture Notes in Computer Science, pp. 1030–1033. Springer, Berlin (2005). ISBN 3-540-28943-7

  • Cena, F., Carmagnola, F., Cortassa, O., Gena, C., Wang, Y., Stash, N., Aroyo, L.: Tag interoperability in cultural web-based applications. In: Brusilovsky, P., Davis, H.C. (eds.) Hypertext, pp. 221–222. ACM, Pittsburgh (2008). ISBN 978-1-59593-985-2

  • Chin, D.N.: Empirical evaluation of user models and user-adapted systems. User Model. User-Adapt. Interact. 11(1–2), 181–194 (2001). ISSN 0924-1868

  • Damiano R., Gena C., Lombardo V., Nunnari F., Pizzo A.: A stroll with carletto: adaptation in drama-based tours with virtual characters. User Model. User-Adapt. Interact. 18(5), 417–453 (2008)

    Article  Google Scholar 

  • deGemmis, M., Lops, P., Semeraro, G., Basile, P.: Integrating tags in a semantic content-based recommender. In: Proceedings of the 2008 ACM conference on Recommender systems, RecSys ’08, pp. 163–170. ACM, New York (2008). ISBN 978-1-60558-093-7. http://doi.acm.org/10.1145/1454008.1454036

  • Diederich, J., Iofciu, T.: Finding communities of practice from user profiles based on folksonomies. In: Proceedings of the 1st International Workshop on Building Technology Enhanced Learning solutions for Communities of Practice (TEL-CoPs’06), October 2006

  • Fidel, R., Crandall, M.: Users’ perception of the performance of a filtering system. In: Proceedings of the 20th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’97, pp. 198–205. ACM, New York (1997). ISBN 0-89791-836-3. http://doi.acm.org/10.1145/258525.258568

  • Freyne, J., Berkovsky, S., Daly, E.M., Geyer, W.: Social networking feeds: recommending items of interest. In: Proceedings of the Fourth ACM Conference on Recommender Systems, RecSys’10, pp. 277–280. ACM, New York (2010). ISBN 978-1-60558-906-0. http://doi.acm.org/10.1145/1864708.1864766

  • Gena C.: Methods and techniques for the evaluation of user-adaptive systems. Knowl. Eng. Rev. 20(1), 1–37 (2005)

    Article  Google Scholar 

  • Gena, C., Weibelzahl, S.: Usability engineering for the adaptive web. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.) The Adaptive Web, vol. 4321 of Lecture Notes in Computer Science, pp. 720–762. Springer, Berlin (2007). ISBN 978-3-540-72078-2

  • Good, N., Schafer, J.B., Konstan, J.A., Borchers, A., Sarwar, B.M., Herlocker, J.L., Riedl, J.: Combining collaborative filtering with personal agents for better recommendations. In: Proceedings of the Sixteenth National Conference on Artificial Intelligence and Eleventh Conference on Innovative Applications of Artificial Intelligence, AAAI/IAAI 1999, pp. 439–446, 18–22 July 1999, Orlando, Florida. AAAI Press/The MIT Press, Menlo Park/Cambridge (1999)

  • Guy, I., Zwerdling, N., Carmel, D., Ronen, I., Uziel, E., Yogev, S., Ofek-Koifman, S.: Personalized recommendation of social software items based on social relations. In: RecSys ’09: Proceedings of the third ACM Conference on Recommender Systems, pp. 53–60. ACM, New York (2009). ISBN 978-1-60558-435-5. http://10.1145/1639714.1639725.

  • Herlocker, J.L., Konstan, J.A., Terveen, L., Riedl, J.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1), 5–53 (2004). ISSN 1046-8188. http://10.1145/963770.963772.

    Google Scholar 

  • Höök, K.:Evaluating the utility and usability of an adaptive hypermedia system. In: Proceedings of International Conference on Intelligent User Interfaces, IUI97, pp. 179–186. ACM Press, New York (1997). http://10.1145/238218.238320.

  • Jameson, A.: Adaptive interfaces and agents. In: The human-computer interaction handbook: fundamentals, Evolving Technologies and Emerging Applications, pp. 305–330. Lawrence Erlbaum Associates Inc., Hillsdale (2003). ISBN 0-8058-3838-4.

  • Karagiannidis, C., Sampson, D.G. : Layered evaluation of adaptive applications and services. In: Proceedings of the International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, AH00, pp. 343–346. Springer, London (2000). ISBN 3-540-67910-3

  • Kay, J.: Scrutable adaptation: Because we can and must. In: Wade, V.P., Ashman, H., Smyth, B. (eds.) AH, vol. 4018 of Lecture Notes in Computer Science, pp. 11–19. Springer, Berlin (2006). ISBN 3-540-34696-1

  • Keppel, G., Saufley, W.H., Tokunaga, H.: Introduction to Design and Analysis: A Student’s Handbook. W.H. Freeman and Co., New York (1998)

  • Kim, H.-N., ElSaddik, A.: Exploring social tagging for personalized community recommendations. Special issue on Personalization in SocialWeb-Based Systems, User Modeling and User-Adapted Interaction, 23(2–3) (2013)

  • Kobsa A., Koenemann J., Pohl W.: Personalized hypermedia presentation techniques for improving online customer relationship. Knowl. Eng. Rev. 16(2), 111–115 (2001)

    Article  MATH  Google Scholar 

  • Kyriacou, D., Davis, H.C., Tiropanis, T.: A (multi‘domain’sional) scrutable user modelling infrastructure for enriching lifelong user modelling. In: Proceedings of Lifelong User Modelling Workshop AT Conference UMAP 2009, Trento, pp. 46–54, 2009

  • Loizou, S.K., Dimitrova, V.: Adaptive notifications to support knowledge sharing in close-knit virtual communities. Special issue on Personalization in SocialWeb-Based Systems, User Modeling and User-Adapted Interaction, 23(2–3) (2013)

  • Marlow, C., Naaman, M., Boyd, D., Davis, M.: Ht06, tagging paper, taxonomy, flickr, academic article, to read. In: 17th Conference on Hypertext and Hypermedia HYPERTEXT ’06, pp. 31–40. ACM, New York (2006)

  • McLaughlin, M.R., Herlocker, J.L.: A collaborative filtering algorithm and evaluation metric that accurately model the user experience. In: Sanderson, M., Jrvelin, K., Allan, J., Bruza, P. (eds.) SIGIR 2004: Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Sheffield, UK, 25–29 July 2004, pp. 329–336. ACM, New York (2004)

  • Michlmayr, E., Cayzer, S.: Learning user profiles from tagging data and leveraging them for personal(ized) information access. In: Proceedings of the Workshop on Tagging and Metadata for Social Information Organization, 16th International World Wide Web Conference, 2007

  • Nakamoto, R.Y., Nakajima, S., Miyazaki, J., Uemura, S., Kato, H., Inagaki, Y.: Reasonable tag-based collaborative filtering for social tagging systems. In: Proceedings of the 2nd ACM Workshop on Information Credibility on the Web, WICOW ’08, pp. 11–18. ACM, New York (2008). ISBN 978-1-60558-259-7. doi:10.1145/1458527.1458533

  • Nauerz, A., Bakalov, F., Welsch, M., Konig-Ries, B.: New tagging paradigms for content recommendation in web 2.0 portals. In: Proceedings of International Workshop on Adaptation and Personalization for Web 2.0 (AP-WEB 2.0 2009), pp. 143–147, 2009

  • Nielsen, J.: Risks of quantitative studies. http://www.useit.com/alertbox/20040301.html. Accessed Jan 2009, March 2004

  • Niu, W.T., Kay, J.: Pervasive personalisation of location information: Personalised context ontology. In: Proceedings of the 5th International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, AH ’08, pp. 143–152. Springer, Berlin (2008). ISBN 978-3-540-70984-8

  • Nonnecke, B., Preece, J., Andrews, D.: What lurkers and posters think of each other. In: Proceedings of the Proceedings of the 37th Annual Hawaii International Conference on System Sciences (HICSS’04)—Track 7—vol. 7, HICSS ’04, p. 70195.1. IEEE Computer Society, Washington (2004). ISBN 0-7695-2056-1. http://dl.acm.org/citation.cfm?id=962755.963094

  • Paramythis, A., Weibelzahl, S.: A decomposition model for the layered evaluation of interactive adaptive systems. In: User Modeling, vol. 3538 of Lecture Notes in Computer Science, pp. 438–442. Springer, Berlin (2005)

  • Paramythis, A., Totter, P., Stephanidis, P.: A modular approach to the evaluation of adaptive user interface. In: Weibelzahl, S., Chin, D., Weber, G. (eds.) Proceedings of the First Workshop on Empirical Evaluation of Adaptive Systems, pp. 9–24, Sonthofen, Germany (2001)

  • Paramythis, A., Weibelzahl, S., Masthoff, J.: Layered evaluation of interactive adaptive systems: framework and formative methods. User Model. User-Adapt. Interact. 20, 383–453 (2010). ISSN 0924-1868. http://10.1007/s11257-010-9082-4

  • Pirolli, P., Kairam, S.: A knowledge-tracing model of learning from a social tagging system. Special issue on Personalization in SocialWeb-Based Systems, User Modeling and User-Adapted Interaction, 23(2–3) (2013)

  • Preece J.J., Nonnecke B., Andrews D.: The top five reasons for lurking: improving community experiences for everyone. Comput. Hum. Behav. 20(2), 201–223 (2004)

    Article  Google Scholar 

  • Prieur, C., Cardon, D., Beuscart, J.-S., Pissard, N., Pons, P.: The strength of weak cooperation: a case study on flickr. CoRR. abs/0802.2317 (2008)

  • Rizzo, P., Lee, H., Shaw, E., Johnson, W.L., Wang, N., Mayer, R.E.: A semi-automated wizard of oz interface for modeling tutorial strategies. In: Proceedings of International Conference on User Modeling, UM 2005, pp. 174–178. doi:10.1007/11527886

  • Sarwar, B., Karypis, G., Konstan, J., Reidl, J.: Item-based collaborative filtering recommendation algorithms. In: Proceedings of the 10th International Conference on World Wide Web, pp. 285–295. ACM Press, New York (2001)

  • Shapira, B., Rokach, L., Freilichman, S.: Utilizing facebook single and cross domain data for recommendation systems. Special issue on Personalization in SocialWeb-Based Systems, User Modeling and User-Adapted Interaction, 23(2–3) (2013)

  • Shepitsen, A., Gemmell, J., Mobasher, B., Burke, R.: Personalized recommendation in social tagging systems using hierarchical clustering. In: Proceedings of the 2008 ACM conference on Recommender Systems, RecSys ’08, pp. 259–266. ACM, New York (2008). ISBN 978-1-60558-093-7. http://doi.acm.org/10.1145/1454008.1454048

  • Sinha, R.R., Swearingen, K.: Comparing recommendations made by online systems and friends. In: DELOS Workshop: Personalisation and Recommender Systems in Digital Libraries, 2001

  • Strauss, A., Corbin, J.: Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. SAGE Publications, Thousand Oaks (1998). ISBN 0803959400

  • Szomszor, M., Alani, H., Cantador, I., O’Hara, K., Shadbolt, N.: Semantic modelling of user interests based on cross-folksonomy analysis. In: Sheth, A.P., Staab, S., Dean, M., Paolucci, M., Maynard, D., Finin, T.W., Thirunarayan, K. (eds.) International Semantic Web Conference, vol. 5318 of Lecture Notes in Computer Science, pp. 632–648. Springer, Berlin (2008). ISBN 978-3-540-88563-4

  • Totterdell, P., Boyle, E.: Adaptive user interfaces. In: Browne, D., Totterdell, A., Norman, D. (eds.) The Evaluation of Adaptive Systems, pp. 161–194. Academic Press, London (1990)

  • van Setten, M., Brussee, R., van Vliet, H., Gazendam, L., van Houten, Y., Veenstra, M.: On the importance of “Who Tagged What”. In: Workshop on the Social Navigation and Community Based Adaptation Technologies, pp. 552–561, Dublin, Ireland, 2006. http://www.sis.pitt.edu/%7Epaws/SNC_BAT06/crc/vansetten.pdf

  • Vig, J., Sen, S., Riedl, J.: Navigating the tag genome. In: Pu, P., Pazzani, M.J., André, E., Riecken, D. (eds.) IUI, pp. 93–102. ACM, New York (2011). ISBN 978-1-4503-0419-1

  • Wang Y., Stash N., Aroyo L., Gorgels P., Rutledge L., Schreiber G.: Recommendations based on semantically enriched museum collections. J. Web Semant. 6(4), 283–290 (2008)

    Article  Google Scholar 

  • Wang, Y., Zhang, J., Vassileva, J.:Personalized recommendation of integrated social data across social networking sites. In: Proceedings of SASweb Workshop, at UMAP 2010, pp. 19–30, 2010

  • Weibelzahl, S.: Evaluation of adaptive systems. In: User Modeling, vol. 2109 of Lecture Notes in Computer Science, pp. 292–294. Springer, Berlin (2001)

  • Weibelzahl, S.: Evaluation of adaptive systems. Dissertation. University of Trier, Trier (2003)

  • Weibelzahl, S., Lauer, C.:Framework for the evaluation of adaptive CBR-systems. In: Reimer, U., Schmitt, S., Vollrath, I. (eds.) Proceedings of the 9th German Workshop on Case-Based Reasoning (GWCBR01), Aachen: Shakerm, pp. 254–263 (2001)

  • Weibelzahl, S., Weber, G.: A database of empirical evaluations of adaptive systems. In: Proceedings of Workshop Lernen–Lehren–Wissen–Adaptivity (LLWA 01), Research Report in Computer Science Nr. 763, University of Dortmund, pp. 302–306, 2001

  • Zanardi, V., Capra, L.: Social ranking: uncovering relevant content using tag-based recommender systems. In: Pu, P., Bridge, D.G., Mobasher, B., Ricci, F. (eds.) Proceedings of International Conference on Recommender Systems, RecSys08, pp. 51–58. ACM, New York (2008). ISBN 978-1-60558-093-7

  • Zhen, Y., Li, W.-J., Yeung, D.-Y.: Tagicofi: tag informed collaborative filtering. In: Bergman, L.D., Tuzhilin, A., Burke, R.D., Felfernig, A., Schmidt-Thieme, L. (eds.) RecSys, pp. 69–76. ACM, New York (2009). ISBN 978-1-60558-435-5

  • Zimmermann, A., Lorenz, A.: Listen: a user-adaptive audio-augmented museum guide. User Model. User-Adapt. Interact. 18, 389–416 (2008). ISSN 0924-1868. doi:10.1007/s11257-008-9049-x. http://portal.acm.org/citation.cfm?id=1455853.1455878

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Federica Cena.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Gena, C., Cena, F., Vernero, F. et al. The evaluation of a social adaptive website for cultural events. User Model User-Adap Inter 23, 89–137 (2013). https://doi.org/10.1007/s11257-012-9129-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11257-012-9129-9

Keywords

Navigation