Abstract
Digital humanities initiatives play an important role in making cultural heritage collections accessible to the global community of researchers and general public for the first time. Further work is needed to provide useful and usable tools to support users in working with those digital contents in virtual environments. The CULTURA project has developed a corpus agnostic research environment integrating innovative services that guide, assist and empower a broad spectrum of users in their interaction with cultural artefacts. This article presents (1) the CULTURA system and services and the two collections that have been used for testing and deploying the digital humanities research environment, and (2) an evaluation methodology and formative evaluation study with apprentice researchers. An evaluation model was developed which has served as a common ground for systematic evaluations of the CULTURA environment with user communities around the two test bed collections. The evaluation method has proven to be suitable for accommodating different evaluation strategies and allows meaningful consolidation of evaluation results. The evaluation outcomes indicate a positive perception of CULTURA. A range of useful suggestions for future improvement has been collected and fed back into the development of the next release of the research environment.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
References
Borgman, C.L.: The digital future is now: A call to action for the humanities. Digital Human. Quart. 3(4). http://www.digitalhumanities.org/dhq/vol/3/4/000077/000077.html (2009). Accessed 3 Oct 2013
Hampson, C., Agosti, M., Orio, N., Bailey, E., Lawless, S., Conlan, O., Wade, V.: The CULTURA project: supporting next generation interaction with digital cultural heritage collections. In: Proceedings of the 4th International Euromed Conference, pp. 668–675. Springer, Heidelberg (2012)
Hampson, C., Lawless, S., Bailey, E., Yogev, S., Zwerdling, N., Carmel, D.: CULTURA: A metadata-rich environment to support the enhanced interrogation of cultural collections. In: Dodero, J.M., Palomo-Duarte, M., Karmpiperis, P. (eds.) Proceedings of the 6th Metadata and Semantics Research Conference, pp. 227–238. Springer, Berlin (2012)
Clarke, A.: The 1641 Depositions. In: Fox, P. (ed.) Treasures of the Library, pp. 111–122. Trinity College, Dublin (1986)
O’Regan, D., Sweetnam, M., Fennell, B., Lawless, S.: A collaborative linguistic research interface for the 1641 Depositions. Poster presented at Digital Humanities 2011. Stanford, CA (2011)
Agosti, M., Benfante, L., Orio, N.: IPSA: A digital archive of herbals to support scientific research. In: Sembok T.M.T. (ed.) Proceedings of the International Conference on Asian Digital Libraries ICADL 2003. LNCS vol. 2911, pp. 253–264. Springer, Berlin (2003)
Bellamy, C.: The sound of many hands clapping: Teaching the digital humanities through virtual research environments (VREs). Digital Human. Quart. 6(2). http://www.digitalhumanities.org/dhq/vol/6/2/000119/000119.html (2012). Accessed 3 Oct 2013
Hunter, J., Gerber, A.: The Aus-e-Lit project: Advanced eResearch services for scholars of Australian literature. VALA 2010. Melbourne, Australia. http://www.itee.uq.edu.au/eresearch/papers/2010/VALA2010_Hunter.pdf (2010). Accessed 3 Oct 2013
Neuroth, H., Lohmeier, F., Smith, K.M.: TextGrid—virtual research environment for the humanities. Int. J. Digit. Curation 6(2), 222–231 (2011)
Agosti, M., Conlan, O., Ferro, N., Hampson, C., Munnelly, G.: Interacting with digital cultural heritage collections via annotations: the CULTURA approach. In: Proceedings of the 2013 ACM Symposium on Document Engineering, pp. 13–22. ACM, New York (2013)
Lawless, S., Hampson, C., Mitankin, P., Gerdjikov, S.: Normalisation in historical text collections. In: Proceedings of Digital Humanities 2013, pp. 507–509. Lincoln, Nebraska (2013)
Yogev, S., Roitman, H., Carmel, D. Zwerdling, N.: Towards expressive exploratory search over entity-relationship data. In: Proceedings of the \(21^{st}\) International Conference Companion on World Wide Web, pp. 83–92. New York (2012)
Hampson, C., Bailey, E., Munnelly, G., Lawless, S., Conlan, O.: Dynamic personalisation for digital cultural heritage collections. In: Proceedings of the 6th International Workshop on Personalized Access to Cultural Heritage. Rome (2013)
Carmel, D., Zwerdling, N., Yogev, S.: Entity oriented search and exploration for cultural heritage collections: the EU CULTURA project. In: Proceedings of the \(21^{st}\) International Conference Companion on World Wide Web, pp. 227–230. New York (2012)
Mariani Canova, G.: Per Cultura: le immagini dei manoscritti della scienza a Padova dal Medioevo al Rinascimento. In: Atti e Memorie dell’Accademia Galileiana di Scienze, Lettere ed Arti. vol. CXXIV, pp. 81–90 (2011–2012)
Saracevic, T.: Digital library evaluation: toward an evolution of concepts. Libr. Trends 49(2), 350–369 (2000)
Zhang, Y.: Developing a holistic model for digital library evaluation. Dissertation. The State University of New Jersey. (2007)
De Jong, M., Schellens, P.J.: Reader-focused text evaluation: an overview of goals and methods. J. Bus. Tech. Commun. 11(4), 401–432 (1997)
Saracevic, T.: Evaluation of digital libraries: An overview. In: Agosti M., Fuhr N. (eds.) DELOS Workshop on the Evaluation of Digital Libraries. Padua, Italy. http://www.scils.rutgers.edu/~tefko/DL_evaluation_Delos.pdf (2004). Accessed 7 Oct 2013
Kovács, L., Micsik, A.: The evaluation computer: A model for structuring evaluation activities. In: Agosti, M., Fuhr, N. (eds.) DELOS Workshop on the Evaluation of Digital Libraries. Padua (2004)
Zhang, Y.: Developing a holistic model for digital library evaluation. J. Am. Soc. Inf. Sci. Technol. 61(1), 88–110 (2010)
Fuhr, N., Tsakonas, G., Aalberg, T., Agosti, M., Hansen, P., Kapidakis, S., Klas, C.-P., Kovács, L., Landoni, M., Micsik, A., Papatheodorou, C., Peters, C., Sølvberg, I.: Evaluation of digital libraries. Int. J. Digit. Libr. 8(1), 21–38 (2007)
Tsakonas, G., Papatheodorou, C.: Analysing and evaluating usefulness and usability in electronic information services. J. Inf. Sci. 32(5), 400–419 (2006)
Tsakonas, G., Kapidakis, S., Papatheodorou, C.: Evaluation of user interaction in digital libraries. In: Agosti, M., Fuhr, N. (eds.) DELOS Workshop on Evaluation of Digital Libraries, pp. 45–60. Padua (2004)
Steiner, C.M., Hillemann, E., Nussbaumer, A., Albert, D., Sweetnam, M., Hampson, C., Conlan, O.: The CULTURA evaluation model: An approach responding to the evaluation needs in an innovative research environment. In: S. Lawless, M. Agosti, P. Clough, O. Conlan (eds.) Proceedings of the First Workshop on Exploration, Navigation and Retrieval of Information in Cultural Heritage, SIGIR 2013, pp. 43–46. ACM, Dublin (2013)
Sweetnam, M., Agosti, M., Orio, N., Ponchia, C., Steiner, C., Hillemann, E.-C., Ó Siochrú, M., Lawless, S.: User needs for enhanced engagement with cultural heritage collections. In: Zaphiris, P., Buchanan, G., Rasmussen, E., Loizides, F. (eds.) Theory and Practice of Digital Libraries, TPDL 2012. LNCS vol. 7489, pp. 64–75. Springer, Berlin (2012)
Davis, F.D., Bagozzi, R.P., Warshaw, P.R.: User acceptance of computer technology: a comparison of two theoretical models. Manag. Sci. 35(8), 982–1003 (1989)
Gerdjikov, S., Mihov, S., Nenchev, V.: Extraction of spelling variations from language structure for noisy text correction. In: Proceedings of the 12th International Conference on Document Analysis and Recognition. IEEE, pp. 324–328 (2013)
Steiner, C.M., Albert, D.: Tailor-made or unfledged? Evaluating the quality of adaptive eLearning. In: Psaromiligkos, Spyridakos, A., Retalis, S. (eds.) Evaluation in e-learning., pp. 111–143. Nova Science, New York (2012)
Brusilovsky, P., Karagiannidis, C., Sampson, D.: Layered evaluation of adaptive learning systems. Int. J. Contin. Eng. Educ. Life-Long Learn. 14, 402–421 (2004)
Brooke, J.: SUS: a “quick and dirty” usability scale. In: Jordan, P.W., Thomas, B., Weerdmeester, B.A., McCleeland, A.L. (eds.) Usability Evaluation in Industry, pp. 189–194. Taylor and Francis, London (1996)
Thong, J.Y.L., Hong, W., Tam, K.-Y.: Understanding user acceptance of digital libraries: what are the roles of interface characteristics, organizational context, and individual differences? Int. J. Hum.-Comput. Stud. 57, 215–242 (2002)
Hampson, C., Lawless, S., Bailey, E., Steiner, C.M., Hillemann, E.-C., Conlan, O.: Metadata-enhanced exploration of digital cultural collections. Int. J. Metadata Semant. Ontol. 9, 155–167 (2014)
Nussbaumer, A., Hillemann, E.-C., Steiner, C.M., Albert, D.: An evaluation system for digital libraries. In: Zaphiris, P., Buchanan, G., Rasmussen, E., Loizides, F. (eds.) Theory and practice of digital libraries. Second International Conference, TPDL 2012. LNCS, vol. 7489, pp. 414–419. Springer, Berlin (2012)
Acknowledgments
The work reported has been partially supported by the CULTURA project, as part of the Seventh Framework Programme of the European Commission, Area “Digital Libraries and Digital Preservation” (ICT-2009.4.1), grant agreement no. 269973, and would not have been possible without the close collaboration between all CULTURA partners. http://www.cultura-strep.eu/partners.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Steiner, C.M., Agosti, M., Sweetnam, M.S. et al. Evaluating a digital humanities research environment: the CULTURA approach. Int J Digit Libr 15, 53–70 (2014). https://doi.org/10.1007/s00799-014-0127-x
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00799-014-0127-x