Skip to main content
SpringerLink
Log in

Extracting relevance and affect information from physiological text annotation

  • Published:
User Modeling and User-Adapted Interaction Aims and scope Submit manuscript

Abstract

We present physiological text annotation, which refers to the practice of associating physiological responses to text content in order to infer characteristics of the user information needs and affective responses. Text annotation is a laborious task, and implicit feedback has been studied as a way to collect annotations without requiring any explicit action from the user. Previous work has explored behavioral signals, such as clicks or dwell time to automatically infer annotations, and physiological signals have mostly been explored for image or video content. We report on two experiments in which physiological text annotation is studied first to (1) indicate perceived relevance and then to (2) indicate affective responses of the users. The first experiment tackles the user’s perception of relevance of an information item, which is fundamental towards revealing the user’s information needs. The second experiment is then aimed at revealing the user’s affective responses towards a -relevant- text document. Results show that physiological user signals are associated with relevance and affect. In particular, electrodermal activity was found to be different when users read relevant content than when they read irrelevant content and was found to be lower when reading texts with negative emotional content than when reading texts with neutral content. Together, the experiments show that physiological text annotation can provide valuable implicit inputs for personalized systems. We discuss how our findings help design personalized systems that can annotate digital content using human physiology without the need for any explicit user interaction.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  • Adomavicius, G., Tuzhilin, A.: Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans. Knowl. Data Eng. 17(6), 734–749 (2005). doi:10.1109/TKDE.2005.99

    Article  Google Scholar 

  • Andreassi, J.L.: Psychophysiology: human Behavior & Physiological Response. Psychology Press, New York (2000)

    Google Scholar 

  • Arapakis, I., Konstas, I., Jose, J.M.: Using facial expressions and peripheral physiological signals as implicit indicators of topical relevance. In: Proceedings of the 17th ACM International Conference on Multimedia, MM ’09, pp. 461–470, New York (2009a). ACM. ISBN 978-1-60558-608-3. doi:10.1145/1631272.1631336

  • Arapakis, I., Moshfeghi, Y., Joho, H., Ren, R., Hannah, D., Jose, J.M.: Enriching user profiling with affective features for the improvement of a multimodal recommender system. In: Proceedings of the ACM International Conference on Image and Video Retrieval, CIVR ’09, pp. 29:1–29: 8, New York (2009b). ACM. ISBN 978-1-60558-480-5. doi:10.1145/1646396.1646433

  • Arapakis, I., Athanasakos, K., Jose, J.M.: A comparison of general vs personalised affective models for the prediction of topical relevance. In: Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’10, pp. 371–378, New York (2010). ACM. ISBN 978-1-4503-0153-4. doi:10.1145/1835449.1835512

  • Bagiella, E., Sloan, R.P., Heitjan, D.F.: Mixed-effects models in psychophysiology. Psychophysiology 37, 13–20, 1 (2000) ISSN 1469-8986. http://journals.cambridge.org/articleS0048577200980648

  • Barral, O., Jacucci, G.: Applying physiological computing methods to study psychological, affective and motivational relevance. In: Jacucci, G., Gamberini, L., Freeman, J., Spagnolli, A. (eds.) Symbiotic Interaction. Lecture Notes in Computer Science, vol. 8820, pp. 35–46. Springer, New York (2014)

    Google Scholar 

  • Barral, O., Eugster, M.J.A., Ruotsalo, T., Spapé, M.M., Kosunen, I., Ravaja, N., Kaski, S., Jacucci, G.: Exploring peripheral physiology as a predictor of perceived relevance in information retrieval. In: Proceedings of the 20th International Conference on Intelligent User Interfaces, IUI ’15, pp. 389–399, New York (2015) ACM. ISBN 978-1-4503-3306-1. doi:10.1145/2678025.2701389

  • Benedek, M, Kaernbach, C.: A continuous measure of phasic electrodermal activity. J. Neurosci. Methods 190(1), 80–91 (2010) ISSN 0165-0270. doi:10.1016/j.jneumeth.2010.04.028. http://www.sciencedirect.com/science/article/pii/S0165027010002335

  • Boucsein, W.: Electrodermal Activity. Springer, Boston (2012). ISBN 9781461411253; 9781461411260. http://edepot.wur.nl/207828

  • Cacioppo, J.T., Petty, R.E., Losch, M.E., Kim, H.S.: Electromyographic activity over facial muscle regions can differentiate the valence and intensity of affective reactions. J. Personal. Soc. Psychol. 50(2), 260–268, February 1986. ISSN 0022-3514. http://view.ncbi.nlm.nih.gov/pubmed/3701577

  • Carver, C.S., White, T.L.: Behavioral inhibition, behavioral activation, and affective responses to impending reward and punishment: the bis/bas scales. J. Personal. Soc. Psychol. 67(2), 319–333 (1994) ISSN 0022-3514. doi:10.1037/0022-3514.67.2.319

  • Celli, F., Ghosh, A., Alam, F., Riccardi, G.: In the mood for sharing contents: emotions, personality and interaction styles in the diffusion of news. Inf. Process. Manag. (2015). ISSN 0306-4573. doi:10.1016/j.ipm.2015.08.002. http://www.sciencedirect.com/science/article/pii/S030645731500103X

  • Conati, C., Maclaren, H.: Empirically building and evaluating a probabilistic model of user affect. User Model. User-Adapt. Interact. 19(3), 267–303 (2009)

    Article  Google Scholar 

  • Cosijn, E., Ingwersen, P.: Dimensions of relevance. Inf. Process. Manag. 36(4), 533–550 (2000). doi:10.1016/S0306-4573(99)00072-2

    Article  Google Scholar 

  • Davidson, J., Liebald, B., Liu, J., Nandy, P., Van Vleet, T., Gargi, U., Gupta, S., He, Y., Lambert, M., Livingston, B., Sampath, D.: The youtube video recommendation system. In: Proceedings of the Fourth ACM Conference on Recommender Systems, RecSys ’10, pp. 293–296, New York (2010). ACM. ISBN 978-1-60558-906-0. doi:10.1145/1864708.1864770

  • Dawson, M.E., Schell, A.M., Filion, D.L., Berntson, G.G.: The electrodermal system. In: Cacioppo, J.T., Tassinary, L.G., Berntson, G. (eds.) Handbook of Psychophysiology, 3rd edn, pp. 157–181. Cambridge University Press, Cambridge (2007)

    Chapter  Google Scholar 

  • Eugster, M.J.A., Ruotsalo, T., Spapé, M.M., Kosunen, I., Barral, O., Ravaja, N., Jacucci, G., Kaski, S.: Predicting term-relevance from brain signals. In: Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR ’14, pp. 425–434, New York (2014) ACM. ISBN 978-1-4503-2257-7. doi:10.1145/2600428.2609594

  • Fridlund, A.J., Cacioppo, J.T.: Guidelines for human electromyographic research. Psychophysiology 23(5), 567–589 (1986)

    Article  Google Scholar 

  • Gonzalez, G., de la Rosa, J.L., Montaner, M., Delfin, S.: Embedding emotional context in recommender systems. In: 2007 IEEE 23rd International Conference on Data Engineering Workshop, pp. 845–852, April 2007. doi:10.1109/ICDEW.2007.4401075

  • Gray, J.A., McNaughton, N.: The neuropsychology of Anxiety: an Enquiry into the Function of the Septo-Hippocampal System, 33rd edn. Oxford University Press, Oxford (2003)

    Book  Google Scholar 

  • Herlocker, J.L., Konstan, J.A., Terveen, L.G., Riedl, J.T.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1), 5–53 (2004). doi:10.1145/963770.963772

    Article  Google Scholar 

  • Ioannou, S.V., Raouzaiou, A.T., Tzouvaras, V.A., Mailis, T.P., Karpouzis, K.C., Kollias, S.D.: Emotion recognition through facial expression analysis based on a neurofuzzy network. Neural Netw. 18(4), 423–435 (2005). ISSN 0893-6080. doi:10.1016/j.neunet.2005.03.004. http://www.sciencedirect.com/science/article/pii/S0893608005000377. Emotion and Brain

  • Kelly, Diane, Fu, Xin: Elicitation of term relevance feedback: An investigation of term source and context. In: Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’06, pp. 453–460, New York (2006) ACM. ISBN 1-59593-369-7. doi:10.1145/1148170.1148249

  • Kelly, D., Teevan, J.: Implicit feedback for inferring user preference: a bibliography. SIGIR Forum 37(2), 18–28 (2003). doi:10.1145/959258.959260

    Article  Google Scholar 

  • Koelstra, S., Muhl, C., Soleymani, M., Lee, J.S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., Patras, I.: A database for emotion analysis;using physiological signals. IEEE Trans. Affect. Comput. 3(1), 18–31 (2012). doi:10.1109/T-AFFC.2011.15

    Article  Google Scholar 

  • Koelstra, S., Yazdani, A., Soleymani, M., Mhl, C., Lee, J.S., Nijholt, A., Pun, T., Ebrahimi, T., Patras, I.: Single trial classification of eeg and peripheral physiological signals for recognition of emotions induced by music videos. In: Yao, Y., Sun, R., Poggio, T., Liu, J., Zhong, N., Huang, J. (eds.) Brain Informatics. Brain Informatics, vol. 6334, pp. 89–100. Springer, Berlin (2010). doi:10.1007/978-3-642-15314-3_9

    Chapter  Google Scholar 

  • Koenemann, J., Belkin, N.J.: A case for interaction: A study of interactive information retrieval behavior and effectiveness. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’96, pP. 205–212, New York (1996). ACM. ISBN 0-89791-777-4. doi:10.1145/238386.238487

  • Leech, g: Introducing corpus annotation. In: Garside, R., Leech, G., Mcenery, A. (eds.) Corpus Annotation: linguistic Information from Computer Text Corpora. Addison Wesley Longman, New York (1997)

    Google Scholar 

  • Lopatovska, I.: Searching for good mood: examining relationships between search task and mood. Proc. Am. Soc. Inf. Sc. Technol. 46(1), 1–13 (2009). doi:10.1002/meet.2009.1450460222

    Article  Google Scholar 

  • Lopatovska, I., Arapakis, I.: Theories, methods and current research on emotions in library and information science, information retrieval and humancomputer interaction. Inf. Process. Manag. 47(4), 575–592 (2011). ISSN 0306-4573. doi:10.1016/j.ipm.2010.09.001. http://www.sciencedirect.com/science/article/pii/S0306457310000737

  • Moshfeghi, Y., Jose, J.M.: An effective implicit relevance feedback technique using affective, physiological and behavioural features. In: Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’13, pp. 133–142, New York (2013). ACM. ISBN 978-1-4503-2034-4. doi:10.1145/2484028.2484074

  • Pantic, M., Rothkrantz, L.J.M.: Toward an affect-sensitive multimodal human-computer interaction. Proc. IEEE 91(9), 1370–1390 (2003). doi:10.1109/JPROC.2003.817122

    Article  Google Scholar 

  • Posner, J., Russell, J.A., Peterson, B.S.: The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development, and psychopathology. Dev. Psychopathol. 17(03), 715–734 (2005)

    Article  Google Scholar 

  • Ravaja, N.: Contributions of psychophysiology to media research: review and recommendations. Media Psychol. 6(2), 193–235 (2004)

    Article  Google Scholar 

  • Saracevic, T.: Relevance reconsidered. In: Proceedings of the 2nd Conference on Conceptions of Library and Information Science (CoLIS 2), pp. 201–218. ACM Press, New York (1996)

  • Soleymani, M., Pantic, M.: Human-centered implicit tagging: overview and perspectives. In: 2012 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 3304–3309, Oct 2012. doi:10.1109/ICSMC.2012.6378301

  • Soleymani, M., Chanel, G., Kierkels, J.J.M., Pun, T.: Affective characterization of movie scenes based on multimedia content analysis and user’s physiological emotional responses. In: Tenth IEEE International Symposium on Multimedia, 2008. ISM 2008, pp. 228–235, Dec 2008. doi:10.1109/ISM.2008.14

  • Sternberg, S.: The discovery of processing stages: Extensions of donders’ method. Acta Psychol. 30(0), 276–315 (1969). ISSN 0001-6918. doi:10.1016/0001-6918(69)90055-9. http://www.sciencedirect.com/science/article/pii/0001691869900559

  • Tkalcic, M., Burnik, U., Kor, A.: Using affective parameters in a content-based recommender system for images. User Model. User-Adapt. Interact. 20(4), 279–311 (2010). doi:10.1007/s11257-010-9079-z

    Article  Google Scholar 

  • Tkalcic, M., Kosir, A., Tasic, J.: Affective recommender systems: the role of emotions in recommender systems. In: Proceedings The RecSys 2011 Workshop on Human Decision Making in Recommender Systems, pp. 9–13. Citeseer (2011)

  • Treacy Solovey, E., Afergan, D., Peck, E.M., Hincks, S.W., Jacob, R.J.K.: Designing implicit interfaces for physiological computing: Guidelines and lessons learned using fnirs. ACM Trans. Comput.-Hum. Interact. 21(6), 35:1–35:27, January 2015. ISSN 1073-0516. doi:10.1145/2687926

  • Van Boxtel, A., Jessurun, M.: Amplitude and bilateral coherency of facial and jaw-elevator emg activity as an index of effort during a two-choice serial reaction task. Psychophysiology 30(6), 589–604 (1993)

    Article  Google Scholar 

  • Veldhuizen, I.J.T., Gaillard, A.W.K., De Vries, J.: The influence of mental fatigue on facial emg activity during a simulated workday. Biol. Psychol. 63(1), 59–78 (2003)

    Article  Google Scholar 

  • Venables, P.H., Mitchell, D.A.: The effects of age, sex and time of testing on skin conductance activity. Biol. Psychol. 43(2), 87–101 (1996). ISSN 0301-0511. doi:10.1016/0301-0511(96)05183-6. http://www.sciencedirect.com/science/article/pii/0301051196051836

  • Waterink, W., Van Boxtel, A.: Facial and jaw-elevator emg activity in relation to changes in performance level during a sustained information processing task. Biol. Psychol. 37(3), 183–198 (1994)

    Article  Google Scholar 

  • Zhai, C., Lafferty, J.: A study of smoothing methods for language models applied to information retrieval. ACM Trans. Inf. Syst. (TOIS) 22(2), 179–214 (2004)

    Article  Google Scholar 

Download references

Acknowledgements

This work has been partly supported by the Academy of Finland (278090, Multivire, 255725; and the Finnish Centre of Excellence in Computational Inference Research COIN) and MindSee (FP7—ICT; Grant Agreement # 611570). Certain data included herein are derived from the Web of Science prepared by THOMSON REUTERS, Inc., Philadelphia, Pennsylvania, USA: Copyright THOMSON REUTERS, 2011. All rights reserved. Data are also included from the Digital Libraries of the ACM, IEEE, and Springer.

Author information

Authors and Affiliations

  1. Department of Computer Science, Helsinki Institute for Information Technology HIIT, University of Helsinki, PO Box 68, 00014, Helsinki, Finland

    Oswald Barral, Ilkka Kosunen & Giulio Jacucci

  2. Department of Computer Science, Helsinki Institute for Information Technology HIIT, Aalto University, PO Box 15400, 00076, Helsinki, Finland

    Tuukka Ruotsalo, Michiel M. Spapé, Manuel J. A. Eugster, Niklas Ravaja, Samuel Kaski & Giulio Jacucci

  3. Department of Social Research, Helsinki Institute for Information Technology HIIT, University of Helsinki, PO Box 54, 00014, Helsinki, Finland

    Niklas Ravaja

Authors
  1. Oswald Barral
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ilkka Kosunen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tuukka Ruotsalo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michiel M. Spapé
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Manuel J. A. Eugster
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Niklas Ravaja
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Samuel Kaski
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Giulio Jacucci
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Oswald Barral.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Barral, O., Kosunen, I., Ruotsalo, T. et al. Extracting relevance and affect information from physiological text annotation. User Model User-Adap Inter 26, 493–520 (2016). https://doi.org/10.1007/s11257-016-9184-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11257-016-9184-8

Keywords

Search

Navigation