Tuukka Ruotsalo
Luis A. Leiva
ABSTRACT
Information retrieval (IR) relies on a general notion of relevance, which is used as the principal foundation for ranking and evaluation methods. However, IR does not account for more a nuanced affective experience. Here, we consider the emotional response decoded directly from the human brain as an alternative dimension of relevance. We report an experiment covering seven different scenarios in which we measure and predict how users emotionally respond to visual image contents by using functional near-infrared spectroscopy (fNIRS) neuroimaging on two commonly used affective dimensions: valence (negativity and positivity) and arousal (boredness and excitedness). Our results show that affective states can be successfully decoded using fNIRS, and utilized to complement the present notion of relevance in IR studies. For example, we achieved 0.39 Balanced accuracy and 0.61 AUC in 4-class classification of affective states (vs. 0.25 Balanced accuracy and 0.5 AUC of a random classifier). Likewise, we achieved 0.684 Precision@20 when retrieving high-arousal images. Our work opens new avenues for incorporating emotional states in IR evaluation, affective feedback, and information filtering.
- Eugene Agichtein, Eric Brill, and Susan Dumais. 2006. Improving Web Search Ranking by Incorporating User Behavior Information. In Proc. SIGIR.Google Scholar
Digital Library
- Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization framework. In Proc. KDD.Google ScholarDigital Library
- Ioannis Arapakis, Joemon M Jose, and Philip D Gray. 2008. Affective feedback: an investigation into the role of emotions in the information seeking process. In Proc. SIGIR.Google Scholar
- Ioannis Arapakis and Luis A. Leiva. 2020. Learning Efficient Representations of Mouse Movements to Predict User Attention. In Proc. SIGIR.Google Scholar
- Pedro Avero and Manuel G Calvo. 2006. Affective priming with pictures of emotional scenes: The role of perceptual similarity and category relatedness. Span. J. Psychol., Vol. 9, 1 (2006).Google ScholarCross Ref
- Hasan Ayaz, Meltem Izzetoglu, Kurtulus Izzetoglu, and Banu Onaral. 2019. The use of functional near-infrared spectroscopy in neuroergonomics. In Neuroergonomics. Elsevier.Google Scholar
- Michela Balconi, Elisabetta Grippa, and Maria Elide Vanutelli. 2015a. Resting lateralized activity predicts the cortical response and appraisal of emotions: an fNIRS study. Soc. Cogn. Affect. Neurosci., Vol. 10, 12 (2015).Google ScholarCross Ref
- Michela Balconi, Elisabetta Grippa, and Maria Elide Vanutelli. 2015b. What hemodynamic (fNIRS), electrophysiological (EEG) and autonomic integrated measures can tell us about emotional processing. Brain Cogn., Vol. 95 (2015).Google ScholarCross Ref
- Oswald Barral, Manuel JA Eugster, Tuukka Ruotsalo, Michiel M Spapé, Ilkka Kosunen, Niklas Ravaja, Samuel Kaski, and Giulio Jacucci. 2015. Exploring peripheral physiology as a predictor of perceived relevance in information retrieval. In Proc. IUI.Google ScholarDigital Library
- Lukas Brückner, Ioannis Arapakis, and Luis A. Leiva. 2021. When Choice Happens: A Systematic Examination of Mouse Movement Length for Decision Making in Web Search. In Proc. SIGIR.Google Scholar
- Scott C Bunce, Meltem Izzetoglu, Kurtulus Izzetoglu, Banu Onaral, and Kambiz Pourrezaei. 2006. Functional near-infrared spectroscopy. IEEE Eng. Med. Biol. Mag., Vol. 25, 4 (2006).Google ScholarCross Ref
- Luis Carretié. 2014. Exogenous (automatic) attention to emotional stimuli: a review. Cogn. Affect. Behav. Neurosci., Vol. 14, 4 (2014).Google ScholarCross Ref
- Ming Chen, Lu Zhang, and Jan P Allebach. 2015. Learning deep features for image emotion classification. In Proc. ICIP.Google ScholarDigital Library
- Xu Cui, Signe Bray, and Allan L Reiss. 2010. Functional near infrared spectroscopy (NIRS) signal improvement based on negative correlation between oxygenated and deoxygenated hemoglobin dynamics. Neuroimage, Vol. 49, 4 (2010), 3039--3046.Google ScholarCross Ref
- Keith M Davis, Carlos de la Torre-Ortiz, and Tuukka Ruotsalo. 2022a. Brain-Supervised Image Editing. In Proc. CVPR.Google Scholar
- Keith M. Davis, Michiel Spapé, and Tuukka Ruotsalo. 2022b. Contradicted by the Brain: Predicting Individual and Group Preferences via Brain-Computer Interfacing. IEEE Trans. Affect. Comput. (2022).Google Scholar
- Keith M. Davis III, Michiel Spapé, and Tuukka Ruotsalo. 2021. Collaborative Filtering with Preferences Inferred from Brain Signals. In Proc. WWW.Google Scholar
- Carlos de la Torre-Ortiz, Michiel M Spapé, Lauri Kangassalo, and Tuukka Ruotsalo. 2020. Brain relevance feedback for interactive image generation. In Proc. UIST.Google ScholarDigital Library
- David T Delpy, Mark Cope, Pieter van der Zee, Simon Arridge, Susan Wray, and JS Wyatt. 1988. Estimation of optical pathlength through tissue from direct time of flight measurement. Phys. Med. Biol., Vol. 33, 12 (1988).Google Scholar
- Manuel JA Eugster, Tuukka Ruotsalo, Michiel M Spapé, Oswald Barral, Niklas Ravaja, Giulio Jacucci, and Samuel Kaski. 2016. Natural brain-information interfaces: Recommending information by relevance inferred from human brain signals. Sci. Rep., Vol. 6, 1 (2016).Google ScholarCross Ref
- Manuel J.A. Eugster, Tuukka Ruotsalo, Michiel M. Spapé, Ilkka Kosunen, Oswald Barral, Niklas Ravaja, Giulio Jacucci, and Samuel Kaski. 2014. Predicting Term-Relevance from Brain Signals. In Proc. SIGIR.Google ScholarDigital Library
- Frank A Fishburn, Ruth S Ludlum, Chandan J Vaidya, and Andrei V Medvedev. 2019. Temporal derivative distribution repair (TDDR): a motion correction method for fNIRS. Neuroimage, Vol. 184 (2019).Google ScholarCross Ref
- Don C Fowles. 1980. The three arousal model: Implications of Gray's two-factor learning theory for heart rate, electrodermal activity, and psychopathy. Int. J. Psychophysiol., Vol. 17, 2 (1980), 87--104.Google ScholarCross Ref
- E Grippa, Maria Elide Vanutelli, I Venturella, E Molteni, and Michela Balconi. 2014. Hemodynamic responses (fNIRS) and EEG modulation of prefrontal cortex during emotion processing. In Proc. SIPF.Google Scholar
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In Proc. CVPR.Google ScholarCross Ref
- Fabian Herold, Patrick Wiegel, Felix Scholkmann, and Notger G Müller. 2018. Applications of functional near-infrared spectroscopy (fNIRS) neuroimaging in exercise-cognition science: a systematic, methodology-focused review. J. Clin. Med., Vol. 7, 12 (2018).Google ScholarCross Ref
- Xin Hu, Chu Zhuang, Fei Wang, Yong-Jin Liu, Chang-Hwan Im, and Dan Zhang. 2019. fNIRS evidence for recognizably different positive emotions. Front. Hum. Neurosci., Vol. 13 (2019).Google ScholarCross Ref
- Giulio Jacucci, Oswald Barral, Pedram Daee, Markus Wenzel, Baris Serim, Tuukka Ruotsalo, Patrik Pluchino, Jonathan Freeman, Luciano Gamberini, Samuel Kaski, et al. 2019. Integrating neurophysiologic relevance feedback in intent modeling for information retrieval. J. Assoc. Inf. Sci. Technol., Vol. 70, 9 (2019).Google ScholarDigital Library
- Thorsten Joachims. 2002. Optimizing search engines using clickthrough data. In Proc. KDD.Google ScholarDigital Library
- Lauri Kangassalo, Michiel Spapé, Giulio Jacucci, and Tuukka Ruotsalo. 2019. Why do users issue good queries? Neural correlates of term specificity. In Proc. SIGIR.Google Scholar
- Lauri Kangassalo, Michiel Spapé, and Tuukka Ruotsalo. 2020. Neuroadaptive modelling for generating images matching perceptual categories. Sci. Rep., Vol. 10, 1 (2020).Google ScholarCross Ref
- Aditya Khosla, Atish Das Sarma, and Raffay Hamid. 2014. What makes an image popular?. In Proc. WWW.Google ScholarDigital Library
- Peter J. Lang, Margaret M. Bradley, and Bruce N. Cuthbert. 2008. International Affective Picture System (IAPS): Affective ratings of pictures and instruction manual.Google Scholar
- Jana Machajdik and Allan Hanbury. 2010. Affective image classification using features inspired by psychology and art theory. In Proc. MM.Google ScholarDigital Library
- Daniel McDuff, Paul Thomas, Nick Craswell, Kael Rowan, and Mary Czerwinski. 2021. Do Affective Cues Validate Behavioural Metrics for Search?. In Proc. SIGIR.Google ScholarDigital Library
- Dominika Michalkova, Mario Parra-Rodriguez, and Yashar Moshfeghi. 2022. Information Need Awareness: an EEG study. In Proc. SIGIR.Google ScholarDigital Library
- Yashar Moshfeghi and Joemon M Jose. 2013. An effective implicit relevance feedback technique using affective, physiological and behavioural features. In Proc. SIGIR.Google ScholarDigital Library
- Sakrapee Paisalnan, Yashar Moshfeghi, and Frank Pollick. 2021. Neural Correlates of Realisation of Satisfaction in a Successful Search Process. Proc. Assoc. Inf. Sci., Vol. 58, 1 (2021).Google ScholarCross Ref
- Zuzana Pinkosova, William J. McGeown, and Yashar Moshfeghi. 2020. The Cortical Activity of Graded Relevance. In Proc. SIGIR.Google ScholarDigital Library
- Luca Pollonini, Cristen Olds, Homer Abaya, Heather Bortfeld, Michael S Beauchamp, and John S Oghalai. 2014. Auditory cortex activation to natural speech and simulated cochlear implant speech measured with functional near-infrared spectroscopy. Hear. Res., Vol. 309 (2014).Google ScholarCross Ref
- Tianrong Rao, Xiaoxu Li, and Min Xu. 2020. Learning multi-level deep representations for image emotion classification. Neural Process. Lett., Vol. 51, 3 (2020).Google ScholarDigital Library
- James A Russell. 1980. A circumplex model of affect. J. Pers. Soc. Psychol., Vol. 39, 6 (1980).Google ScholarCross Ref
- Stefanie Schmidt and Wolfgang G Stock. 2009. Collective indexing of emotions in images. A study in emotional information retrieval. J. Assoc. Inf. Sci. Technol., Vol. 60, 5 (2009).Google Scholar
- Lucas R Trambaiolli, Juliana Tossato, André M Cravo, Claudinei E Biazoli Jr, and Jo ao R Sato. 2021. Subject-independent decoding of affective states using functional near-infrared spectroscopy. Plos one, Vol. 16, 1 (2021), e0244840.Google ScholarCross Ref
- Anirudh Unni, Klas Ihme, Henrik Surm, Lars Weber, Andreas Lüdtke, Daniela Nicklas, Meike Jipp, and Jochem W Rieger. 2015. Brain activity measured with fNIRS for the prediction of cognitive workload. In Proc. CogInfoCom. IEEE.Google ScholarCross Ref
- Carmen Vidaurre and Benjamin Blankertz. 2010. Towards a Cure for BCI Illiteracy. Brain Topogr., Vol. 23, 2 (2010).Google ScholarCross Ref
- Ji Wan, Dayong Wang, Steven Chu Hong Hoi, Pengcheng Wu, Jianke Zhu, Yongdong Zhang, and Jintao Li. 2014. Deep learning for content-based image retrieval: A comprehensive study. In Proc. MM.Google ScholarDigital Library
- Jiang Wang, Yang Song, Thomas Leung, Chuck Rosenberg, Jingbin Wang, James Philbin, Bo Chen, and Ying Wu. 2014. Learning fine-grained image similarity with deep ranking. In Proc. CVPR.Google ScholarDigital Library
- Victoria Yanulevskaya, Jan C van Gemert, Katharina Roth, Ann-Katrin Herbold, Nicu Sebe, and Jan-Mark Geusebroek. 2008. Emotional valence categorization using holistic image features. In Proc. ICIP.Google ScholarCross Ref
- Sicheng Zhao, Guiguang Ding, Qingming Huang, Tat-Seng Chua, Björn W Schuller, and Kurt Keutzer. 2018. Affective Image Content Analysis: A Comprehensive Survey. In Proc. IJCAI.Google ScholarCross Ref
- Sicheng Zhao, Yue Gao, Xiaolei Jiang, Hongxun Yao, Tat-Seng Chua, and Xiaoshuai Sun. 2014. Exploring principles-of-art features for image emotion recognition. In Proc. MM.Google ScholarDigital Library
- Sicheng Zhao, Xingxu Yao, Jufeng Yang, Guoli Jia, Guiguang Ding, Tat-Seng Chua, Bjoern W Schuller, and Kurt Keutzer. 2021. Affective image content analysis: Two decades review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intellig. (2021).Google Scholar
Index Terms
- Affective Relevance: Inferring Emotional Responses via fNIRS Neuroimaging
Recommendations
Psychological responses to simulated displays of mismatched emotional expressions
Embodied agents are often designed with the ability to simulate human emotion. This paper investigates the psychological impact of simulated emotional expressions on computer users with a particular emphasis on how mismatched facial and audio ...
Affective Characterization of Movie Scenes Based on Multimedia Content Analysis and User's Physiological Emotional Responses
ISM '08: Proceedings of the 2008 Tenth IEEE International Symposium on MultimediaIn this paper, we propose an approach for affective representation of movie scenes based on the emotions that are actually felt by spectators. Such a representation can be used for characterizing the emotional content of video clips for e.g. affective ...
Affective agents for enhancing emotional experience
HAI '14: Proceedings of the second international conference on Human-agent interactionWe propose shared emotional experience agents. They enhance the user's emotional experience by emotional contagion. Our experiment has 12 participants watch videos together with a robot that expresses an emotional state by body and voice. The results ...
Comments