Yomna Abdelrahman
Anam Ahmad Khan
Joshua Newn
Eduardo Velloso
Andreas Bulling
Frank Vetere
Albrecht Schmidt
Skip Abstract Section
Abstract
Despite the importance of attention in user performance, current methods for attention classification do not allow to discriminate between different attention types. We propose a novel method that combines thermal imaging and eye tracking to unobtrusively classify four types of attention: sustained, alternating, selective, and divided. We collected a data set in which we stimulate these four attention types in a user study (N = 22) using combinations of audio and visual stimuli while measuring users' facial temperature and eye movement. Using a Logistic Regression on features extracted from both sensing technologies, we can classify the four attention types with high AUC scores up to 75.7% for the user independent-condition independent, 87% for the user-independent-condition dependent, and 77.4% for the user-dependent prediction. Our findings not only demonstrate the potential of thermal imaging and eye tracking for unobtrusive classification of different attention types but also pave the way for novel applications for attentive user interfaces and attention-aware computing.
- Yomna Abdelrahman, Mariam Hassib, Maria Guinea Marquez, Markus Funk, and Albrecht Schmidt. 2015. Implicit Engagement Detection for Interactive Museums Using Brain-Computer Interfaces. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct (MobileHCI '15). ACM, New York, NY, USA, 838--845. Google Scholar
Digital Library
- Yomna Abdelrahman, Eduardo Velloso, Tilman Dingler, Albrecht Schmidt, and Frank Vetere. 2017. Cognitive Heat: Exploring the Usage of Thermal Imaging to Unobtrusively Estimate Cognitive Load. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 3, Article 33 (Sept. 2017), 20 pages. Google ScholarDigital Library
- David W. Aha. 1997. Lazy Learning. Kluwer Academic Publishers, Norwell, MA, USA, Chapter Lazy Learning, 7--10. http://dl.acm.org/citation.cfm?id=273530.273534 Google ScholarDigital Library
- James Rowland Angell. 1910. Psychology: An Introductory Study of the Structure and Function of Human Consciousness. H. Holt.Google Scholar
- T Armstrong and BO Olatunji. 2009. What They See Is What You Get: Eye Tracking of Attention in the Anxiety Disorders. Psychological Science Agenda 23, 3 (2009).Google Scholar
- Othman Asiry, Haifeng Shen, and Paul Calder. 2015. Extending Attention Span of ADHD Children Through an Eye Tracker Directed Adaptive User Interface. In Proceedings of the ASWEC 2015 24th Australasian Software Engineering Conference (ASWEC ' 15 Vol. II). ACM, New York, NY, USA, 149--152. Google ScholarDigital Library
- Jonas Auda, Dominik Weber, Alexandra Voit, and Stefan Schneegass. 2018. Understanding User Preferences Towards Rule-based Notification Deferral. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (CHI EA '18). ACM, New York, NY, USA, Article LBW584, 6 pages. Google ScholarDigital Library
- Andreas Bulling. 2016. Pervasive Attentive User Interfaces. IEEE Computer 49, 1 (2016), 94--98. Google ScholarDigital Library
- Carlos Carreiras, André Lourenço, Helena Aidos, Hugo Plácido da Silva, and Ana L. N. Fred. 2016. Unsupervised Analysis of Morphological ECG Features for Attention Detection. Springer International Publishing, Cham, 437--453.Google Scholar
- O. T.. Chen, P. Chen, and Y. Tsai. 2017. Attention Estimation System via Smart Glasses. (Aug 2017), 1--5.Google Scholar
- Thomas H. Davenport and John C. Beck. 2001. The Attention Economy. Ubiquity 2001, May, Article 6 (May 2001). Google ScholarDigital Library
- Tilman Dingler, Albrecht Schmidt, and Tonja Machulla. 2017. Building Cognition-Aware Systems: A Mobile Toolkit for Extracting Time-of-Day Fluctuations of Cognitive Performance. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 3, Article 47 (Sept. 2017), 15 pages. Google ScholarDigital Library
- Stephan Dreiseitl and Lucila Ohno-Machado. 2002. Logistic Regression and Artificial Neural Network Classification Models: A Methodology Review. Journal of Biomedical Informatics 35, 5 (2002), 352--359. Google ScholarDigital Library
- Albert Hoang Duc, Paul Bays, and Masud Husain. 2008. Chapter 5.5 - Eye movements as a Probe of Attention. In Using Eye Movements as an Experimental Probe of Brain Function, Christopher Kennard and R.John Leigh (Eds.). Progress in Brain Research, Vol. 171. Elsevier, 403--411.Google Scholar
- Ami Eidels, James T. Townsend, and Daniel Algom. 2010. Comparing perception of Stroop stimuli in focused versus divided attention paradigms: Evidence for dramatic processing differences. Cognition 114, 2 (2010), 129--150.Google ScholarCross Ref
- Mai ElKomy, Yomna Abdelrahman, Markus Funk, Tilman Dingler, Albrecht Schmidt, and Slim Abdennadher. 2017. ABBAS: An Adaptive Bio-sensors Based Assistive System. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '17). ACM, New York, NY, USA, 2543--2550. Google ScholarDigital Library
- Charles W Eriksen and James E Hoffman. 1972. Temporal and spatial characteristics of selective encoding from visual displays. Perception & Psychophysics 12, 2 (1972), 201--204.Google ScholarCross Ref
- Josef Falkinger. {n.d.}. Limited Attention as a Scarce Resource in Information-Rich Economies. The Economic Journal 118, 532 ({n. d.}), 1596--1620.Google Scholar
- Josef Falkinger. 2007. Attention economies. Journal of Economic Theory 133, 1 (2007), 266--294.Google ScholarCross Ref
- Maite Frutos-Pascual and Begonya Garcia-Zapirain. 2015. Assessing Visual Attention Using Eye Tracking Sensors in Intelligent Cognitive Therapies Based on Serious Games. Sensors 15, 5 (2015), 11092--11117.Google ScholarCross Ref
- Markus Funk, Tilman Dingler, Jennifer Cooper, and Albrecht Schmidt. 2015. Stop Helping Me - I'M Bored!: Why Assembly Assistance Needs to Be Adaptive. In Adjunct Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM International Symposium on Wearable Computers (UbiComp/ISWC'15 Adjunct). ACM, New York, NY, USA, 1269--1273. Google ScholarDigital Library
- John M. Gardiner and Alan J. Parkin. 1990. Attention and Recollective Experience in Recognition Memory. Memory & Cognition 18, 6 (01 Nov 1990), 579--583.Google Scholar
- Gongde Guo, Hui Wang, David Bell, Yaxin Bi, and Kieran Greer. 2003. KNN Model-Based Approach in Classification. In On The Move to Meaningful Internet Systems 2003: CoopIS, DOA, and ODBASE, Robert Meersman, Zahir Tari, and Douglas C.Schmidt (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 986--996.Google Scholar
- Sandra G. Hart. 2006. Nasa-Task Load Index (NASA-TLX); 20 Years Later. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 50, 9, 904--908.Google ScholarCross Ref
- Mariam Hassib, Stefan Schneegass, Philipp Eiglsperger, Niels Henze, Albrecht Schmidt, and Florian Alt. 2017. EngageMeter: A System for Implicit Audience Engagement Sensing Using Electroencephalography. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 5114--5119. Google ScholarDigital Library
- James E. Hoffman and Baskaran Subramaniam. 1995. The role of visual attention in saccadic eye movements. Perception & Psychophysics 57, 6 (01 Jan 1995), 787--795.Google Scholar
- Stephanos Ioannou, Sjoerd Ebisch, Tiziana Aureli, Daniela Bafunno, Helene Alexi Ioannides, Daniela Cardone, Barbara Manini, Gian Luca Romani, Vittorio Gallese, and Arcangelo Merla. 2013. The autonomic signature of guilt in children: a thermal infrared imaging study. PloS one 8, 11 (2013), e79440.Google ScholarCross Ref
- Stephanos Ioannou, Vittorio Gallese, and Arcangelo Merla. 2014. Thermal infrared imaging in psychophysiology: potentialities and limits. Psychophysiology 51, 10 (2014), 951--963.Google ScholarCross Ref
- Daniel Kahneman. 1973. Attention and Effort. Vol. 1063. Prentice-Hall Englewood Cliffs, NJ.Google Scholar
- Kimberly A. Kerns, Karen Eso, and Jennifer Thomson. 1999. Investigation of a Direct Intervention for Improving Attention in Young Children With ADHD. Developmental Neuropsychology 16, 2 (1999), 273--295.Google ScholarCross Ref
- Thomas Kosch, Yomna Abdelrahman, Markus Funk, and Albrecht Schmidt. 2017. One Size Does Not Fit All: Challenges of Providing Interactive Worker Assistance in Industrial Settings. In Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2017 ACM International Symposium on Wearable Computers (UbiComp '17). ACM, New York, NY, USA, 1006--1011. Google ScholarDigital Library
- Nataliya Kosmyna, Caitlin Morris, Utkarsh Sarawgi, and Pattie Maes. 2019. AttentivU: A Biofeedback System for Real-time Monitoring and Improvement of Engagement. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA '19). ACM, New York, NY, USA, Article VS07, 2 pages. Google ScholarDigital Library
- P.Kuyper. 1972. The cocktail party effect. Audiology 11, 5-6 (1972), 277--282.Google Scholar
- Martijn J. M. Lamers, Ardi Roelofs, and Inge M. Rabeling-Keus. 2010. Selective attention and response set in the Stroop task. Memory & Cognition 38, 7 (01 Oct 2010), 893--904.Google Scholar
- Muhamad Hafiz Abd Latif, Hazlina Md Yusof, S. Naim Sidek, and Nazreen Rusli. 2015. Implementation of GLCM Features in Thermal Imaging for Human Affective State Detection. Procedia Computer Science 76 (2015), 308--315.Google ScholarCross Ref
- Sophie Leroy. 2009. Why is it so hard to do my work? The challenge of attention residue when switching between work tasks. Organizational Behavior and Human Decision Processes 109, 2 (2009), 168--181.Google ScholarCross Ref
- Yongchang Li, Xiaowei Li, Martyn Ratcliffe, Li Liu, Yanbing Qi, and Quanying Liu. 2011. A Real-time EEG-based BCI System for Attention Recognition in Ubiquitous Environment. In Proceedings of 2011 International Workshop on Ubiquitous Affective Awareness and Intelligent Interaction (UAAII '11). ACM, New York, NY, USA, 33--40. Google ScholarDigital Library
- Lars Lischke, Sven Mayer, Andreas Preikschat, Markus Schweizer, Ba Vu, Paweł W. Woźniak, and Niels Henze. 2018. Understanding Large Display Environments: Contextual Inquiry in a Control Room. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (CHI EA '18). ACM, New York, NY, USA, Article LBW134, 6 pages. Google ScholarDigital Library
- Ning-Han Liu, Cheng-Yu Chiang, and Hsuan-Chin Chu. 2013. Recognizing the Degree of Human Attention Using EEG Signals from Mobile Sensors. Sensors 13, 8 (2013), 10273--10286.Google ScholarCross Ref
- Jun Ma, Du Lei, Xingming Jin, Xiaoxia Du, Fan Jiang, Fei Li, Yiwen Zhang, and Xiaoming Shen. 2012. Compensatory brain activation in children with attention deficit/hyperactivity disorder during a simplified Go/No-go task. Journal of Neural Transmission 119, 5 (2012), 613--619.Google ScholarCross Ref
- Matei Mancas, Vincent P Ferrera, Nicolas Riche, and John G Taylor. 2016. From Human Attention to Computational Attention: A Multidisciplinary Approach. Vol. 10. Springer. Google ScholarDigital Library
- Gloria Mark, Mary Czerwinski, and Shamsi T. Iqbal. 2018. Effects of Individual Differences in Blocking Workplace Distractions. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 92, 12 pages. Google ScholarDigital Library
- Allan F. Mirsky, Bruno J. Anthony, Connie C. Duncan, Mary Beth Ahearn, and Sheppard G. Kellam. 1991. Analysis of the elements of attention: A neuropsychological approach. Neuropsychology Review 2, 2 (01 Jun 1991), 109--145.Google ScholarCross Ref
- Mahnas Jean Mohammadi-Aragh, John E. Ball, and Donna Jaison. 2016. Using wavelets to categorize student attention patterns. In 2016 IEEE Frontiers in Education Conference (FIE). 1--8.Google ScholarCross Ref
- Mona Moisala, Viljami Salmela, Emma Salo, Synnöve Carlson, Virve Vuontela, Oili Salonen, and Kimmo Alho. 2015. Brain activity during divided and selective attention to auditory and visual sentence comprehension tasks. Frontiers in Human Neuroscience 9 (2015), 86.Google ScholarCross Ref
- Moshe Naveh-Benjamin, Jonathan Guez, Yoko Hara, Matthew S. Brubaker, and Iris Lowenschuss-Erlich. 2014. The Effects of Divided Attention on Encoding Processes under Incidental and Intentional Learning Instructions: Underlying Mechanisms? Quarterly Journal of Experimental Psychology 67, 9 (2014), 1682--1696.Google ScholarCross Ref
- Joshua Newn, Fraser Allison, Eduardo Velloso, and Frank Vetere. 2018. Looks Can Be Deceiving: Using Gaze Visualisation to Predict and Mislead Opponents in Strategic Gameplay. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 261, 12 pages. Google ScholarDigital Library
- Joshua Newn, Eduardo Velloso, Fraser Allison, Yomna Abdelrahman, and Frank Vetere. 2017. Evaluating Real-Time Gaze Representations to Infer Intentions in Competitive Turn-Based Strategy Games. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play (CHI PLAY '17). ACM, New York, NY, USA, 541--552. Google ScholarDigital Library
- Anat Ninio and Daniel Kahneman. 1974. Reaction time in focused and in divided attention. Journal of Experimental Psychology 103, 3 (1974), 394.Google ScholarCross Ref
- Michael I. Posner. 1980. Orienting of Attention. Quarterly Journal of Experimental Psychology 32, 1 (1980), 3--25.Google ScholarCross Ref
- Michael I. Posner and Steven E. Petersen. 1990. The Attention System of the Human Brain. Annual Review of Neuroscience 13, 1 (1990), 25--42.Google ScholarCross Ref
- Penelope J Qualls and Peter W Sheehan. 1981. Role of the feedback signal in electromyograph biofeedback: The relevance of attention. Journal of Experimental Psychology: General 110, 2 (1981), 204.Google ScholarCross Ref
- Jailan Salah, Yomna Abdelrahman, Yasmeen Abdrabou, Khaled Kassem, and Slim Abdennadher. 2018. Exploring the Usage of Commercial Bio-Sensors for Multitasking Detection. In Proceedings of the 17th International Conference on Mobile and Ubiquitous Multimedia (MUM 2018). ACM, New York, NY, USA, 265--277. Google ScholarDigital Library
- Jailan Salah, Yomna Abdelrahman, Ahmed Dakrouni, and Slim Abdennadher. 2018. Judged by the Cover: Investigating the Effect of Adaptive Game Interface on the Learning Experience. In Proceedings of the 17th International Conference on Mobile and Ubiquitous Multimedia (MUM 2018). ACM, New York, NY, USA, 215--225. Google ScholarDigital Library
- Dario D. Salvucci and Joseph H. Goldberg. 2000. Identifying Fixations and Saccades in Eye-tracking Protocols. In Proceedings of the 2000 Symposium on Eye Tracking Research & Applications (ETRA '00). ACM, New York, NY, USA, 71--78. Google ScholarDigital Library
- Jerritta Selvaraj, Murugappan M, R Nagarajan, and Khairunizam Wan. 2011. Physiological signals based human emotion Recognition: a review. (03 2011).Google Scholar
- Dvijesh Shastri, Arcangelo Merla, Panagiotis Tsiamyrtzis, and Ioannis Pavlidis. 2009. Imaging facial signs of neurophysiological responses. IEEE Transactions on Biomedical Engineering 56, 2 (2009), 477--484.Google ScholarCross Ref
- Rajita Sinha, William R Lovallo, and Oscar A Parsons. 1992. Cardiovascular differentiation of emotions. Psychosomatic Medicine 54, 4 (1992), 422--435.Google ScholarCross Ref
- McKay Moore Sohlberg and Catherine A. Mateer. 1987. Effectiveness of an attention-training program. Journal of Clinical and Experimental Neuropsychology 9, 2 (1987), 117--130.Google ScholarCross Ref
- Namrata Srivastava, Joshua Newn, and Eduardo Velloso. 2018. Combining Low and Mid-Level Gaze Features for Desktop Activity Recognition. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 4, Article 189 (Dec. 2018), 27 pages. Google ScholarDigital Library
- Julian Steil, Philipp Müller, Yusuke Sugano, and Andreas Bulling. 2018. Forecasting User Attention During Everyday Mobile Interactions Using Device-integrated and Wearable Sensors. In Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '18). ACM, New York, NY, USA, Article 1, 13 pages. Google ScholarDigital Library
- J Ridley Stroop. 1992. Studies of interference in serial verbal reactions. Journal of Experimental Psychology: General 121, 1 (1992), 15.Google ScholarCross Ref
- Walter Sturm, Klaus Willmes, Bernt Orgass, and Wolfgang Hartje. 1997. Do Specific Attention Deficits Need Specific Training? Neuropsychological Rehabilitation 7, 2 (1997), 81--103.Google ScholarCross Ref
- Elizabeth A. Styles. 1997. The Psychology of Attention. Psychology Press.Google Scholar
- Benjamin Tag, Ryan Mannschreck, Kazunori Sugiura, George Chernyshov, Naohisa Ohta, and Kai Kunze. 2017. Facial Thermography for Attention Tracking on Smart Eyewear: An Initial Study. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '17). ACM, New York, NY, USA, 2959--2966. Google ScholarDigital Library
- Dereck Toker, Cristina Conati, Ben Steichen, and Giuseppe Carenini. 2013. Individual User Characteristics and Information Visualization: Connecting the Dots Through Eye Tracking. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 295--304. Google ScholarDigital Library
- Geertje van Daalen, Tineke M Willemsen, Karin Sanders, and Marc JPM van Veldhoven. 2009. Emotional exhaustion and mental health problems among employees doing "people work": The impact of job demands, job resources and family-to-work conflict. International archives of occupational and environmental health 82, 3 (2009), 291--303.Google Scholar
- Mélodie Vidal, Jayson Turner, Andreas Bulling, and Hans Gellersen. 2012. Wearable eye tracking for mental health monitoring. Computer Communications 35, 11 (2012), 1306--1311. Google ScholarDigital Library
- Alexandra Voit, Benjamin Poppinga, Dominik Weber, Matthias Böhmer, Niels Henze, Sven Gehring, Tadashi Okoshi, and Veljko Pejovic. 2016. UbiTtention: Smart & Ambient Notification and Attention Management. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct (UbiComp '16). ACM, New York, NY, USA, 1520--1523. Google ScholarDigital Library
- Dominik Weber, Alireza Sahami Shirazi, Sven Gehring, Niels Henze, Benjamin Poppinga, Martin Pielot, and Tadashi Okoshi. 2016. Smarttention, Please!: 2nd Workshop on Intelligent Attention Management on Mobile Devices. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct (MobileHCI '16). ACM, New York, NY, USA, 914--917. Google ScholarDigital Library
- Pawel W. Wozniak, Lars Lischke, Sven Mayer, Andreas Preikschat, Markus Schweizer, Ba Vu, Carlo von Molo, and Niels Henze. 2017. Understanding Work in Public Transport Management Control Rooms. In Companion of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17 Companion). ACM, New York, NY, USA, 339--342. Google ScholarDigital Library
- Richard D Wright and Lawrence M Ward. 2008. Orienting of attention. Oxford University Press.Google Scholar
- Johannes Zagermann, Ulrike Pfeil, and Harald Reiterer. 2018. Studying Eye Movements As a Basis for Measuring Cognitive Load. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (CHI EA '18). ACM, New York, NY, USA, Article LBW095, 6 pages. Google ScholarDigital Library
- Janez Zaletelj and Andrej Košir. 2017. Predicting students' attention in the classroom from Kinect facial and body features. EURASIP Journal on Image and Video Processing 2017, 1 (01 Dec 2017), 80.Google ScholarCross Ref
- Qiushi Zhou, Joshua Newn, Namrata Srivastava, Tilman Dingler, Jorge Goncalves, and Eduardo Velloso. 2019. Cognitive Aid: Task Assistance Based On Mental Workload Estimation. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA '19). ACM, New York, NY, USA, Article LBW2315, 6 pages. Google ScholarDigital Library
Index Terms
- Classifying Attention Types with Thermal Imaging and Eye Tracking
Recommendations
Spatio-Temporal Detection of Divided Attention in Reading Applications Using EEG and Eye Tracking
IUI '15: Proceedings of the 20th International Conference on Intelligent User InterfacesReading is central to learning and communicating, however, divided attention in the form of distraction may be present in learning environments, resulting in a limited understanding of the reading material. This paper presents a novel system that can ...
An eye tracking dataset for point of gaze detection
ETRA '12: Proceedings of the Symposium on Eye Tracking Research and ApplicationsThis paper presents a new, publicly available eye tracking dataset, aimed to be used as a benchmark for Point of Gaze (PoG) detection algorithms. The dataset consists of a set of videos recording the eye motion of human test subjects as they were ...
Eye detection and coarse localization of pupil for video-based eye tracking systems
AbstractA video-based eye tracking system generally captures NIR images, each of which contains one or two eyes of a subject. The subject’s point of gaze is then determined using 3D eye model and pupil centre corneal reflection technique. Eye detection ...
Comments