Skip to main content
SpringerLink
Log in

Investigating the suitability of online eye tracking for psychological research: Evidence from comparisons with in-person data using emotion–attention interaction tasks

  • Published:
Behavior Research Methods Aims and scope Submit manuscript

Abstract

The future is bound to bring rapid methodological changes to psychological research. One such promising candidate is the use of webcam-based eye tracking. Earlier research investigating the quality of online eye-tracking data has found increased spatial and temporal error compared to infrared recordings. Our studies expand on this work by investigating how this spatial error impacts researchers’ abilities to study psychological phenomena. We carried out two studies involving emotion–attention interaction tasks, using four participant samples. In each study, one sample involved typical in-person collection of infrared eye-tracking data, and the other involved online collection of webcam-based data. We had two main findings: First, we found that the online data replicated seven of eight in-person results, although the effect sizes were just 52% [42%, 62%] the size of those seen in-person. Second, explaining the lack of replication in one result, we show how online eye tracking is biased toward recording more gaze points near the center of participants’ screen, which can interfere with comparisons if left unchecked. Overall, our results suggest that well-powered online eye-tracking research is highly feasible, although researchers must exercise caution, collecting more participants and potentially adjusting their stimulus designs or analytic procedures.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  • Ahler, D. J., Roush, C. E., & Sood, G. (2019). The micro-task market for lemons: Data quality on Amazon’s mechanical Turk Paper presented at the Meeting of the Midwest Political Science Association.

  • Amasino, D. R., Sullivan, N. J., Kranton, R. E., & Huettel, S. A. (2019). Amount and time exert independent influences on intertemporal choice. Nature Human Behaviour, 3(4), 383–392.

    Article  PubMed  PubMed Central  Google Scholar 

  • Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68(3), 255–278. https://doi.org/10.1016/j.jml.2012.11.001

    Article  Google Scholar 

  • Bates, D., Mächler, M., Bolker, B., & Walker, S. (2014). Fitting linear mixed-effects models using lme4. arXiv. https://doi.org/10.18637/jss.v067.i01

  • Bogdan, P. C., Dolcos, F., Katsumi, Y., Dolcos, S., O'Brien, M., Iordan, A. D., et al. (2023). Reconciling opposing effects of emotion on relational memory: Behavioral, eye-tracking, & brain imaging investigations Submitted.

  • Brunyé, T. T., & Gardony, A. L. (2017). Eye tracking measures of uncertainty during perceptual decision making. International Journal of Psychophysiology, 120, 60–68.

    Article  PubMed  Google Scholar 

  • Carretié, L. (2014). Exogenous (automatic) attention to emotional stimuli: A review. Cognitive, Affective, & Behavioral Neuroscience, 14(4), 1228–1258.

    Article  Google Scholar 

  • Dan-Glauser, E. S., & Scherer, K. R. (2011). The Geneva affective picture database (GAPED): A new 730-picture database focusing on valence and normative significance. Behavior Research Methods, 43(2), 468–477. https://doi.org/10.3758/s13428-011-0064-1

    Article  PubMed  Google Scholar 

  • Degen, J., Kursat, L., & Leigh, D. D. (2021). Seeing is believing: Testing an explicit linking assumption for visual world eye-tracking in psycholinguistics Paper presented at the Proceedings of the Annual Meeting of the Cognitive Science Society.

  • Dolcos, F., Bogdan, P. C., O'Brien, M., Iordan, A. D., Madison, A., Buetti, S., & Dolcos, S. (2022). The impact of focused attention on emotional evaluation: An eye-tracking investigation. Emotion, 22(5), 1088–1099.

    Article  PubMed  Google Scholar 

  • Dolcos, F., Katsumi, Y., Bogdan, P. C., Shen, C., Jun, S., Buetti, S., & Dolcos, S. (2020a). The impact of focused attention on subsequent emotional recollection: A functional MRI investigation. Neuropsychologia, 138, 107338.

    Article  PubMed  Google Scholar 

  • Dolcos, F., Katsumi, Y., Moore, M., Berggren, N., de Gelder, B., Derakshan, N., & Dolcos, S. (2020b). Neural correlates of emotion–attention interactions: From perception, learning, and memory to social cognition, individual differences, and training interventions. Neuroscience & Biobehavioral Reviews, 108, 559–601.

    Article  Google Scholar 

  • Dolcos, F., Katsumi, Y., Shen, C., Bogdan, P. C., Jun, S., Larsen, R., & Dolcos, S. (2020c). The impact of focused attention on emotional experience: A functional MRI investigation. Cognitive, Affective, & Behavioral Neuroscience, 20(5), 1011–1026.

    Article  Google Scholar 

  • Ehinger, B. V., Groß, K., Ibs, I., & König, P. (2019). A new comprehensive eye-tracking test battery concurrently evaluating the pupil labs glasses and the EyeLink 1000. PeerJ, 7, e7086.

    Article  PubMed  PubMed Central  Google Scholar 

  • Fiedler, S., & Glöckner, A. (2012). The dynamics of decision making in risky choice: An eye-tracking analysis. Frontiers in Psychology, 3, 335.

    Article  PubMed  PubMed Central  Google Scholar 

  • Gignac, G. E., & Szodorai, E. T. (2016). Effect size guidelines for individual differences researchers. Personality and Individual Differences, 102, 74–78.

    Article  Google Scholar 

  • Goodman, A. M., Katz, J. S., & Dretsch, M. N. (2016). Military affective picture system (MAPS): A new emotion-based stimuli set for assessing emotional processing in military populations. Journal of Behavior Therapy and Experimental Psychiatry, 50, 152–161. https://doi.org/10.1016/j.jbtep.2015.07.006

    Article  PubMed  Google Scholar 

  • Gross, J. J., & John, O. P. (2003). Individual differences in two emotion regulation processes: Implications for affect, relationships, and well-being. Journal of Personality and Social Psychology, 85(2), 348–362. https://doi.org/10.1037/0022-3514.85.2.348

    Article  PubMed  Google Scholar 

  • Hutt, S., Wong, A., Papoutsaki, A., Baker, R. S., Gold, J. I., & Mills, C. (2023). Webcam-based eye tracking to detect mind wandering and comprehension errors. Behavior Research Methods, 1–17.

  • Kees, J., Berry, C., Burton, S., & Sheehan, K. (2017). An analysis of data quality: Professional panels, student subject pools, and Amazon's mechanical Turk. Journal of Advertising, 46(1), 141–155.

    Article  Google Scholar 

  • Kragel, J. E., & Voss, J. L. (2022). Looking for the neural basis of memory. Trends in Cognitive Sciences, 26(1), 53–65.

    Article  PubMed  Google Scholar 

  • Krajbich, I., Armel, C., & Rangel, A. (2010). Visual fixations and the computation and comparison of value in simple choice. Nature Neuroscience, 13(10), 1292–1298.

    Article  PubMed  Google Scholar 

  • Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. B. (2017). lmerTest package: Tests in linear mixed effects models. Journal of Statistical Software, 82(13). https://doi.org/10.18637/jss.v082.i13

  • Kwak, Y., Payne, J. W., Cohen, A. L., & Huettel, S. A. (2015). The rational adolescent: Strategic information processing during decision making revealed by eye tracking. Cognitive Development, 36, 20–30.

    Article  PubMed  PubMed Central  Google Scholar 

  • Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, 863.

    Article  PubMed  PubMed Central  Google Scholar 

  • Lang, P. J., Bradley, M. M., & Cuthbert, B. N. (2008). International affective picture system (IAPS): Affective ratings of pictures and instruction manual.

  • Lorah, J. (2018). Effect size measures for multilevel models: Definition, interpretation, and TIMSS example. Large-Scale Assessments in Education, 6(1), 1–11.

    Article  Google Scholar 

  • Lovakov, A., & Agadullina, E. R. (2021). Empirically derived guidelines for effect size interpretation in social psychology. European Journal of Social Psychology.

  • Luke, S. G. (2017). Evaluating significance in linear mixed-effects models in R. Behavior Research Methods, 49(4), 1494–1502. https://doi.org/10.3758/s13428-016-0809-y

    Article  PubMed  Google Scholar 

  • Marchewka, A., Zurawski, L., Jednorog, K., & Grabowska, A. (2014). The Nencki affective picture system (NAPS): Introduction to a novel, standardized, wide-range, high-quality, realistic picture database. Behavior Research Methods, 46(2), 596–610. https://doi.org/10.3758/s13428-013-0379-1

    Article  PubMed  Google Scholar 

  • Meteyard, L., & Davies, R. A. (2020). Best practice guidance for linear mixed-effects models in psychological science. Journal of Memory and Language, 112, 104092. https://doi.org/10.1016/j.jml.2020.104092

    Article  Google Scholar 

  • Öhman, A., Flykt, A., & Esteves, F. (2001). Emotion drives attention: Detecting the snake in the grass. Journal of Experimental Psychology: General, 130(3), 466.

    Article  PubMed  Google Scholar 

  • Papoutsaki, A., Sangkloy, P., Laskey, J., Daskalova, J., Huang, J., & Hays, J. (2016). WebGazer: Scalable webcam eye tracking using user interactions. In Paper presented at the IJCAI-16: International joint conference on artificial intelligence. USA https://par.nsf.gov/biblio/10024076

    Google Scholar 

  • R Core Team(2013) R: A language and environment for statistical computing.

  • Riggs, L., McQuiggan, D. A., Farb, N., Anderson, A. K., & Ryan, J. D. (2011). The role of overt attention in emotion-modulated memory. Emotion, 11(4), 776.

    Article  PubMed  Google Scholar 

  • Sassenberg, K., & Ditrich, L. (2019). Research in social psychology changed between 2011 and 2016: Larger sample sizes, more self-report measures, and more online studies. Advances in Methods and Practices in Psychological Science, 2(2), 107–114.

    Article  Google Scholar 

  • Schneegans, T., Bachman, M. D., Huettel, S. A., & Heekeren, H. (2021). Exploring the potential of online webcam-based eye tracking in decision-making research and influence factors on data quality.

    Book  Google Scholar 

  • Semmelmann, K., & Weigelt, S. (2018). Online webcam-based eye tracking in cognitive science: A first look. Behavior Research Methods, 50(2), 451–465.

    Article  PubMed  Google Scholar 

  • Slim, M. S., & Hartsuiker, R. J. (2022). Moving visual world experiments online? A web-based replication of Dijkgraaf, Hartsuiker, and Duyck (2017) using PCIbex and WebGazer. Js. Behavior Research Methods, 1–19.

  • Strauss, G. P., Ossenfort, K. L., & Whearty, K. M. (2016). Reappraisal and distraction emotion regulation strategies are associated with distinct patterns of visual attention and differing levels of cognitive demand. PLoS One, 11(11), e0162290.

    Article  PubMed  PubMed Central  Google Scholar 

  • Strohmaier, A. R., MacKay, K. J., Obersteiner, A., & Reiss, K. M. (2020). Eye-tracking methodology in mathematics education research: A systematic literature review. Educational Studies in Mathematics, 104, 147–200.

    Article  Google Scholar 

  • Vos, M., Minor, S., & Ramchand, G. C. (2022). Comparing infrared and webcam eye tracking in the visual world paradigm.

  • Voss, J. L., Bridge, D. J., Cohen, N. J., & Walker, J. A. (2017). A closer look at the hippocampus and memory. Trends in Cognitive Sciences, 21(8), 577–588.

    Article  PubMed  PubMed Central  Google Scholar 

  • Wessa, M., Kanske, P., Neumeister, P., Bode, K., Heissler, J., & Schönfelder, S. (2010). EmoPicS: Subjective and psychophysiological evaluation of new imagery for clinical biopsychological research. Z. Klin. Psychol. Psychother. Suppl, 1, 11–77.

    Google Scholar 

  • Yang, X., & Krajbich, I. (2021). Webcam-based online eye-tracking for behavioral research. Judgment and Decision making, 16(6).

Download references

Acknowledgments

This research was carried out in part at the University of Illinois’ Beckman Institute for Advanced Science & Technology. During the preparation of this manuscript, P.C.B. was supported by a Predoctoral Fellowship provided by the Beckman Foundation and a Dissertation Completion Fellowship provided by the University of Illinois, and F.D. was supported by an Emanuel Donchin Professorial Scholarship in Psychology from the University of Illinois. The authors thank Margaret O’Brien, Anna Madison, and Chen Shen for their assistance with data collection for the in-person versions of the tasks. The authors also thank Dolcos Lab members for their help with stimulus creation.

Code availability

The analysis code has been deposited in a public GitHub repository, alongside the data (https://github.com/paulcbogdan/Suitability_Online_ET).

Funding

This research was supported by research funds from the University of Illinois to F.D. and S.D.

Author information

Authors and Affiliations

  1. Beckman Institute for Advanced Science and Technology, University of Illinois Urbana-Champaign, Urbana, IL, USA

    Paul C. Bogdan, Sanda Dolcos, Alejandro Lleras & Florin Dolcos

  2. Department of Psychology, University of Illinois at Urbana-Champaign, Champaign, IL, USA

    Paul C. Bogdan, Sanda Dolcos, Simona Buetti, Alejandro Lleras & Florin Dolcos

  3. Neuroscience Program, University of Illinois at Urbana-Champaign, Urbana, IL, USA

    Florin Dolcos

Authors
  1. Paul C. Bogdan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sanda Dolcos
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Simona Buetti
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alejandro Lleras
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Florin Dolcos
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

F.D. and S.D. conceived of the tasks; P.C.B. implemented the online versions of the task and the online eye tracking with guidance from F.D. and S.D.; P.C.B. collected the online data; P.C.B. designed the analytic approach with feedback from F.D., S.D., S.B., and A.L.; P.C.B. performed the analyses; P.C.B. wrote the first draft of the manuscript and revised it based on feedback from F.D, S.B., A.L., and S.D. All authors approved of the final submission.

Corresponding authors

Correspondence to Paul C. Bogdan or Florin Dolcos.

Ethics declarations

Conflict of interest

The authors declare no conflicts of interest.

Ethics approval

Protocols performed by the current research were approved by the University of Illinois Institutional Review Board.

Consent to participate

Informed consent was obtained from all individual participants included in the study.

Consent for publication

Not applicable.

Additional information

Open Practices Statement

The data and analysis code will be deposited in a publicly available repository, pending IRB approval and upon article acceptance. The experiments were not pre-registered.

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

ESM 1

(PDF 103 kb)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bogdan, P.C., Dolcos, S., Buetti, S. et al. Investigating the suitability of online eye tracking for psychological research: Evidence from comparisons with in-person data using emotion–attention interaction tasks. Behav Res 56, 2213–2226 (2024). https://doi.org/10.3758/s13428-023-02143-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.3758/s13428-023-02143-z

Keywords

Search

Navigation