Skip to main content
SpringerLink
Log in

The Affective Dynamics of Everyday Digital Life: Opening Computational Possibility

  • RESEARCH ARTICLE
  • Published:
Affective Science Aims and scope Submit manuscript

Abstract

Up to now, there was no way to observe and track the affective impacts of the massive amount of complex visual stimuli that people encounter “in the wild” during their many hours of digital life. In this paper, we propose and illustrate how recent advances in AI—trained ensembles of deep neural networks—can be deployed on new data streams that are long sequences of screenshots of study participants’ smartphones obtained unobtrusively during everyday life. We obtained affective valence and arousal ratings of hundreds of images drawn from existing picture repositories often used in psychological studies, and a new screenshot repository chronicling individuals’ everyday digital life from both N = 832 adults and an affect computation model (Parry & Vuong, 2021). Results and analysis suggest that (a) our sample rates images similarly to other samples used in psychological studies, (b) the affect computation model is able to assign valence and arousal ratings similarly to humans, and (c) the resulting computational pipeline can be deployed at scale to obtain detailed maps of the affective space individuals travel through on their smartphones. Leveraging innovative methods for tracking the emotional content individuals encounter on their smartphones, we open the possibility for large-scale studies of how the affective dynamics of everyday digital life shape individuals’ moment-to-moment experiences and well-being.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Anderson, C. A., & Bushman, B. J. (2002). Human aggression. Annual Review of Psychology, 53(1), 27–51.

    Article  PubMed  Google Scholar 

  • Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., ... & Liang, P. (2021). On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258

  • Bradley, M., & Lang, P. (1994). Measuring emotion: The self-assessment manikin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry, 25, 49–59. https://doi.org/10.1016/0005-7916(94)90063-9

    Article  PubMed  Google Scholar 

  • Brinberg, M., Ram, N., Yang, X., Cho, M. J., Sundar, S. S., Robinson, T. N., & Reeves, B. (2021). The idiosyncrasies of everyday digital lives: Using the Human Screenome Project to study user behavior on smartphones. Computers in Human Behavior, 114, 106570.

    Article  PubMed  Google Scholar 

  • Carnagey, N. L., Anderson, C. A., & Bushman, B. J. (2007). The effect of video game violence on physiological desensitization to real-life violence. Journal of Experimental Social Psychology, 43(3), 489–496.

    Article  Google Scholar 

  • Crone, D. L., Bode, S., Murawski, C., & Laham, S. M. (2018). The Socio-Moral Image Database (SMID): A novel stimulus set for the study of social, moral and affective processes. PloS One, 13(1), e0190954. https://doi.org/10.1371/journal.pone.0190954

  • Detenber, B. H., Han, J., & Lang, A. (2021). The influence of form and presentation attributes of traditional media on emotion. In K. Döveling & E. A. Konijn, Routledge International Handbook of Emotions and Media (pp. 147–163). Routledge.

  • Dricu, M., & Frühholz, S. (2016). Perceiving emotional expressions in others: Activation likelihood estimation meta-analyses of explicit evaluation, passive perception and incidental perception of emotions. Neuroscience & Biobehavioral Reviews, 71, 810–828.

    Article  Google Scholar 

  • Fikkers, K. M., & Piotrowski, J. T. (2020). Content and person effects in media research: Studying differences in cognitive, emotional, and arousal responses to media content. Media Psychology, 23(4), 493–520.

    Article  Google Scholar 

  • Hald, G. M., & Malamuth, N. N. (2015). Experimental effects of exposure to pornography: The moderating effect of personality and mediating effect of sexual arousal. Archives of Sexual Behavior, 44(1), 99–109.

    Article  PubMed  Google Scholar 

  • Harari, G. M., Müller, S. R., Aung, M. S., & Rentfrow, P. J. (2017). Smartphone sensing methods for studying behavior in everyday life. Current Opinion in Behavioral Sciences, 18, 83–90.

    Article  Google Scholar 

  • Kensinger, E., & Mickley, K. (2008). Emotional valence influences the neural correlates associated with remembering and knowing. Cognitive, Affective, & Behavioral Neuroscience, 8, 143–152. https://doi.org/10.3758/CABN.8.2.143

    Article  Google Scholar 

  • Kim, H. R., Kim, Y. S., Kim S., Lee, I. K. (2017). Building emotional machines: recognizing image emotions through deep neural networks. IEEE Transactions on Multimedia. https://doi.org/10.1109/TMM.2018.2827782

  • Knobloch-Westerwick, S. (2015). The selective exposure self-and affect-management (SESAM) model: Applications in the realms of race, politics, and health. Communication Research, 42(7), 959–985.

    Article  Google Scholar 

  • Koo, T., Li, M. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. National Library of Medicine. https://doi.org/10.1016/j.jcm.2016.02.012

  • Kurdi, B., Lozano, S., & Banaji, M. (2016). Introducing the Open Affective Standardized Image Set (OASIS). Behavior Research Methods, 49, 457–470. https://doi.org/10.3758/s13428-016-0715-3

    Article  Google Scholar 

  • Lang, A. (2000). The limited capacity model of mediated message processing. Journal of Communication, 50(1), 46–70. https://doi.org/10.1111/j.1460-2466.2000.tb02833.x

  • Lawrence, I., & Lin, K. (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics, 45(1), 255–268. https://doi.org/10.2307/2532051

  • Luo, M., & Hancock, J. T. (2020). Self-disclosure and social media: Motivations, mechanisms and psychological well-being. Current Opinion in Psychology, 31, 110–115.

    Article  PubMed  Google Scholar 

  • Myrick, J. G., & Nabi, R. L. (2017). Fear arousal and health and risk messaging. In Oxford Research Encyclopedia of Communication. NY: Oxford University Press.

    Google Scholar 

  • Nummenmaa, L., Hyönä, J., & Calvo, M. G. (2010). Semantic categorization precedes affective evaluation of visual scenes. Journal of Experimental Psychology: General, 139(2), 222.

    Article  PubMed  Google Scholar 

  • Oliver, M. B. (2008). Tender affective states as predictors of entertainment preference. Journal of Communication, 58(1), 40–61.

    Article  Google Scholar 

  • Parry, B., & Vuong, Q. (2021). Deep affect: Using objects, scenes and facial expressions in a deep neural network to predict arousal and valence values of images. PsyArXiv. https://doi.org/10.31234/osf.io/t9p3f

  • Perrin, A. (2021). Mobile Technology and Home Broadband 2021. Pew Research Center. https://www.pewresearch.org/internet/2021/06/03/mobile-technology-and-home-broadband-2021/

  • Ram, N., Yang, X., Cho, M.-J., Brinberg, M., Muirhead, F., Reeves, B., & Robinson, T. N. (2020). Screenomics: a new approach for observing and studying individuals’ digital lives. Journal of Adolescent Research, 35(1), 16–50. https://doi.org/10.1177/0743558419883362

    Article  PubMed  Google Scholar 

  • Ram, N., Haber, N., Robinson, T. N., & Reeves, B. (2023). Binding the person-specific approach to modern AI in the human screenome project: Moving past generalizability to transferability. Multivariate Behavioral Research. https://doi.org/10.1080/00273171.2023.2229305

    Article  PubMed  Google Scholar 

  • R Core Team. (2020). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/

  • Reeves, B., Ram, N., Robinson, T. N., Cummings, J. J., Giles, C. L., Pan, J., … Yeykelis, L. (2021). Screenomics: A framework to capture and analyze personal life experiences and the ways that technology shapes them. Human–Computer Interaction, 36(2), 150–201.

    Article  PubMed  Google Scholar 

  • Reeves, B., Robinson, T., & Ram, N. (2020). Time for the human screenome project. Nature, 577, 314–317. https://doi.org/10.1038/d41586-020-00032-5

    Article  PubMed  Google Scholar 

  • Revelle, W. (2022). Psych: Procedures for Psychological, Psychometric, and Personality Research. Northwestern University, Evanston, Illinois. R package version 2.2.3. https://CRAN.R-project.org/package=psych

  • Schreurs, L., & Vandenbosch, L. (2021). Introducing the social media literacy (SMILE) model with the case of the positivity bias on social media. Journal of Children and Media, 15(3), 320–337. https://doi.org/10.1080/17482798.2020.1809481

    Article  Google Scholar 

  • Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420.

    Article  PubMed  Google Scholar 

  • Sievert C (2020). Interactive web-based data visualization with R, plotly, and shiny. Chapman and Hall/CRC. ISBN 9781138331457, https://plotly-r.com

  • Signorell, A., et al. (2022). DescTools: Tools for descriptive statistics. R package version, 0(99), 47. https://cran.r-project.org/package=DescTools

    Google Scholar 

  • Simonyan, K., Zisserman, A. (2014) Very deep convolutional networks for large-scale image recognition. https://doi.org/10.48550/arXiv.1409.1556

  • Stevens, H. R., Oh, Y. J., & Taylor, L. D. (2021). Desensitization to fear-inducing COVID-19 health news on Twitter: observational study. JMIR Infodemiology, 1(1), e26876.

    Article  PubMed  Google Scholar 

  • Wahl-Jorgensen, K. (2019). Emotions, media and politics. John Wiley & Sons.

    Google Scholar 

  • Wickham, H. (2016). ggplot2: Elegant graphics for data analysis. Springer-Verlag New York. ISBN 978-3-319-24277-4, https://ggplot2.tidyverse.org

  • Wright, P. J., Tokunaga, R. S., & Kraus, A. (2016). A meta-analysis of pornography consumption and actual acts of sexual aggression in general population studies. Journal of Communication, 66(1), 183–205.

    Article  Google Scholar 

  • Yeykelis, L., Cummings, J. J., & Reeves, B. (2018). The fragmentation of work, entertainment, E-Mail, and news on a personal computer: Motivational predictors of switching between media content. Media Psychology, 21(3), 377–402.

    Article  Google Scholar 

  • Zacks, J. M. (2015). Flicker: Your brain on movies. Oxford University Press.

    Google Scholar 

  • Zavershynskyi, M., Skidanov, A., Polosukhin, I. (2018). NAPS: Natural program synthesis dataset. https://doi.org/10.48550/arXiv.1807.03168

  • Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., & Torralba, A. (2017). Places: A 10 Million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40, 1452–1464. https://doi.org/10.1109/TPAMI.2017.2723009

    Article  PubMed  Google Scholar 

  • Zillmann, D. (1983). Transfer of excitation in emotional behavior. In J.T. Cacioppa & R.E. Petty (Eds.), Social psychophysiology: A sourcebook (pp. 215–240). New York: Guilford.

  • Zillmann, D. (2000). Mood management in the context of selective exposure theory. Annals of the International Communication Association, 23(1), 103–123.

    Article  Google Scholar 

Download references

Acknowledgements

Thanks very much to the research staff for creating the simulated day-in-digital-life screenshots, the study participants for providing many ratings, and to George Parry and Quoc Vuong for inspiration and practical possibility.

Author information

Authors and Affiliations

  1. Department of Psychology, Stanford University, Stanford, CA, 94305, USA

    Maia L. Rocklin, Anna Angelina Garròn Torres & Nilam Ram

  2. Department of Communication, Stanford University, Stanford, 300-A Building 120, 450 Jane Stanford Way, Stanford, CA, 94305, USA

    Byron Reeves & Nilam Ram

  3. Department of Pediatrics, Stanford University, Stanford, CA, 94305, USA

    Thomas N. Robinson

  4. Department of Medicine, Stanford University, Stanford, CA, 94305, USA

    Thomas N. Robinson

Authors
  1. Maia L. Rocklin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anna Angelina Garròn Torres
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Byron Reeves
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Thomas N. Robinson
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Nilam Ram
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nilam Ram.

Ethics declarations

Funding

This work was supported by Stanford VPUE and HAI.

Conflict of interest

The authors declare no competing interests.

Data availability

The data and analysis code are available on Github at https://github.com/The-Change-Lab/affectivedynamics.

Code availability

Not applicable.

Authors' Contribution

Not applicable.

Ethics approval

Not applicable.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Additional information

Handling Editor: Ralph Adolphs

Supplementary Information

ESM 1

(DOCX 20 kb)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rocklin, M.L., Garròn Torres, A.A., Reeves, B. et al. The Affective Dynamics of Everyday Digital Life: Opening Computational Possibility. Affec Sci 4, 529–540 (2023). https://doi.org/10.1007/s42761-023-00202-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42761-023-00202-4

Keywords

Search

Navigation