Yasmeen Abdrabou
Sheikh Radiah Rivu
Jonathan Liebers
Uwe Gruenefeld
Pascal Knierim
Mohamed Khamis
Stefan Schneegass
Florian Alt
ABSTRACT
In this work, we explore attacker behavior during shoulder surfing. As such behavior is often opportunistic and difficult to observe in real world settings, we leverage the capabilities of virtual reality (VR). We recruited 24 participants and observed their behavior in two virtual waiting scenarios: at a bus stop and in an open office space. In both scenarios, participants shoulder surfed private screens displaying different types of content. From the results we derive an understanding of factors influencing shoulder surfing behavior, reveal common attack patterns, and sketch a behavioral shoulder surfing model. Our work suggests directions for future research on shoulder surfing and can serve as a basis for creating novel approaches to mitigate shoulder surfing.
- Yasmeen Abdrabou, Radiah Rivu, Tarek Ammar, Jonathan Liebers, Alia Saad, Carina Liebers, Uwe Gruenefeld, Pascal Knierim, Mohamed Khamis, Ville Maekelae, Stefan Schneegass, and Florian Alt. 2022. Understanding Shoulder Surfer Behavior Using Virtual Reality. In Adjunct Proceedings of the IEEE Conference on Virtual Reality and 3D User Interfaces.Google Scholar
- Florian Alt and Emanuel von Zezschwitz. 2019. Emerging Trends in Usable Security and Privacy. i-com 18, 3 (2019), 189–195. https://doi.org/doi:10.1515/icom-2019-0019Google Scholar
- Adam J. Aviv, John T. Davin, Flynn Wolf, and Ravi Kuber. 2017. Towards Baselines for Shoulder Surfing on Mobile Authentication. In Proceedings of the 33rd Annual Computer Security Applications Conference (Orlando, FL, USA) (ACSAC 2017). Association for Computing Machinery, New York, NY, USA, 486–498.Google ScholarDigital Library
- Jay Ayres, Jason Flannick, Johannes Gehrke, and Tomi Yiu. 2002. Sequential PAttern Mining Using a Bitmap Representation. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining(Edmonton, Alberta, Canada) (KDD ’02). Association for Computing Machinery, New York, NY, USA, 429–435.Google ScholarDigital Library
- Mihai Bâce, Alia Saad, Mohamed Khamis, Stefan Schneegass, and Andreas Bulling. 2022. PrivacyScout: Assessing Vulnerability to Shoulder Surfing on Mobile Devices. In Proc. on Privacy Enhancing Technologies (PETs). Sciendo.Google ScholarCross Ref
- Gilbert Beyer, Florian Alt, Jörg Müller, Albrecht Schmidt, Karsten Isakovic, Stefan Klose, Manuel Schiewe, and Ivo Haulsen. 2011. Audience Behavior around Large Interactive Cylindrical Screens. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI ’11). Association for Computing Machinery, New York, NY, USA, 1021–1030.Google ScholarDigital Library
- Leon Bošnjak and Boštjan Brumen. 2020. Shoulder surfing experiments: A systematic literature review. Computers & Security 99(2020), 102023.Google ScholarCross Ref
- Harry Brignull and Yvonne Rogers. 2003. Enticing people to interact with large public displays in public spaces. In Human-computer interaction - INTERACT ’03: IFIP TC 13 International Conference on Human-Computer Interaction, 1st - 5th September 2003, Zurich, Switzerland, Vol. 3. IOS Press, Amsterdam and Berlin.Google Scholar
- Frederik Brudy, David Ledo, Saul Greenberg, and Andreas Butz. 2014. Is Anyone Looking? Mitigating Shoulder Surfing on Public Displays through Awareness and Protection. In Proceedings of The International Symposium on Pervasive Displays (Copenhagen, Denmark) (PerDis ’14). Association for Computing Machinery, New York, NY, USA, 1–6.Google ScholarDigital Library
- Ceenu George, Mohamed Khamis, Emanuel von Zezschwitz, Henri Schmidt, Marinus Burger, Florian Alt, and Heinrich Hussmann. 2017. Seamless and Secure VR: Adapting and Evaluating Established Authentication Systems for Virtual Reality. In Proceedings 2017 Workshop on Usable Security. Internet Society, San Diego, CA, USA.Google ScholarCross Ref
- Yi-Lun Chen, Wei-Chi Ku, Yu-Chang Yeh, and Dun-Min Liao. 2013. A simple text-based shoulder surfing resistant graphical password scheme. In 2013 International Symposium on Next-Generation Electronics. IEEE, Kaohsiung, Taiwan, 161–164.Google ScholarCross Ref
- Malin Eiband, Mohamed Khamis, Emanuel von Zezschwitz, Heinrich Hussmann, and Florian Alt. 2017. Understanding Shoulder Surfing in the Wild: Stories from Users and Observers. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 4254–4265.Google ScholarDigital Library
- Habiba Farzand, Kinshuk Bhardwaj, Karola Marky, and Mohamed Khamis. 2021. The Interplay between Personal Relationships & Shoulder Surfing Mitigation. In Mensch Und Computer 2021 (Ingolstadt, Germany) (MuC ’21). Association for Computing Machinery, New York, NY, USA, 338–343.Google Scholar
- Christina Katsini, Yasmeen Abdrabou, George E. Raptis, Mohamed Khamis, and Florian Alt. 2020. The Role of Eye Gaze in Security and Privacy Applications: Survey and Future HCI Research Directions. Association for Computing Machinery, New York, NY, USA, 1–21.Google ScholarDigital Library
- Mohamed Khamis, Ludwig Trotter, Ville Mäkelä, Emanuel von Zezschwitz, Jens Le, Andreas Bulling, and Florian Alt. 2018. CueAuth: Comparing Touch, Mid-Air Gestures, and Gaze for Cue-Based Authentication on Situated Displays. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 4, Article 174 (dec 2018).Google ScholarDigital Library
- Mun-Kyu Lee. 2014. Security notions and advanced method for human shoulder-surfing resistant PIN-entry. IEEE Transactions on Information Forensics and Security 9, 4(2014), 695–708.Google ScholarDigital Library
- Nizar R. Mabroukeh and C. I. Ezeife. 2010. A Taxonomy of Sequential Pattern Mining Algorithms. ACM Comput. Surv. 43, 1, Article 3 (dec 2010), 41 pages. https://doi.org/10.1145/1824795.1824798Google ScholarDigital Library
- Ville Mäkelä, Rivu Radiah, Saleh Alsherif, Mohamed Khamis, Chong Xiao, Lisa Borchert, Albrecht Schmidt, and Florian Alt. 2020. Virtual Field Studies: Conducting Studies on Public Displays in Virtual Reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems(CHI’20). Association for Computing Machinery, New York, NY, USA, 1–15.Google ScholarDigital Library
- Diogo Marques, Tiago Guerreiro, Luis Carriço, Ivan Beschastnikh, and Konstantin Beznosov. 2019. Vulnerability & Blame: Making Sense of Unauthorized Access to Smartphones. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–13.Google ScholarDigital Library
- Florian Mathis, Kami Vaniea, and Mohamed Khamis. 2021. Observing Virtual Avatars: The Impact of Avatars’ Fidelity on Identifying Interactions. In Academic Mindtrek 2021 (Tampere/Virtual, Finland) (Mindtrek 2021). Association for Computing Machinery, New York, NY, USA, 154–164.Google Scholar
- Florian Mathis, Kami Vaniea, and Mohamed Khamis. 2021. RepliCueAuth: Validating the Use of a Lab-Based Virtual Reality Setup for Evaluating Authentication Systems. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 534, 18 pages. https://doi.org/10.1145/3411764.3445478Google ScholarDigital Library
- Jörg Müller, Florian Alt, Daniel Michelis, and Albrecht Schmidt. 2010. Requirements and Design Space for Interactive Public Displays. In Proceedings of the 18th ACM International Conference on Multimedia (Firenze, Italy) (MM ’10). Association for Computing Machinery, New York, NY, USA, 1285–1294.Google ScholarDigital Library
- Ildar Muslukhov, Yazan Boshmaf, Cynthia Kuo, Jonathan Lester, and Konstantin Beznosov. 2013. Know Your Enemy: The Risk of Unauthorized Access in Smartphones by Insiders. In Proceedings of the 15th International Conference on Human-Computer Interaction with Mobile Devices and Services (Munich, Germany) (MobileHCI ’13). Association for Computing Machinery, New York, NY, USA, 271–280.Google ScholarDigital Library
- Rivu Radiah, Ville Mäkelä, Sarah Prange, Sarah Delgado Rodriguez, Robin Piening, Yumeng Zhou, Kay Köhle, Ken Pfeuffer, Yomna Abdelrahman, Matthias Hoppe, Albrecht Schmidt, and Florian Alt. 2021. Remote VR Studies: A Framework for Running Virtual Reality Studies Remotely Via Participant-Owned HMDs. ACM Trans. Comput.-Hum. Interact. 28, 6, Article 46 (nov 2021), 36 pages.Google ScholarDigital Library
- Philipp A. Rauschnabel, Reto Felix, Chris Hinsch, Hamza Shahab, and Florian Alt. 2022. What is XR? Towards a Framework for Augmented and Virtual Reality. Computers in Human Behavior 133 (2022), 107289.Google ScholarDigital Library
- Mudassar Raza, Muhammad Iqbal, Muhammad Sharif, and Waqas Haider. 2012. A survey of password attacks and comparative analysis on methods for secure authentication. World Applied Sciences Journal 19, 4 (2012), 439–444.Google Scholar
- Volker Roth, Kai Richter, and Rene Freidinger. 2004. A PIN-Entry Method Resilient against Shoulder Surfing. In Proceedings of the 11th ACM Conference on Computer and Communications Security (Washington DC, USA) (CCS ’04). Association for Computing Machinery, New York, NY, USA, 236–245.Google ScholarDigital Library
- Aitor Rovira, Richard Southern, David Swapp, Claire Campbell, Jian J Zhang, Mark Levine, and Mel Slater. 2021. Bystander Affiliation Influences Intervention Behavior: A Virtual Reality Study. SAGE Open 11, 3 (2021), 21582440211040076.Google ScholarCross Ref
- Aitor Rovira, David Swapp, Bernhard Spanlang, and Mel Slater. 2009. The use of virtual reality in the study of people’s responses to violent incidents. Frontiers in behavioral neuroscience 3 (2009), 59.Google Scholar
- A Rovira i Pérez. 2016. Simulating Social Situations in Immersive Virtual Reality-A Study of Bystander Responses to Violent Emergencies. Ph.D. Dissertation. UCL (University College London).Google Scholar
- Alia Saad, Michael Chukwu, and Stefan Schneegass. 2018. Communicating Shoulder Surfing Attacks to Users. In Proceedings of the 17th International Conference on Mobile and Ubiquitous Multimedia (Cairo, Egypt) (MUM 2018). Association for Computing Machinery, New York, NY, USA, 147–152.Google ScholarDigital Library
- Alia Saad, Michael Chukwu, and Stefan Schneegass. 2018. Communicating Shoulder Surfing Attacks to Users. In Proceedings of the 17th International Conference on Mobile and Ubiquitous Multimedia (Cairo, Egypt) (MUM 2018). Association for Computing Machinery, New York, NY, USA, 147–152.Google ScholarDigital Library
- Alia Saad, Dina Hisham Elkafrawy, Slim Abdennadher, and Stefan Schneegass. 2020. Are They Actually Looking? Identifying Smartphones Shoulder Surfing Through Gaze Estimation. In ACM Symposium on Eye Tracking Research and Applications (Stuttgart, Germany) (ETRA ’20 Adjunct). Association for Computing Machinery, New York, NY, USA, Article 42, 3 pages.Google ScholarDigital Library
- Alia Saad, Jonathan Liebers, Uwe Gruenefeld, Florian Alt, and Stefan Schneegass. 2021. Understanding Bystanders’ Tendency to Shoulder Surf Smartphones Using 360-Degree Videos in Virtual Reality. In Proceedings of the 23rd International Conference on Mobile Human-Computer Interaction. Association for Computing Machinery, New York, NY, USA, Article 35, 8 pages.Google ScholarDigital Library
- Dario D Salvucci and Joseph H Goldberg. 2000. Identifying fixations and saccades in eye-tracking protocols. In Proceedings of the 2000 symposium on Eye tracking research & applications. Association for Computing Machinery, New York, NY, USA, 71–78. https://doi.org/10.1145/355017.355028Google ScholarDigital Library
- Ludwig Sidenmark and Hans Gellersen. 2019. Eye, Head and Torso Coordination During Gaze Shifts in Virtual Reality. ACM Trans. Comput.-Hum. Interact. 27, 1, Article 4 (Dec. 2019), 40 pages.Google ScholarDigital Library
- Emanuel von Zezschwitz, Alexander De Luca, Bruno Brunkow, and Heinrich Hussmann. 2015. SwiPIN: Fast and Secure PIN-Entry on Smartphones. Association for Computing Machinery, New York, NY, USA, 1403–1406.Google ScholarDigital Library
- Keita Watanabe, Fumito Higuchi, Masahiko Inami, and Takeo Igarashi. 2012. CursorCamouflage: Multiple Dummy Cursors as a Defense against Shoulder Surfing. In SIGGRAPH Asia 2012 Emerging Technologies. Association for Computing Machinery, New York, NY, USA, 1–2.Google Scholar
- Oliver Wiese and Volker Roth. 2015. Pitfalls of Shoulder Surfing Studies. In Proceedings 2015 Workshop on Usable Security, Jens Grossklags (Ed.). Internet Society, Reston, VA.Google ScholarCross Ref
- Bob G. Witmer and Michael J. Singer. 1998. Measuring Presence in Virtual Environments: A Presence Questionnaire. Presence: Teleoper. Virtual Environ. 7, 3 (jun 1998), 225–240. https://doi.org/10.1162/105474698565686Google ScholarDigital Library
- Dhruv Kumar Yadav, Beatrice Ionascu, Sai Vamsi Krishna Ongole, Aditi Roy, and Nasir Memon. 2015. Design and Analysis of Shoulder Surfing Resistant PIN Based Authentication Mechanisms on Google Glass. In Financial Cryptography and Data Security, Michael Brenner, Nicolas Christin, Benjamin Johnson, and Kurt Rohloff (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 281–297.Google Scholar
Recommendations
Understanding Shoulder Surfing in the Wild: Stories from Users and Observers
CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing SystemsResearch has brought forth a variety of authentication systems to mitigate observation attacks. However, there is little work about shoulder surfing situations in the real world. We present the results of a user survey (N=174) in which we investigate ...
Understanding Bystanders’ Tendency to Shoulder Surf Smartphones Using 360-degree Videos in Virtual Reality
MobileHCI '21: Proceedings of the 23rd International Conference on Mobile Human-Computer InteractionShoulder surfing is an omnipresent risk for smartphone users. However, investigating these attacks in the wild is difficult because of either privacy concerns, lack of consent, or the fact that asking for consent would influence people’s behavior (e.g.,...
Reducing shoulder-surfing by using gaze-based password entry
SOUPS '07: Proceedings of the 3rd symposium on Usable privacy and securityShoulder-surfing -- using direct observation techniques, such as looking over someone's shoulder, to get passwords, PINs and other sensitive personal information -- is a problem that has been difficult to overcome. When a user enters information using a ...
Comments