Skip to main content
SpringerLink
Log in

Privacy Protection Against Shoulder Surfing in Mobile Environments

  • Conference paper
  • First Online:
Mobile and Ubiquitous Systems: Computing, Networking and Services (MobiQuitous 2022)

  • 281 Accesses

Abstract

Smartphones and other mobile devices have seen an unprecedented rise in use among consumers. These devices are widely used in public locations where traditional computers could hardly be accessed. Although such ubiquitous computing is desirable for users, the use of mobile devices in public locations has led to rising privacy concerns. Malicious individuals can easily glean personal data from a mobile device screen by visual eavesdropping without a user’s knowledge. In this paper, we propose two schemes to identify and protect private user data displayed on mobile device screens in public environments. The first scheme considers generic mobile applications’ complex user interfaces as an image, and uses a deep, convolutional object detection network to automatically identify sensitive content displayed by mobile applications. Such content is then blurred against shoulder surfing attacks. To allow users to identify custom fields in applications that they think should be hidden, we introduce methods for dynamic sample generation and model retraining that only need users to provide a small number of seed samples. The second scheme focuses on web applications due to the popularity of the web platform, and automates the detection and blurring of sensitive web fields through HTML (HyperText Markup Language) parsing and CSS (Cascading Style Sheets) style modification as showcased via a Chromium-based browser extension. Evaluations show the effectiveness of our schemes.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Gartner Says Global Smartphone Sales Grew 6% in 2021. https://www.gartner.com/en/newsroom/press-releases/2022-03-01-4q21-smartphone-market-share. Accessed 31 Aug 2022

  2. Market Share: Smartphones, Worldwide, 4Q07 and 2007. https://www.gartner.com/en/documents/619509/market-share-smartphones-worldwide-4q07-and-2007. Accessed 31 Aug 2022

  3. Saad, A., Liebers, J., Gruenefeld, U., Alt, F., Schneegass, S.: Understanding Bystanders’ tendency to shoulder surf smartphones using 360-degree videos in virtual reality. In: Proceedings of the 23rd International Conference on Mobile Human-Computer Interaction (MobileHCI 2021), Article 35, pp. 1–8. Association for Computing Machinery. New York, NY, USA (2021). https://doi.org/10.1145/3447526.3472058

  4. How Americans Use Their Cellphones in Public. https://www.pewresearch.org/internet/2015/08/26/chapter-2-phone-use-in-public-areas/. Accessed 31 Aug 2022

  5. Redmon, J., Farhadi, A.: YOLOv3: An Incremental Improvement (2018)

    Google Scholar 

  6. Darling, D., Li, A., Li, Q.: Automated bystander detection and anonymization in mobile photography. In: EAI International Conference on Security and Privacy in Communication Networks (SecureComm) (2020)

    Google Scholar 

  7. Darling, D., Li, A., Li, Q.: Feature-based model for automated identification of subjects and bystanders in photos. In: IEEE International Workshop on the Security, Privacy, and Digital Forensics of Mobile Systems and Networks (MobiSec) (2019)

    Google Scholar 

  8. Li, A., Darling, D., Li, Q.: PhotoSafer: content-based and context-aware private photo protection for smartphones. In: IEEE Symposium on Privacy-Aware Computing (PAC) (2018)

    Google Scholar 

  9. Li, A., Du, W., Li, Q.: PoliteCamera: respecting strangers’ privacy in mobile photographing. In: International Conference on Security and Privacy in Communication Networks (SecureComm) (2018)

    Google Scholar 

  10. Li, A., Li, Q., Gao, W.: PrivacyCamera: privacy-aware photographing with mobile phones. In: IEEE International Conference on Sensing, Communication and Networking (SECON) (2016)

    Google Scholar 

  11. Kumar, M., Garfinkel, T., Boneh, D., Winograd, T.: Reducing shoulder-surfing by using gaze-based password entry. In Proceedings of the 3rd symposium on Usable privacy and security (SOUPS 2007), 13–19. Association for Computing Machinery, New York, NY, USA (2017). https://doi.org/10.1145/1280680.1280683

  12. Chakraborty, N., Mondal, S.: Tag digit based honeypot to detect shoulder surfing attack. In: Mauri, J.L., Thampi, S.M., Rawat, D.B., Jin, D. (eds.) SSCC 2014. CCIS, vol. 467, pp. 101–110. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-662-44966-0_10

    Chapter  Google Scholar 

  13. Yu, X., Wang, Z., Li, Y., Li, L., Zhu, W.T., Song, L.: EvoPass: evolvable graphical password against shoulder-surfing attacks. Comput. Secur. 179–198 (2017)

    Google Scholar 

  14. Zhang, R., Zhang, N., Du, C., Lou, W., Hou, Y.T., Kawamoto, Y.: AugAuth: shoulder-surfing resistant authentication for augmented reality. In: 2017 IEEE International Conference on Communications (ICC), pp. 1–6 (2017). https://doi.org/10.1109/ICC.2017.7997251

  15. Sun, H.-M., Chen, S.-T., Yeh, J.-H., Cheng, C.-Y.: A shoulder surfing resistant graphical authentication system. IEEE Trans. Dependabl. Secur. Comput. 15(2), 180–193 (2018). https://doi.org/10.1109/TDSC.2016.2539942

    Article  Google Scholar 

  16. Zezschwitz, E., Ebbinghaus, S., Hussmann, H., Luca, A.: You can’t watch this! Privacy-respectful photo browsing on smartphones. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI 2016). Association for Computing Machinery, New York, NY, USA, 4320–4324 (2016). https://doi.org/10.1145/2858036.2858120

  17. Eiband, M., Zezschwitz, E., Buschek, D., Hußmann, H.: My scrawl hides it all: protecting text messages against shoulder surfing with handwritten fonts. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA 2016). Association for Computing Machinery, New York, NY, USA, 2041–2048 (2016). https://doi.org/10.1145/2851581.2892511

  18. Zhou, H., et al.: Enhancing mobile content privacy with proxemics aware notifications and protection. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI 2016). Association for Computing Machinery, New York, NY, USA, 1362–1373 (2016). https://doi.org/10.1145/2858036.2858232

  19. Khamis, M., Eiband, M., Zürn, M., Hussmann, H.: EyeSpot: leveraging gaze to protect private text content on mobile devices from shoulder surfing. Multimod. Technol. Interact. 2(3), 45 (2008). https://doi.org/10.3390/mti2030045

  20. Ragozin, K., Pai, Y., Augereau, O., Kise, K., Kerdels, J., Kunze, K.: Private reader: using eye tracking to improve reading privacy in public spaces. In Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI 2019). Association for Computing Machinery, New York, NY, USA, Article 18, pp. 1–6 (2019). https://doi.org/10.1145/3338286.3340129

  21. Saad, A., Chukwu, M., Schneegass, S.: Communicating shoulder surfing attacks to users. In: Proceedings of the 17th International Conference on Mobile and Ubiquitous Multimedia (MUM 2018), pp. 147–152. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3282894.3282919

  22. Lian, S., Hu, W., Song, X., Liu, Z.: Smart privacy-preserving screen based on multiple sensor fusion. In IEEE Trans. Consum. Electron. 59(1), 136–143 (2013). https://doi.org/10.1109/TCE.2013.6490252

    Article  Google Scholar 

  23. Ali, M.E., Anwar, A., Ahmed, I., Hashem, T., Kulik, L., Tanin, E.: Protecting mobile users from visual privacy attacks. In: Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp 2014 Adjunct), pp. 1–4. Association for Computing Machinery, New York, NY, USA (2014). https://doi.org/10.1145/2638728.2638788

  24. Watanabe, K., Higuchi, F., Inami, M., Igarashi, T.: CursorCamouflage: multiple dummy cursors as a defense against shoulder surfing. In: SIGGRAPH Asia 2012 Emerging Technologies (2012)

    Google Scholar 

  25. Li, C., Liang, M., Xiao, K., Fong, S., Wang, Q., Song, W.: Human body and face detection based anti-shoulder attack system on ATM. In: Proceedings of the International Conference on Big Data and Internet of Thing (BDIOT2017), pp. 145–148. Association for Computing Machinery, New York, NY, USA (2017). https://doi.org/10.1145/3175684.3175706

  26. Brudy, F., Ledo, D., Greenberg, S., Butz, A.: Is anyone looking? Mitigating shoulder surfing on public displays through awareness and protection. In: Proceedings of The International Symposium on Pervasive Displays (PerDis 2014), pp. 1–6. Association for Computing Machinery, New York, NY, USA (2014). https://doi.org/10.1145/2611009.2611028

  27. Tan, M., Pang, R., Le, Q.V. : EfficientDet: Scalable and Efficient Object Detection (2020)

    Google Scholar 

  28. Jocher, G., e al.: ultralytics/yolov5: v5.0 - YOLOv5-P6 1280 models, AWS, Supervise.ly and YouTube integrations (2021)

    Google Scholar 

  29. Wang, C. -Y., Mark Liao, H. -Y., Wu, Y. -H., Chen, P.-Y., Hsieh, J.-W., Yeh, I.-H.: CSPNet: a new backbone that can enhance learning capability of CNN. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1571–1580 (2020). https://doi.org/10.1109/CVPRW50498.2020.00203

  30. He, K., Zhang, X., Ren, S., Sun, J.: Spatial pyramid pooling in deep convolutional networks for visual recognition. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8691, pp. 346–361. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10578-9_23

    Chapter  Google Scholar 

  31. Liu, S., Qi, L., Qin, H., Shi, J., Jia, J.: Path Aggregation Network for Instance Segmentation (2018)

    Google Scholar 

  32. OpenCV: Smoothing Images. https://docs.opencv.org/3.4/d4/d13/tutorial_py_filtering.html. Accessed 31 Aug 2022

  33. OpenCV: Image Filtering. https://docs.opencv.org/4.x/d4/d86/group__imgproc__filter.html. Accessed 31 Aug 2022

  34. Browser Market Share Worldwide -December 2021. https://gs.statcounter.come/browser-market-share. Accessed 31 Aug 2022

  35. Top 100 Websites Ranking on the Web. https://rankranger.com/top-websites. Accessed 31 Aug 2022

  36. Kivilohkare, G.: Optimizing the Critical Rendering Path for Decreased Website Loading Time. Åbo Akademi (2020)

    Google Scholar 

  37. Lighthouse - Chrome Developers. https://developer.chrome.com/docs/lighthouse/. Accessed 31 Aug 2022

  38. Total Blocking Time - Chrome Developer. https://developer.chrome.com/docs/lighthouse/performance/lighthouse-total-blocking-time/. Accessed 31 Aug 2022

Download references

Author information

Authors and Affiliations

  1. University of Arkansas, Fayetteville, AR, USA

    David Darling, Yaling Liu & Qinghua Li

Authors
  1. David Darling
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yaling Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qinghua Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qinghua Li .

Editor information

Editors and Affiliations

  1. University of Pittsburgh, Pittsburgh, PA, USA

    Shangguan Longfei

  2. Microsoft Research, Redmond, WA, USA

    Priyantha Bodhi

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Darling, D., Liu, Y., Li, Q. (2023). Privacy Protection Against Shoulder Surfing in Mobile Environments. In: Longfei, S., Bodhi, P. (eds) Mobile and Ubiquitous Systems: Computing, Networking and Services. MobiQuitous 2022. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 492. Springer, Cham. https://doi.org/10.1007/978-3-031-34776-4_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-34776-4_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-34775-7

  • Online ISBN: 978-3-031-34776-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation