skip to main content
10.1145/3611643.3613867acmconferencesArticle/Chapter ViewAbstractPublication PagesfseConference Proceedingsconference-collections
research-article

AG3: Automated Game GUI Text Glitch Detection Based on Computer Vision

Published:30 November 2023Publication History

ABSTRACT

With the advancement of device software and hardware performance, and the evolution of game engines, an increasing number of emerging high-quality games are captivating game players from all around the world who speak different languages. However, due to the vast fragmentation of the device and platform market, a well-tested game may still experience text glitches when installed on a new device with an unseen screen resolution and system version, which can significantly impact the user experience. In our testing pipeline, current testing techniques for identifying multilingual text glitches are laborious and inefficient. In this paper, we present AG3, which offers intelligent game traversal, precise visual text glitch detection, and integrated quality report generation capabilities. Our empirical evaluation and internal industrial deployment demonstrate that AG3 can detect various real-world multilingual text glitches with minimal human involvement.

References

  1. D. Adamo, M. K. Khan, S. Koppula, and R. Bryce. 2018. Reinforcement learning for android gui testing. In 9th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation. 2–8. Google ScholarGoogle Scholar
  2. D. Amalfitano, A. R. Fasolino, P. Tramontana, S. D. Carmine, and A. M. Memon. 2012. Using GUI ripping for automated testing of android applications. In IEEE/ACM International Conference on Automated Software Engineering. 258–261. Google ScholarGoogle Scholar
  3. Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse engineering mobile application user interfaces with remaui. In 30th IEEE/ACM International Conference in Automated Software Engineering (ASE). 248–259. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Young-Min Baek and Doo-Hwan Bae. 2016. Automated model-based android gui testing using multi-level gui comparison criteria. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering. 238–249. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Farnaz Behrang, Steven P Reiss, and Alessandro Orso. 2018. GUIfetch: sup- porting app design and development through GUI search. In 5th International Conference on Mobile Software Engineering and Systems. 236–246. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. N. P. Borges, J. Hotzkow, and A. Zeller. 2018. Droidmate-2: a platform for android test generation. In 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE). 916–919. Google ScholarGoogle Scholar
  7. Chunyang Chen, Sidong Feng, Zhengyang Liu, Zhenchang Xing, Shengdong Zhao, and Linda Liu. 2020. From Lost to Found: Discover Missing UI Design Semantics through Recovering Missing Tags. In the ACM on Human-Computer Interaction, Volume. 4, No. CSCW. Google ScholarGoogle Scholar
  8. Chunyang Chen, Sidong Feng, Zhenchang Xing, Linda Liu, and Shengdong Zhao. 2019. Gallery DC: Design Search and Knowledge Discovery through Auto-created GUI Component Gallery. In ACM on Human-Computer Interaction 3, CSCW (2019). 1–22. Google ScholarGoogle Scholar
  9. Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and Yang Liu. 2018. From ui design image to gui skeleton: a neural machine translator to bootstrap mobile gui implementation. In 40th International Conference on Software Engineering. 665–676. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xin Xia, Liming Zhu, John Grundy, and Jinshui Wang. 2020. Wireframe-based UI design search through image autoencoder. In ACM Transactions on Software Engineering and Methodology (TOSEM) 29, 3 (2020). 1–31. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Jieshan Chen, Mulong Xie, Zhenchang Xing, Chunyang Chen, Xiwei Xu, Liming Zhu, and Guoqiang Li. 2020. Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination? In ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Ke Chen, Yufei Li, Yingfeng Chen, Changjie Fan, Zhipeng Hu, and Wei Yang. 2021. Glib: towards automated test oracle for graphically-rich applications. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1093–1104. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Sen Chen, Lingling Fan, Chunyang Chen, Ting Su, Wenhe Li, Yang Liu, and Lihua Xu. 2019. Storydroid: Automated generation of storyboard for Android apps. In 41st International Conference on Software Engineering (ICSE). 596–607. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Sen Chen, Lingling Fan, Chunyang Chen, Minhui Xue, Yang Liu, and Lihua Xu. 2019. GUI-Squatting Attack: Automated Generation of Android Phishing Apps. In IEEE Transactions on Dependable and Secure Computing (2019). Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. S. R. Choudhary, A. Gorla, and A. Orso. 2015. Automated test input generation for android: Are we there yet? In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). 429–440. Google ScholarGoogle Scholar
  16. Riccardo Coppola, Maurizio Morisio, and Marco Torchiano. 2017. Scripted gui testing of android apps: A study on diffusion, evolution and fragility. In Proceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering. 22–32. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. F. Y. B. Daragh and S. Malek. 2021. Deep gui: Black-box gui input generation with deep learning. In 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). 905–916. Google ScholarGoogle Scholar
  18. Giovanni Denaro, Luca Guglielmo, Leonardo Mariani, and Oliviero Riganelli. 2019. GUI testing in production: challenges and opportunities. In Proceedings of the Conference Companion of the 3rd International Conference on Art, Science, and Engineering of Programming. 1–3. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Google Developers. 2023. UI/Application Exerciser Monkey. https://developer.android.com/studio/test/other-testing-tools/monkey Google ScholarGoogle Scholar
  20. Z. Dong, M. Bohme, L. Cojocaru, and A. Roychoudhury. 2020. Time-travel testing of android apps. In 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE). 481–492. Google ScholarGoogle Scholar
  21. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, and Sylvain Gelly. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929. Google ScholarGoogle Scholar
  22. Yi Gao, Yang Luo, Daqing Chen, Haocheng Huang, Wei Dong, Mingyuan Xia, Xue Liu, and Jiajun Bu. 2017. Every pixel counts: Fine-grained UI rendering analysis for mobile applications. In 2017-IEEE Conference on Computer Communications. 1–9. Google ScholarGoogle ScholarCross RefCross Ref
  23. Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. 2021. Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv:2107.08430. Google ScholarGoogle Scholar
  24. Tianxiao Gu, Chengnian Sun, Xiaoxing Ma, Chun Cao, Chang Xu, Yuan Yao, Qirun Zhang, Jian Lu, and Zhendong Su. 2019. Practical GUI testing of Android applications via model abstraction and refinement. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). 269–280. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. T. Gu, C. Sun, X. Ma, C. Cao, C. Xu, Y. Yao, Q. Zhang, J. Lu, and Z. Su. 2019. Practical gui testing of android applications via model abstraction and refinement. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). 269–280. Google ScholarGoogle Scholar
  26. Bernard J Jansen. 1998. The graphical user interface. ACM SIGCHI Bulletin, 30, 2 (1998), 22–26. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. P. Kong, L. Li, J. Gao, K. Liu, T. F. Bissyande, and J. Klein. 2018. Automated testing of android apps: A systematic literature review. In IEEE Transactions on Reliability. 45–66. Google ScholarGoogle Scholar
  28. Wenjie Li, Yanyan Jiang, Chang Xu, Yepang Liu, Xiaoxing Ma, and Jian Lu. 2019. Characterizing and Detecting Inefficient Image Displaying Issues in Android Apps. In 26th IEEE International Conference on Software Analysis, Evolution and Reengineering. 355–365. Google ScholarGoogle Scholar
  29. Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2017. Droidbot: a lightweight ui-guided test input generator for android. In 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C). 23–26. Google ScholarGoogle Scholar
  30. Y. Li, Z. Yang, Y. Guo, and X. Chen. 2019. Humanoid: A deep learningbased approach to automated black-box android app testing. In 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). 1070–1073. Google ScholarGoogle Scholar
  31. Guoqing Liu, Mengzhang Cai, Li Zhao, Tao Qin, Adrian Brown, Jimmy Bischoff, and Tie-Yan Liu. 2022. Inspector: Pixel-Based Automated Game Testing via Exploration, Detection, and Investigation. In 2022 IEEE Conference on Games (CoG). 237–244. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Zhe Liu, Chunyang Chen, Junjie Wang, Yuekai Huang, Jun Hu, and Qing Wang. 2020. Owl eyes: Spotting ui display issues via visual understanding. In 2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE). 398–409. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Zhengwei Lv, Chao Peng, Zhao Zhang, Ting Su, Kai Liu, and Ping Yang. 2022. Fastbot2: Reusable Automated Model-based GUI Testing for Android Enhanced by Reinforcement Learning. In 37th IEEE/ACM International Conference on Automated Software Engineering (ASE 2022). Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Yun Ma, Yangyang Huang, Ziniu Hu, Xusheng Xiao, and Xuanzhe Liu. 2019. Paladin: Automated generation of reproducible test cases for android apps. In Proceedings of the 20th International Workshop on Mobile Computing Systems and Applications. 99–104. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Aravind Machiry, Rohan Tahiliani, and Mayur Naik. 2013. Dynodroid: An input generation system for android apps. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. 224–234. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. K. Mao, MG. Harman, and Y. Jia. 2016. Sapienz: Multi-objective automated testing for android applications. In 25th International Symposium on Software Testing and Analysis. 94–105. Google ScholarGoogle Scholar
  37. Leonardo Mariani, Mauro Pezzè, and Daniele Zuddas. 2018. Augusto: Exploiting popular functionalities for the generation of semantic gui tests with oracles. In Proceedings of the 40th International Conference on Software Engineering. 280–290. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Atif M Memon and Myra B Cohen. 2013. Automated testing of GUI applications: models, tools, and controlling flakiness. In 2013 35th International Conference on Software Engineering (ICSE). 1479–1480. Google ScholarGoogle ScholarCross RefCross Ref
  39. Nariman Mirzaei, Joshua Garcia, Hamid Bagheri, Alireza Sadeghi, and Sam Malek. 2016. Reducing combinatorics in GUI testing of android applications. In 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE). 559–570. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Kevin Moran, Carlos Bernal-Cárdenas, Michael Curcio, Richard Bonett, and Denys Poshyvanyk. 2018. Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps. arXiv preprint arXiv:1802.02312. Google ScholarGoogle Scholar
  41. Kevin Moran, Boyang Li, Carlos Bernal-Cárdenas, Dan Jelf, and Denys Poshy-vanyk. 2018. Automated reporting of GUI design violations for mobile apps. In 40th International Conference on Software Engineering. 165–175. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Kevin Moran, Mario Linares-Vásquez, Carlos Bernal-Cárdenas, Christopher Vendome, and Denys Poshyvanyk. 2017. Crashscope: A practical tool for automated testing of android applications. In 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C). 15–18. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Kevin Moran, Mario Linares Vásquez, and Denys Poshyvanyk. 2017. Automated GUI testing of Android apps: from research to practice. In 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C). 505–506. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Kevin Moran, Cody Watson, John Hoskins, George Purnell, and Denys Poshy-vanyk. 2018. Detecting and Summarizing GUI Changes in Evolving Mobile Apps. arXiv preprint arXiv:1807.09440. Google ScholarGoogle Scholar
  45. Alfredo Nantes, Ross Brown, and Frederic Maire. 2008. A framework for the semi-automatic testing of video games. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment. 4, 197–202. Google ScholarGoogle Scholar
  46. Ciprian Paduraru, Miruna Paduraru, and Alin Stefanescu. 2021. Automated game testing using computer vision methods. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW). 65–72. Google ScholarGoogle ScholarCross RefCross Ref
  47. Chao Peng, Zhao Zhang, Zhengwei Lv, and Ping Yang. 2022. MUBot: Learning to Test Large-Scale Commercial Android Apps like a Human. In 38th International Conference on Software Maintenance and Evolution (ICSME 2022). Google ScholarGoogle ScholarCross RefCross Ref
  48. Steven PReiss, Yun Miao, and Qi Xin. 2018. Seeking the User Interface. In Automated Software Engineering. 157–193. Google ScholarGoogle Scholar
  49. A. Romdhana, A. Merlo, M. Ceccato, and P. Tonella. 2021. Deep reinforcement learning for black-box testing of android apps. In 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). arXiv preprint arXiv:2101.02636. Google ScholarGoogle Scholar
  50. Frank Rosenblatt. 1958. The perceptron: a probabilistic model for information storage and organization in the brain.. Psychological review, 65, 6 (1958), 386. Google ScholarGoogle Scholar
  51. T. Su, G. Meng, Y. Chen, K. Wu, W. Yang, Yao Y., G. Pu, and Y. Liu. 2017. Guided, stochastic model-based gui testing of android apps. In 2017 11th Joint Meeting on Foundations of Software Engineering. 245–256. Google ScholarGoogle Scholar
  52. Ting Su, Guozhu Meng, Yuting Chen, Ke Wu, Weiming Yang, Yao Yao, Geguang Pu, Yang Liu, and Zhendong Su. 2017. Guided, stochastic model-based GUI testing of Android apps. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. 245–256. Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Yuhui Su, Zhe Liu, Chunyang Chen, Junjie Wang, and Qing Wang. 2021. OwlEyes-online: a fully automated platform for detecting and localizing UI display issues. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1500–1504. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. TestBird. 2023. Mobile Internet Testing Expert. https://www.testbird.com/ Google ScholarGoogle Scholar
  55. Torchvision. 2023. The pre-training model weights of VGG16. https://download.pytorch.org/models/vgg16-397923af.pth Google ScholarGoogle Scholar
  56. T. A. T. Vuong, S. Takada, S. Koppula, and R. Bryce. 2018. A reinforcement learning based approach to automated testing of android applications. In 9th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation. 31–37. Google ScholarGoogle Scholar
  57. W. Wang, D. Li, W. Yang, Y. Cao, Z. Zhang, Y. Deng, and T. Xie. 2018. An empirical study of android test generation tools in industrial cases. In 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE). 738–748. Google ScholarGoogle Scholar
  58. Wikipedia. 2023. Cosine similarity. https://en.wikipedia.org/wiki/Cosine_similarity Google ScholarGoogle Scholar
  59. Wikipedia. 2023. Levenshtein Distance. https://en.wikipedia.org/wiki/Levenshtein_distance Google ScholarGoogle Scholar
  60. Wikipedia. 2023. Long tail. https://en.wikipedia.org/wiki/Long_tail Google ScholarGoogle Scholar
  61. Dehai Zhao, Zhenchang Xing, Chunyang Chen, Xin Xia, and Guoqiang Li. 2019. ActionNet: vision-based workflow action recognition from programming screencasts. In 41st International Conference on Software Engineering (ICSE). 350–361. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. AG3: Automated Game GUI Text Glitch Detection Based on Computer Vision

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in
      • Published in

        cover image ACM Conferences
        ESEC/FSE 2023: Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
        November 2023
        2215 pages
        ISBN:9798400703270
        DOI:10.1145/3611643

        Copyright © 2023 ACM

        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 30 November 2023

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article

        Acceptance Rates

        Overall Acceptance Rate112of543submissions,21%
      • Article Metrics

        • Downloads (Last 12 months)106
        • Downloads (Last 6 weeks)12

        Other Metrics

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader