ABSTRACT
With the advancement of device software and hardware performance, and the evolution of game engines, an increasing number of emerging high-quality games are captivating game players from all around the world who speak different languages. However, due to the vast fragmentation of the device and platform market, a well-tested game may still experience text glitches when installed on a new device with an unseen screen resolution and system version, which can significantly impact the user experience. In our testing pipeline, current testing techniques for identifying multilingual text glitches are laborious and inefficient. In this paper, we present AG3, which offers intelligent game traversal, precise visual text glitch detection, and integrated quality report generation capabilities. Our empirical evaluation and internal industrial deployment demonstrate that AG3 can detect various real-world multilingual text glitches with minimal human involvement.
- D. Adamo, M. K. Khan, S. Koppula, and R. Bryce. 2018. Reinforcement learning for android gui testing. In 9th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation. 2–8. Google Scholar
- D. Amalfitano, A. R. Fasolino, P. Tramontana, S. D. Carmine, and A. M. Memon. 2012. Using GUI ripping for automated testing of android applications. In IEEE/ACM International Conference on Automated Software Engineering. 258–261. Google Scholar
- Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse engineering mobile application user interfaces with remaui. In 30th IEEE/ACM International Conference in Automated Software Engineering (ASE). 248–259. Google ScholarDigital Library
- Young-Min Baek and Doo-Hwan Bae. 2016. Automated model-based android gui testing using multi-level gui comparison criteria. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering. 238–249. Google ScholarDigital Library
- Farnaz Behrang, Steven P Reiss, and Alessandro Orso. 2018. GUIfetch: sup- porting app design and development through GUI search. In 5th International Conference on Mobile Software Engineering and Systems. 236–246. Google ScholarDigital Library
- N. P. Borges, J. Hotzkow, and A. Zeller. 2018. Droidmate-2: a platform for android test generation. In 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE). 916–919. Google Scholar
- Chunyang Chen, Sidong Feng, Zhengyang Liu, Zhenchang Xing, Shengdong Zhao, and Linda Liu. 2020. From Lost to Found: Discover Missing UI Design Semantics through Recovering Missing Tags. In the ACM on Human-Computer Interaction, Volume. 4, No. CSCW. Google Scholar
- Chunyang Chen, Sidong Feng, Zhenchang Xing, Linda Liu, and Shengdong Zhao. 2019. Gallery DC: Design Search and Knowledge Discovery through Auto-created GUI Component Gallery. In ACM on Human-Computer Interaction 3, CSCW (2019). 1–22. Google Scholar
- Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and Yang Liu. 2018. From ui design image to gui skeleton: a neural machine translator to bootstrap mobile gui implementation. In 40th International Conference on Software Engineering. 665–676. Google ScholarDigital Library
- Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xin Xia, Liming Zhu, John Grundy, and Jinshui Wang. 2020. Wireframe-based UI design search through image autoencoder. In ACM Transactions on Software Engineering and Methodology (TOSEM) 29, 3 (2020). 1–31. Google ScholarDigital Library
- Jieshan Chen, Mulong Xie, Zhenchang Xing, Chunyang Chen, Xiwei Xu, Liming Zhu, and Guoqiang Li. 2020. Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination? In ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Google ScholarDigital Library
- Ke Chen, Yufei Li, Yingfeng Chen, Changjie Fan, Zhipeng Hu, and Wei Yang. 2021. Glib: towards automated test oracle for graphically-rich applications. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1093–1104. Google ScholarDigital Library
- Sen Chen, Lingling Fan, Chunyang Chen, Ting Su, Wenhe Li, Yang Liu, and Lihua Xu. 2019. Storydroid: Automated generation of storyboard for Android apps. In 41st International Conference on Software Engineering (ICSE). 596–607. Google ScholarDigital Library
- Sen Chen, Lingling Fan, Chunyang Chen, Minhui Xue, Yang Liu, and Lihua Xu. 2019. GUI-Squatting Attack: Automated Generation of Android Phishing Apps. In IEEE Transactions on Dependable and Secure Computing (2019). Google ScholarDigital Library
- S. R. Choudhary, A. Gorla, and A. Orso. 2015. Automated test input generation for android: Are we there yet? In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). 429–440. Google Scholar
- Riccardo Coppola, Maurizio Morisio, and Marco Torchiano. 2017. Scripted gui testing of android apps: A study on diffusion, evolution and fragility. In Proceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering. 22–32. Google ScholarDigital Library
- F. Y. B. Daragh and S. Malek. 2021. Deep gui: Black-box gui input generation with deep learning. In 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). 905–916. Google Scholar
- Giovanni Denaro, Luca Guglielmo, Leonardo Mariani, and Oliviero Riganelli. 2019. GUI testing in production: challenges and opportunities. In Proceedings of the Conference Companion of the 3rd International Conference on Art, Science, and Engineering of Programming. 1–3. Google ScholarDigital Library
- Google Developers. 2023. UI/Application Exerciser Monkey. https://developer.android.com/studio/test/other-testing-tools/monkey Google Scholar
- Z. Dong, M. Bohme, L. Cojocaru, and A. Roychoudhury. 2020. Time-travel testing of android apps. In 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE). 481–492. Google Scholar
- Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, and Sylvain Gelly. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929. Google Scholar
- Yi Gao, Yang Luo, Daqing Chen, Haocheng Huang, Wei Dong, Mingyuan Xia, Xue Liu, and Jiajun Bu. 2017. Every pixel counts: Fine-grained UI rendering analysis for mobile applications. In 2017-IEEE Conference on Computer Communications. 1–9. Google ScholarCross Ref
- Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. 2021. Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv:2107.08430. Google Scholar
- Tianxiao Gu, Chengnian Sun, Xiaoxing Ma, Chun Cao, Chang Xu, Yuan Yao, Qirun Zhang, Jian Lu, and Zhendong Su. 2019. Practical GUI testing of Android applications via model abstraction and refinement. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). 269–280. Google ScholarDigital Library
- T. Gu, C. Sun, X. Ma, C. Cao, C. Xu, Y. Yao, Q. Zhang, J. Lu, and Z. Su. 2019. Practical gui testing of android applications via model abstraction and refinement. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). 269–280. Google Scholar
- Bernard J Jansen. 1998. The graphical user interface. ACM SIGCHI Bulletin, 30, 2 (1998), 22–26. Google ScholarDigital Library
- P. Kong, L. Li, J. Gao, K. Liu, T. F. Bissyande, and J. Klein. 2018. Automated testing of android apps: A systematic literature review. In IEEE Transactions on Reliability. 45–66. Google Scholar
- Wenjie Li, Yanyan Jiang, Chang Xu, Yepang Liu, Xiaoxing Ma, and Jian Lu. 2019. Characterizing and Detecting Inefficient Image Displaying Issues in Android Apps. In 26th IEEE International Conference on Software Analysis, Evolution and Reengineering. 355–365. Google Scholar
- Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2017. Droidbot: a lightweight ui-guided test input generator for android. In 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C). 23–26. Google Scholar
- Y. Li, Z. Yang, Y. Guo, and X. Chen. 2019. Humanoid: A deep learningbased approach to automated black-box android app testing. In 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). 1070–1073. Google Scholar
- Guoqing Liu, Mengzhang Cai, Li Zhao, Tao Qin, Adrian Brown, Jimmy Bischoff, and Tie-Yan Liu. 2022. Inspector: Pixel-Based Automated Game Testing via Exploration, Detection, and Investigation. In 2022 IEEE Conference on Games (CoG). 237–244. Google ScholarDigital Library
- Zhe Liu, Chunyang Chen, Junjie Wang, Yuekai Huang, Jun Hu, and Qing Wang. 2020. Owl eyes: Spotting ui display issues via visual understanding. In 2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE). 398–409. Google ScholarDigital Library
- Zhengwei Lv, Chao Peng, Zhao Zhang, Ting Su, Kai Liu, and Ping Yang. 2022. Fastbot2: Reusable Automated Model-based GUI Testing for Android Enhanced by Reinforcement Learning. In 37th IEEE/ACM International Conference on Automated Software Engineering (ASE 2022). Google ScholarDigital Library
- Yun Ma, Yangyang Huang, Ziniu Hu, Xusheng Xiao, and Xuanzhe Liu. 2019. Paladin: Automated generation of reproducible test cases for android apps. In Proceedings of the 20th International Workshop on Mobile Computing Systems and Applications. 99–104. Google ScholarDigital Library
- Aravind Machiry, Rohan Tahiliani, and Mayur Naik. 2013. Dynodroid: An input generation system for android apps. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. 224–234. Google ScholarDigital Library
- K. Mao, MG. Harman, and Y. Jia. 2016. Sapienz: Multi-objective automated testing for android applications. In 25th International Symposium on Software Testing and Analysis. 94–105. Google Scholar
- Leonardo Mariani, Mauro Pezzè, and Daniele Zuddas. 2018. Augusto: Exploiting popular functionalities for the generation of semantic gui tests with oracles. In Proceedings of the 40th International Conference on Software Engineering. 280–290. Google ScholarDigital Library
- Atif M Memon and Myra B Cohen. 2013. Automated testing of GUI applications: models, tools, and controlling flakiness. In 2013 35th International Conference on Software Engineering (ICSE). 1479–1480. Google ScholarCross Ref
- Nariman Mirzaei, Joshua Garcia, Hamid Bagheri, Alireza Sadeghi, and Sam Malek. 2016. Reducing combinatorics in GUI testing of android applications. In 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE). 559–570. Google ScholarDigital Library
- Kevin Moran, Carlos Bernal-Cárdenas, Michael Curcio, Richard Bonett, and Denys Poshyvanyk. 2018. Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps. arXiv preprint arXiv:1802.02312. Google Scholar
- Kevin Moran, Boyang Li, Carlos Bernal-Cárdenas, Dan Jelf, and Denys Poshy-vanyk. 2018. Automated reporting of GUI design violations for mobile apps. In 40th International Conference on Software Engineering. 165–175. Google ScholarDigital Library
- Kevin Moran, Mario Linares-Vásquez, Carlos Bernal-Cárdenas, Christopher Vendome, and Denys Poshyvanyk. 2017. Crashscope: A practical tool for automated testing of android applications. In 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C). 15–18. Google ScholarDigital Library
- Kevin Moran, Mario Linares Vásquez, and Denys Poshyvanyk. 2017. Automated GUI testing of Android apps: from research to practice. In 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C). 505–506. Google ScholarDigital Library
- Kevin Moran, Cody Watson, John Hoskins, George Purnell, and Denys Poshy-vanyk. 2018. Detecting and Summarizing GUI Changes in Evolving Mobile Apps. arXiv preprint arXiv:1807.09440. Google Scholar
- Alfredo Nantes, Ross Brown, and Frederic Maire. 2008. A framework for the semi-automatic testing of video games. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment. 4, 197–202. Google Scholar
- Ciprian Paduraru, Miruna Paduraru, and Alin Stefanescu. 2021. Automated game testing using computer vision methods. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW). 65–72. Google ScholarCross Ref
- Chao Peng, Zhao Zhang, Zhengwei Lv, and Ping Yang. 2022. MUBot: Learning to Test Large-Scale Commercial Android Apps like a Human. In 38th International Conference on Software Maintenance and Evolution (ICSME 2022). Google ScholarCross Ref
- Steven PReiss, Yun Miao, and Qi Xin. 2018. Seeking the User Interface. In Automated Software Engineering. 157–193. Google Scholar
- A. Romdhana, A. Merlo, M. Ceccato, and P. Tonella. 2021. Deep reinforcement learning for black-box testing of android apps. In 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). arXiv preprint arXiv:2101.02636. Google Scholar
- Frank Rosenblatt. 1958. The perceptron: a probabilistic model for information storage and organization in the brain.. Psychological review, 65, 6 (1958), 386. Google Scholar
- T. Su, G. Meng, Y. Chen, K. Wu, W. Yang, Yao Y., G. Pu, and Y. Liu. 2017. Guided, stochastic model-based gui testing of android apps. In 2017 11th Joint Meeting on Foundations of Software Engineering. 245–256. Google Scholar
- Ting Su, Guozhu Meng, Yuting Chen, Ke Wu, Weiming Yang, Yao Yao, Geguang Pu, Yang Liu, and Zhendong Su. 2017. Guided, stochastic model-based GUI testing of Android apps. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. 245–256. Google ScholarDigital Library
- Yuhui Su, Zhe Liu, Chunyang Chen, Junjie Wang, and Qing Wang. 2021. OwlEyes-online: a fully automated platform for detecting and localizing UI display issues. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1500–1504. Google ScholarDigital Library
- TestBird. 2023. Mobile Internet Testing Expert. https://www.testbird.com/ Google Scholar
- Torchvision. 2023. The pre-training model weights of VGG16. https://download.pytorch.org/models/vgg16-397923af.pth Google Scholar
- T. A. T. Vuong, S. Takada, S. Koppula, and R. Bryce. 2018. A reinforcement learning based approach to automated testing of android applications. In 9th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation. 31–37. Google Scholar
- W. Wang, D. Li, W. Yang, Y. Cao, Z. Zhang, Y. Deng, and T. Xie. 2018. An empirical study of android test generation tools in industrial cases. In 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE). 738–748. Google Scholar
- Wikipedia. 2023. Cosine similarity. https://en.wikipedia.org/wiki/Cosine_similarity Google Scholar
- Wikipedia. 2023. Levenshtein Distance. https://en.wikipedia.org/wiki/Levenshtein_distance Google Scholar
- Wikipedia. 2023. Long tail. https://en.wikipedia.org/wiki/Long_tail Google Scholar
- Dehai Zhao, Zhenchang Xing, Chunyang Chen, Xin Xia, and Guoqiang Li. 2019. ActionNet: vision-based workflow action recognition from programming screencasts. In 41st International Conference on Software Engineering (ICSE). 350–361. Google ScholarDigital Library
Index Terms
- AG3: Automated Game GUI Text Glitch Detection Based on Computer Vision
Recommendations
An empirical study of GUI widget detection for industrial mobile games
ESEC/FSE 2021: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software EngineeringWith the widespread adoption of smartphones in our daily life, mobile games experienced increasing demand over the past years. Meanwhile, the quality of mobile games has been continuously drawing more and more attention, which can greatly affect the ...
How computer gamers experience the game situation: a behavioral study
Theoretical and Practical Computer Applications in EntertainmentVery little is known about computer gamers' playing experience. Most social scientific research has treated gaming as an undifferentiated activity associated with various factors outside the gaming context. This article considers computer games as ...
Automated Evaluation of Game Experience based on Game Dynamics and Motives for Play
CHI PLAY '22: Extended Abstracts of the 2022 Annual Symposium on Computer-Human Interaction in PlayThis paper describes an automated evaluation of the overall game experience using a synthetic agent, that we contextualize for First-Person Shooter games. This evaluation method is based on the characterization of the game experience through dynamics ...
Comments