Zhe Liu
ABSTRACT
The complexity of GUI and some combinations of personalized settings make the UI display issues occur frequently. Unfortunately, little is known about the causes of UI display issues. The Android fragmentation and variety of UI components post a great challenge to repair the issue. Based on the our empirical study, this paper proposes Woodpecker to automatically detect, localize and repair UI display issues in Android apps. It detects screenshots with issues with computer vision technology, localizes buggy source code from screenshot, and repairs issues with pre-defined templates automatically. We evaluate Woodpecker with 30 real-world UI display issues, it can successfully detect 87% and repair 77% issues. We further apply Woodpecker to another 256 popular open-source Android apps, and successfully uncover 112 previously-undetected issues. It can automatically repair 106 (94%) issues, with 76 of them accepted by developers so far, while others pending (none of them is rejected).
- 2016. Mobile Internet use passes desktop for the first time. https://techcrunch.com/2016/11/01/mobile-internet-use-passes-desktop-for-the-first-time-study-finds/.Google Scholar
- 2020. Number of available applications in the Google Play Store from December 2009 to June 2020. https://www.statista.com/statistics/266210/number-of-available-applications-in-the-google-play-store.Google Scholar
- 2021. http://tools.android.com/tips/lint.Google Scholar
- 2021. https://github.com/stylelint/stylelint.Google Scholar
- 2021. Common UI bugs. https://www.qamadness.com/common-user-interface-bugs-or-why-ui-testing-matters/.Google Scholar
- 2021. F-droid. http://f-droid.org/.Google Scholar
- 2021. GPSlogger. https://github.com/mendhak/gpslogger.Google Scholar
- Abdulmajeed Alameer, Paul T. Chiou, and William G. J. Halfond. 2019. Efficiently Repairing Internationalization Presentation Failures by Solving Layout Constraints. In ICST 2019. IEEE, 172--182. Google ScholarCross Ref
- Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xiwei Xu, Liming Zhu, Guoqiang Li, and Jinshui Wang. 2020. Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep Learning. arXiv preprint arXiv:2003.00380 (2020).Google Scholar
- Ke Chen, Yufei Li, Yingfeng Chen, Changjie Fan, Zhipeng Hu, and Wei Yang. 2021. GLIB: Towards Automated Test Oracle for Graphically-Rich Applications. arXiv preprint arXiv:2106.10507 (2021).Google Scholar
- Liushan Chen, Yu Pei, and Carlo Alberto Furia. 2020. Contract-based program repair without the contracts: An extended study. IEEE Transactions on Software Engineering (2020).Google Scholar
- Paul T. Chiou, Ali S. Alotaibi, and William G. J. Halfond. 2021. Detecting and localizing keyboard accessibility failures in web applications. In ESEC/FSE '21. ACM, 855--867. Google ScholarDigital Library
- Barthelemy Dagenais and Martin P Robillard. 2009. SemDiff: Analysis and recommendation support for API evolution. In 2009 IEEE 31st International Conference on Software Engineering. IEEE, 599--602.Google ScholarDigital Library
- Favio DeMarco, Jifeng Xuan, Daniel Le Berre, and Martin Monperrus. 2014. Automatic repair of buggy if conditions and missing preconditions with SMT. In Proceedings of the 6th international workshop on constraints in software testing, verification, and analysis. 30--39.Google ScholarDigital Library
- Qing Gao, Hansheng Zhang, Jie Wang, Yingfei Xiong, Lu Zhang, and Hong Mei. 2015. Fixing recurring crash bugs via analyzing q&a sites (T). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).Google ScholarDigital Library
- Yi Gao, Yang Luo, Daqing Chen, Haocheng Huang, Wei Dong, Mingyuan Xia, Xue Liu, and Jiajun Bu. 2017. Every pixel counts: Fine-grained UI rendering analysis for mobile applications. In IEEE INFOCOM 2017-IEEE Conference on Computer Communications. IEEE, 1--9.Google ScholarCross Ref
- Linda Di Geronimo, Larissa Braz, Enrico Fregnan, Fabio Palomba, and Alberto Bacchelli. [n.d.]. UI Dark Patterns and Where to Find Them: A Study on Mobile Applications and User Perception.Google Scholar
- Claire Le Goues, Michael Pradel, and Abhik Roychoudhury. 2019. Automated program repair. Commun. ACM 62, 12 (2019), 56--65.Google ScholarDigital Library
- Bernard J. Jansen. 1998. The Graphical User Interface. SIGCHI Bull. 30, 2 (April 1998), 22--26. Google ScholarDigital Library
- Pingfan Kong, Li Li, Jun Gao, Tegawendé F Bissyandé, and Jacques Klein. 2019. Mining android crash fixes in the absence of issue-and change-tracking systems. In ISSTA. 78--89.Google Scholar
- Anil Koyuncu, Kui Liu, Tegawendé F. Bissyandé, Dongsun Kim, Jacques Klein, Martin Monperrus, and Yves Le Traon. 2020. FixMiner: Mining relevant fix patterns for automated program repair. Empir. Softw. Eng. 25, 3 (2020), 1980--2024. Google ScholarDigital Library
- Anil Koyuncu, Kui Liu, Tegawendé F. Bissyandé, Dongsun Kim, Martin Monperrus, Jacques Klein, and Yves Le Traon. 2019. iFixR: bug report driven program repair. In ESEC/SIGSOFT FSE 2019. ACM, 314--325.Google ScholarDigital Library
- Wenjie Li, Yanyan Jiang, Chang Xu, Yepang Liu, Xiaoxing Ma, and Jian Lu. 2019. Characterizing and detecting inefficient image displaying issues in Android apps. In 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 355--365.Google ScholarCross Ref
- Kui Liu, Anil Koyuncu, Tegawendé F Bissyandé, Dongsun Kim, Jacques Klein, and Yves Le Traon. 2019. You cannot fix what you cannot find! an investigation of fault localization bias in benchmarking automated program repair systems. In 2019 12th IEEE conference on software testing, validation and verification (ICST).Google ScholarCross Ref
- Kui Liu, Anil Koyuncu, Dongsun Kim, and Tegawendé F Bissyandé. 2019. Tbar: revisiting template-based automated program repair. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. 31--42.Google ScholarDigital Library
- Zhe Liu, Chunyang Chen, Junjie Wang, Yuekai Huang, Jun Hu, and Qing Wang. 2020. Owl Eyes: Spotting UI Display Issues via Visual Understanding. In 2020 35rd IEEE/ACM International Conference on Automated Software Engineering (ASE).Google ScholarDigital Library
- Fan Long and Martin Rinard. 2015. Staged program repair with condition synthesis. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. 166--178.Google ScholarDigital Library
- Siqi Ma, David Lo, Teng Li, and Robert H Deng. 2016. Cdrep: Automatic repair of cryptographic misuses in android applications. In Proceedings of the 11th ACM on Asia Conference on Computer and Communications Security. 711--722.Google ScholarDigital Library
- Fernanda Madeiral, Simon Urli, Marcelo de Almeida Maia, and Martin Monperrus. 2019. BEARS: An Extensible Java Bug Benchmark for Automatic Program Repair Studies. In SANER 2019, Hangzhou, China, February 24--27, 2019. IEEE, 468--478.Google Scholar
- Sonal Mahajan, Negarsadat Abolhassani, Phil McMinn, and William G. J. Halfond. 2018. Automated repair of mobile friendly problems in web pages. In ICSE 2018. ACM, 140--150. Google ScholarDigital Library
- Sonal Mahajan, Abdulmajeed Alameer, Phil McMinn, and William G. J. Halfond. 2017. Automated repair of layout cross browser issues using search-based techniques. In ISSTA. ACM, 249--260. Google ScholarDigital Library
- Sonal Mahajan, Abdulmajeed Alameer, Phil McMinn, and William G. J. Halfond. 2018. Automated Repair of Internationalization Presentation Failures in Web Pages Using Style Similarity Clustering and Search-Based Techniques. In ICST. IEEE Computer Society, 215--226.Google Scholar
- Sonal Mahajan, Abdulmajeed Alameer, Phil McMinn, and William G. J. Halfond. 2021. Effective automated repair of internationalization presentation failures in web applications using style similarity clustering and search-based techniques. Softw. Test. Verification Reliab. 31, 1--2 (2021). Google ScholarCross Ref
- Sonal Mahajan and William G. J. Halfond. 2015. Detection and Localization of HTML Presentation Failures Using Computer Vision-Based Techniques. In ICST 2015. IEEE, 1--10. Google ScholarCross Ref
- Sergey Mechtaev, Jooyong Yi, and Abhik Roychoudhury. 2016. Angelix: Scalable multiline program patch synthesis via symbolic analysis. In Proceedings of the 38th international conference on software engineering. 691--701.Google ScholarDigital Library
- Martin Monperrus. 2018. Automatic software repair: a bibliography. ACM Computing Surveys (CSUR) 51, 1 (2018), 1--24.Google ScholarDigital Library
- Fabiano Pecorelli, Gemma Catolino, Filomena Ferrucci, Andrea De Lucia, and Fabio Palomba. 2020. Testing of mobile applications in the wild: A large-scale empirical study on android apps. In Proceedings of the 28th International Conference on Program Comprehension. 296--307.Google ScholarDigital Library
- Yuhua Qi, Xiaoguang Mao, Yan Lei, Ziying Dai, and Chengsong Wang. 2014. The strength of random search on automated program repair. In Proceedings of the 36th International Conference on Software Engineering. 254--265.Google ScholarDigital Library
- Alireza Sadeghi, Reyhaneh Jabbarvand, and Sam Malek. 2017. PATDroid: permission-aware GUI testing of Android. In ESEC/FSE 2017. ACM, 220--232. Google ScholarDigital Library
- Shin Hwei Tan, Zhen Dong, Xiang Gao, and Abhik Roychoudhury. 2018. Repairing crashes in android apps. In 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE). IEEE, 187--198.Google ScholarDigital Library
- Mian Wan, Yuchen Jin, Ding Li, Jiaping Gui, Sonal Mahajan, and William G. J. Halfond. 2017. Detecting display energy hotspots in Android apps. Softw. Test. Verification Reliab. 27, 6 (2017). Google ScholarCross Ref
- Z. Wang, J. Gao, X. Chen, F. U. Hao-Jie, and X. Y. Fan. 2018. Automatic Program Repair Techniques: A Survey. Chinese Journal of Computers 41, 3 (2018), 588--610.Google Scholar
- Westley Weimer, Zachary P Fry, and Stephanie Forrest. 2013. Leveraging program equivalence for adaptive program repair: Models and first results. In ASE. IEEE.Google Scholar
- Thomas D White, Gordon Fraser, and Guy J Brown. 2019. Improving random GUI testing with image-based widget detection. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM.Google ScholarDigital Library
- Tianyong Wu, Xi Deng, Jun Yan, and Jian Zhang. 2019. Analyses for specific defects in android applications: A survey. Frontiers of Computer Science (2019).Google ScholarDigital Library
- Jifeng Xuan, Matias Martinez, Favio Demarco, Maxime Clement, Sebastian Lamelas Marcote, Thomas Durieux, Daniel Le Berre, and Martin Monperrus. 2016. Nopol: Automatic repair of conditional statement bugs in java programs. IEEE Transactions on Software Engineering 43, 1 (2016), 34--55.Google ScholarDigital Library
- Shengqian Yang, Haowei Wu, Hailong Zhang, Yan Wang, Chandrasekar Swaminathan, Dacong Yan, and Atanas Rountev. 2018. Static window transition graphs for Android. Automated Software Engineering 25, 4 (2018), 833--873.Google ScholarDigital Library
- Jooyong Yi, Shin Hwei Tan, Sergey Mechtaev, Marcel Böhme, and Abhik Roychoudhury. 2018. A correlation study between automated program repair and test-suite metrics. Empir. Softw. Eng. 23, 5 (2018), 2948--2979.Google ScholarDigital Library
- Shengcheng Yu, Chunrong Fang, Yexiao Yun, Zhenfei Cao, Kai Mei, Zhihao Cao, and Zhenyu Chen. 2021. Detecting Crowdsourced Test Report Consistency for Mobile Apps with Deep Image Understanding and Text Analysis. ICSE (2021).Google Scholar
- Fiorella Zampetti, Saghan Mudbhari, Venera Arnaoudova, Massimiliano Di Penta, Sebastiano Panichella, and Giuliano Antoniol. 2022. Using code reviews to automatically configure static analysis tools. Empir. Softw. Eng. 27, 1 (2022), 28. Google ScholarDigital Library
- Peilun Zhang, Yanjie Jiang, Anjiang Wei, Victoria Stodden, Darko Marinov, and August Shi. 2021. Domain-Specific Fixes for Flaky Tests with Wrong Assumptions on Underdetermined Specifications. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 50--61.Google Scholar
- Dehai Zhao, Zhenchang Xing, Chunyang Chen, Xiwei Xu, Liming Zhu, Guoqiang Li, and Jinshui Wang. 2020. Seenomaly: Vision-Based Linting of GUI Animation Effects Against Design-Don't Guidelines. In 42nd International Conference on Software Engineering (ICSE'20). ACM, New York, NY.Google ScholarDigital Library
Index Terms
- Woodpecker: identifying and fixing Android UI display issues
Index terms have been assigned to the content through auto-classification.
Recommendations
An Explorative Study of the Mobile App Ecosystem from App Developers' Perspective
WWW '17: Proceedings of the 26th International Conference on World Wide WebWith the prevalence of smartphones, app markets such as Apple App Store and Google Play has become the center stage in the mobile app ecosystem, with millions of apps developed by tens of thousands of app developers in each major market. This paper ...
Comments