Marco Guarnieri
David Basin
ABSTRACT
Test isolation is a prerequisite for the correct execution of test suites on web applications. We present Test Execution Checkpointing, a method for efficient test isolation. Our method instruments web applications to support checkpointing and exploits this support to isolate and optimize tests. We have implemented and evaluated this method on five popular PHP web applications. The results show that our method not only provides test isolation essentially for free, it also reduces testing time by 44% on average.
- Derek Allard. Bamboo Invoice. http://github.com/derekallard/BambooInvoice (Retrieved: January 2016).Google Scholar
- Anneliese A. Andrews, Jeff Offutt, and Roger T. Alexander. 2005. Testing web applications by modeling with FSMs. Software and Systems Modeling 4, 3 (2005), 326–345. Google ScholarDigital Library
- Jonathan Bell. 2014. Detecting, isolating, and enforcing dependencies among and within test cases. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 799–802. Google ScholarDigital Library
- Jonathan Bell and Gail Kaiser. 2014. Unit test virtualization with VMVM. In Proceedings of the 36th International Conference on Software Engineering. ACM, 550–561. Google ScholarDigital Library
- Jonathan Bell, Gail Kaiser, Eric Melski, and Mohan Dattatreya. 2015. Efficient dependency detection for safe Java test acceleration. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. ACM, 770–781. Google ScholarDigital Library
- Francesco Bolis, Angelo Gargantini, Marco Guarnieri, and Eros Magri. 2012. Evolutionary testing of PHP web applications with WETT. In International Symposium on Search Based Software Engineering. Springer, 285–291. Google ScholarDigital Library
- Francesco Bolis, Angelo Gargantini, Marco Guarnieri, Eros Magri, and Lorenzo Musto. 2012. Model-driven testing for web applications using abstract state machines. In Current Trends in Web Engineering: ICWE 2012 International Workshops. Springer, 71–78. Google ScholarDigital Library
- Greg Bronevetsky, Daniel Marques, Keshav Pingali, and Paul Stodghill. 2003. Automated Application-level Checkpointing of MPI Programs. In Proceedings of the 9th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP ’03). ACM, New York, NY, USA, 84–94. https:// Google ScholarDigital Library
- 781513Google Scholar
- Greg Bronevetsky, Daniel Marques, Keshav Pingali, Peter Szwed, and Martin Schulz. 2004. Application-level Checkpointing for Shared Memory Programs. In Proceedings of the 11th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS XI). ACM, New York, NY, USA, 235–247. https:// Google ScholarDigital Library
- Tristan Buchs. 2015. Checkpointing-based testing. Master’s thesis. ETH Zurich, Switzerland.Google Scholar
- Tsong Yueh Chen and Man Fai Lau. 1998. A new heuristic for test suite reduction. Information and Software Technology 40, 5-6 (1998), 347–354.Google Scholar
- Tsong Yueh Chen and Man Fai Lau. 1998. A simulation study on some heuristics for test suite reduction. Information and Software Technology 40, 13 (1998), 777–787.Google ScholarCross Ref
- Jong-Deok Choi and Harini Srinivasan. 1998. Deterministic Replay of Java Multithreaded Applications. In Proceedings of the SIGMETRICS Symposium on Parallel and Distributed Tools (SPDT ’98). ACM, New York, NY, USA, 48–59. https:// Google ScholarDigital Library
- CRIU. Checkpoint/Restore In Userspace. http://criu.org (Retrieved: January 2016).Google Scholar
- Pranavadatta Devaki, Suresh Thummalapenta, Nimit Singhania, and Saurabh Sinha. 2013. Efficient and Flexible GUI Test Execution via Test Merging. In Proceedings of the 2013 International Symposium on Software Testing and Analysis (ISSTA 2013). ACM, New York, NY, USA, 34–44. https:// Google ScholarDigital Library
- 2483781Google Scholar
- E. N. Elnozahy, D. B. Johnson, and W. Zwaenepoel. 1992. The performance of consistent checkpointing. In Proceedings 11th Symposium on Reliable Distributed Systems. 39–47. https://Google Scholar
- Elmootazbellah N. Elnozahy and James S. Plank. 2004. Checkpointing for Peta-Scale Systems: A Look into the Future of Practical Rollback-Recovery. IEEE Transactions on Dependable and Secure Computing 1, 2 (2004), 97–108. Google ScholarDigital Library
- Sequel: The Database Toolkit for Ruby. Testing with Sequel. http://sequel. jeremyevans.net/rdoc/files/doc/testing_rdoc.html (Retrieved: January 2016).Google Scholar
- Gordon Fraser and Franz Wotawa. 2007. Redundancy based test-suite reduction. In Proceedings of the 10th International Conference on Fundamental Approaches to Software Engineering (FASE 2007). 291–305. Google ScholarDigital Library
- Gallery. Gallery - Your photos on your website. http://galleryproject.org/ (Retrieved: January 2016).Google Scholar
- Alex Gyori, August Shi, Farah Hariri, and Darko Marinov. 2015. Reliable Testing: Detecting State-polluting Tests to Prevent Test Dependency. In Proceedings of the 2015 International Symposium on Software Testing and Analysis (ISSTA 2015). ACM, New York, NY, USA, 223–233. https:// Google ScholarDigital Library
- Dan Hao, Lu Zhang, Xingxia Wu, Hong Mei, and Gregg Rothermel. 2012. Ondemand Test Suite Reduction. In Proceedings of the 34th International Conference on Software Engineering (ICSE ’12). IEEE Press, Piscataway, NJ, USA, 738–748. Google ScholarDigital Library
- M Jean Harrold, Rajiv Gupta, and Mary Lou Soffa. 1993. A methodology for controlling the size of a test suite. ACM Transactions on Software Engineering and Methodology (TOSEM) 2, 3 (1993), 270–285. Google ScholarDigital Library
- Mats PE Heimdahl and Devaraj George. 2004. Test-suite reduction for model based tests: Effects on test quality and implications for testing. In Proceedings of the 19th IEEE International Conference on Automated Software Engineering. IEEE, 176–185. Google ScholarDigital Library
- Dennis Jeffrey and Neelam Gupta. 2007. Improving fault detection capability by selectively retaining test cases during test suite reduction. IEEE Transactions on software Engineering 33, 2 (2007). Google ScholarDigital Library
- James A Jones and Mary Jean Harrold. 2003. Test-suite reduction and prioritization for modified condition/decision coverage. IEEE Transactions on software Engineering 29, 3 (2003), 195–209. Google ScholarDigital Library
- S. A. Khalek and S. Khurshid. 2011. Efficiently Running Test Suites Using Abstract Undo Operations. In Proceedings of the 22nd IEEE International Symposium on Software Reliability Engineering. 110–119. https:// Google ScholarDigital Library
- Donald E. Knuth. 1998. The Art of Computer Programming (2nd ed.). Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA. Google ScholarDigital Library
- Berkeley Lab. Berkeley Lab Checkpoint/Restart (BLCR) for LINUX. http://crd.lbl. gov/departments/computer-science/CLaSS/research/BLCR/ (Retrieved: January 2016).Google Scholar
- Wing Lam, Sai Zhang, and Michael D. Ernst. 2015. When tests collide: Evaluating and coping with the impact of test dependence. Technical Report UW-CSE-15-03- 01. University of Washington Department of Computer Science and Engineering, Seattle, WA, USA.Google Scholar
- J. L. Lawall and G. Muller. 2000. Efficient incremental checkpointing of Java programs. In Proceedings of the 2000 International Conference on Dependable Systems and Networks. 61–70. https:// Google ScholarDigital Library
- LibCkpt. A Portable Checkpointer for Unix. http://web.eecs.utk.edu/plank/plank/ www/libckpt.html (Retrieved: January 2016).Google Scholar
- Giuseppe Antonio Di Lucca, Anna Rita Fasolino, Francesco Faralli, and Ugo De Carlini. 2002. Testing web applications. In ICSM.Google Scholar
- Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, and Darko Marinov. 2014. An Empirical Analysis of Flaky Tests. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2014). ACM, New York, NY, USA, 643–653. https:// Google ScholarDigital Library
- Gerard Meszaros. 2007. xUnit test patterns: Refactoring test code. Pearson Education. Google ScholarDigital Library
- James Mickens, Jeremy Elson, and Jon Howell. 2010. Mugshot: Deterministic Capture and Replay for JavaScript Applications. In 7th USENIX Symposium on Networked Systems Design and Implementation (NSDI 10). USENIX Association, San Jose, CA. https://www.usenix.org/conference/nsdi10-0/ mugshot-deterministic-capture-and-replay-javascript-applications Google ScholarDigital Library
- Microsoft MSDN. How to: Write a SQL Server Unit Test that Runs within the Scope of a Single Transaction. http://msdn.microsoft.com/en-US/library/ jj851217(v=vs.103).aspx (Retrieved: January 2016).Google Scholar
- Kivanç Muşlu, Bilge Soran, and Jochen Wuttke. 2011. Finding Bugs by Isolating Unit Tests. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (ESEC/FSE ’11). ACM, New York, NY, USA, 496–499. https:// Google ScholarDigital Library
- OSCommerce. OSCommerce - Creating Online Stores Worldwide. http://www. oscommerce.com/ (Retrieved: January 2016).Google Scholar
- phpBB. phpBB - Free and Open Source Forum Software. http://www.phpbb.com/ (Retrieved: January 2016).Google Scholar
- Feng Qin, Joseph Tucek, Jagadeesan Sundaresan, and Yuanyuan Zhou. 2005. Rx: Treating Bugs As Allergies—a Safe Method to Survive Software Failures. In Proceedings of the 20th ACM Symposium on Operating Systems Principles (SOSP ’05). ACM, New York, NY, USA, 235–248. https:// Google ScholarDigital Library
- Filippo Ricca and Paolo Tonella. 2001. Analysis and Testing of Web Applications. In Proceedings of the 23rd International Conference on Software Engineering (ICSE ’01). IEEE Computer Society, Washington, DC, USA, 25–34. Google ScholarDigital Library
- Gregg Rothermel, Mary Jean Harrold, Jeffery Ostrin, and Christie Hong. 1998. An empirical study of the effects of minimization on the fault detection capabilities of test suites. In Proceedings of the 14th International Conference on Software Maintenance. IEEE, 34–43. Google ScholarDigital Library
- Sreedevi Sampath, Valentin Mihaylov, Amie Souter, and Lori Pollock. 2004. Composing a Framework to Automate Testing of Operational Web-Based Software. In Proceedings of the 20th IEEE International Conference on Software Maintenance (ICSM ’04). IEEE Computer Society, Washington, DC, USA, 104–113. Google ScholarDigital Library
- Sara Sprenkle, Emily Gibson, Sreedevi Sampath, and Lori Pollock. 2005. Automated Replay and Failure Detection for Web Applications. In Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering (ASE ’05). ACM, New York, NY, USA, 253–262. https:// Google ScholarDigital Library
- Sriraman Tallam and Neelam Gupta. 2005. A Concept Analysis Inspired Greedy Algorithm for Test Suite Minimization. In Proceedings of the 6th ACM SIGPLANSIGSOFT Workshop on Program Analysis for Software Tools and Engineering (PASTE ’05). ACM, New York, NY, USA, 35–42. https:// Google ScholarDigital Library
- tSQLt. tSQLt – Database Unit Testing for SQL Server. http://tsqlt.org/ (Retrieved: January 2016).Google Scholar
- A. Vahabzadeh, A. M. Fard, and A. Mesbah. 2015. An empirical study of bugs in test code. In 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME). 101–110. https:// Google ScholarDigital Library
- w3af. Open Source Web Application Security Scanner. http://w3af.org/ (Retrieved: January 2016). ISSTA’17, July 2017, Santa Barbara, CA, USA M. Guarnieri, P. Tsankov, T. Buchs, M. Torabi Dashti, and D. BasinGoogle Scholar
- W. Eric Wong, Joseph R. Horgan, Saul London, and Hira Agrawal Bellcore. 1997. A Study of Effective Regression Testing in Practice. In Proceedings of the 8th International Symposium on Software Reliability Engineering (ISSRE ’97). IEEE Computer Society, Washington, DC, USA, 264–274. Google ScholarDigital Library
- Wordpress. Wordpress - Blog, Tool, Publishing Platform, and CMS. http://wordpress.org/ (Retrieved: January 2016).Google Scholar
- Guoqing Xu, Atanas Rountev, Yan Tang, and Feng Qin. 2007. Efficient Checkpointing of Java Software Using Context-sensitive Capture and Replay. In Proceedings of the the 6th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on The Foundations of Software Engineering (ESEC-FSE ’07). ACM, New York, NY, USA, 85–94. https: // Google ScholarDigital Library
- Sai Zhang, Darioush Jalali, Jochen Wuttke, Kç Muşlu, Wing Lam, Michael D. Ernst, and David Notkin. 2014. Empirically Revisiting the Test Independence Assumption. In Proceedings of the 2014 International Symposium on Software Testing and Analysis (ISSTA 2014). ACM, New York, NY, USA, 385–396. https: // Google ScholarDigital Library
- Hao Zhong, Lu Zhang, and Hong Mei. 2006. An Experimental Comparison of Four Test Suite Reduction Techniques. In Proceedings of the 28th International Conference on Software Engineering (ICSE ’06). ACM, New York, NY, USA, 636–640. https:// Google ScholarDigital Library
- Qian Zhongsheng. 2010. Test case generation and optimization for user sessionbased web application testing. Journal of Computers 5, 11 (2010), 1655–1662.Google Scholar
Index Terms
- Test execution checkpointing for web applications
Recommendations
Leveraging existing tests in automated test generation for web applications
ASE '14: Proceedings of the 29th ACM/IEEE International Conference on Automated Software EngineeringTo test web applications, developers currently write test cases in frameworks such as Selenium. On the other hand, most web test generation techniques rely on a crawler to explore the dynamic states of the application. The first approach requires much ...
DOM-based test adequacy criteria for web applications
ISSTA 2014: Proceedings of the 2014 International Symposium on Software Testing and AnalysisTo assess the quality of web application test cases, web developers currently measure code coverage. Although code coverage has traditionally been a popular test adequacy criterion, we believe it alone is not adequate for assessing the quality of web ...
Parallel execution of prioritized test cases for regression testing of web applications
ACSC '13: Proceedings of the Thirty-Sixth Australasian Computer Science Conference - Volume 135We present a new approach for automatically prioritizing and distributing test cases on multiple machines. Our approach is based on a functional dependency graph (FDG) of a web application. We partition the test suite into test sets according to the ...
Comments