ABSTRACT
Constraint-based input generation enables systematic testing for effective bug finding, but requires exploration of very large spaces of candidate inputs. This paper introduces a novel approach to optimize input generation using Korat – a solver for constraints written as imperative predicates in Java – when Korat is executed more than once for the same constraint solving problem. Our key insight is that in certain application scenarios the Korat search over the same state space and constraint is repeated across separate runs of Korat, and an earlier run can be summarized to optimize a later run. We introduce invalid ranges to represent parts of the exploration space that do not contain any valid inputs but must be explicitly explored by Korat. Our approach directly prunes these parts in a future run of Korat over the same search problem. We develop our approach for two settings: a sequential setting where the Korat search is run using one worker (i.e., processing unit), and a parallel setting where the search is distributed to several workers. In the parallel setting, we build on a previous technique for parallel Korat, namely SEQ-ON, and integrate invalid ranges with it. Experimental evaluation using 6 subjects show that our approach achieves: in the sequential setting, a speedup of up to 2.82X over sequential Korat (in comparison, SEQ-ON does not provide any speedup in the sequential setting); and in the distributed setting, using up to 32 workers, a speedup of up to 38.84X over sequential Korat (using 1 worker), and up to 3.04X over SEQ-ON in terms of total execution time across the workers.
- Thomas Ball, Daniel Hoffman, Frank Ruskey, Richard Webber, and Lee White. 2000. State generation and automated class testing. Software Testing, Verification and Reliability 10, 3 (2000), 149–170.Google ScholarCross Ref
- Dirk Beyer, Thomas A. Henzinger, M. Erkan Keremoglu, and Philipp Wendler. 2012. Conditional Model Checking: A Technique to Pass Information Between Verifiers. In Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering (FSE ’12). ACM, New York, NY, USA, 57:1–57:11. Google ScholarDigital Library
- Chandrasekhar Boyapati, Sarfraz Khurshid, and Darko Marinov. 2002. Korat: Automated testing based on Java predicates. In ISSTA. ACM, New York, NY, USA, 123–133. Google ScholarDigital Library
- Stefan Bucur, Vlad Ureche, Cristian Zamfir, and George Candea. 2011. Parallel Symbolic Execution for Automated Real-world Software Testing. In Proceedings of the Sixth Conference on Computer Systems (EuroSys). 183–198. Google ScholarDigital Library
- Cristian Cadar and Dawson R. Engler. 2005. Execution Generated Test Cases: How to Make Systems Code Crash Itself. In 12th International Conference on Model Checking Software (SPIN’05). Springer-Verlag, Berlin, Heidelberg, 2–23. Google ScholarDigital Library
- Edmund M. Clarke, Jr., Orna Grumberg, and Doron A. Peled. 1999. Model Checking. MIT Press, Cambridge, MA, USA. Google ScholarDigital Library
- Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein. 2009. Introduction to Algorithms, Third Edition (3rd ed.). The MIT Press, Cambridge, MA, USA. Google ScholarDigital Library
- Nima Dini. 2016. MKorat: A Novel Approach for Memoizing the Korat Search and Some Potential Applications.Google Scholar
- Matthew B. Dwyer, Sebastian Elbaum, Suzette Person, and Rahul Purandare. 2007. Parallel Randomized State-Space Search. In Proceedings of the 29th International Conference on Software Engineering (ICSE ’07), Vol. 37. IEEE Computer Society, Washington, DC, USA, 3–12. Google ScholarDigital Library
- Niklas Een and Niklas Sorensson. 2003. An Extensible SAT-solver. In 6th Conference on Theory and Applications of Satisfiability Testing (SAT). Santa Margherita Ligure, Italy.Google Scholar
- Antonio Filieri, Marcelo F. Frias, Corina S. Pă£Să£Reanu, and Willem Visser. 2015. Model Counting for Complex Data Structures. In Proceedings of the 22nd International Symposium on Model Checking Software (SPIN 2015), Vol. 9232. Springer-Verlag New York, Inc., New York, NY, USA, 222–241. Google ScholarDigital Library
- Diego Funes, Junaid Haroon Siddiqui, and Sarfraz Khurshid. 2012. Ranged Model Checking. ACM SIGSOFT Software Engineering Notes 37, 6 (2012), 1–5. Google ScholarDigital Library
- Milos Gligoric, Tihomir Gvero, Vilas Jagannath, Sarfraz Khurshid, Viktor Kuncak, and Darko Marinov. 2010. Test generation through programming in UDITA. In 32Nd ACM/IEEE International Conference on Software Engineering - Volume 1 (ICSE ’10). ACM, New York, NY, USA, 225–234. Google ScholarDigital Library
- Patrice Godefroid. 1997. Model Checking for Programming Languages using Verisoft. In Conference Record of POPL’97: The 24th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, Papers Presented at the Symposium. ACM, New York, NY, USA, 174–186. Google ScholarDigital Library
- Patrice Godefroid. 2007. Compositional Dynamic Test Generation. In Proceedings of the 34th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL ’07). ACM, New York, NY, USA, 47–54. Google ScholarDigital Library
- Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: directed automated random testing. In Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’05). ACM, New York, NY, USA, 213–223. Google ScholarDigital Library
- M.J. Harrold and M.L. Souffa. 1988. An incremental approach to unit testing during maintenance. In Conference on Software Maintenance.Google Scholar
- Gerald Holzmann. 1997. The Model Checker SPIN. IEEE Transactions on Software Engineering 23, 5 (May 1997). Google ScholarDigital Library
- Gerard J. Holzmann, Rajeev Joshi, and Alex Groce. 2011. Swarm Verification Techniques. IEEE Trans. Softw. Eng. 37, 6 (Nov 2011), 845–857. Google ScholarDigital Library
- Daniel Jackson and Mandana Vaziri. 2000. Finding Bugs with a Constraint Solver. In International Symposium oAn Software Testing and Analysis (ISSTA). Google ScholarDigital Library
- Sarfraz Khurshid, Corina S. Pasareanu, and Willem Visser. 2003. Generalized Symbolic Execution for Model Checking and Testing. In Proceedings of the 9th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS’03). Springer-Verlag, Berlin, Heidelberg, 553–568. Google ScholarDigital Library
- James C. King. 1976. Symbolic Execution and Program Testing. Commun. ACM 19, 7 (1976), 385–394. Google ScholarDigital Library
- KoratWebPage. Korat Home Page. http://korat.sourceforge.net/index.html.Google Scholar
- Shuvendu K Lahiri, Robert Nieuwenhuis, and Albert Oliveras. 2006. SMT techniques for fast predicate abstraction. In International Conference on Computer Aided Verification. Springer, 424–437. Google ScholarDigital Library
- Steven Lauterburg, Ahmed Sobeih, Darko Marinov, and Mahesh Viswanathan. 2008. Incremental state-space exploration for programs with dynamically allocated data. In 30th international conference on Software engineering. Google ScholarDigital Library
- Barbara Liskov and John Guttag. 2000. Program Development in Java: Abstraction, Specification, and Object-Oriented Design. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA. Google ScholarDigital Library
- Darko Marinov and Sarfraz Khurshid. 2001. TestEra: A Novel Framework for Automated Testing of Java Programs. In 16th IEEE International Conference on Automated Software Engineering (ASE ’01). IEEE Computer Society, Washington, DC, USA, 22–31. Google ScholarDigital Library
- Sasa Misailovic, Aleksandar Milicevic, Nemanja Petrovic, Sarfraz Khurshid, and Darko Marinov. 2007. Parallel test generation and execution with Korat. In 6th joint meeting of the European Software Engineering Conference and the ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/FSE). ACM, New York, NY, USA, 135–144. Google ScholarDigital Library
- Rui Qiu. 2016. Scaling and Certifying Symbolic Execution. Ph.D. Dissertation. University of Texas at Austin.Google Scholar
- Junaid Haroon Siddiqui and Sarfraz Khurshid. 2009. PKorat: Parallel Generation of Structurally Complex Test Inputs. In Second International Conference on Software Testing Verification and Validation, ICST. IEEE, Denver, CO, USA, 250–259. Google ScholarDigital Library
- J. H. Siddiqui and S. Khurshid. 2010. ParSym: Parallel symbolic execution. In 2nd International Conference on Software Technology and Engineering. V1–405–V1– 409.Google Scholar
- Junaid Haroon Siddiqui and Sarfraz Khurshid. 2012. Scaling Symbolic Execution Using Ranged Analysis. In Proceedings of the 27th Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA, part of SPLASH. ACM, New York, NY, USA, 523–536. Google ScholarDigital Library
- O. Sokolsky and S. A. Smolka. 1994. Incremental model checking in the modal mu-calculus. In International Conference on Computer Aided Verification. Google ScholarDigital Library
- Matt Staats and Corina Pˇasˇareanu. 2010. Parallel Symbolic Execution for Structural Test Generation. In Proceedings of the Nineteenth International Symposium on Software Testing and Analysis (ISSTA ’10). ACM, New York, NY, USA, 183–194. Google ScholarDigital Library
- Ulrich Stern and David L. Dill. 1997. Parallelizing the Murphi Verifier. In Proceedings of the 9th International Conference on Computer Aided Verification (CAV ’97). Springer-Verlag, London, UK, UK, 256–278. Google ScholarDigital Library
- Engin Uzuncaova. 2008. Efficient Specification-based Testing Using Incremental Techniques. Ph.D. Dissertation. University of Texas at Austin. Google ScholarDigital Library
- Willem Visser, Jaco Geldenhuys, and Matthew B. Dwyer. 2012. Green: Reducing, Rusing and Recycling Constraints in Program Analysis. In ESEC/FSE. Google ScholarDigital Library
- Willem Visser, Klaus Havelund, Guillaume P. Brat, and Seungjoon Park. 2000. Model Checking Programs. In The Fifteenth IEEE International Conference on Automated Software Engineering, ASE,Grenoble, France (ASE ’00). IEEE Computer Society, Washington, DC, USA, 3–12. Google ScholarDigital Library
- G. Yang, M. B. Dwyer, and G. Rothermel. 2009. Regression model checking. In ICSM. 115–124.Google Scholar
Index Terms
- Optimizing parallel Korat using invalid ranges
Recommendations
Parallel test generation and execution with Korat
ESEC-FSE '07: Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineeringWe present novel algorithms for parallel testing of code that takes structurally complex test inputs. The algorithms build on the Korat algorithm for constraint-based generation of structurally complex test inputs. Given an imperative predicate that ...
Korat-API: a framework to enhance korat to better support testing and reliability techniques
SAC '18: Proceedings of the 33rd Annual ACM Symposium on Applied ComputingLogical constraints play an important role in software testing and reliability. For example, constraints written by users allow automating test case generation and systematic bug finding, and constraints computed using data-flow of a program allow ...
Korat: automated testing based on Java predicates
This paper presents Korat, a novel framework for automated testing of Java programs. Given a formal specification for a method, Korat uses the method precondition to automatically generate all (nonisomorphic) test cases up to a given small size. Korat ...
Comments