Abstract
We present a system called Smyth for program sketching in a typed functional language whereby the concrete evaluation of ordinary assertions gives rise to input-output examples, which are then used to guide the search to complete the holes. The key innovation, called live bidirectional evaluation, propagates examples "backward" through partially evaluated sketches. Live bidirectional evaluation enables Smyth to (a) synthesize recursive functions without trace-complete sets of examples and (b) specify and solve interdependent synthesis goals. Eliminating the trace-completeness requirement resolves a significant limitation faced by prior synthesis techniques when given partial specifications in the form of input-output examples.
To assess the practical implications of our techniques, we ran several experiments on benchmarks used to evaluate Myth, a state-of-the-art example-based synthesis tool. First, given expert examples (and no partial implementations), we find that Smyth requires on average 66% of the number of expert examples required by Myth. Second, we find that Smyth is robust to randomly-generated examples, synthesizing many tasks with relatively few more random examples than those provided by an expert. Third, we create a suite of small sketching tasks by systematically employing a simple sketching strategy to the Myth benchmarks; we find that user-provided sketches in Smyth often further reduce the total specification burden (i.e. the combination of partial implementations and examples). Lastly, we find that Leon and Synquid, two state-of-the-art logic-based synthesis tools, fail to complete several tasks on which Smyth succeeds.
Supplemental Material
- Aws Albarghouthi, Sumit Gulwani, and Zachary Kincaid. 2013. Recursive Program Synthesis. In Computer Aided Verification (CAV).Google Scholar
- Rajeev Alur, Rastislav Bodik, Garvit Juniwal, Milo M. K. Martin, Mukund Raghothaman, Sanjit A. Seshia, Rishabh Singh, Armando Solar-Lezama, Emina Torlak, and Abhishek Udupa. 2013. Syntax-Guided Synthesis. In Formal Methods in Computer-Aided Design (FMCAD).Google Scholar
- Alan Ross Anderson, Nuel D. Belnap Jr., and J. Michael Dunn. 1992. Entailment, Vol. II: The Logic of Relevance and Necessity. Princeton University Press.Google Scholar
- James Bornholt and Emina Torlak. 2018. Finding Code That Explodes under Symbolic Evaluation. Proceedings of the ACM on Programming Languages (PACMPL), Issue OOPSLA ( 2018 ).Google ScholarDigital Library
- John Byrnes. 1999. Proof Search and Normal Forms in Natural Deduction. Ph.D. Dissertation. Carnegie Mellon University.Google Scholar
- Sarah E. Chasins, Maria Mueller, and Rastislav Bodik. 2018. Rousillon: Scraping Distributed Hierarchical Web Data. In Symposium on User Interface Software and Technology (UIST).Google ScholarDigital Library
- Adam Chlipala, Leaf Petersen, and Robert Harper. 2005. Strict Bidirectional Type Checking. In Workshop on Types in Languages Design and Implementation (TLDI).Google Scholar
- Yu Feng, Ruben Martins, Jacob Van Gefen, Isil Dillig, and Swarat Chaudhuri. 2017a. Component-Based Synthesis of Table Consolidation and Transformation Tasks from Examples. In Conference on Programming Language Design and Implementation (PLDI).Google ScholarDigital Library
- Yu Feng, Ruben Martins, Yuepeng Wang, Isil Dillig, and Thomas W. Reps. 2017b. Component-Based Synthesis for Complex APIs. In Symposium on Principles of Programming Languages (POPL).Google Scholar
- John Feser. 2016. Inductive Program Synthesis from Input-Output Examples. Master's Thesis, Rice University.Google Scholar
- John Feser. 2020. Personal communication, February 2020.Google Scholar
- John K. Feser, Swarat Chaudhuri, and Isil Dillig. 2015. Synthesizing Data Structure Transformations from Input-Output Examples. In Conference on Programming Language Design and Implementation (PLDI).Google ScholarDigital Library
- Jonathan Frankle, Peter-Michael Osera, David Walker, and Steve Zdancewic. 2016. Example-Directed Synthesis: A TypeTheoretic Interpretation. In Symposium on Principles of Programming Languages (POPL).Google Scholar
- Sumit Gulwani. 2011. Automating String Processing in Spreadsheets Using Input-Output Examples. In Symposium on Principles of Programming Languages (POPL).Google Scholar
- Sumit Gulwani, Mikaël Mayer, Filip Niksic, and Ruzica Piskac. 2015. StriSynth: Synthesis for Live Programming. In International Conference on Software Engineering (ICSE).Google Scholar
- Sumit Gulwani, Oleksandr Polozov, and Rishabh Singh. 2017. Program Synthesis. Foundations and Trends in Programming Languages 4, 1-2 ( 2017 ), 1-119. https://doi.org/10.1561/2500000010 Google ScholarCross Ref
- Zheng Guo, David Justo, Michael James, Jiaxiao Zhou, Ziteng Wang, Ranjit Jhala, and Nadia Polikarpova. 2020. Program Synthesis by Type-Guided Abstraction Refinement. Proceedings of the ACM on Programming Languages (PACMPL), Issue POPL ( 2020 ).Google ScholarDigital Library
- Tihomir Gvero, Viktor Kuncak, Ivan Kuraj, and Ruzica Piskac. 2013. Complete Completion Using Types and Weights. In Conference on Programming Language Design and Implementation (PLDI).Google Scholar
- Brian Hempel, Justin Lubin, and Ravi Chugh. 2019. Output-Directed Programming for SVG. In Symposium on User Interface Software and Technology (UIST).Google Scholar
- Jeevana Priya Inala, Nadia Polikarpova, Xiaokang Qiu, Benjamin S. Lerner, and Armando Solar-Lezama. 2017. Synthesis of Recursive ADT Transformations from Reusable Templates. In Tools and Algorithms for the Construction and Analysis of Systems (TACAS).Google Scholar
- Gilles Kahn. 1987. Natural Semantics. In Symposium on Theoretical Aspects of Computer Sciences (STACS).Google Scholar
- Etienne Kneuss, Manos Koukoutos, and Viktor Kuncak. 2015. Deductive Program Repair. In Computer Aided Verification (CAV).Google Scholar
- Etienne Kneuss, Ivan Kuraj, Viktor Kuncak, and Philippe Suter. 2013. Synthesis Modulo Recursive Functions. In Conference on Object-Oriented Programming Languages, Systems, and Applications (OOPSLA).Google Scholar
- Juraj Kubelka, Romain Robbes, and Alexandre Bergel. 2018. The Road to Live Programming: Insights from the Practice. In International Conference on Software Engineering (ICSE).Google ScholarDigital Library
- Justin Lubin, Nick Collins, Cyrus Omar, and Ravi Chugh. 2020. Program Sketching with Live Bidirectional Evaluation. Extended version of this ICFP 2020 paper available as CoRR abs/ 1911.00583 (https://arxiv.org/abs/ 1911.00583).Google Scholar
- Kazutaka Matsuda and Meng Wang. 2018. HOBiT: Programming Lenses Without Using Lens Combinators. In European Symposium on Programming (ESOP).Google Scholar
- Mikaël Mayer, Viktor Kunčak, and Ravi Chugh. 2018. Bidirectional Evaluation with Direct Manipulation. Proceedings of the ACM on Programming Languages (PACMPL), Issue OOPSLA ( 2018 ).Google ScholarDigital Library
- Anders Miltner, Solomon Maina, Kathleen Fisher, Benjamin C. Pierce, David Walker, and Steve Zdancewic. 2019. Synthesizing Symmetric Lenses. Proceedings of the ACM on Programming Languages (PACMPL), Issue ICFP ( 2019 ).Google ScholarDigital Library
- Anders Miltner, Saswat Padhi, Todd D. Millstein, and David Walker. 2020. Data-Driven Inference of Representation Invariants. In Conference on Programming Language Design and Implementation (PLDI).Google Scholar
- Aleksandar Nanevski, Frank Pfenning, and Brigitte Pientka. 2008. Contextual Modal Type Theory. ACM Transactions on Computational Logic (TOCL) ( 2008 ).Google Scholar
- Cyrus Omar, Ian Voysey, Ravi Chugh, and Matthew A. Hammer. 2019. Live Functional Programming with Typed Holes. Proceedings of the ACM on Programming Languages (PACMPL), Issue POPL ( 2019 ).Google Scholar
- Peter-Michael Osera. 2015. Program Synthesis with Types. Ph.D. Dissertation. University of Pennsylvania.Google Scholar
- Peter-Michael Osera and Steve Zdancewic. 2015. Type-and-Example-Directed Program Synthesis. In Conference on Programming Language Design and Implementation (PLDI).Google Scholar
- Roly Perera, Umut A. Acar, James Cheney, and Paul Blain Levy. 2012. Functional Programs That Explain Their Work. In International Conference on Functional Programming (ICFP).Google Scholar
- Benjamin C. Pierce and David N. Turner. 2000. Local Type Inference. ACM Transactions on Programming Languages and Systems (TOPLAS) ( 2000 ).Google Scholar
- Nadia Polikarpova. 2020. Personal communication, February and May 2020.Google Scholar
- Nadia Polikarpova, Ivan Kuraj, and Armando Solar-Lezama. 2016. Program Synthesis from Polymorphic Refinement Types. In Conference on Programming Language Design and Implementation (PLDI).Google ScholarDigital Library
- Patrick M. Rondon, Ming Kawaguci, and Ranjit Jhala. 2008. Liquid Types. In Conference on Programming Language Design and Implementation (PLDI).Google Scholar
- Jeremy G. Siek and Walid Taha. 2006. Gradual Typing for Functional Languages. In Scheme and Functional Programming Workshop.Google Scholar
- Jeremy G. Siek, Michael M. Vitousek, Matteo Cimini, and John Tang Boyland. 2015. Refined Criteria for Gradual Typing. In Summit on Advances in Programming Languages (SNAPL).Google Scholar
- Calvin Smith and Aws Albarghouthi. 2016. MapReduce Program Synthesis. In Conference on Programming Language Design and Implementation (PLDI).Google Scholar
- Armando Solar-Lezama. 2008. Program Synthesis by Sketching. Ph.D. Dissertation. UC Berkeley.Google ScholarDigital Library
- Armando Solar-Lezama. 2009. The Sketching Approach to Program Synthesis. In Asian Symposium on Programming Languages and Systems (APLAS).Google ScholarDigital Library
- Armando Solar-Lezama, Rodric Rabbah, Rastislav Bodík, and Kemal Ebcioğlu. 2005. Programming by Sketching for Bit-Streaming Programs. In Conference on Programming Language Design and Implementation (PLDI).Google ScholarDigital Library
- Armando Solar-Lezama, Liviu Tancau, Rastislav Bodik, Sanjit Seshia, and Vijay Saraswat. 2006. Combinatorial Sketching for Finite Programs. In International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS).Google Scholar
- Steven L. Tanimoto. 2013. A Perspective on the Evolution of Live Programming. In Workshop on Live Programming (LIVE).Google ScholarCross Ref
- Emina Torlak and Rastislav Bodik. 2013. Growing Solver-Aided Languages with Rosette. In Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software (Onward!).Google Scholar
- Emina Torlak and Rastislav Bodik. 2014. A Lightweight Symbolic Virtual Machine for Solver-Aided Host Languages. In Conference on Programming Language Design and Implementation (PLDI).Google ScholarDigital Library
- Niki Vazou, Patrick M. Rondon, and Ranjit Jhala. 2013. Abstract rRefinement Types. In European Conference on Programming Languages and Systems (ESOP).Google ScholarDigital Library
- Chenglong Wang, Yu Feng, Rastislav Bodik, Alvin Cheung, and Isil Dillig. 2020. Visualization by Example. Proceedings of the ACM on Programming Languages (PACMPL), Issue POPL ( 2020 ).Google ScholarDigital Library
Index Terms
- Program sketching with live bidirectional evaluation
Recommendations
Program Sketching by Automatically Generating Mocks from Tests
Computer Aided VerificationAbstractSketch is a popular program synthesis tool that solves for unknowns in a sketch or partial program. However, while Sketch is powerful, it does not directly support modular synthesis of dependencies, potentially limiting scalability. In this paper, ...
Program sketching
Sketching is a synthesis methodology that aims to bridge the gap between a programmer's high-level insights about a problem and the computer's ability to manage low-level details. In sketching, the programmer uses a partial program, a sketch, to ...
Algorithmic program synthesis: introduction
Program synthesis is a process of producing an executable program from a specification. Algorithmic synthesis produces the program automatically, without an intervention from an expert. While classical compilation falls under the definition of ...
Comments