Hostname: page-component-848d4c4894-5nwft Total loading time: 0 Render date: 2024-05-03T07:19:41.324Z Has data issue: false hasContentIssue false
  • English
  • Français

Dynamic witnesses for static type errors (or, Ill-Typed Programs Usually Go Wrong)

Published online by Cambridge University Press:  21 May 2018

ERIC L. SEIDEL Open the ORCID record for ERIC L. SEIDEL [Opens in a new window] ,
RANJIT JHALA  and
WESTLEY WEIMER
ERIC L. SEIDEL
Affiliation:
Department of Computer Science and Engineering, University of California, San Diego, CA, USA
RANJIT JHALA
Affiliation:
Department of Computer Science and Engineering, University of California, San Diego, CA, USA
WESTLEY WEIMER
Affiliation:
Department of Computer Science, University of Virginia, Charlottesville, VA, USA (e-mail: eseidel@cs.ucsd.edu, jhala@cs.ucsd.edu, weimer@virginia.edu)
Rights & Permissions [Opens in a new window]

Abstract

Core share and HTML view are not available for this content. However, as you have access to this content, a full PDF is available via the ‘Save PDF’ action button.

Static type errors are a common stumbling block for newcomers to typed functional languages. We present a dynamic approach to explaining type errors by generating counterexample witness inputs that illustrate how an ill-typed program goes wrong. First, given an ill-typed function, we symbolically execute the body to synthesize witness values that make the program go wrong. We prove that our procedure synthesizes general witnesses in that if a witness is found, then for all inhabited input types, there exist values that can make the function go wrong. Second, we show how to extend this procedure to produce a reduction graph that can be used to interactively visualize and debug witness executions. Third, we evaluate the coverage of our approach on two data sets comprising over 4,500 ill-typed student programs. Our technique is able to generate witnesses for around 85% of the programs, our reduction graph yields small counterexamples for over 80% of the witnesses, and a simple heuristic allows us to use witnesses to locate the source of type errors with around 70% accuracy. Finally, we evaluate whether our witnesses help students understand and fix type errors, and find that students presented with our witnesses show a greater understanding of type errors than those presented with a standard error message.

Type
Research Article
Information
Journal of Functional Programming , Volume 28 , 2018 , e13
Copyright
Copyright © Cambridge University Press 2018 

References

Bayne, M., Cook, R. & Ernst, M. D. (2011) Always-available static and dynamic feedback. In Proceedings of the 33rd International Conference on Software Engineering. ICSE '11. New York, NY, USA: ACM, pp. 521–530.Google Scholar
Cadar, C., Dunbar, D. & Engler, D. (2008) KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs. In Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation. OSDI'08. Berkeley, CA, USA: USENIX Association, pp. 209–224.Google Scholar
Chamarthi, H. R., Dillinger, P. C., Kaufmann, M. & Manolios, P. (2011) Integrating testing and interactive theorem proving. In Proceedings of the 10th International Workshop on the ACL2 Theorem Prover and its Applications. ACL2, pp. 4–19.CrossRefGoogle Scholar
Charguéraud, A. (2014) Improving type error messages in ocaml. In Proceedings of the ML Family/OCaml Users and Developers Workshops. Electronic Proceedings in Theoretical Computer Science, vol. 198. Open Publishing Association, pp. 80–97.Google Scholar
Chen, S. & Erwig, M. (2014) Counter-factual typing for debugging type errors. In Proceedings of the 41st ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages. POPL. New York, NY, USA: ACM, pp. 583–594.CrossRefGoogle Scholar
Christiansen, D. R. (2014) Reflect on your mistakes! lightweight domain-specific error messages. In Proceedings of the 15th Symposium on Trends in Functional Programming.Google Scholar
Claessen, K. & Hughes, J. (2000) QuickCheck: A lightweight tool for random testing of haskell programs. In Proceedings of the 5th ACM SIGPLAN International Conference on Functional Programming. New York, NY, USA: ACM, pp. 268–279.Google Scholar
Clements, J., Flatt, M. & Felleisen, M. (2001) Modeling an algebraic stepper. In Programming Languages and Systems. Lecture Notes in Computer Science. Berlin, Heidelberg: Springer, pp. 320–334.Google Scholar
Csallner, C. & Smaragdakis, Y. (2004) JCrasher: An automatic robustness tester for java. Softw. Pract. Exp. 34 (11), 1025–1050.CrossRefGoogle Scholar
Damas, L & Milner, R. (1982) Principal type-schemes for functional programs. In Proceedings of the 9th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages. New York, NY, USA: ACM, pp. 207–212.CrossRefGoogle Scholar
Felleisen, M., Findler, R. B. & Flatt, M. (2009) Semantics Engineering with PLT Redex. 1st ed. The MIT Press.Google Scholar
Flanagan, C., Flatt, M., Krishnamurthi, S., Weirich, S. & Felleisen, M. (1996) Catching bugs in the web of program invariants. In Proceedings of the ACM SIGPLAN 1996 Conference on Programming Language Design and Implementation. New York, NY, USA: ACM, pp. 23–32.Google Scholar
Fleiss, J. L. (1971) Measuring nominal scale agreement among many raters. Psychol. Bull. 76 (5), 378.CrossRefGoogle Scholar
Gast, H. (2004) Explaining ML type errors by data flows. In Implementation and Application of Functional Languages. Lecture Notes in Computer Science. Berlin Heidelberg: Springer, pp. 72–89.Google Scholar
Godefroid, P., Klarlund, N. & Sen, K. (2005) DART: Directed automated random testing. In Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation. New York, NY, USA: ACM, pp. 213–223.Google Scholar
Guo, P. J. (2013) Online Python Tutor: Embeddable web-based program visualization for CS education. In Proceedings of the 44th ACM Technical Symposium on Computer Science Education. SIGCSE '13. New York, NY, USA: ACM, pp. 579–584.CrossRefGoogle Scholar
Haack, C. & Wells, J. B. (2003) Type error slicing in implicitly typed Higher-Order languages. In Programming Languages and Systems. Lecture Notes in Computer Science. Berlin Heidelberg: Springer, pp. 284–301.Google Scholar
Hage, J. & Heeren, B. (2006) Heuristics for type error discovery and recovery. In Implementation and Application of Functional Languages. Lecture Notes in Computer Science. Berlin Heidelberg: Springer, pp. 199–216.Google Scholar
Hage, J. & Heeren, B. (2009) Strategies for solving constraints in type and effect systems. Electron. Notes Theor. Comput. Sci. 236 (2), 163–183.Google Scholar
Heeren, B., Hage, J. & Swierstra, S. D. (2003) Scripting the type inference process. In Proceedings of the 8th ACM SIGPLAN International Conference on Functional Programming, vol. 38. ACM, pp. 3–13.Google Scholar
Krippendorff, K. (2012) Content Analysis: An Introduction to Its Methodology. SAGE Publications.Google Scholar
Landis, J. R & Koch, G. G. (1977) The measurement of observer agreement for categorical data. Biometrics 33 (1), 159–174.CrossRefGoogle ScholarPubMed
Lee, O. & Yi, K. (1998) Proofs about a folklore let-polymorphic type inference algorithm. ACM Trans. Program. Lang. Syst. 20 (4), 707–723.CrossRefGoogle Scholar
Lempsink, E. (2009) Generic Type-Safe Diff and Patch for Families of Datatypes. M.Phil. thesis, Universiteit Utrecht.Google Scholar
Lerner, B., Grossman, D. & Chambers, C. (2006) Seminal: Searching for ML type-error messages. In Proceedings of the 2006 Workshop on ML. New York, NY, USA: ACM, pp. 63–73.Google Scholar
Lerner, B. S., Flower, M., Grossman, D. & Chambers, C. (2007) Searching for type-error messages. In Proceedings of the 28th ACM SIGPLAN Conference on Programming Language Design and Implementation. New York, NY, USA: ACM, pp. 425–434.CrossRefGoogle Scholar
Lindblad, F. (2007) Property directed generation of first-order test data. In Proceedings of the Eighth Symposium on Trends in Functional Programming. Morazán, M. T. (ed), vol. 8, pp. 105–123.Google Scholar
Loncaric, C., Chandra, S., Schlesinger, C. & Sridharan, M. (2016) A practical framework for type inference error explanation. In Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications. ACM, pp. 781–799.Google Scholar
Mann, H. B. & Whitney, D. R. (1947) On a test of whether one of two random variables is stochastically larger than the other. Ann. Math. Stat. 18 (1), 50–60.Google Scholar
Marceau, G., Fisler, K. & Krishnamurthi, S. (2011a) Measuring the effectiveness of error messages designed for novice programmers. In Proceedings of the 42Nd ACM Technical Symposium on Computer Science Education. New York, NY, USA: ACM, pp. 499–504.CrossRefGoogle Scholar
Marceau, G., Fisler, K. & Krishnamurthi, S. (2011b) Mind your language: On novices' interactions with error messages. In Proceedings of the 10th SIGPLAN Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software. Onward! 2011. New York, NY, USA: ACM, pp. 3–18.Google Scholar
McAdam, B. J. (1998) On the unification of substitutions in type inference. In Implementation of Functional Languages, Hammond, K., Davie, T. & Clack, C. (eds), Lecture Notes in Computer Science. Berlin Heidelberg: Springer, pp. 137–152.Google Scholar
Naylor, M. & Runciman, C. (2007) Finding inputs that reach a target expression. In Proceedings of the 7th IEEE International Working Conference on Source Code Analysis and Manipulation. pp. 133–142.Google Scholar
Nelson, G. & Oppen, D. C. (1979) Simplification by cooperating decision procedures. ACM Trans. Program. Lang. Syst. 1 (2), 245–257.Google Scholar
Neubauer, M. & Thiemann, P. (2003) Discriminative sum types locate the source of type errors. In Proceedings of the 8th ACM SIGPLAN International Conference on Functional Programming. New York, NY, USA: ACM, pp. 15–26.Google Scholar
Nguyen, P. C, & Van Horn, D. (2015) Relatively complete counterexamples for higher-order programs. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation. New York, NY, USA: ACM, pp. 446–456.Google Scholar
Pacheco, C., Lahiri, S. K, Ernst, M. D. & Ball, T. (2007) Feedback-Directed random test generation. In Proceedings of the 29th International Conference on Software Engineering. ICSE '07, pp. 75–84.Google Scholar
Pavlinovic, Z., King, T. & Wies, T. (2014) Finding minimum type error sources. In Proceedings of the 2014 ACM International Conference on Object Oriented Programming Systems Languages & Applications. New York, NY, USA: ACM, pp. 525–542.Google Scholar
Pavlinovic, Z., King, T. & Wies, T. (2015) Practical SMT-based type error localization. In Proceedings of the 20th ACM SIGPLAN International Conference on Functional Programming. New York, NY, USA: ACM, pp. 412–423.Google Scholar
Perera, R., Acar, U. A., Cheney, J. & Levy, P. B. (2012) Functional programs that explain their work. In Proceedings of the 17th ACM SIGPLAN International Conference on Functional Programming. New York, NY, USA: ACM, pp. 365–376.Google Scholar
Rahli, V., Wells, J., Pirie, J. & Kamareddine, F. (2015) Skalpel: A type error slicer for standard ML. Electron. Notes Theor. Comput. Sci. 312 (24), 197–213.Google Scholar
Runciman, C., Naylor, M. & Lindblad, F. (2008) Smallcheck and lazy smallcheck: Automatic exhaustive testing for small values. In Proceedings of the 1st ACM SIGPLAN Symposium on Haskell. New York, NY, USA: ACM, pp. 37–48.Google Scholar
Sagonas, K., Silva, J. & Tamarit, S. (2013) Precise explanation of success typing errors. In Proceedings of the ACM SIGPLAN 2013 Workshop on Partial Evaluation and Program Manipulation. New York, NY, USA: ACM, pp. 33–42.Google Scholar
Schilling, T. (2011) Constraint-free type error slicing. In Trends in Functional Programming. Lecture Notes in Computer Science. Berlin Heidelberg: Springer, pp. 1–16.Google Scholar
Seidel, E. L., Jhala, R. & Weimer, W. (2016b June) Dynamic Witnesses for Static Type Errors.Google Scholar
Seidel, E. L., Jhala, R. & Weimer, W. (2016a) Dynamic witnesses for static type errors (or, ill-typed programs usually go wrong) In Proceedings of the 21st ACM SIGPLAN International Conference on Functional Programming. ACM, pp. 228–242.Google Scholar
Seidel, E. L., Vazou, N. & Jhala, R. (2015) Type targeted testing. In Proceedings of the 24th European Symposium on Programming on Programming Languages and Systems. ESOP '15. New York, NY, USA: Springer-Verlag New York, Inc, pp. 812–836.Google Scholar
Serrano, A. & Hage, J. (2016) Type error diagnosis for embedded DSLs by Two-Stage specialized type rules. In Programming Languages and Systems. Lecture Notes in Computer Science. Berlin Heidelberg: Springer, pp. 672–698.CrossRefGoogle Scholar
Seven, D. (2014 17 Apr.) Knightmare: A DevOps Cautionary Tale. https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/. Accessed: 2017-4-24.Google Scholar
Tillmann, N. & de Halleux, J. (2008) Pex–White box test generation for .NET. In Tests and Proofs, Beckert, B. & Hähnle, R. (eds), Lecture Notes in Computer Science. Berlin Heidelberg: Springer, pp. 134–153.Google Scholar
Vytiniotis, D., Peyton Jones, S. & Magalhães, J. P. (2012) Equality proofs and deferred type errors: A compiler pearl. In Proceedings of the 17th ACM SIGPLAN International Conference on Functional Programming. ICFP '12. New York, NY, USA: ACM, pp. 341–352.Google Scholar
Wheeler, D. A. (2014 23 Nov.) The apple goto fail vulnerability: lessons learned. https://www.dwheeler.com/essays/apple-goto-fail.html. Accessed: 2017-4-24.Google Scholar
Zhang, D. & Myers, A. C. (2014) Toward general diagnosis of static errors. In Proceedings of the 41st ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages. POPL '14. New York, NY, USA: ACM, pp. 569–581.Google Scholar
Zhang, D., Myers, A. C., Vytiniotis, D. & Peyton-Jones, S. (2015) Diagnosing type errors with class. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation. PLDI 2015. New York, NY, USA: ACM, pp. 12–21.Google Scholar
Submit a response

Discussions

No Discussions have been published for this article.