Guillaume Marceau
Kathi Fisler
Shriram Krishnamurthi
ABSTRACT
Good error messages are critical for novice programmers. Re-cognizing this, the DrRacket programming environment provides a series of pedagogically-inspired language subsets with error messages customized to each subset. We apply human-factors research methods to explore the effectiveness of these messages. Unlike existing work in this area, we study messages at a fine-grained level by analyzing the edits students make in response to various classes of errors. We present a rubric (which is not language specific) to evaluate student responses, apply it to a course-worth of student lab work, and describe what we have learned about using the rubric effectively. We also discuss some concrete observations on the effectiveness of these messages.
- Chabert, J.M. and Higginbotham, T.F. An Investigation of Novice Programmer Errors in IBM 370 (OS) Assembly Lan-guage. In Proceedings of the ACM Southeast Regional Confe-rence, pages 319--323. 1976. Google Scholar
Digital Library
- Cohen, J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37--46, 1960.Google ScholarCross Ref
- Coull, N.J. SNOOPIE: Development Of A Learning Support Tool For Novice Programmers Within A Conceptual Framework. PhD Thesis, School of Computer Science, University Of St. Andrews, 2008.Google Scholar
- Fagin, B.S. and Merkle, L. Quantitative analysis of the effects of robots on introductory Computer Science education. Journal on Educational Resources in Computing, 2(4):1--18, 2002. Google ScholarDigital Library
- Findler, R.B., Clements, J., Flanagan, C., Flatt, M., Krishna-murthi, S., Steckler, P., and Felleisen, M. DrScheme: A programming environment for Scheme. Journal of Functional Programming, 12(02):159--182, 2002. Google ScholarDigital Library
- Gross, P. and Powers, K. Evaluating assessments of novice programming environments. Proceedings of the International Workshop on Computing Education Research, :99--110, 2005. Google ScholarDigital Library
- Hagan, D. and Markham, S. Teaching Java with the BlueJ environment. In Proceedings of Australasian Society for Computers in Learning in Tertiary Education Conference. Citeseer, 2000.Google Scholar
- Hage, J. and Keeken, P.V. Mining Helium programs with Neon. In Technical Report, Department of Information and Computing Sciences, Utrecht University.Google Scholar
- Holt, R.C., Wortman, D.B., Barnard, D.T., and Cordy, J.R. SP/k: a system for teaching computer programming. Communications of the ACM, 20(5):301--309, 1977. Google ScholarDigital Library
- Hristova, M., Misra, A., Rutter, M., and Mercuri, R. Identifying and correcting Java programming errors for introductory computer science students. In Proceedings of the Symposium on Computer Science Education, pages 153--156. 2003. Google ScholarDigital Library
- Jackson, J., Cobb, M., and Carver, C. Identifying top Java errors for novice programmers. In Proceedings of the Frontiers in Education Conference, pages T4C--24. 2005.Google ScholarCross Ref
- Jadud, M.C. A First Look at Novice Compilation Behaviour Using BlueJ. Computer Science Education, 15(1):25--40, 2005.Google ScholarCross Ref
- Jadud, M.C. Methods and tools for exploring novice compilation behaviour. In Proceedings of the International Work-shop on Computing Education Research, pages 73--84. ACM, 2006. Google ScholarDigital Library
- Köksal, M.F., Baar, R.E., and Üsküdarl1, S. Screen-Replay: A Session Recording and Analysis Tool for DrScheme. In Proceedings of the Scheme and Functional Programming Workshop, Technical Report, California Polytechnic State University, CPSLO-CSC-09-03, pages 103--110. 2009.Google Scholar
- Lane, H.C. and VanLehn, K. Intention-based scoring: An approach to measuring success at solving the composition problem. ACM SIGCSE Bulletin, 37(1):373--377, 2005. Google ScholarDigital Library
- Levy, R.B., Ben-Ari, M., and Uronen, P.A. The Jeliot 2000 program animation system. Computers & Education, 40(1):1--15, 2003. Google ScholarDigital Library
- Moskal, B., Lurie, D., and Cooper, S. Evaluating the effectiveness of a new instructional approach. Proceedings of the Symposium on Computer Science Education, 35:75--79, 2004. Google ScholarDigital Library
- Murphy, L., Lewandowski, G., McCauley, R., Simon, B., Thomas, L., and Zander, C. Debugging: the good, the bad, and the quirky - a qualitative analysis of novices' strategies. ACM SIGCSE Bulletin, 40(1):163--167, 2008. Google ScholarDigital Library
- Ng Cheong Vee, M., Mannock, K., and Meyer, B. Empirical study of novice errors and error paths in object-oriented programming. In Proceedings of the Conference of the Higher Education Academy, Subject Centre for Information and Computer Sciences, pages 54--58. 2006.Google Scholar
- Nienaltowski, M., Pedroni, M., and Meyer, B. Compiler Error Messages: What Can Help Novices? Proceedings of the Technical Symposium on Computer Science Education, 39:168--172, 2008. Google ScholarDigital Library
- Pears, A., Seidman, S., Malmi, L., Mannila, L., Adams, E., Bennedsen, J., Devlin, M., and Paterson, J. A survey of literature on the teaching of introductory programming. ACM SIGCSE Bulletin, 39(4):204--223, 2007. Google ScholarDigital Library
- Ragonis, N. and Ben-Ari, M. On understanding the statics and dynamics of object-oriented programs. ACM SIGCSE Bulletin, 37(1):226--230, 2005. Google ScholarDigital Library
- Rey, J.S. From Alice To BlueJ: A Transition To Java. Master's thesis, School of Computing, Robert Gordon University, 2009.Google Scholar
- Spohrer, J.C. and Soloway, E. Novice mistakes: are the folk wisdoms correct? Communications of the ACM, 29(7)1986. Google ScholarDigital Library
- Van Haaster, K. and Hagan, D. Teaching and Learning with BlueJ: an Evaluation of a Pedagogical Tool. In Information Science + Information Technology Education Joint Conference. Citeseer, 2004.Google Scholar
Index Terms
- Measuring the effectiveness of error messages designed for novice programmers
Recommendations
Compiler Error Messages Considered Unhelpful: The Landscape of Text-Based Programming Error Message Research
ITiCSE-WGR '19: Proceedings of the Working Group Reports on Innovation and Technology in Computer Science EducationDiagnostic messages generated by compilers and interpreters such as syntax error messages have been researched for over half of a century. Unfortunately, these messages which include error, warning, and run-time messages, present substantial difficulty ...
Mind your language: on novices' interactions with error messages
Onward! 2011: Proceedings of the 10th SIGPLAN symposium on New ideas, new paradigms, and reflections on programming and softwareError messages are one of the most important tools that a language offers its programmers. For novices, this feed-back is especially critical. Error messages typically contain both a textual description of the problem and an indication of where in the ...
Teaching Students To Use Programming Error Messages
CompEd 2023: Proceedings of the ACM Conference on Global Computing Education Vol 2Research shows many students struggle to use programming error and warning messages effectively. Instead of using these messages as aids to debug and fix their code, some students have negative emotional reactions to seeing 'angry red text'. Not ...
Comments