Eric Wiebe
Jennifer London
ABSTRACT
The recognition of middle grades as a critical juncture in CS education has led to the widespread development of CS curricula and integration efforts. The goal of many of these interventions is to develop a set of underlying abilities that has been termed computational thinking (CT). This goal presents a key challenge for assessing student learning: we must identify assessment items associated with an emergent understanding of key cognitive abilities underlying CT that avoid specialized knowledge of specific programming languages. In this work we explore the psychometric properties of assessment items appropriate for use with middle grades (US grades 6-8; ages 11-13) students. We also investigate whether these items measure a single ability dimension. Finally, we strive to recommend a "lean" set of items that can be completed in a single 50-minute class period and have high face validity. The paper makes the following contributions: 1) adds to the literature related to the emerging construct of CT, and its relationship to the existing CTt and Bebras instruments, and 2) offers a research-based CT assessment instrument for use by both researchers and educators in the field.
- Goode, J. and Chapman, G. Exploring Computer Science. University of Oregon, Eugene, OR, 2016.Google Scholar
- Lee, I., Martin, F. and Apone, K. Integrating computational thinking across the K-8 curriculum. ACM Inroads, 5, 4 (2014), 64--71. Google ScholarDigital Library
- Mannila, L., Dagiene, V., Demo, B., Grgurina, N., Mirolo, C., Rolandsson, L. and Settle, A. 2014. Computational Thinking in K-9 Education. In Proceedings of the Proceedings of the Working Group Reports of the 2014 on Innovation and Technology in Computer Science Education Conference. ACM, 1--29. Google ScholarDigital Library
- Zur-Bargury, I., Parv, B. and Lanzberg, D. 2013. A nationwide exam as a tool for improving a new curriculum. In Proceedings of the Proceedings of the 18th ACM conference on Innovation and technology in computer science education. ACM, 267--272. Google ScholarDigital Library
- Witherspoon, E. B., Higashi, R. M., Schunn, C. D., Baehr, E. C. and Shoop, R. Developing Computational Thinking through a Virtual Robotics Programming Curriculum. ACM Trans. Comput. Educ., 18, 1 (2017), 1--20. Google ScholarDigital Library
- Martin, L. The promise of the Maker Movement for education. Journal of Pre-College Engineering Education Research (J-PEER), 5, 1 (2015), 4.Google ScholarCross Ref
- Rodger, S. H., Hayes, J., Lezin, G., Qin, H., Nelson, D., Tucker, R., Lopez, M., Cooper, S., Dann, W. and Slater, D. 2009. Engaging middle school teachers and students with alice in a diverse set of subjects. In Proceedings of the Proceedings of the 40th ACM technical symposium on Computer science education (SIGCSE). ACM, 271--275. Google ScholarDigital Library
- Bienkowski, M., Snow, E., Rutstein, D. W. and Grover, S. Assessment design patterns for computational thinking practices in secondary computer science: A first look. SRI International, Menlo Park, CA, 2015.Google Scholar
- Grover, S. and Pea, R. Computational Thinking in K--12: A Review of the State of the Field. Educational Researcher, 42, 1 (2013), 38--43.Google ScholarCross Ref
- {10Wing, J. M. Computational Thinking. Communications of the ACM, 49, 3 (2006), 33--35. Google ScholarDigital Library
- K12CS K--12 Computer Science Framework. 2016.Google Scholar
- NRC, N. R. C. A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas. The National Academies, Washington, DC, 2011.Google Scholar
- Buffum, P. S., Martinez-Arocho, A. G., Frankosky, M. H., Rodriguez, F. J., Wiebe, E. N. and Boyer, K. E. 2014. CS principles goes to middle school: learning how to teach Big Data. Proceedings of the 45th ACM technical symposium on computer science education (SIGCSE '14). ACM, 151--156. Google ScholarDigital Library
- Jona, K., Wilensky, U., Trouille, L., Horn, M., Orton, K., Weintrop, D. and Beheshti, E. 2014. Embedding computational thinking in science, technology, engineering, and math (CT-STEM). Future Directions in Computer Science Education Summit Meeting.Google Scholar
- Weintrop, D., Beheshti, E., Horn, M., Orton, K., Jona, K., Trouille, L. and Wilensky, U. Defining computational thinking for mathematics and science classrooms. Journal of Science Education and Technology, 25, 1 (2016), 127--147.Google ScholarCross Ref
- Grover, S. 2017. Assessing Algorithmic and Computational Thinking in K-12: Lessons from a Middle School Classroom. In Emerging Research, Practice, and Policy on Computational Thinking. Educational Communications and Technology: Issues and Innovations. Springer, 269--288.Google Scholar
- Grover, S., Cooper, S. and Pea, R. 2014. Assessing computational learning in K-12. In Proceedings of the 2014 conference on Innovation & technology in computer science education (ITTICSE '14). ACM, 57--62. Google ScholarDigital Library
- Shute, V. J., Sun, C. and Asbell-Clarke, J. Demystifying computational thinking. Educational Research Review, 22 (2017), 142--158.Google ScholarCross Ref
- Brennan, K. and Resnick, M. 2012. New frameworks for studying and assessing the development of computational thinking. In Proceedings of the 2012 annual meeting of the American Educational Research Association. AERA.Google Scholar
- Denner, J., Werner, L. and Ortiz, E. Computer games created by middle school girls: Can they be used to measure understanding of computer science concepts? Computers & Education, 58, 1 (1// 2012), 240--249. Google ScholarDigital Library
- Tew, A. E. and Guzdial, M. 2011. The FCS1: a language independent assessment of CS1 knowledge. In Proceedings of the Proceedings of the 42nd ACM technical symposium on Computer science education. ACM, 111--116. Google ScholarDigital Library
- Weintrop, D. and Wilensky, U. 2015. Using Commutative Assessments to Compare Conceptual Understanding in Blocks-based and Text-based Programs. In International Computing Education Research Conference (ICER '15). ACM, 101--110. Google ScholarDigital Library
- Taylor, C., Zingaro, D., Porter, L., Webb, K. C., Lee, C. B. and Clancy, M. Computer science concept inventories: past and future. Computer Science Education, 24, 4 (2014), 253--276.Google ScholarCross Ref
- Curzon, P., McOwan, P. W., Plant, N. and Meagher, L. R. 2014. Introducing teachers to computational thinking using unplugged storytelling. In Proceedings of the 9th Workshop in Primary and Secondary Computing Education. ACM, 89--92. Google ScholarDigital Library
- Román-González, M., Moreno-León, J. and Robles, G. 2017. Complementary tools for computational thinking assessment. In Proceedings of international conference on computational thinking education (CTE 2017). The Education University of Hong Kong, 154--159.Google Scholar
- Román-González, M., Pérez-González, J.-C. and Jiménez-Fernández, C. Which cognitive abilities underlie computational thinking? Criterion validity of the Computational Thinking Test. Computers in Human Behavior 72 (2016), 678--691. Google ScholarDigital Library
- Román-González, M., Pérez-González, J.-C., Moreno-León, J. and Robles, G. Extending the nomological network of computational thinking with non-cognitive factors. Computers in Human Behavior, 80 (2018), 441--459. Google ScholarDigital Library
- Alfonso, V. C., Flanagan, D. P. and Radwan, S. The impact of the Cattell-Horn-Carroll theory on test development and interpretation of cognitive and academic abilities. Guilford Publications, City, 2005.Google Scholar
- McGrew, K. S. CHC theory and the human cognitive abilities project: Standing on the shoulders of the giants of psychometric intelligence research. Intelligence, 37, 1 (2009), 1--10.Google ScholarCross Ref
- Ambrósio, A. P., Xavier, C. and Georges, F. 2014. Digital ink for cognitive assessment of computational thinking. In Proceedings of the 2014 IEEE Frontiers in Education Conference (FIE). IEEE, 1--7.Google Scholar
- Basawapatna, A. R., Koh, K. H. and Repenning, A. 2010. Using scalable game design to teach computer science from middle school to graduate school. In Proceedings of the fifteenth annual conference on Innovation and technology in computer science education. ACM, 224--228. Google ScholarDigital Library
- Werner, L., Denner, J. and Campe, S. 2012. The Fairy Performance Assessment: Measuring Computational Thinking in Middle School. In Proceeding of the 44th ACM technical symposium on computer science education (SIGCSE '12). ACM, 421--426. Google ScholarDigital Library
- Dagiene, V. and Futschek, G. 2008. Bebras international contest on informatics and computer literacy: Criteria for good tasks. In International Conference on Informatics in Secondary Schools-Evolution and Perspectives. Springer, 19--30. Google ScholarDigital Library
- Dagiene, V. and Sentance, S. 2016. It's Computational Thinking! Bebras Tasks in the Curriculum. In International Conference on Informatics in Schools: Situation, Evolution, and Perspectives. Springer, 28--39.Google Scholar
- Aksit, O. Enhancing Science Learning through Computational Thinking and Modeling in Middle School Classrooms: A Mixed Methods Study. Dissertation, North Carolina State University, Raleigh, NC, 2018.Google Scholar
- Moreno-León, J. and Robles, G. 2015. Dr. Scratch: A web tool to automatically evaluate Scratch projects. In Proceedings of the workshop in primary and secondary computing education (WiPSCE '15 ). ACM, 132--133. Google ScholarDigital Library
- Blokhuis, D., Millican, P., Roffey, C., Schrijvers, E. and Sentance, S. UK Bebras Computational Thinking Challenge 2016. University of Oxford, Oxford, UK, 2015.Google Scholar
- Barendsen, E., Mannila, L., Demo, B., Nata, Grgurina, A., Izu, C., Mirolo, C., Sentance, S., Settle, A., Gabriel, S. 2015. Concepts in K-9 Computer Science Education. In Proceedings of the 2015 ITiCSE on Working Group Reports. ACM, 85--116. Google ScholarDigital Library
- Dagiene, V., Stupurien, G. and Vinikien, L. 2016. Promoting Inclusive Informatics Education Through the Bebras Challenge to All K-12 Students. In Proceedings of the Proceedings of the 17th International Conference on Computer Systems and Technologies 2016. ACM, 407--414. Google ScholarDigital Library
- Bellettini, C., Lonati, V., Malchiodi, D., Monga, M., Morpurgo, A. and Torelli, M. 2015. How Challenging are Bebras Tasks?: An IRT Analysis Based on the Performance of Italian Students. In Proceedings of the 2015 ACM Conference on Innovation and Technology in Computer Science Education. ACM, 27--32. Google ScholarDigital Library
- Gujberova, M. and Kalas, I. 2013. Designing productive gradations of tasks in primary programming education. In Proceedings of the 8th Workshop in Primary and Secondary Computing Education. ACM, 108--117. Google ScholarDigital Library
- Hubwieser, P. and Muhling, A. 2014. Playing PISA with Bebras. In Proceedings of the 9th Workshop in Primary and Secondary Computing Education. ACM, 128--129. Google ScholarDigital Library
- Izu, C., Mirolo, C., Settle, A., Mannila, L. and Stupuriene, G. Exploring Bebras Tasks Content and Performance: A Multinational Study. Informatics in Education, 16, 1 (2017), 39--59.Google ScholarCross Ref
- Dagiene, V., Mannila, L., Poranen, T., Rolandsson, L. and Derhjelm, S. 2014. Students' performance on programming-related tasks in an informatics contest in Finland, Sweden and Lithuania. In Proceedings of the 2014 conference on Innovation; technology in computer science education. ACM, 153--158. Google ScholarDigital Library
- Fischer, G. H. and Molenaar, I. W. Rasch models: Foundations, recent developments, and applications. Springer Science, 2012.Google Scholar
- Linacre, J. M. Winsteps®. Winsteps.com, 2018.Google Scholar
- de Ayala, R. J. The Theory and Practice of Item Response Theory. Guilford Press, New York, 2009.Google Scholar
- Chalmers, R. P. mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48, 6 (2012), 1--29.Google ScholarCross Ref
Index Terms
- Development of a Lean Computational Thinking Abilities Assessment for Middle Grades Students
Recommendations
Infusing computational thinking into middle grade science classrooms: lessons learned
WiPSCE '18: Proceedings of the 13th Workshop in Primary and Secondary Computing EducationThere is a growing need to present all students with an opportunity to learn computer science and computational thinking (CT) skills during their primary and secondary education. Traditionally, these opportunities are available outside of the core ...
Teaching how to teach computational thinking
ITiCSE 2018: Proceedings of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science EducationComputational Thinking is argued to be an essential skill for the workforce of the 21st century. As a skill, Computational Thinking should be taught in all schools, employing computational ideas integrated into other disciplines. Up until now, questions ...
Computational thinking outreach: reaching across the K-12 curriculum
Recruiting a precollege audience into computing disciplines can be challenging. One approach is to engage those that have a strong influence with the precollege students, K-12 teachers [16]. To engage these teachers, we held a Google-sponsored Computer ...
Comments