Suad Alaofi
Seán Russell
ABSTRACT
This research examines the validity and efficacy of an English computer terminology multiple-choice test. The test was developed to investigate the relationship between non-native English-speaking students’ knowledge of computer terminology and their academic performance in CS1 courses. In this research, the Rasch model was used to find empirical evidence of test validity. Six aspects of construct validity were evaluated: content, substantive, structural, generalisability, external and consequential aspects. Based on analyses of 150 students’ responses, the test shows an acceptable level of validity in all six aspects of construct validity. Moreover, the efficacy of using this test to predict the students’ academic performance in CS1 courses was investigated. Statistical analysis of the data shows that the computer terminology test can predict approximately 50% of final exam performance in CS1 courses after controlling the impact of English proficiency. The aim of this research (along with other ongoing research) is to aid in the identification of non-native English-speaking students who require more specific language support. Additionally, the research will guide the development of resources and tools that can help these students.
- Suad Alaofi. 2020. The Impact of English Language on Non-Native English Speaking Students’ Performance in Programming Class. In Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education (Trondheim, Norway) (ITiCSE ’20). Association for Computing Machinery, New York, NY, USA, 585–586. https://doi.org/10.1145/3341525.3394008Google Scholar
Digital Library
- Suad Alaofi and Seán Russell. 2021. Computer Terminology Test for Non-native English Speaking CS1 Students. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education. Association for Computing Machinery, New York, NY, USA, 1304–1304.Google ScholarDigital Library
- Vahid Aryadoust, Hannah Ann Hui Tan, and Li Ying Ng. 2019. A Scientometric review of Rasch measurement: The rise and progress of a specialty. Frontiers in psychology 10 (2019), 2197.Google Scholar
- P Baghaei. 2008. The Rasch model as a construct validation tool. Rasch Measurement Transactions 22, 1 (2008), 1145–1146.Google Scholar
- Purya Baghaei and Nazila Amrahi. 2011. Validation of a multiple choice English vocabulary test with the Rasch model. Journal of Language Teaching & Research 2, 5 (2011).Google ScholarCross Ref
- David Beglar. 2010. A Rasch-based validation of the Vocabulary Size Test. Language testing 27, 1 (2010), 101–118.Google Scholar
- Jia Bi. 2020. How large a vocabulary do Chinese computer science undergraduates need to read English-medium specialist textbooks?English for Specific Purposes 58 (2020), 77–89. https://doi.org/10.1016/j.esp.2020.01.001Google ScholarCross Ref
- Maryna Bogachyk and Dmytro Bihunov. 2020. The Structural-Semantic Features of Computer Terms in English. Cognitive Studies| Études cognitives20 (2020).Google Scholar
- Trevor Bond. 2004. Validity and assessment: a Rasch measurement perspective. Metodología de las Ciencias del Comportamiento 5 (2004), 179–194.Google Scholar
- T Bond and C Fox. 2001. Applying the Rasch model. Mahwah, NJ: L.Google Scholar
- William J Boone and Kathryn Scantlebury. 2006. The role of Rasch analysis when conducting science education research utilizing multiple-choice tests. Science Education 90, 2 (2006), 253–269.Google ScholarCross Ref
- Dennis Bouvier, Ellie Lovellette, John Matta, Bedour Alshaigy, Brett A Becker, Michelle Craig, Jana Jackova, Robert McCartney, Kate Sanders, and Mark Zarb. 2016. Novice programmers and the problem description effect. In Proceedings of the 2016 ITiCSE Working Group Reports. 103–118.Google ScholarDigital Library
- Stephanie Buono and Eunice Eunhee Jang. 2021. The Effect of Linguistic Factors on Assessment of English Language Learners’ Mathematical Ability: A Differential Item Functioning Analysis. Educational Assessment 26, 2 (2021), 125–144.Google ScholarCross Ref
- Nell Dale. 2005. Content and emphasis in CS1. ACM SIGCSE Bulletin 37, 4 (2005), 69–73.Google ScholarDigital Library
- Aintzane Doiz, David Lasagabaster, and Juan Manuel Sierra. 2012. English-medium instruction at universities: Global challenges. Multilingual matters.Google Scholar
- EF-Education. [n.d.]. English levels and English proficiency scores: EF SET. https://www.efset.org/english-score/Google Scholar
- Education First. 2020. EF Standard English Test. https://www.efset.org/.Google Scholar
- Huda Gedawy, Saquib Razak, and Hanan Alshikhabobakr. 2019. The Effectiveness of Creating Localized Content for Middle School Computing Curriculum. In Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education. 478–484.Google ScholarDigital Library
- F Gravetter and L Wallnau. 2015. Statistics for the behavioral sciences. Cengage Learning.Google Scholar
- Philip J Guo. 2018. Non-native english speakers learning computer programming: Barriers, desires, and design opportunities. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–14.Google ScholarDigital Library
- Matthew Hertz. 2010. What do” CS1” and” CS2” mean? Investigating differences in the early courses. In Proceedings of the 41st ACM technical symposium on Computer science education. 199–203.Google ScholarDigital Library
- Ed.D James Sick. 2009. Rasch Measurement in Language Education Part 4: Rasch Analysis Software Programs., (p. 13 ‐ 16) pages.Google Scholar
- Kevin J Keen and Letha Etzkorn. 2009. Predicting students’ grades in computer science courses based on complexity measures of teacher’s lecture notes. Journal of Computing Sciences in Colleges 24, 5 (2009), 44–48.Google ScholarDigital Library
- Truman Lee Kelley. 1927. Interpretation of educational measurements. (1927).Google Scholar
- Abdullah Aied Khuwaileh. 2010. IT terminology and translation: Cultural, lexicographic and linguistic problems. LSP Journal-Language for special purposes, professional communication, knowledge management and cognition 1, 2 (2010).Google Scholar
- Jane Knight. 2013. The changing landscape of higher education internationalisation – for better or worse?Perspectives: Policy and Practice in Higher Education 17, 3 (2013), 84–90. https://doi.org/10.1080/13603108.2012.753957Google ScholarCross Ref
- Carl A Lager. 2006. Types of mathematics-language reading interactions that unnecessarily hinder algebra learning and assessment. Reading Psychology 27, 2-3 (2006), 165–204.Google ScholarCross Ref
- John M. Linacre. 2021. A user’s guide to Winsteps/Ministeps Rasch-Model programs, Program Manual 5.0.0.Google Scholar
- John M. Linacre. 2021. Winsteps. http://www.winsteps.comGoogle Scholar
- Ernesto Macaro, Samantha Curle, Jack Pun, Jiangshan An, and Julie Dearden. 2018. A systematic review of English medium instruction in higher education. Language Teaching 51, 1 (2018), 36–76.Google ScholarCross Ref
- Zunita Mohamad Maskor, Harun Baharudin, and Maimun Aqsha Lubis. 2018. Measurement validity and reliability of the productive vocabulary knowledge instrument for Arabic learners in Malaysian secondary schools. Advanced Science Letters 24, 5 (2018), 3423–3426.Google ScholarCross Ref
- Raina Mason and Carolyn Seton. 2020. Assessing International Students: The Role of Cognitive Load. In Proceedings of the Twenty-Second Australasian Computing Education Conference. 160–166.Google ScholarDigital Library
- Raina Mason and Carolyn Seton. 2021. Leveling the playing field for international students in IT courses. In Australasian Computing Education Conference. 138–146.Google ScholarDigital Library
- Tim McNamara and Ute Knoch. 2012. The Rasch wars: The emergence of Rasch measurement in language testing. Language Testing 29, 4 (2012), 555–576.Google ScholarCross Ref
- Samuel Messick. 1989. Validity. (3ed ed.). Macmillan, New York, 13–103.Google Scholar
- Samuel Messick. 1996. Validity and washback in language testing. Language testing 13, 3 (1996), 241–256.Google Scholar
- Daniel E Minshall. 2013. A Computer Science Word List. Master’s thesis. Swansea University, Swansea.Google Scholar
- Yali Mu and Hao Bai. 2020. A Study on Influencing Factors of the Application Level of Computer in Chinese for International Students. In 2020 The 4th International Conference on Education and Multimedia Technology. 244–249.Google ScholarDigital Library
- Robert R Pagano. 2012. Understanding statistics in the behavioral sciences. Cengage Learning.Google Scholar
- Georg Rasch. 1960. Studies in mathematical psychology: I. Probabilistic models for some intelligence and attainment tests. (1960).Google Scholar
- Kenneth D Royal. 2009. Making Meaningful Measurement in Survey Research: The Use of Person and Item Maps.Online Submission (2009).Google Scholar
- Ivan Ruby and Bojana Krsmanovic. 2017. Does learning a programming language require learning English? A comparative analysis between English and programming languages. In EdMedia+ Innovate Learning. Association for the Advancement of Computing in Education (AACE), 420–427.Google Scholar
- Judith Runnels. 2011. Evaluation of an achievement English vocabulary test using Rasch analysis. (2011).Google Scholar
- Judith Runnels. 2012. Using the Rasch model to validate a multiple choice English achievement test. International Journal of Language Studies 6, 4 (2012), 141–155.Google Scholar
- Chris R. Sawyer, Kurk Gayle, Andrew Topa, and William G. Powers. 2014. Listening Fidelity Among Native and Nonnative English-Speaking Undergraduates as a Function of Listening Apprehension and Gender. Communication Research Reports 31, 1 (2014), 62–71. https://doi.org/10.1080/08824096.2013.844119Google ScholarCross Ref
- Judy Sheard, Angela Carbone, Selby Markham, Angas John Hurst, Des Casey, and Chris Avram. 2008. Performance and progression of first year ICT students. In Proceedings of the tenth conference on Australasian computing education-Volume 78. 119–127.Google ScholarDigital Library
- Everett V Smith Jr. 2001. Evidence for the reliability of measures and validity of measure interpretation: a Rasch measurement perspective. Journal of applied measurement(2001).Google Scholar
- Purwo Susongko. 2016. Validation of science achievement test with the rasch model. Jurnal Pendidikan IPA Indonesia 5, 2 (2016), 268–277.Google Scholar
- Edward W Wolfe and Everett V Smith Jr. 2007. Instrument development tools and activities for measure validation using Rasch models: part II–validation activities. Journal of applied measurement 8, 2 (2007), 204–234.Google Scholar
- Benjamin D Wright and Mark Stone. 1999. Measurement essentials.Google Scholar
Index Terms
- A Validated Computer Terminology Test for Predicting Non-native English-speaking CS1 Students’ Academic Performance
Index terms have been assigned to the content through auto-classification.
Recommendations
The Influence of Foreign Language Classroom Anxiety on Academic Performance in English-based CS1 Courses
UKICER '22: Proceedings of the 2022 Conference on United Kingdom & Ireland Computing Education ResearchThe use of the English language as a medium of instruction is becoming the norm in many higher education institutions around the world. This means all subjects, especially in science disciplines, are taught in English regardless of the native language ...
Computer Terminology Test for Non-native English Speaking CS1 Students
SIGCSE '21: Proceedings of the 52nd ACM Technical Symposium on Computer Science EducationThis study describes the development of a word list for CS1 and the creation of a Computer Terminology Test. The CS1 word list contains 123 most frequent words compiled from CS1 textbooks and lecture notes. This list is beneficial for students, teachers,...
The Impact of English Language on Non-Native English Speaking Students' Performance in Programming Class
ITiCSE '20: Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science EducationThe study focuses on investigating the impact of English on Non-native English speaking students' performance in computer programming class. Four related factors will be examined including English proficiency, knowledge of English computer terminology, ...
Comments