Arav Agarwal
Majd Sakr
ABSTRACT
We conduct a preliminary study of the effect of GPT's temperature parameter on the diversity of GPT4-generated questions. We find that using higher temperature values leads to significantly higher diversity, with different temperatures exposing different types of similarity between generated sets of questions. We also demonstrate that diverse question generation is especially difficult for questions targeting lower levels of Bloom's Taxonomy.
- Benjamin S Bloom, Max D Englehart, Edward J Furst, Walker H Hill, David R Krathwohl, et al. 1956. Taxonomy of educational objectives, handbook I: the cognitive domain. New York: David McKay Co.Google Scholar
- Paul Denny, Hassan Khosravi, Arto Hellas, Juho Leinonen, and Sami Sarsa. 2023. Can We Trust AI-Generated Educational Content? Comparative Analysis of Human and AI-Generated Learning Resources. arXiv:2306.10509 [cs.HC]Google Scholar
- Jacob Doughty, Zipiao Wan, Anishka Bompelli, Jubahed Qayum, Taozhi Wang, Juran Zhang, Yujia Zheng, Aidan Doyle, Pragnya Sridhar, Arav Agarwal, Christopher Bogart, Eric Keylor, Can Kultur, Jaromir Savelka, and Majd Sakr. 2024. A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education. In Proceedings of the 26th Australasian Computing Education Conference ACE '24). Association for Computing Machinery, New York, NY, USA, 114--123. https://doi.org/10.1145/3636243.3636256Google ScholarDigital Library
- J. Richard Landis and Gary G. Koch. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics 33, 1 (1977), 159--174. http://www.jstor.org/stable/2529310Google ScholarCross Ref
- Stephen MacNeil, Andrew Tran, Arto Hellas, Joanne Kim, Sami Sarsa, Paul Denny, Seth Bernstein, and Juho Leinonen. 2023. Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). Association for Computing Machinery, NewYork, NY, USA, 931--937. https://doi.org/10.1145/3545945.3569785Google ScholarDigital Library
- Pranjal Dilip Naringrekar, Ildar Akhmetov, and Eleni Stroulia. 2023. Generating CS1 Coding Questions Using OpenAI. In Proceedings of the 25th Western Canadian Conference on Computing Education (Vancouver, BC, Canada) (WCCCE '23). Association for Computing Machinery, New York, NY, USA, Article 11, 2 pages. https://doi.org/10.1145/3593342.3593348Google ScholarDigital Library
Index Terms
- Understanding the Role of Temperature in Diverse Question Generation by GPT-4
Recommendations
A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education
ACE '24: Proceedings of the 26th Australasian Computing Education ConferenceThere is a constant need for educators to develop and maintain effective up-to-date assessments. While there is a growing body of research in computing education on utilizing large language models (LLMs) in generation and engagement with coding ...
Automating Question Generation From Educational Text
Artificial Intelligence XLAbstractThe use of question-based activities (QBAs) is wide-spread in education, traditionally forming an integral part of the learning and assessment process. In this paper, we design and evaluate an automated question generation tool for formative and ...
Improving the Readability of Generated Tests Using GPT-4 and ChatGPT Code Interpreter
Search-Based Software EngineeringAbstractA major challenge in automated test generation is the readability of generated tests. Emerging large language models (LLMs) excel at language analysis and transformation tasks. We propose that improving test readability is such a task and explore ...
Comments