Abstract
Large language models (LLMs), such as ChatGPT, have shown remarkable performance on various natural language processing (NLP) tasks, including educational question answering (EQA). However, LLMs generate text entirely based on knowledge obtained during pre-training, which means they struggle with recent information or domain-specific knowledge bases. Moreover, only providing answers to questions posed to LLMs without any grounding materials makes it difficult for students to judge their validity. We therefore propose a method for integrating information retrieval systems with LLMs when developing EQA systems, which in addition to improving EQA performance grounds the answers in the educational context. Our experiments show that the proposed system outperforms vanilla ChatGPT with a significant margin of 110.9%, 67.8%, 43.3%, and 9.2% on BLEU, ROUGE, METEOR and BERTScore. In addition, we argue that the use of the retrieved educational context enhances the transparency and reliability of the EQA process, making it easier to determine the correctness of the answers.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
https://huggingface.co/docs/datasets/index. Last access 15 March 2023.
- 2.
https://openai.com/blog/introducing-chatgpt-and-whisper-apis. Last accessed 15 March 2023.
- 3.
Due to the computational power limitation, we experimented with the 7B version in this project. https://github.com/facebookresearch/LLaMA. Last accessed 10 March 2023.
- 4.
https://huggingface.co/evaluate-metric. Last Access 15 March 2023.
- 5.
https://perspectiveapi.com/how-it-works/. Last Access 15 March 2023.
- 6.
Although there are many comparable instances in the SciQ datasets, we only present two single examples as part of the current project’s scope. However, we plan to conduct an extensive quantitative analysis as future work.
References
Banerjee, S., Lavie, A.: METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In: Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65–72. Association for Computational Linguistics, Ann Arbor, Michigan, June 2005. https://aclanthology.org/W05-0909
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186. Association for Computational Linguistics, Minneapolis, Minnesota, June 2019. https://doi.org/10.18653/v1/N19-1423. https://aclanthology.org/N19-1423
Hoffmann, J., et al.: Training compute-optimal large language models (2022)
Hsu, H.H., Huang, N.F.: Xiao-shih: a self-enriched question answering bot with machine learning on Chinese-based moocs. IEEE Trans. Learn. Technol. 15(2), 223–237 (2022). https://doi.org/10.1109/TLT.2022.3162572
Ji, Z., et al.: Survey of hallucination in natural language generation. ACM Comput. Surv. 55(12) (2023). https://doi.org/10.1145/3571730
Khattab, O., Zaharia, M.: Colbert: efficient and effective passage search via contextualized late interaction over bert. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2020, pp. 39–48. Association for Computing Machinery, New York (2020). https://doi.org/10.1145/3397271.3401075
Komeili, M., Shuster, K., Weston, J.: Internet-augmented dialogue generation. In: Annual Meeting of the Association for Computational Linguistics (2021)
Lin, C.Y.: ROUGE: a package for automatic evaluation of summaries. In: Text Summarization Branches Out, pp. 74–81. Association for Computational Linguistics, Barcelona, July 2004. https://aclanthology.org/W04-1013
Ocaña-Fernández, Y., Valenzuela-Fernández, L.A., Garro-Aburto, L.L.: Artificial intelligence and its implications in higher education. J. Educ. Psychology-Propositos y Representaciones 7(2), 553–568 (2019)
OpenAI: Chatgpt: Optimizing language models for dialogue (2022). https://openai.com/blog/chatgpt/
Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. pp. 311–318. Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, July 2002. https://doi.org/10.3115/1073083.1073135. https://aclanthology.org/P02-1040
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al.: Improving language understanding by generative pre-training (2018)
Rae, J.W., et al.: Scaling language models: Methods, analysis and insights from training gopher (2022)
Touvron, H., et al.: Llama: open and efficient foundation language models (2023)
Welbl, J., Liu, N.F., Gardner, M.: Crowdsourcing multiple choice science questions. In: Proceedings of the 3rd Workshop on Noisy User-generated Text, pp. 94–106. Association for Computational Linguistics, Copenhagen, Denmark, September 2017. https://doi.org/10.18653/v1/W17-4413
Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: Bertscore: evaluating text generation with bert (2020)
Zylich, B., Viola, A., Toggerson, B., Al-Hariri, L., Lan, A.: Exploring automated question answering methods for teaching assistance. In: Bittencourt, I.I., Cukurova, M., Muldner, K., Luckin, R., Millán, E. (eds.) AIED 2020. LNCS (LNAI), vol. 12163, pp. 610–622. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-52237-7_49
Acknowledgement
This research was funded by Swedish Research Council with grant number 2019-05049. The computations/data handling were enabled by resources provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS) at C3SE partially funded by the Swedish Research Council through grant agreement no. 2022-06725.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Wu, Y., Henriksson, A., Duneld, M., Nouri, J. (2023). Towards Improving the Reliability and Transparency of ChatGPT for Educational Question Answering. In: Viberg, O., Jivet, I., Muñoz-Merino, P., Perifanou, M., Papathoma, T. (eds) Responsive and Sustainable Educational Futures. EC-TEL 2023. Lecture Notes in Computer Science, vol 14200. Springer, Cham. https://doi.org/10.1007/978-3-031-42682-7_32
Download citation
DOI: https://doi.org/10.1007/978-3-031-42682-7_32
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-42681-0
Online ISBN: 978-3-031-42682-7
eBook Packages: Computer ScienceComputer Science (R0)