Integration of ChatGPT Into a Course for Medical Students: Explorative Study on Teaching Scenarios, Students’ Perception, and Applications

Abstract Background Text-generating artificial intelligence (AI) such as ChatGPT offers many opportunities and challenges in medical education. Acquiring practical skills necessary for using AI in a clinical context is crucial, especially for medical education. Objective This explorative study aimed to investigate the feasibility of integrating ChatGPT into teaching units and to evaluate the course and the importance of AI-related competencies for medical students. Since a possible application of ChatGPT in the medical field could be the generation of information for patients, we further investigated how such information is perceived by students in terms of persuasiveness and quality. Methods ChatGPT was integrated into 3 different teaching units of a blended learning course for medical students. Using a mixed methods approach, quantitative and qualitative data were collected. As baseline data, we assessed students’ characteristics, including their openness to digital innovation. The students evaluated the integration of ChatGPT into the course and shared their thoughts regarding the future of text-generating AI in medical education. The course was evaluated based on the Kirkpatrick Model, with satisfaction, learning progress, and applicable knowledge considered as key assessment levels. In ChatGPT-integrating teaching units, students evaluated videos featuring information for patients regarding their persuasiveness on treatment expectations in a self-experience experiment and critically reviewed information for patients written using ChatGPT 3.5 based on different prompts. Results A total of 52 medical students participated in the study. The comprehensive evaluation of the course revealed elevated levels of satisfaction, learning progress, and applicability specifically in relation to the ChatGPT-integrating teaching units. Furthermore, all evaluation levels demonstrated an association with each other. Higher openness to digital innovation was associated with higher satisfaction and, to a lesser extent, with higher applicability. AI-related competencies in other courses of the medical curriculum were perceived as highly important by medical students. Qualitative analysis highlighted potential use cases of ChatGPT in teaching and learning. In ChatGPT-integrating teaching units, students rated information for patients generated using a basic ChatGPT prompt as “moderate” in terms of comprehensibility, patient safety, and the correct application of communication rules taught during the course. The students’ ratings were considerably improved using an extended prompt. The same text, however, showed the smallest increase in treatment expectations when compared with information provided by humans (patient, clinician, and expert) via videos. Conclusions This study offers valuable insights into integrating the development of AI competencies into a blended learning course. Integration of ChatGPT enhanced learning experiences for medical students.


Median IQR
Openness to digital innovation I often test new digital technologies.4.0 3.00-5.00 I am very inquisitive about new digital technologies.4.0 3.00-5.00 I enjoy testing new digital technologies.4.0 3.00-5.00 I am interested in using new digital technologies for myself.5.0 3.25-5.00 I am curious about the further development of digital technologies in the future.5.0 4.00-6.00

Satisfaction
Overall, I am satisfied that ChatGPT has been addressed in this course.5.5 4.25-6.00 Overall, I am satisfied with the topics that ChatGPT has been covered in this course.5.0 4.00-6.00 The use of ChatGPT has made the corresponding teaching units more interesting.5.0 4.25-6.00 The integration of ChatGPT in the corresponding teaching units has been implemented well.

Progress
Using ChatGPT has increased my understanding the the respective content regarding placebo and nocebo conveyed.
4.0 3.00-5.00 By using practical examples it has become clearer to me how to use ChatGPT.5.0 4.00-5.75 By using practical examples it has become clearer to me which possibilities the use of ChatGPT can offer in my current student or later professional everyday life.
5.0 4.00-5.00 On the basis of practical examples it has become clearer to me which limits the use of ChatGPT can have in my current student or later professional everyday life 5.0 4.00-6.00

Application
The application of text-generating AI tools taught using the example of ChatGPT will be useful to me in my everyday student life.
4.0 3.25-5.00 The application of text-generating AI tools taught using the example of ChatGPT will be useful to me in my future professional life.
4.0 3.25-5.00 It became clear to me that a critical evaluation of the generated texts is necessary when using ChatGPT.
5.0 5.00-6.00 The learning units have raised or increased my interest in ChatGPT.5.0 4.00-6.00 Each item was evaluated on a scale ranging from 1 (no agreement) to 6 (full agreement).
Patient experience I suffered from headaches for over 10 years, several times a month, sometimes so severe that my everyday life was very restricted.Sometimes I was even unable to work.I tried prophylactic medication, but they did not help and I did not tolerate it well.I took painkillers for acute pain, but they did not help enough either.Then a friend told me about acupuncture and recommended that I try it.I had a total of 10 sessions, and at the beginning, I didn't really notice any improvement.
However, after about four sessions, the headaches were much better.Since then, I've been largely headache-free.Even if my headaches occasionally resurface, the painkillers effectively manage them, allowing me to maintain my daily routine without disruption.Acupuncture has really been an eye-opener for me.Should the headaches intensify again, I would not hesitate to undergo acupuncture treatment once more.

Information from the Internet
Acupuncture is a treatment method in which thin needles are placed in specific areas of the body to relieve pain.Studies have shown that acupuncture can help with tension headaches by reducing the frequency and intensity of headaches.The treatment can also help improve quality of life by increasing well-being and sleep quality.
Acupuncture is a very safe treatment method when performed by an experienced and qualified acupuncturist.Possible side effects are usually mild and temporary, such as slight bruising or pain at the injection site.
The mode of action of acupuncture is not fully understood, but it is believed to work by releasing the body's own painkillers and improving blood flow to the tissues.If you suffer from tension headaches, acupuncture may be an effective treatment option for you.However, it is important to consult a qualified acupuncturist to ensure that the treatment is safe and effective.

Screenshots from interactive whiteboard discussion
Students were instructed to review the text in nine subgroups regarding three criteria comprehensibility (lay language), patient safety and correct application of the communication method (positive framing).Each subgroup reviewed the same text.Review decision of each group was documented by placing coloured dots in the respective sections for each criterium.Additionally, students were asked to label problematic content in the texts by using coloured markers (Table 1).

After basic prompt
(Dots indicate judgments of groups for the three criteria; labeled text indicate unclear, ambiguous and critical terms as identified by the students.)

Table 1 :
Terms labeled as problematic by students group after basic

prompt German English N (Group decision counts)
After extended prompt (Dots indicate judgments of groups for the three criteria; labeled text indicate unclear, ambiguous and critical terms as identified by the students.)

Table 2 :
Terms labeled as problematic by students group after extended prompt