Building Up While Shutting Down: An Academic Health System Educational Response to the COVID-19 Pandemic

The disruptive influence of the novel Coronavirus Disease 2019 (COVID-19) pandemic was felt worldwide, shutting down global activities and with it standard medical educational processes of face-to-face classroom sessions and clinical clerkship education. In the midst of suspending our face-to-face educational events to support safety for students, faculty, patients, and the community, we recognized the urgent need to build and implement online medical education coursework focused on the COVID-19 pandemic. In this innovative educational experience, we created a comprehensive approach to learning about the COVID-19 pandemic, including not only basic science and clinical knowledge, but also incorporating information in health system science, public health, and health equity. This study examined that perspective of third and fourth year medical students enrolled in a rapidly created two-week online COVID-19 Pandemic course. A course evaluation was completed by 266 of 347 students enrolled in the course. The evaluation included 10 Likert-scale items and 3 open response items. Results indicated an appreciation of basic science and clinical content of the course and serve to inform quality improvement for future iterations of the course or other institutions seeking to develop or improve COVID-19 coursework.


Introduction
The disruptive influence of the novel Coronavirus Disease 2019  pandemic was felt worldwide, shutting down activities across the globe and effectively eliminating the traditional educational operations. The impact on medical education was striking, as face-to-face classroom and clinical clerkship activities were rapidly suspended for the safety of students, faculty, patients, and the community. For the third and fourth year medical students who will In response to the COVID-19 pandemic, a growing number of medical schools have made changes in medical education. The American Association of Medical Colleges (AAMC) Response Dashboard indicated that at least 61% of medical schools reported developing a COVID-19 course in order to address student needs (AAMC, 2020). Though one study assessed students' perceptions of on an online learning platform created to allow virtual learning during the COVID-19 pandemic (Singh et al., 2020), to our knowledge, this is the first reported description and corresponding evaluation of the educational content of a COVID-19 pandemic course in a medical school setting. While many medical schools have implemented COVID-19 coursework, there is a need to research the perceived quality of such course materials from the students' perspectives.
The primary objective of this study was to assess whether the COVID-19 course materials were perceived as useful to students. A secondary objective was to use the evaluation data for course improvement. The course material would be used again within a month by second year (M2) medical students who would shortly enter the clinical clerkship phase of their medical education to treat patients potentially exposed to COVID-19.

Methods
The COVID-19 Pandemic online course was developed over a of two week period for initial use by third year (M3) and fourth year (M4) medical students immediately after these students were removed from their clinical clerkship rotations. Traditional clerkships were suspended the second week of March and this course began the fourth week of March. Course evaluations were sent electronically to all enrolled students (n=347) at the completion of the course. The course evaluation process was completely anonymous and voluntary. Course evaluations included Likert-scale rating items and open response questions.
It was a priority for faculty that student received well-rounded education that addressed not only the medical implications of COVID-19, but also elucidated the broad impact of this public health crisis beyond clinical care. We designed the course to address four major areas of learning: basic science and pathophysiology, clinical care, health system science, and health equity. "Fireside Chats" were moderated, live sessions using the video conferencing platform Zoom. The course began with a live session to acknowledge the emotional impact of the rapidly evolving COVID-19 pandemic and student stress management. This was moderated by course co-directors and representatives from faculty wellness. The course was concluded with a "Lessons for the Future" wrap up session (Table 1). To allow students maximum flexibility to complete course assignments, given unexpected disruptions to study time, space, and methods, the course was designed with mixed asynchronous and synchronous-recorded learning events. Asynchronous events allowed students to complete the learning event when convenient, considering other educational, personal, and family responsibilities. The faculty also recognized that many students may feel isolated due to rapidly enacted social distancing requirements and thus scheduled synchronous-recorded leaning events via the video conferencing software Zoom. This allowed facultystudent interaction through both the live video and the typed chat feature during the learning event. These synchronous, interactive events were also recorded and posted to the online learning management software for all students to view at another time, if more convenient.
Exams: Two scheduled online exams included approximately 3 questions per learning event. Students were allowed flexibility to complete each exam in their own testing location. Each exam was available for student to initiate within a 2.5 day period following the week of learning events. The time available for each test was calculated by 90 seconds per question which mirrors the average length of time allotted per question for the United States Medical Licensure Exam (USMLE) Step 1. After submitting their answers, students were allotted an additional 5 minutes to review incorrect answers, and read answer rationales. No test security monitoring software was used, though students were reminded of the college honor code prohibiting consulting with others or other resources during the exam. It was also anticipated that a short amount of time allotted per question would reduce the likelihood of cheating.

Course Evaluation Instrument:
The 13-item course evaluation was sent to students electronically and included 10 questions that asked students to rate each item using a 5-point Likert-scale of Strongly Disagree, Disagree, Neither Agree nor Disagree, Agree, and Strongly Agree, and 3 open-ended questions that prompted students to write about course strengths, opportunities for improvement, and additional comments. The course evaluation was anonymous and voluntary.

Results/Analysis
A total of 266 students completed the course evaluation of 347 students enrolled in the course, giving a response rate of 76.7%.
Quantitative: Data indicate that the majority of students ranked all course evaluation prompts with responses of "Strongly Agree" and "Agree" combined ( Table 2). The items that students ranked the highest in combined ratings of "Strongly Agree" and "Agree" were "How well do think this course prepared you to understand the scientific background of the COVID-19 pandemic" (94.23%), "Testing: Material on exams was related to material covered either in class or in course assignments; students were treated equitably" (90.97%), and "How well do you think the course prepared you with knowledge of appropriate protocols for use of Personal Protective Equipment" (89.96%). The items that students ranked the lowest in combined rating of "Strongly Agree" and "Agree" were "How well do you think this course prepared you to manage patients with COVID-19?" (80.76%), "How well do you think this course prepared you to understand the social determinants of health that contribute to public health crises?" (84.11%), and "Clarity: Goals and educational objectives of course were clear, faculty expectations of students were  Describe some of the strengths of the course in your opinion. 1.
Describe possible areas of improvement for the course in your opinion.

2.
Enter any additional comments in the space provided. 3.
Themes that emerged from the wide range of comments included (1) an appreciation of timely course development to allow relevant continuity of medical education during the pandemic, (2) an appreciation of COVID-19 specific basic science educational content to provide vital knowledge for medical students during the pandemic, and (3) constructive comments that called for more course learning events that covered clinical applications. Student comments included, "Very informative material and helped me feel more knowledgeable about the virus and confident in being able to intelligently discuss the pandemic and share information," and, "I would have loved a few more lectures on the specifics of actually managing a COVID patient, i.e. ventilator settings, working through a few vignettes, etc. Overall, thank you for organizing such a wonderful course so quickly."

Discussion
It was challenging to create a course on evolving content as knowledge, expectations, protocols, and daily routines were also changing for all. Nonetheless, we realized that the charge to create timely, relevant medical educational content was imperative. To the best of our knowledge, this is the first published survey of medical students' evaluation of a rapidly created COVID-19 Pandemic course that sought to balance salient basic science, clinical, ethical, wellness, public health, and health equity topics surrounding the COVID-19 Pandemic.
As expected, quantitative rating and qualitative comment themes aligned in regard to strengths and opportunities for the course in future iterations. Students recognized the monumental effort involved to create and curate educational content in such a short amount of time. Additionally, students appreciated the inclusion of high yield basic science information, and lectures related to the scientific underpinnings of the COVID-19 pandemic received the highest evaluations in ratings and in comments.
Also expected, student feedback indicated that the greatest potential area for course improvement was related to clinical application of knowledge, specifically case presentation, case management, and personal safety. It is Clemmons K, Boehme K, Cannon M, Graham J, Lipschitz R MedEdPublish https://doi.org/10.15694/mep.2020.000165.1 Page | 10 important to note that students enrolled in the course were in their third and fourth years of medical school and had been abruptly removed from their clinical clerkship rotations for their and others' safety. These students felt a great amount of concern and fear as they were quickly going to be reintroduced to the clinical environment and any COVID-19 related knowledge would be immediately relevant.
Addressing the clinical concerns posed two distinct challenges. First, course content required clinician input and was difficult to develop quickly given the two-week development time frame and many clinician educators were largely unavailable due to patient treatment responsibilities. Second, the clinical implications for treatment and management of COVID-19 patients were rapidly changing from day-to-day during the early phase of this pandemic and keeping abreast of clinically appropriate information while simultaneously building a curriculum proved challenging. The planning processes for future iterations of the course for future groups of students will incorporate these findings and will have the benefit of clinical treatment experiences at the initial stages of this noel COVID-19 pandemic.
One unexpected finding was not related to the results of the student responses in the course evaluation. Faculty review of student results on Exam 1 and Exam 2 seemed to indicate individual effort, i.e., that the students were not collaborating with others or using outside materials during the testing. Exam 1 percentage correct scores ranged from a low of 79% to high of 100%. Exam 2 percentage correct scores ranged from a low of 84% to a high of 100%. This may have resulted from the brief average time allotted for the exam questions, the result of social distancing, or intrinsic motivation to test one's knowledge related to such an important societal medical concern.
A limitation of this study is that this course evaluation was a one-time survey of students in a single institution faced with unprecedented and rapidly evolving circumstances of the COVID-19 pandemic. It is possible that perceived needs may change over time. Nevertheless, the findings of this study provide valuable insight to schools who want to initiate or improve their COVID-19 pandemic education. This evaluation will be used to make future improvements to our COVID-19 education for medical students and other students across health care programs at our university, as well as continuing education for physicians and other health care workers across our state.

Conclusion
Building up educational content for medical students while in the midst of shutting down traditional educational processes at our academic health system proved a formidable challenge; but one of vital importance for those medical students who will shortly become practicing physicians in a world forever changed by the COVID-19 pandemic.
Several findings of the study would be helpful for other schools as they plan the development or improvement of coursework that covers the COVID-19 pandemic. Education related to the virology and epidemiology of the COVID-19 virus is likely to be well accepted. However, medical students are acutely aware of the need for thorough clinical education related to COVID-19, specifically in the areas of diagnosis, management, safety protocols, and personal protective equipment. In addition, addressing the larger context of health equity, physician mental health, and community response beyond the clinical environment offers an important opportunity to prepare well rounded future physicians.
Clinician input is essential to COVID-19 education, but clinical physician time is stretched thin during a pandemic, making it less likely that clinicians will have the ability to set aside time to leave clinical service in order to prepare separate educational content on short notice for medical students. Medical education programs may need to explore novel options that allow students to learn from clinical experiences without adding additional time and effort to practicing clinicians. One such option may a "Day in the Life" type video recording that maintains patient anonymity Clemmons K, Boehme K, Cannon M, Graham J, Lipschitz R MedEdPublish https://doi.org/10.15694/mep.2020.000165.1 Page | 11 but allows medical students to be virtually present for the clinical experience. A recording could then be edited by clinical educators working with educational technology experts to develop instructional material that could be organized by topic and include imbedded learning objectives and associated assignments for students.

Take Home Messages
COVID-19 medical education should seek to balance basic science content, clinical applications as well as public health and health equity considerations. Student feedback indicates that practical clinical knowledge and applications for managing patients with COVID-19 is valued. Innovative educational content should prioritize designing virtual clinical content that seamlessly integrates into practicing clinicians' regular clinical responsibilities.