Implementing a training program for resident-led virtual clinics during the COVID-19 pandemic

Given the impact of the COVID-19 pandemic on healthcare, physicians in the USA, Canada, and globally have begun to utilize virtual care more widely to provide safe, timely, and accessible ambulatory care. While the COVID-19 pandemic prompted its use more widely, it is very likely that this mode of healthcare delivery will continue after the pandemic as it is an eﬀective consultation tool and allows physicians to better serve their patient care populations, particularly those in rural and remote communities. But physicians are not alone in providing virtual care. In medical teaching institutions, residents provide much of the frontline, ambulatory care and, being a newer mode of service, residents require education and training on this method of healthcare delivery; however, no standardized guidelines for training, supervision, or assessment exist. While there is overlap in the process used to conduct in-person and virtual consults, residents require training in speciﬁc areas that are unique to the online setting, particularly in conducting the physical exam. This paper outlines a structured process for developing and implementing a training program for resident-led virtual clinics, including the supervision, performance, and assessment expectations of residents. The recommendations provided are intentionally generic to facilitate use by other programs and institutions.


Introduction
The applicability and use of virtual care has increased since the onset of the COVID-19 pandemic in North America as a way to continue to offer safe, timely, and accessible care (Centers for Disease Control and Prevention, 2020; Public Health Agency of Canada, 2020). While residents provide much of the frontline ambulatory care in teaching

The Use of Virtual Care
Traditionally, telemedicine involved remote delivery of healthcare services and the sharing of clinical information between two hospitals/healthcare facilities (American Telemedicine Association, 2020). More recently, however, advancements such as improved utility and availability of video conferencing devices (e.g. smartphones and tablets), patients and their family member(s) can participate in virtual care from their home or location of choice (Appireddy et al., 2019). The purpose of this mode of healthcare delivery is to facilitate high quality and efficient patient care, especially for those patients who cannot access in-person health care (Shaw et al., 2018;Lougheed, 2019;Canadian Medical Association, 2019a;Canadian Medical Association, 2019b). One of the well-known limitations of virtual care is the inability to do a full physical examination; however, this limitation can be overcome through the use of a modified physical exam (Supplementary File 1) (Wechsler, 2015). Virtual physical exams can be just as informative and directive for general examinations and many neurological conditions as an in-person visit in select clinical scenarios (Wechsler, 2015). Further, the use of remote monitoring devices can help with monitoring of vital signs and other physical examination findings (Andres et al., 2018;Volterrani and Sposato, 2019;Steth IO, 2020).

Development and Implementation
There is a lack of residency training and accreditation requirements for virtual care globally, including in the USA, Canada, and Australia with only one Accreditation Council for Graduate Medical Education (ACGME) accredited residency training program that includes virtual health in their assessment milestones (Canadian Medical Association, 2019a;Pourmand et al., 2020). While there are e-health milestones in CanMEDS 2015 (e.g. ME 1.4 "Use technology to facilitate consultation for patients who may have limited or delayed access to care"), this milestone and others are not included in the milestone requirements for Canadian Neurology training programs (Canadian Medical Association, 2019a; Royal College of Physicians and Surgeons of Canada, 2020c). Moreover, there is a gap in research guiding the assessment of residency training in the area of virtual care (Canadian Medical Association, 2019a). As such, we sought to develop our own virtual care training program and assessments. Drawing from the stages of learning proposed by Dreyfus and clinical competence ranking based on Miller's pyramid, we have developed a resident virtual care training program (Dreyfus and Dreyfus, 1980;Miller, 1990). This program was led by the residency program director (LBL), virtual care physician lead (RA), resident lead (AW), and the educational consultant (LC). Much like the clinical skills learned in medical schools, such as history taking and the physical exam, we found that there is a process to becoming an expert in virtual care that follows a similar pattern. When we began implementing this program, all of our residents were new to virtual care but, as Miller's pyramid suggests, with clear objectives (Competencies/Entrustable Professional Activities (EPAs)/tasks), direct observation and ongoing measurement (assessment and coaching) residents will gain the knowledge and competence needed to integrate virtual care into their clinical practice (Miller, 1990). Our approach can be broken down into 4 phases.

Phase 1 -Introduction to Virtual Care
During phase 1, the "knows" phase in Miller's Pyramid, residents are introduced to virtual care (Miller, 1990). This Boissé Lomax L, Crawford L, Wong A, Balasubramaniam G, Appireddy R MedEdPublish https://doi.org/10.15694/mep.2021.000149.1 Page | 3 phase is conducted in the form of didactic lectures with numerous opportunities for discussion. The initial discussions focus on changes in communication styles, virtual care etiquette, optimal use of technology, and virtual care clinical skills. Residents are provided a communication script, outlining communication, and etiquette considerations, as well as the modified physical exam that includes prompts to guide patient positioning and proximity to the camera. Discussions focus on the performance of a modified physical exam, utility, and scope of virtual care, patient selection, triaging, clinical conditions for virtual care, and advantages of virtual care as well as limitations. Future cohorts of residents entering the Neurology training program at Queen's University will go through the virtual care training program, starting with Phase 1 during their orientation.

Phase 2 -Exposing Residents to Virtual Care
During this phase, the "knowns how" phase, residents participate in simulated virtual care encounters with a standardized patient in a controlled environment where experienced observers provide feedback (more on this in the "Assessment Considerations" section) (Miller, 1990). The simulations are done virtually to best simulate a real-life virtual visit and to easily facilitate direct observation (via remote login), a key component in competency-based medical education (Hawkins et al., 2015;Holmboe, 2015). The domains that require feedback and coaching most are virtual care etiquette, creating the ideal environment, and the physical exam. We found that all the residents, including the most junior, have many transferrable skills that they intuitively integrate into their first standardized visits, namely verbal communication and history taking. These simulated sessions provide the resident with a safe environment to practice skills and consolidate knowledge, with the understanding that their performance will be monitored and assessed. These sessions are also an excellent opportunity for residents to receive feedback, not only from their supervising attending but also from the simulated patients who can identify communication strengths as well as areas for improvement. Residents are expected to attain 3 "achieving" assessments (Table 1) in the simulated sessions before they can participate in virtual care encounters with actual patients.

Phase 3 -Integrating Residents into Virtual Care
As residents become comfortable running virtual visits, the focus should move toward teaching safe and effective use of virtual visits for patient care. At this stage, the resident may "show how" to conduct a virtual visit with real patients but not have reached the "does" phase, yet. (Miller, 1990). Depending on the degree of entrustment (the level of independence entrusted to the resident for a specific Competency/EPA/task), the attending may be conferenced in to 1. directly observe the entire visit, 2. review and confirm with the patient at the end of each visit, or 3. review the case with the resident at the end of the clinic day (see section on "Guidelines for Resident Supervision, Performance, and Assessment"). This is a critical phase as patient safety is dependent on the geographical location of the supervisor and their ability to intervene in patient care decisions if needed (Canadian Medical Association, 2019a). Compared to traditional models for in-person care, virtual care lends itself well to a variety of remote supervision scenarios ((Canadian Medical Association, 2019a). Decisions regarding the level of resident supervision should centre on ensuring patient safety and resident learning (Canadian Medical Association, 2019a). Through practice and working with different attendings, the most important skill the resident develops at this stage is their ability to judge whether a patient is safe to be assessed virtually. Ultimately, through experience and ongoing coaching from their supervisor, the resident will adjust their risk tolerance to what they will ultimately use for deciding when to apply a virtual over an in person visit.

Phase 4 -Mastery
The final stage is the mastery of residents' virtual visit skills, the "does" phase (Miller, 1990). This stage will ultimately be obtained over multiple months to years of experience and practice where residents will learn the nuances that allow them to most effectively, and intuitively apply this valuable skill. This stage will continue on past graduation from residency and involves a commitment to lifelong learning and professional development.

Resident Supervision
Given the novelty of virtual care for residents, it is important to have appropriate levels of supervision to ensure patient needs are met, and that any gaps in communication, etiquette, and clinical assessment are addressed (Table 2) (Canadian Medical Association, 2019a; Wolfram, 2020). A significant advantage offered by the virtual care platform is the ability to remotely supervise residents which works well with the busy schedules of physicians and also ensures safe physical distancing during the COVID-19 pandemic. 2. Direct observation is always recommended but the level of supervision should be based on a resident's experience and level of training (all initial virtual visits should be directly supervised) 3. The supervisor must be readily available in instances of remote supervision (e.g., available by phone) 4. Rounds, pre-clinic meetings, or other routine standard-setting and checkpoints should be maintained or initiated due to the limitations imposed by virtual care. These meetings may be accomplished through conference calls, video chats, or messaging apps 5. Charts should be reviewed and signed off during or after each clinic or shift 6. Residents should document all interactions regardless of the means of communication used

Performance Expectations of Residents
A key element when supervising residents is establishing performance expectations that align with their experience and level of training (Table 3) (UNC School of Medicine, 2020; Wolfram, 2020). While we cannot replace a clinical educator's ability to make judgments about a resident's readiness to participate in a task, we have developed a general guideline on resident performance expectations by stage of training to aid in the observation and assessment of residents. The guideline was based on the RIME Framework which uses a developmental approach to describe resident performance along four stages of development (UNC School of Medicine, 2020). While this framework was originally developed for undergraduate medical training programs, it aligns well with post-graduate developmental expectations. We chose to base the guideline on the staged progression implicit in CBME because all programs at Queen's University implemented CBME in July 2017, and many programs in Canada, the USA, and Australia have already transitioned to or are about to launch CBME (Dagnone et al., 2017; Royal College of Physicians and Surgeons of Canada, 2019b).

Assessment Considerations
Whether patient care is delivered in person or virtually, the goal is the same and, thus, we modified existing assessments for various Competencies/EPAs/tasks in our program to suit the virtual context (Table 1). The Neurology training program at Queen's University is currently using the national Competence By Design (CBD) EPAs (competencies/tasks), milestones, and assessments outlined by the Royal College of Physicians and Surgeons of Canada (RCPSC) so we had existing templates to work from (Royal College of Physicians and Surgeons of Canada, 2020d). The milestones required by the RCPSC were not changed but, instead, additional milestones were added, as needed, to better assess the nuances of virtual care.
As suggested in Phase 4 of our resident virtual clinic training program, intuitive integration of knowledge and skills into professional practice comes after months to years of experience; thus, we included a self-reflection component to our training program where residents assess their perceived successes and potential areas of growth (Miller, 1990). This not only permits them to consider their growing competencies, but it also provides active training in the cognitive processes of assessing one's performance, a vital skill in a self-regulated profession (Duffy and Holmboe, 2006).
As a supervisor/attending providing feedback or completing workplace-based assessments on residents, consider the guidelines laid out in Table 3. These guidelines are intended to inform the assessor of the knowledge and skills that are appropriate for the resident's current stage of training. They are also framed in such a way as to facilitate feedback that coaches for improvement toward the desired performance and expectations of the stage. This feedback will not only inform the resident where they performed well but will also generate clear next steps for further improvement. The "red flags" were adopted from the key indicators established by the Department of Family promptly to ensure future behaviour or knowledge vulnerabilities can be managed appropriately. We recommend that red flags and the actions taken to address them are documented clearly to track recurrence or support decisions such as implementing a modified learning plan or remediation, if needed.

Resident Experiences to Date
There are three Neurology residents (including GB) currently completing virtual epilepsy clinics. Review of clinical encounters (by LBL, RA, AW) revealed that all relevant aspects of clinical care during an epilepsy clinic such as the physical exam, seizure frequency monitoring, antiepileptic drug (AED) compliance, side effects, dose titration, review of results of EEG and imaging could be done successfully by residents in the virtual environment. Virtual visits by residents are also being completed in General Neurology, Multiple Sclerosis, Neuromuscular, and Movement Disorder clinics. Resident assessment data is still being compiled but has been organized in such a way that we will be able to establish if there are performance variations between in-person and virtual visits among residents.

Conclusion
While the COVID-19 pandemic prompted an immediate need for residents to participate in virtual clinics, the need has always been present and will continue to play an important role in healthcare delivery. Due to the novel nature of virtual care, carefully planned measures need to be put in place to support residents while ensuring patient safety and effective health care delivery. By providing clear objectives (competencies/EPAs/tasks), direct observation, and ongoing measurement (assessments), residents can be positive contributors in this much-needed service while ensuring they develop the virtual care skills needed for future independent practice. A future version of this paper will highlight our ongoing quality improvement efforts as well as compare resident performance in in-person and virtual clinics based on the assessment data collected.

Take Home Messages
There will be an ongoing need/demand for virtual care after the COVID-19 pandemic A structured approach to education and training is needed to ensure patient safety and that residents receive timely and effective feedback A modified physical exam protocol is necessary to support virtual care Coaching and assessment practices need to align with the competencies outlined by certifying colleges (e.g. the Royal College of Physicians and Surgeons of Canada)