Assessing Surgical Residents; Challenges and Future Options

Surgical education has changed significantly since the first formal program was established by William Halstead. Specifically, in the last ten years, surgical education has moved from the popular "See One, Do One, Teach One" model to a competency based model using distinct milestones. While these milestones and competencies have been successfully implemented in many fields, including Otolaryngology, there remains significant problems that need to be addressed. Specifically, the lack of assessment tools able to accurately measure whether trainee surgeons can perform steps or indeed an entire operation is a significant and underreported problem.


Introduction
Over the past 100 years' surgical education has undergone a drastic change from the first formal surgical training program established in 1890 by William Halstead at Johns Hopkins Hospital (Kotsis & Chung, 2013). Over the years the training model colloquially became known as the "See One, Do One, Teach One" model. However, at the turn of the 20th century, patients and doctors had newfound regulations and expectations. Specifically, in 1999, the Institute of Medicine published a report which suggested that patient's fears of being "practiced on" were perhaps founded; as there was a high rate of preventable medical errors (Kohn & M, 2000). In response, the ACGME began to revise the traditional surgical training model, and created six competencies; patient care, professionalism, medical knowledge, practice based learning and improvement, system based practice, and interpersonal and communication skills that residents must fulfill in order to enter practice as an independent surgeon (Mery, Greenberg, Patel, & Jaik, 2008;Nasca, Philibert, Brigham, & Flynn, 2012).
In 2003, a 80hr/week work restriction was instituted on all residents in an attempt to reduce medical errors that were a result of being overworked and tired (Kotsis & Chung, 2013). In 2009, the ACGME began to remodel their existing accreditation processes to better match the clinical competencies set back in 1999. As a result the Next Accreditation System (NAS) was unveiled and was put in place in July, 2013 for 7 out of 26 ACGME accredited specialties; for the others the rollout of NAS was July 2014 (Nasca et al., 2012). While the new accreditation system was developed in order to allow for safer medical practice and teaching there are many problems and concerns with the new system; one issue that is of the utmost importance is the assessment of whether current residents meet milestones for the competencies set by the ACGME in 1999. Given that approximately 27.5% of general surgery residents do not feel that they will be able to operate independently before finishing training; all specialties must ensure that appropriate tools and methods are available to assess whether residents are fully capable of performing core specialty specific procedures as an independent physicians (Yeo, Viola, Berg, & et al., 2009). While various tools are available, currently, few accurately assess whether residents can perform an entire operation.

Teaching Models and Milestones
While the Halstead model has largely gone out of favor as the sole method for teaching residents, the milestone approach has become largely integrated into most residency programs. In fields such as Otolaryngology the levels have been appropriately defined and noting progress from a simple grasping of the competency to a mastery of the competency is far clearer than it has been in the past. As an example, in the field of Otolaryngology, Level 4 has been identified as the graduation target ("The Otolaryngology Milestone Project," 2014). While milestones have been more appropriately defined, the question still remains as to a viable teaching/learning model for attending surgeons and residents, respectively.
One proposed model is the Zwisch model which is comprised of four main stages of learning; Show and Tell, Smart Help, Dumb Help, and No Help (DaRosa et al., 2012). The Show and Tell phase is represented by the attending surgeon completing the operation while divulging relevant tips while operating. After some time, the resident graduates to the Smart Help stage at which point they complete a majority of the operation under the directed guidance of the attending surgeon. The attending surgeon may switch with the resident and provide further counseling and guidance. During the Dumb Help stage, the resident will complete most of the procedure and is only interrupted if it is deemed that the residents action may cause direct harm. In contrast to the Smart Help stage, there is no direct guidance by the attending surgeon other than the aforementioned case. The last stage involves the attending surgeon assuming a completely assisting role and providing very little guidance, if any (DaRosa et al., 2012). While this particular model is useful in order to teach and in some cases, assess resident performance, it has limitations and drawbacks. One potential limitation of this model is when the procedure may be more complicated, examples include bariatric surgery and surgeries in which advanced fellowships are often required. In those types of surgeries, residents may not have the extensive operative experience required to complete those cases without assistance . Another limitation is that literature has shown that the perception of autonomy differs between residents and surgeons; such that while a resident may feel they completed very little of the operation; the attending surgeon may feel the opposite .

Assessment Tools
Global Tools: While the Zwisch model can be used as an assessment tool in itself, over the last 20 years many other tools have Sandher S, Gibber M MedEdPublish https://doi.org/10.15694/mep.2017.000177 Page | 3 been developed to specifically measure resident's technical and intraoperative skills (DaRosa et al., 2012;. While there are at least 13 different tools currently in use, many of them use a global rating scale to assess technical skills of residents. Five of the global tools are; Objective Structured Assessment of Technical Skills (OSATS), Global Operative Assessment of Laparoscopic Skills (GOALS), Global Rating Scale (GRS), Surgical Efficiency Score (SES), and Global Rating Index for Technical Skills (GRITS), and ASCRS (Ahmed, Miskovic, Darzi, Athanasiou, & Hanna, 2011;Champagne et al., 2017;Datta, Bann, Mandalia, & Darzi, 2006;Doyle, Webber, & Sidhu, 2007;Martin et al., 1997;Melina C. Vassiliou et al., 2005). Examples of technical skills that these tools measure include, respect for tissues, efficiency of movement, use of assistants, etc. Additionally, one of these tools the ASCRS tool; rated video footage of laparoscopic colectomies using an internally validated scale to in order to further enhance the safety and quality of the procedure (Champagne et al., 2017). While these tools provide a platform for assessing residents; they do not provide sufficient information regarding a resident's ability to perform a specific operation or steps of an operation.

Non-Global Assessment Tools:
In contrast to the global tools mentioned previously; other tools have been developed to measure non-technical surgical competence while operating. Currently non-global assessment tools that are available include the Surgical Procedure Feedback Rubric ( Vassiliou et al., 2006;Wohaibi et al., 2007). While some of the tools, most notably the PBA tool, do provide some insight into whether the resident is able to correctly and fully complete steps of a particular operation, most do not. Instead most of the tools measure other general aspects of surgical competency, such as communication skills, pre-operative planning, post-operative planning, knowledge of the particular procedure, patient knowledge, etc. While all of these are important skill sets for trainee surgeons to gain; these tools again fail to measure competency in particular steps of an operation as well as an entire operation. Additionally, the PBA tool has some particular drawbacks; one being that it is very difficult at least at the time of publishing to easily track the progress of residents in terms of how much of a particular operation they can perform. Due to the rather large number of data points, assessment would be most likely challenging and unintuitive. In addition, most tools listed above have not been adapted to be used concurrently while the operation is taking/or has recently taken place. However, Wagner and his group tested a web-based application that used features of the OPRS to rank how well residents completed certain parts of operations. They used a numerical scale as part of their ranking and also made it a point to complete the evaluations within 45 minutes of the procedure ending; enhancing the ability for reflection and feedback (Wagner et al., 2014). Even more recently, another tool has been developed called the ROCTES which uses a data based website that both attending and residents log-on and provide feedback regarding the knowledge, attitude and skills of the resident physician using a numerical scale from 1-5. While this format does have utility considering it is a webbased tool; the categories do not accurately reflect the ability of the resident physician to perform all steps of a procedure but simply generalize it into one rating system. Therefore, getting a true understanding of the progression in skill of a resident over the course of the program will be challenging.
Alternative Assessment Tools: While the above categories encompass the majority of the tools that exist, three other tools have been developed that Sandher S, Gibber M MedEdPublish https://doi.org/10.15694/mep.2017.000177 Page | 4 do not neatly fit into either category. The first is the Surgical Learning and Instructional Portfolio (SLIP), which essentially involves each resident writing a report on a case seen within the previous month and reflecting on aspects of the case (Webb, Aprahamian, Weigelt, & Brasel, 2011). Another tool currently being employed is the Objective Structured Clinical Evaluation (OSCE) which has been successfully used in many settings to determine the ability of medical students and residents to interact in a standardized way (McAleer & Walker, 1990). Many medical students have experienced this particular type of examination during their STEP 2 boards that are taken usually at the end of third year of medical school. Another tool, Eubank's Checklist, was used to assess the technical skills of surgical residents during laparoscopic cholecystectomies. It utilized videotapes of recorded laparoscopic cholecystectomies and three different experienced surgeons to evaluate the technical skill of surgical residents using a points system (Ahmed et al., 2011;Eubanks et al., 1999). While this method does provide some level of information regarding a resident's technical skills it requires a large amount of expertise and outside time to assess resident skills. Therefore, this tool may not be suited for the current healthcare environment. The last assessment utility was actually a methodology, which was referenced in Glarner's 2013 APDS Spring Meeting and aimed to be a framework of how other programs could develop tools specifically for their institution (Glarner et al., 2013). It non-only analyzes resident technical skills but also non-technical skills. One of the major strengths of this methodology is that it in contrast to others; it breaks down steps of each operation and assigns a numerical scale based on how well the resident performs each step. The major weakness is that this methodology would need to be adapted to every type of surgery in the form of a tool to be used at all other institutions. Given that all of the above tools have drawbacks and limitations; clearly another tool is needed to measure whether residents can successfully perform a step or entire operation. Additionally, it is clear that tool must be readily adaptable to assess surgical residents on a variety of procedures in a variety of specialties.
New Assessment Tools: While the above tools do provide some level of resident skill assessment; there are two new tools which aim to address many of these issues. One such tool is Explorer Surgical which aims to provide a surgical playbook to enhance teamwork, and improve flow. By providing information regarding steps of a particular procedure, the tool hopes to improve both OR outcomes and efficacy. Another tool is Sigma, which is currently being used at Montefiore Medical Center. This new intraoperative tool tracks how long each step of a particular operation takes as well as who performs the step, attending surgeon or resident. More importantly, the tool is also able to measure what have been defined as micro competencies. Essentially within each step of a particular procedure, there are sub steps that must be completed. As an example, during a parotidectomy the last step is closing the wound. However, to close the wound correctly many other technical skills are required (micro competencies). The resident may need to provide hemostasis, insert a drain, and then close the skin using a variety of suturing methods, such as interrupted sutures. Therefore, by evaluating what sub steps and techniques (micro competencies) a resident can complete/utilize we will be better equipped to target deficiencies in any surgical field/operation. It is our hope that further monitoring using these tool swill aid in identifying areas of weakness in order to improve our resident's performance during surgery [30,31].
Sandher S, Gibber M MedEdPublish https://doi.org/10.15694/mep.2017.000177 Figure 1: A) The panel shows data regarding the length of time taken to complete the procedure on a given day. B) When timing a new operation, the residents name can be added in order to help track the progress of each resident in a particular residency program. C) Each procedure is divided into steps and for each step, a user must indicate who completed a majority of the step, either resident or attending surgeon. The four bubbles located in the bottom right of the screen correspond to the Zwisch model of learning which provides increased granularity to the program director regarding how competency each resident is at performing a particular procedure. D) Each surgeon may complete the surgery in a slightly different order than is what is written in the protocol, therefore certain steps can be skipped and returned to at a later time depending on the surgeon.

Conclusion
With the new regulations and competencies set forth by the ACGME as part of the NAS; it is essential that residency programs and residency program directors have methods and tools to assess the level of competency each surgical resident has (Nasca et al., 2012). However, due to the lack of effective tools for measuring the competency of residents; many programs rely solely qualitative data for determining whether residents meet competencies. Sandher S, Gibber M MedEdPublish https://doi.org/10.15694/mep.2017.000177 Page | 6 Therefore, continued development of assessment tools are required for objective evaluation of surgical residents.

Take Home Messages
In the past two decades, surgical education has changed significantly Residency programs must have tools to gauge resident's operative skill It is imperative that feedback is available for residents to continually improve Two emerging tools, Sigma and ExplORer surgical show early promise

Notes On Contributors
Sukhjot Sandher is currently a third year medical student at Albert Einstein College of Medicine. He obtained an undergraduate degree from the University of California, Davis in 2012 in Neurobiology, Physiology and Behavior.
Marc Gibber, MD, is an assistant professor in the Department of Otolaryngology -Head and Neck Surgery and is Director of Research and Innovation at the Montefiore/Einstein Center for Innovation in Simulation (MECIS). He obtained his medical degree from Albert Einstein College of Medicine and completed a residency in Otolaryngology from Montefiore Medical Center.